# Recommendation Newsletter — End-to-End

## Overview

Daily personalized research recommendation emails sent to eligible users via **Listmonk transactional API**, with full click/open tracking and an admin analytics dashboard.

---

## Pipeline

### 1. User Eligibility

- Users must have `enabled = 1` (verified) and `recommendation_newsletter_enabled = 1`
- Ordered by most recent login; default batch: **500 users** (override with `--limit`)
- Users unsubscribed in Listmonk (blocklisted or removed from list 6) are **skipped**

### 2. Recommendations (Elasticsearch MLT)

- `RecommendationService::getRecommendations()` uses **More Like This** queries against `arabic_research` / `english_research` indices
- Input: user's reading history from `user_read_history` table (tracked by `ReadHistoryService`)
- If fewer than 2 personalized results → **fallback** to recent articles for the user's locale via `getRecentArticles()`

### 3. Email Assembly (`SendRecommendationsCommand`)

For each user:

1. Build `txRecommendations[]` array (title, slug, abstract, publisher, date, downloads, tags)
2. Generate **tracked click URLs** per recommendation — HMAC-signed redirect through `/nl/c`
3. Generate **tracking pixel URL** — HMAC-signed 1×1 GIF via `/nl/o`
4. Generate **unsubscribe URL** — HMAC-signed link to `/newsletter/unsubscribe?type=recommendations`
5. Send via `ListmonkService::sendTransactional()` (template ID 6)
6. Log result to `newsletter_log` table (sent / failed / skipped)

### 4. Listmonk Transactional Template

- **Template ID 6** (`Shamra Recommendations TX`) — Go template with RTL/LTR support
- Subject line from `txData.subject` (locale-aware)
- Article links use `{{ .trackedUrl }}` (routed through click tracker)
- Hidden `<img>` pixel at bottom loads `{{ .Tx.Data.trackingPixelUrl }}` for open tracking
- Footer: unsubscribe link + profile settings link

### 5. Click & Open Tracking

**Controller:** `NewsletterTrackingController` (public, no auth)

| Route | Purpose | Method |
|-------|---------|--------|
| `/nl/c?u=&e=&url=&sig=` | Click tracking | Verify HMAC → log `newsletter_event` (type=click) → 302 redirect to target URL |
| `/nl/o?u=&e=&sig=` | Open tracking | Verify HMAC → log `newsletter_event` (type=open, deduplicated to 1 per 30 min) → return 1×1 GIF |

- **HMAC signatures** use `kernel.secret` — signs `userId:email:click:url` or `userId:email:open`
- Invalid signatures still redirect / return pixel (never block the user)
- Events stored in `newsletter_event` table with: campaign, event_type, user_id, email, url, ip_address, user_agent, created_at

### 6. Admin Dashboard

**URL:** `https://shamra-academia.com/jim19ud83/newsletters` (ROLE_ADMIN only)

**Stats cards:** Sent, Unique Recipients, Opens (with rate), Clicks (with rate), Failed, Skipped, Fallback, Last Sent

**Charts:** Stacked bar chart — Sends (sent/failed) + Engagement (opens/clicks) by day, with 7d/30d/90d toggle

**Sections:**
- Locale breakdown (ar/en)
- Top clicked articles (ranked by click count)
- Send Log tab — paginated, filterable by status and date range
- Click & Open Events tab — paginated, filterable by event type, shows user, email, article URL, IP, timestamp

**API endpoints** (all under `/jim19ud83/newsletters/api/`):

| Endpoint | Returns |
|----------|---------|
| `/stats` | Delivery totals (sent/failed/skipped/unique/fallback) |
| `/daily` | Daily send counts for charting |
| `/logs` | Paginated send log entries |
| `/locales` | Locale breakdown |
| `/event-stats` | Click/open totals + unique counts |
| `/daily-events` | Daily click/open counts |
| `/events` | Paginated click/open event log |
| `/top-urls` | Top clicked article URLs |

---

## Running

```bash
# Daily cron (9:00 CET)
0 9 * * * cd /var/www/html/academia_v2 && sudo -u www-data php bin/console app:send-recommendations --env=prod

# Manual test (single user)
sudo -u www-data php bin/console app:send-recommendations --user=35799 --env=prod

# Dry run
sudo -u www-data php bin/console app:send-recommendations --dry-run --limit=10 --env=prod
```

## Unsubscribe

- **Recommendation-only:** `/newsletter/unsubscribe?email=...&token=...&type=recommendations` → sets `recommendation_newsletter_enabled = 0` in DB
- **All newsletters:** `/newsletter/unsubscribe?email=...&token=...` → removes from Listmonk list

---

## Key Files

| File | Purpose |
|------|---------|
| `src/Command/SendRecommendationsCommand.php` | Main send loop, builds tracked URLs, logs results |
| `src/Controller/NewsletterTrackingController.php` | `/nl/c` click + `/nl/o` open tracking |
| `src/Controller/NewsletterAdminController.php` | Dashboard page + 8 API endpoints |
| `src/Controller/NewsletterController.php` | Unsubscribe handling |
| `src/Entity/NewsletterLog.php` | Send log entity |
| `src/Entity/NewsletterEvent.php` | Click/open event entity |
| `src/Service/RecommendationService.php` | ES MLT recommendations |
| `src/Service/ReadHistoryService.php` | User reading history + eligibility |
| `src/Service/ListmonkService.php` | Listmonk API client |
| `templates/admin/newsletters/dashboard.html.twig` | Dashboard UI |
| `templates/listmonk/recommendation_template.html` | Go email template (synced to Listmonk ID 6) |
| `migrations/Version20260228180000.php` | `newsletter_log` table |
| `migrations/Version20260301100000.php` | `newsletter_event` table |