# 27 — Newsletter V2 (AWS SES)

> **Status**: Deployed  
> **Date**: 2026-04-03  
> **Dashboard**: `/jim19ud83/newsletter`

---

## Overview

A fully self-hosted newsletter system replacing Listmonk for marketing/campaign emails. Built on AWS SES (us-east-1) with flexible audience segmentation, click/open tracking, opt-out management, and analytics — all within the existing Symfony app.

The system is completely isolated from the existing recommendation/digest newsletter infrastructure (Listmonk). It has its own entities, controllers, services, and admin UI.

---

## AWS SES Configuration

| Setting | Value |
|---------|-------|
| Region | us-east-1 |
| Daily quota | 50,000 messages |
| Max send rate | 14 messages/second |
| Verified identity | `shamra-academia.com` (domain) + `info@shamra-academia.com` (email) |
| Default From | `info@shamra-academia.com` |
| DKIM | 3 CNAME records added to DNS |

### Credentials

- IAM user: `shamra-ses-sender` with `AmazonSESFullAccess` policy
- Access key stored in GitHub Secrets (`SES_ACCESS_KEY`, `SES_SECRET_KEY`)
- Automatically written to `.env.local` on production during deploy via GitHub Actions

### Env Vars

```
SES_ACCESS_KEY=AKIA...
SES_SECRET_KEY=...
SES_REGION=us-east-1
SES_FROM_EMAIL=info@shamra-academia.com
```

---

## Architecture

### Database Tables (6)

| Table | Purpose |
|-------|---------|
| `email_list` | List definitions with JSON filter config |
| `email_list_member` | Users belonging to each list |
| `email_campaign` | Campaign metadata, counters (sent, opens, clicks, unsubs) |
| `email_campaign_send` | Per-recipient send record with tracking token |
| `email_campaign_click` | Individual click events |
| `email_opt_out` | Global opt-out registry |

Migration: `migrations/Version20260403120000.php`  
Collation: `utf8mb4_unicode_ci` (matching existing tables).

### Entities

- `src/Entity/EmailList.php`
- `src/Entity/EmailListMember.php`
- `src/Entity/EmailCampaign.php`
- `src/Entity/EmailCampaignSend.php`
- `src/Entity/EmailCampaignClick.php`
- `src/Entity/EmailOptOut.php`

### Services

| Service | File | Purpose |
|---------|------|---------|
| `SesMailerService` | `src/Service/Newsletter/SesMailerService.php` | AWS SES client wrapper — `send()`, `getSendQuota()` |
| `EmailListBuilderService` | `src/Service/Newsletter/EmailListBuilderService.php` | Builds/refreshes lists from filter definitions, excludes opted-out users |
| `CampaignService` | `src/Service/Newsletter/CampaignService.php` | Campaign lifecycle — prepare sends, send batches, record clicks/opens, analytics |

### Controllers

| Controller | Route Prefix | Auth | Purpose |
|------------|-------------|------|---------|
| `NewsletterV2Controller` | `/jim19ud83/newsletter` | `ROLE_ADMIN` | Admin dashboard + all API endpoints |
| `CampaignTrackingController` | `/em/` | Public (no auth) | Click tracking, open tracking, unsubscribe |

### Messenger (Async Sending)

- **Message**: `src/Message/SendCampaignBatch.php`
- **Handler**: `src/MessageHandler/SendCampaignBatchHandler.php`
- Routed to `async` transport (Doctrine)
- Sends 50 emails per batch, then dispatches another batch if more remain
- 80ms sleep between sends to respect SES rate limit (14/sec)

### Cron Command

```bash
# Daily list refresh — removes opted-out users, adds new matching users
php bin/console app:newsletter:refresh-lists --env=prod
```

Recommended cron: `0 3 * * *`

---

## Features

### Email List Builder

Create lists based on flexible filters queried against `fos_user`:

| Filter | DB Column | Example |
|--------|-----------|---------|
| `country` | `fos_user.country` | `"SY"`, `"US"` |
| `locale` | `fos_user.preferred_locale` | `"ar"`, `"en"` |
| `registered_after` | `fos_user.created_at` | `"2024-01-01"` |
| `registered_before` | `fos_user.created_at` | `"2025-12-31"` |
| `last_login_after` | `fos_user.last_login` | `"2025-06-01"` |
| `last_login_before` | `fos_user.last_login` | `"2026-01-01"` |
| `study_degree` | `fos_user.study_degree` | exact match |
| `has_stripe` | `playground_subscription` join | users with active Stripe sub |
| `has_academic_sub` | `academic_subscription` join | users with active academic sub |

Filters are stored as JSON on the `email_list` row, allowing daily re-evaluation.

### Campaign System

1. Create campaign (draft) — set name, subject, HTML body, assign a list
2. Send test emails to up to 10 addresses
3. Launch — creates `email_campaign_send` rows, dispatches batches via Messenger
4. Batch handler sends 50/batch via SES, self-dispatches until done
5. Real-time stats update on the dashboard

### Personalization

HTML body supports placeholders:

| Placeholder | Replaced With |
|-------------|---------------|
| `{{first_name}}` | Recipient's first name (or "Friend" if null) |
| `{{email}}` | Recipient's email address |
| `{{unsubscribe_url}}` | HMAC-signed unsubscribe link |
| `{{tracking_pixel}}` | 1×1 transparent GIF for open tracking |

### Link Tracking

All `<a href="...">` links in the HTML body are automatically rewritten to pass through the click tracker:

```
Original:  https://shamra-academia.com/some-feature
Rewritten: https://shamra-academia.com/em/c?t={token}&url={base64(original)}
```

The tracker logs the click, then 302-redirects to the original URL. Links to `/em/` paths and `mailto:` are excluded from rewriting.

### Open Tracking

A 1×1 transparent GIF is appended to every email:

```
https://shamra-academia.com/em/o?t={tracking_token}
```

First open per recipient is recorded; subsequent loads are ignored.

### Opt-out / Unsubscribe

Every email includes an unsubscribe link (HMAC-signed, unforgeable, never expires per RFC 8058):

```
https://shamra-academia.com/em/unsub?e={base64(email)}&c={campaignId}&sig={hmac}
```

- Records the opt-out in `email_opt_out` table
- Increments the campaign's unsubscribe counter
- User sees a clean confirmation page
- Opted-out emails are excluded from all future list refreshes and sends

### Analytics

The admin dashboard shows:

- **Overview stats**: total sent, opens, clicks, opt-outs, SES daily quota usage
- **Per-campaign**: open rate, click rate, top clicked URLs, send log
- **Opt-out list**: recent unsubscribes with campaign attribution

---

## Admin Dashboard UI

URL: `https://shamra-academia.com/jim19ud83/newsletter`

Four tabs:

1. **Campaigns** — Create/edit/send campaigns, view status, send test emails
2. **Lists** — Create lists with filters, preview count, refresh, view members
3. **Analytics** — Select a campaign to see detailed open/click/URL analytics
4. **Opt-outs** — View all opted-out email addresses

Modern SPA-style UI (vanilla JS + fetch API), styled similar to SendGrid/Mailchimp with purple accent matching Shamra branding.

---

## API Endpoints

All under `/jim19ud83/newsletter/api/` (admin-only):

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/lists` | List all email lists |
| POST | `/lists/create` | Create a new list with filters |
| POST | `/lists/{id}/refresh` | Rebuild list members |
| DELETE | `/lists/{id}` | Delete a list |
| GET | `/lists/{id}/members` | Paginated member list |
| POST | `/lists/preview` | Preview filter match count |
| GET | `/filters` | Available filter options (countries, degrees) |
| GET | `/campaigns` | List all campaigns |
| POST | `/campaigns/create` | Create a new campaign |
| GET | `/campaigns/{id}` | Campaign details + analytics |
| POST | `/campaigns/{id}/update` | Update draft campaign |
| POST | `/campaigns/{id}/send` | Launch campaign sending |
| POST | `/campaigns/{id}/test` | Send test to specific emails |
| DELETE | `/campaigns/{id}` | Delete a campaign |
| GET | `/campaigns/{id}/sends` | Paginated send log |
| GET | `/stats` | Overall stats + SES quota |
| GET | `/optouts` | Recent opt-outs |

Public tracking endpoints (no auth):

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/em/c` | Click tracker (302 redirect) |
| GET | `/em/o` | Open tracker (1×1 GIF) |
| GET/POST | `/em/unsub` | Unsubscribe page |

---

## Security

- All admin routes require `ROLE_ADMIN`
- Unsubscribe links use HMAC-SHA256 signed with `kernel.secret`
- Click tracker validates URL scheme (http/https only) to prevent open redirects
- SES credentials stored in GitHub Secrets, injected via deploy workflow
- Opt-out is permanent and checked at both list-refresh and send time

---

## Files Summary

```
src/Entity/EmailList.php
src/Entity/EmailListMember.php
src/Entity/EmailCampaign.php
src/Entity/EmailCampaignSend.php
src/Entity/EmailCampaignClick.php
src/Entity/EmailOptOut.php
src/Repository/EmailListRepository.php
src/Repository/EmailListMemberRepository.php
src/Repository/EmailCampaignRepository.php
src/Repository/EmailCampaignSendRepository.php
src/Repository/EmailCampaignClickRepository.php
src/Repository/EmailOptOutRepository.php
src/Service/Newsletter/SesMailerService.php
src/Service/Newsletter/EmailListBuilderService.php
src/Service/Newsletter/CampaignService.php
src/Controller/NewsletterV2Controller.php
src/Controller/CampaignTrackingController.php
src/Message/SendCampaignBatch.php
src/MessageHandler/SendCampaignBatchHandler.php
src/Command/RefreshEmailListsCommand.php
templates/admin/newsletter_v2/dashboard.html.twig
templates/newsletter_v2/unsubscribe.html.twig
migrations/Version20260403120000.php
.github/workflows/deploy-prod.yml (updated)
config/services.yaml (updated)
config/packages/messenger.yaml (updated)
.env (updated with SES defaults)
```