# Shamra Academia - Developer Documentation

## Table of Contents
- [Local Setup](#local-setup)
- [Recent Updates (February 2026)](#recent-updates-february-2026)
- [Creating New Pages](#creating-new-pages)
- [User Authentication & Subscription](#user-authentication--subscription)
- [Database Management](#database-management)
- [ElasticSearch Integration](#elasticsearch-integration)
- [User Reference Library](#user-reference-library)
- [Playground AI Service](#playground-ai-service)
  - [Features](#features)
  - [Architecture Overview](#architecture-overview)
  - [Core Services & Libraries](#core-services--libraries)
  - [API Endpoints](#api-endpoints)
  - [Environment Configuration](#environment-configuration)
  - [AI Prompt Templates](#ai-prompt-templates)
  - [Database Schema](#database-schema)
  - [Subscription & Access Control](#subscription--access-control)
  - [Grounding Data for AI Writing](#grounding-data-for-ai-writing)
  - [Security Considerations](#security-considerations)

---

## Recent Updates (February 2026)

### User Interest Tracking System
Activity-based interest tracking that learns from user behavior to personalize content.

**Key Features:**
- Tracks user interests based on research viewing activity
- FIFO limit of 10 interests per user (oldest removed when limit exceeded)
- Supports both database Tag entities and Elasticsearch string tags
- Personalized homepage recommendations based on interests

**Modified Files:**
- `src/Service/TagInterestsUserService.php` - Native SQL queries for FIFO enforcement
- `src/syndex/AcademicBundle/Module/ElasticResearch.php` - Fallback to string tags when `tag_id` is null
- `src/syndex/AcademicBundle/Service/SqlSearch.php` - Handle both Tag entities and string tags
- `src/syndex/AcademicBundle/Controller/HomepageController.php` - Interest-based personalization

### Homepage Personalization
The homepage now shows personalized research recommendations based on user interests.

**Features:**
- Welcome message with user's name
- "Research matching your interests" section when user has interests
- Falls back to trending content for users without interests
- Bilingual support (Arabic/English) via translation files

**Translation Keys Added:**
- `homepage.welcome` - Personalized welcome message
- `homepage.personalized.title/subtitle` - Personalized section headers
- `homepage.trending.title/subtitle` - Trending section headers

### Communities/Spaces System Redesign

#### 1. Communities Feed Page (`/communities/`)
Complete redesign with modern tabbed interface integrating all community features.

**Tabs:**
- **Feed** - Latest posts from all spaces (default)
- **Following** - Spaces user follows (badge shows count)
- **My Spaces** - Spaces user created
- **Requests** - Pending follow requests (only shown when requests exist)
- **Discover** - All public spaces

**URL Parameters:**
```
/communities/              → Feed tab (default)
/communities/?tab=following → Following tab
/communities/?tab=my-spaces → My Spaces tab
/communities/?tab=requests  → Requests tab
/communities/?tab=discover  → Discover tab
```

**Modified Files:**
- `src/Controller/CommunityController.php` - Added tab support and optimized queries
- `templates/community/communities_feed.html.twig` - Complete redesign with tabs

#### 2. Community/Space Homepage (`/community/{alias}`)
Modern hero section with responsive card-based question listing.

**Features:**
- Hero banner with community cover image
- Community stats (posts, members, views)
- Keywords display
- Admin controls (edit, delete)
- Follow/Unfollow buttons
- Collapsible new post form with Quill editor
- Filter by date or views
- Responsive sidebar with admin stats and member list

**Modified Files:**
- `templates/community/index.html.twig` - Complete redesign

#### 3. Question/Post Detail Page (`/community/{alias}/post/{slug}`)
Stack Overflow-inspired Q&A layout.

**Features:**
- Question card with title, meta, keywords, content
- Answer cards with author info and timestamps
- Collapsible answer form with Quill editor
- Sidebar with question stats and author info
- Admin controls (edit, delete)
- Pagination for answers
- Schema.org structured data for SEO

**Modified Files:**
- `templates/community/question_show.html.twig` - Complete redesign

### Database Query Optimizations

#### CommunityQuestionRepository
New optimized methods with eager loading to prevent N+1 queries:

```php
// Single query for public community questions with relations
findRecentQuestionsOptimized(int $limit = 20): array

// Batch query for specific communities
findRecentFromCommunities(array $communityIds, int $limit = 20): array

// Trending questions (most views in last 30 days)
findTrendingQuestions(int $limit = 10): array

// Questions for single community with eager loading
findByCommunityOptimized(int $communityId, string $orderBy, int $limit, int $offset): array
```

**File:** `src/Repository/CommunityQuestionRepository.php`

### Navigation Updates
- Removed "Tracks" and "My Tracks" from navbar (commented out)
- Removed separate "Followed Communities" link (integrated into tabs)
- Communities link now active for both `/communities/` and `/followed-communities/`

**File:** `templates/header.html.twig`

### New Translations Added

**Arabic (`translations/community.ar.yml`):**
```yaml
community:
  question:
    edit: تعديل
    stats: إحصائيات السؤال
    asked: طُرح
    askedBy: طُرح بواسطة
  back: العودة للمساحة
  no_answers: لا توجد إجابات بعد. كن أول من يجيب!
  trending: الأكثر رواجاً
form:
  cancel: إلغاء
  delete-confirm: هل أنت متأكد؟
```

**English (`translations/community.en.yml`):**
```yaml
community:
  question:
    edit: Edit
    stats: Question Stats
    asked: Asked
    askedBy: Asked by
  back: Back to Community
  no_answers: No answers yet. Be the first to respond!
  trending: Trending
form:
  cancel: Cancel
  delete-confirm: Are you sure?
```

### CSS Design System
All redesigned pages use consistent CSS variables:

```css
:root {
    --primary-color: #00A550;
    --primary-hover: #008844;
    --accent-color: #ff8600;
    --text-color: #333;
    --text-muted: #666;
    --border-color: #e0e0e0;
    --bg-light: #f8f9fa;
    --shadow-sm: 0 2px 8px rgba(0,0,0,0.08);
    --shadow-md: 0 4px 16px rgba(0,0,0,0.12);
    --radius: 12px;
}
```

**Responsive Breakpoints:**
- Desktop: > 991px
- Tablet: 768px - 991px
- Mobile: < 767px

---

## Dev Server Access

The dev/test server at `https://test.shamra-academia.com/` is password-protected with HTTP Basic Auth. Google and all other visitors are blocked with a 401.

- **Username:** `shamra`
- **Password:** `Shamra2026Dev!`

---

## Local Setup

### Prerequisites
- PHP 7.2.5 or higher
- Composer
- MySQL/MariaDB
- Node.js & NPM
- ElasticSearch 7.x

### Installation Steps

1. **Clone the repository**
   ```bash
   git clone <repository-url>
   cd academia_v2
   ```

2. **Install dependencies**
   ```bash
   composer install
   npm install
   ```

3. **Environment configuration**
   ```bash
   cp .env.example .env
   ```
   
   Configure your `.env` file with:
   ```env
   DATABASE_URL="mysql://username:password@localhost:3306/database_name"
   ELASTIC_HOST=localhost:9200
   sk_key=your_stripe_secret_key
   firebase_notification_key=your_firebase_key
   ```

4. **Database setup**
   ```bash
   php bin/console doctrine:database:create
   php bin/console doctrine:schema:update --force
   ```

5. **Build assets**
   ```bash
   npm run build
   ```

6. **Start development server**
   ```bash
   symfony serve -d
   # or
   php -S localhost:8000 -t public/
   ```

---

## Creating New Pages

To create a new page like `shamra-academia.com/playground`, follow these steps:

### 1. Create Controller

Create a new controller in `src/Controller/` or use an existing one:

```php
<?php
// src/Controller/PlaygroundController.php

namespace App\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;

class PlaygroundController extends AbstractController
{
    /**
     * @Route("/playground", name="app_playground")
     */
    public function index(Request $request): Response
    {
        return $this->render('playground/index.html.twig', [
            'title' => 'Playground',
        ]);
    }
}
```

### 2. Configure Routes

Add route configuration to `config/routes.yaml`:

```yaml
playground:
    path:
        en: en/playground
        ar: /playground
    defaults:
        _controller: App\Controller\PlaygroundController::index
    options:
        expose: true
```

### 3. Create Template

Create Twig template in `templates/playground/`:

```twig
{# templates/playground/index.html.twig #}
{% extends 'base.html.twig' %}

{% block title %}{{ title }}{% endblock %}

{% block body %}
    <div class="container">
        <h1>Welcome to Playground</h1>
        <p>Your playground content here...</p>
    </div>
{% endblock %}
```

### 4. Add Navigation (Optional)

Update `templates/header.html.twig` to include navigation link:

```twig
<li>
    <a href="{{ path('app_playground') }}">Playground</a>
</li>
```

### Files Created/Modified:
- Controller: `src/Controller/PlaygroundController.php`
- Route: `config/routes.yaml`
- Template: `templates/playground/index.html.twig`

---

## User Authentication & Subscription

### Checking User Authentication & Subscription

Use the `SubscribeService` to verify user authentication and subscription status:

```php
<?php
// Example usage in a controller

use App\syndex\AcademicBundle\Service\SubscribeService;
use Symfony\Component\Security\Core\Exception\AccessDeniedException;

class ExampleController extends AbstractController
{
    /**
     * @Route("/protected-content", name="protected_content")
     */
    public function protectedContent(SubscribeService $subscribeService): Response
    {
        // Check if user is authenticated
        if (!$this->isUserAuthenticated()) {
            return $this->redirect($this->generateUrl('app_login'));
        }
        
        // Check if user has active subscription
        if (!$this->isUserAuthorized($subscribeService)) {
            throw new AccessDeniedException('Subscription required');
        }
        
        return $this->render('protected/content.html.twig');
    }
    
    private function isUserAuthenticated(): bool
    {
        return $this->getUser() !== null;
    }
    
    private function isUserAuthorized(SubscribeService $subscribeService): bool
    {
        $user = $this->getUser();
        
        // Check if user has active subscription
        if (!$subscribeService->isSubscribed($user)) {
            return false;
        }
        
        // For ChatBot features, check specific subscription type
        if ($this->needsChatBotAccess()) {
            return $subscribeService->isChatBotSubscribed($user);
        }
        
        return true;
    }
    
    private function needsChatBotAccess(): bool
    {
        // Implement your logic to determine if ChatBot access is needed
        return false;
    }
}
```

### Subscription Types and Methods

The `SubscribeService` provides these key methods:

```php
// Check if user has any active subscription
$subscribeService->isSubscribed($user);

// Check if user has ChatBot subscription specifically
$subscribeService->isChatBotSubscribed($user);

// Get subscription type
$subscribeService->getSubscriptionType($user);

// Get remaining tokens for ChatBot users
$subscribeService->getRemaingToken($user);
```

### Authentication in Twig Templates

In templates, use:

```twig
{% if app.user %}
    {# User is logged in #}
    <p>Welcome, {{ app.user.username }}!</p>
    
    {% if is_granted('IS_AUTHENTICATED_REMEMBERED') %}
        {# User has persistent authentication #}
    {% endif %}
{% else %}
    {# User is not logged in #}
    <a href="{{ path('app_login') }}">Login</a>
{% endif %}
```

### Security Configuration

Security rules are defined in `config/packages/security.yaml`. Protected routes require `IS_AUTHENTICATED_FULLY` or `IS_AUTHENTICATED_REMEMBERED`.

---

## Database Management

### Adding New Tables

#### 1. Create Entity

```bash
php bin/console make:entity
```

Or create manually in `src/Entity/`:

```php
<?php
// src/Entity/NewTable.php

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity(repositoryClass="App\Repository\NewTableRepository")
 * @ORM\Table(name="new_table")
 */
class NewTable
{
    /**
     * @ORM\Id
     * @ORM\GeneratedValue
     * @ORM\Column(type="integer")
     */
    private $id;

    /**
     * @ORM\Column(type="string", length=255)
     */
    private $name;

    // Getters and setters...
}
```

#### 2. Create Repository

```bash
php bin/console make:repository NewTable
```

### Adding Columns to Existing Tables

#### 1. Update Entity

Add new property to existing entity:

```php
/**
 * @ORM\Column(type="text", nullable=true)
 */
private $newColumn;

public function getNewColumn(): ?string
{
    return $this->newColumn;
}

public function setNewColumn(?string $newColumn): self
{
    $this->newColumn = $newColumn;
    return $this;
}
```

### Schema Updates

#### Development Environment

```bash
# Generate migration
php bin/console doctrine:migrations:generate

# Or update schema directly (NOT recommended for production)
php bin/console doctrine:schema:update --force

# Check schema status
php bin/console doctrine:schema:validate
```

#### Production Environment

**Always use migrations in production:**

```bash
# 1. Generate migration locally
php bin/console doctrine:migrations:generate

# 2. Edit the generated migration file if needed
# migrations/VersionYYYYMMDDHHMMSS.php

# 3. Test migration locally
php bin/console doctrine:migrations:execute --up VersionYYYYMMDDHHMMSS

# 4. Deploy to production and run:
php bin/console doctrine:migrations:migrate --no-interaction

# 5. Check migration status
php bin/console doctrine:migrations:status
```

### Example Migration

```php
<?php
// migrations/Version20240124000000.php

declare(strict_types=1);

namespace DoctrineMigrations;

use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;

final class Version20240124000000 extends AbstractMigration
{
    public function up(Schema $schema): void
    {
        $this->addSql('ALTER TABLE new_table ADD new_column TEXT DEFAULT NULL');
    }

    public function down(Schema $schema): void
    {
        $this->addSql('ALTER TABLE new_table DROP new_column');
    }
}
```

### Production Database Update Process

1. **Backup database**
2. **Test migrations locally**
3. **Deploy code with migrations**
4. **Run migrations:** `php bin/console doctrine:migrations:migrate`
5. **Verify changes**

---

## ElasticSearch Integration

### Search Implementation Example

Here's how to create a search function that queries ElasticSearch for research papers:

#### 1. Create Search Service

```php
<?php
// src/Service/ResearchSearchService.php

namespace App\Service;

use App\syndex\AcademicBundle\Service\Elasticsearch\Elasticsearch;
use App\syndex\AcademicBundle\Service\Elasticsearch\QueryBuilder;
use App\syndex\AcademicBundle\Service\Elasticsearch\BooleanQuery\BooleanGroupQuery;
use App\syndex\AcademicBundle\Service\Elasticsearch\BooleanQuery\Must;
use App\syndex\AcademicBundle\Service\Elasticsearch\SimpleQuery\SimpleQuery;
use App\syndex\AcademicBundle\Module\ElasticMapper;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface;
use Doctrine\ORM\EntityManagerInterface;

class ResearchSearchService
{
    private $elasticsearch;
    private $entityManager;
    private $parameterBag;

    public function __construct(
        Elasticsearch $elasticsearch, 
        EntityManagerInterface $entityManager,
        ParameterBagInterface $parameterBag
    ) {
        $this->elasticsearch = $elasticsearch;
        $this->entityManager = $entityManager;
        $this->parameterBag = $parameterBag;
    }

    public function searchResearches(
        string $query = '', 
        array $filters = [],
        int $page = 0, 
        int $limit = 10
    ): array {
        // Configure pagination
        $this->elasticsearch->setPage($page);
        $this->elasticsearch->setMaxItemsNumber($limit);

        // Build search query
        $searchQuery = $this->buildSearchQuery($query, $filters);
        
        try {
            // Search Arabic research index
            $arabicResults = $this->searchInIndex(
                $searchQuery, 
                $this->parameterBag->get('elastic_arabic_research_index')
            );
            
            // Search English research index
            $englishResults = $this->searchInIndex(
                $searchQuery, 
                $this->parameterBag->get('elastic_english_research_index')
            );
            
            return [
                'arabic_researches' => $arabicResults,
                'english_researches' => $englishResults,
                'total_count' => count($arabicResults) + count($englishResults)
            ];
            
        } catch (\Exception $e) {
            throw new \Exception('Search failed: ' . $e->getMessage());
        }
    }

    private function buildSearchQuery(string $query, array $filters): array
    {
        $queryBuilder = new QueryBuilder();
        $booleanQuery = new BooleanGroupQuery();
        $mustQuery = new Must();

        // Always exclude deleted researches
        $mustQuery->add(new SimpleQuery('deleted', false, 'term'));

        // Add text search if query provided
        if (!empty($query)) {
            $textQuery = [
                "bool" => [
                    "should" => [
                        ["match" => ["arabic_full_title" => $query]],
                        ["match" => ["english_full_title" => $query]],
                        ["match" => ["arabic_abstract" => $query]],
                        ["match" => ["english_abstract" => $query]],
                        ["match" => ["authors" => $query]]
                    ],
                    "minimum_should_match" => 1
                ]
            ];
            $mustQuery->add($textQuery);
        }

        // Add field filter
        if (!empty($filters['field'])) {
            $mustQuery->add(new SimpleQuery('field', $filters['field'], 'term'));
        }

        // Add tag filter
        if (!empty($filters['tags'])) {
            $tagQuery = [
                "bool" => [
                    "should" => [
                        ["match_phrase" => ["tag" => $filters['tags']]],
                        ["match_phrase" => ["ai_keywords" => $filters['tags']]]
                    ],
                    "minimum_should_match" => 1
                ]
            ];
            $mustQuery->add($tagQuery);
        }

        // Add date range filter
        if (!empty($filters['date_from']) || !empty($filters['date_to'])) {
            $dateQuery = ["range" => ["created_at" => []]];
            if (!empty($filters['date_from'])) {
                $dateQuery["range"]["created_at"]["gte"] = $filters['date_from'];
            }
            if (!empty($filters['date_to'])) {
                $dateQuery["range"]["created_at"]["lte"] = $filters['date_to'];
            }
            $mustQuery->add($dateQuery);
        }

        $booleanQuery->add($mustQuery);
        $queryBuilder->addQuery($booleanQuery);

        return $queryBuilder->buildQuery();
    }

    private function searchInIndex(array $query, string $index): array
    {
        $elasticResponse = $this->elasticsearch->search($query, $index);
        
        if ($elasticResponse['hits']['total']['value'] > 0) {
            $language = strpos($index, 'english') !== false ? 'english' : 'arabic';
            return ElasticMapper::fromElasticToObject(
                $elasticResponse, 
                $this->entityManager, 
                $language
            );
        }
        
        return [];
    }
}
```

#### 2. Create Controller

```php
<?php
// src/Controller/SearchController.php

namespace App\Controller;

use App\Service\ResearchSearchService;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Routing\Annotation\Route;

class SearchController extends AbstractController
{
    /**
     * @Route("/api/research/search", name="api_research_search", methods={"GET"})
     */
    public function searchResearches(
        Request $request, 
        ResearchSearchService $searchService
    ): JsonResponse {
        $query = $request->query->get('q', '');
        $page = (int) $request->query->get('page', 0);
        $limit = (int) $request->query->get('limit', 10);
        
        $filters = [
            'field' => $request->query->get('field'),
            'tags' => $request->query->get('tags'),
            'date_from' => $request->query->get('date_from'),
            'date_to' => $request->query->get('date_to'),
        ];

        try {
            $results = $searchService->searchResearches($query, $filters, $page, $limit);
            
            return new JsonResponse([
                'success' => true,
                'data' => $results,
                'query' => $query,
                'page' => $page
            ]);
        } catch (\Exception $e) {
            return new JsonResponse([
                'success' => false,
                'error' => $e->getMessage()
            ], 500);
        }
    }
}
```

#### 3. Create Frontend Page

```php
<?php
// src/Controller/PlaygroundController.php - Updated

class PlaygroundController extends AbstractController
{
    /**
     * @Route("/playground", name="app_playground")
     */
    public function index(): Response
    {
        return $this->render('playground/search.html.twig');
    }
    
    /**
     * @Route("/playground/search", name="playground_search", methods={"POST"})
     */
    public function search(Request $request, ResearchSearchService $searchService): Response
    {
        $query = $request->request->get('query', '');
        $page = (int) $request->request->get('page', 0);
        
        $results = $searchService->searchResearches($query, [], $page);
        
        return $this->render('playground/results.html.twig', [
            'query' => $query,
            'results' => $results,
            'page' => $page
        ]);
    }
}
```

#### 4. Create Search Template

```twig
{# templates/playground/search.html.twig #}
{% extends 'base.html.twig' %}

{% block title %}Research Search{% endblock %}

{% block body %}
<div class="container mt-4">
    <h1>Research Search</h1>
    
    <form method="post" action="{{ path('playground_search') }}">
        <div class="form-group">
            <input type="text" name="query" class="form-control" 
                   placeholder="Search for research papers..." required>
        </div>
        <button type="submit" class="btn btn-primary">Search</button>
    </form>
    
    <div id="search-results">
        <!-- Results will be loaded here -->
    </div>
</div>

<script>
// AJAX search implementation
$(document).ready(function() {
    $('#search-form').on('submit', function(e) {
        e.preventDefault();
        
        const query = $(this).find('input[name="query"]').val();
        
        $.get('{{ path("api_research_search") }}', {
            q: query,
            page: 0,
            limit: 20
        })
        .done(function(response) {
            if (response.success) {
                displayResults(response.data);
            } else {
                alert('Search failed: ' + response.error);
            }
        });
    });
    
    function displayResults(data) {
        let html = '<div class="search-results">';
        
        // Display Arabic research results
        if (data.arabic_researches && data.arabic_researches.length > 0) {
            html += '<h3>Arabic Research</h3>';
            data.arabic_researches.forEach(research => {
                html += `
                    <div class="research-item">
                        <h4><a href="/show/${research.slug}">${research.arabicFullTitle}</a></h4>
                        <p>${research.arabicAbstract || ''}</p>
                        <small>Authors: ${research.authors || 'Unknown'}</small>
                    </div>
                `;
            });
        }
        
        // Display English research results
        if (data.english_researches && data.english_researches.length > 0) {
            html += '<h3>English Research</h3>';
            data.english_researches.forEach(research => {
                html += `
                    <div class="research-item">
                        <h4><a href="/show/${research.slug}">${research.englishFullTitle}</a></h4>
                        <p>${research.englishAbstract || ''}</p>
                        <small>Authors: ${research.authors || 'Unknown'}</small>
                    </div>
                `;
            });
        }
        
        html += '</div>';
        $('#search-results').html(html);
    }
});
</script>
{% endblock %}
```

### ElasticSearch Configuration

The ElasticSearch configuration is handled through:

- **Service Configuration**: `config/services.yaml` - Line 121
- **Elasticsearch7 Implementation**: `src/syndex/AcademicBundle/Service/Elasticsearch/Elasticsearch7.php`
- **Query Builder**: `src/syndex/AcademicBundle/Service/Elasticsearch/QueryBuilder.php`

### Key ElasticSearch Files:
- Base service: `src/syndex/AcademicBundle/Service/Elasticsearch/Elasticsearch.php`
- ES7 implementation: `src/syndex/AcademicBundle/Service/Elasticsearch/Elasticsearch7.php`
- Query builders: `src/syndex/AcademicBundle/Service/Elasticsearch/`
- Elastic mapper: `src/syndex/AcademicBundle/Module/ElasticMapper.php`

---

## User Reference Library

The User Reference Library allows users to upload and manage their personal collection of PDF research papers with full citation metadata. These references can be used as grounding data in the "Write with AI" feature.

### Access URL
`/myreferences`

### Features

#### 1. PDF Upload & Storage
- Upload PDF files up to 50MB each
- Maximum 500MB storage per user
- Secure storage with unique filenames
- Automatic file size tracking

#### 2. Citation Metadata
- **Core Fields**: Title, Authors, Year, Publication Type
- **Publication Details**: Journal/Book name, Volume, Issue, Pages, Publisher, Edition
- **Academic Identifiers**: DOI, ISBN, ISSN, arXiv ID, PMID
- **Additional Info**: Abstract, Keywords, Notes, URL

#### 3. Publication Types
- Journal Article
- Book
- Book Chapter
- Conference Paper
- Thesis/Dissertation
- Report
- Website
- Other

#### 4. Multi-Language Support
Predefined languages: English, Arabic, German, French, Spanish, Russian, Turkish, Chinese, Japanese, Czech, Italian. Users can also specify custom languages.

#### 5. Organization
- Folder-based organization
- Tags support
- Full-text search across title, authors, abstract, keywords

#### 6. Text Extraction
- Automatic PDF text extraction using `pdftotext`
- Extracted text used for AI grounding in "Write with AI"

### Architecture

#### Entity
- `src/Entity/UserReference.php` - Doctrine entity with 30+ fields

#### Repository
- `src/Repository/UserReferenceRepository.php` - Query methods for search, folders, storage

#### Service
- `src/Service/Playground/UserReferenceService.php` - Business logic for upload, CRUD, text extraction

#### Controller
- `src/Controller/UserReferenceController.php` - API endpoints and page controller

#### Template
- `templates/references/index.html.twig` - Vue.js 3 single-page application

### API Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/myreferences` | Main library page |
| GET | `/myreferences/api/list` | List references (with search/folder filter) |
| POST | `/myreferences/api/upload` | Upload new reference |
| GET | `/myreferences/api/{id}` | Get reference details |
| PUT | `/myreferences/api/{id}` | Update reference metadata |
| DELETE | `/myreferences/deleteEntry/{id}` | Delete reference |
| GET | `/myreferences/download/{id}` | Download PDF file |
| GET | `/myreferences/api/folders` | Get user's folders |
| GET | `/myreferences/api/storage` | Get storage statistics |
| POST | `/myreferences/api/bulk/delete` | Bulk delete |
| POST | `/myreferences/api/bulk/move` | Bulk move to folder |
| GET | `/myreferences/api/{id}/citation` | Get formatted citation |
| POST | `/myreferences/api/{id}/extract` | Extract text from PDF |

### Database Table
```sql
CREATE TABLE user_reference (
    id INT AUTO_INCREMENT PRIMARY KEY,
    user_id INT NOT NULL,
    original_filename VARCHAR(255),
    stored_filename VARCHAR(255),
    file_path VARCHAR(500),
    file_size INT,
    mime_type VARCHAR(100),
    title VARCHAR(1000),
    authors LONGTEXT,
    year INT,
    publication_type VARCHAR(50),
    publication_name VARCHAR(500),
    volume VARCHAR(50),
    issue VARCHAR(50),
    pages VARCHAR(50),
    publisher VARCHAR(255),
    doi VARCHAR(255),
    isbn VARCHAR(50),
    arxiv_id VARCHAR(50),
    abstract LONGTEXT,
    keywords LONGTEXT,
    extracted_text LONGTEXT,
    notes LONGTEXT,
    language VARCHAR(10),
    tags VARCHAR(500),
    folder VARCHAR(100),
    is_processed TINYINT(1),
    created_at DATETIME,
    updated_at DATETIME,
    FOREIGN KEY (user_id) REFERENCES fos_user(id) ON DELETE CASCADE
);
```

---

## Playground AI Service

The Playground AI Service is an advanced academic writing assistant that leverages Azure OpenAI to help researchers with various writing and research tasks. It provides an interactive document editor with AI-powered features for rephrasing, proofreading, translation, and intelligent Q&A capabilities.

### Features

#### 1. AI-Powered Text Processing
- **Rephrase**: Improve clarity and flow while maintaining academic tone and original meaning
- **Proofread**: Detect and correct grammar, spelling, and style issues
- **Criticize**: Provide constructive academic criticism and suggestions for improvement
- **Ask**: Answer questions about selected text with context-aware responses
- **Translate**: Translate text between languages while preserving academic terminology

#### 2. Research Assistant Chat
- Interactive chat with an AI research assistant
- Context-aware responses based on project content and keywords
- Multi-source RAG (Retrieval-Augmented Generation) support:
  - **Shamra Library**: Search internal research database via ElasticSearch
  - **arXiv**: Search academic papers from arXiv API
  - **Web Search**: General web search for broader context
- Real-time streaming responses using Server-Sent Events (SSE)
- Citation generation in multiple formats (APA, MLA, Chicago, etc.)

#### 3. Document Management
- Create and manage multiple research projects
- Auto-save functionality for content preservation
- Upload documents (TXT, DOC, DOCX) with automatic content extraction
- Export documents to DOCX and PDF formats with RTL/Arabic support
- Project keywords for enhanced search context
- Public/private project visibility settings

#### 4. Specialized Features
- **Tashkeel**: Arabic text vowelization (تشكيل)
- **Keyword Extraction**: AI-powered keyword generation from research titles
- **Content Summarization**: Generate concise summaries of research content
- **Selection Analysis**: Answer questions about highlighted/selected text

#### 5. Write with AI (Content Generation)
A powerful feature for generating academic content using research sources as grounding data:

**Workflow:**
1. Click "Write with AI" button in the keywords area (requires 3+ keywords and project title)
2. Enter optional writing instructions (e.g., "Write an introduction about the importance of AI in education")
3. System searches Shamra research index for related papers
4. Select relevant research sources to use as reference material
5. AI generates academic content with proper citations [Author, Year]
6. Refine the generated content with follow-up instructions
7. Insert the final content into the editor

**Key Features:**
- Uses RAG (Retrieval-Augmented Generation) with Shamra research library
- Grounded content generation with proper academic citations
- Multi-turn conversation for iterative refinement
- Support for Arabic and English academic writing
- Real-time streaming for responsive UX

**Requirements:**
- Minimum 3 keywords in project
- Project title required
- At least 1 source must be selected for generation

#### 6. Academic Translation Service
A dedicated translation feature with specialized academic personas:
- **General**: Standard academic translation
- **Scientific**: Precise scientific terminology
- **Medical**: Medical terminology and procedures
- **Legal**: Legal terminology and formal language
- **Technical**: Technology and engineering terms

Translation features include:
- Automatic language detection
- Large document chunked translation (for texts > 3000 characters)
- File upload support (PDF, DOCX, TXT)
- Translation history per project

#### 7. Notebook System
Projects include a notebook system for organizing content:
- **Text**: Regular text content
- **AI Response**: Saved AI-generated content
- **Citation**: Formatted citations
- **Heading**: Section headers

---

### Architecture Overview

The Playground service follows a layered architecture with clear separation of concerns:

```
┌─────────────────────────────────────────────────────────────────┐
│                         Controllers                              │
│  ┌─────────────────────┐ ┌─────────────────┐ ┌────────────────┐ │
│  │PlaygroundController │ │PlaygroundAPI   │ │Playground      │ │
│  │(UI Routes)          │ │Controller      │ │TranslationCtrl │ │
│  └─────────────────────┘ └─────────────────┘ └────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│                          Services                                │
│  ┌───────────────────┐  ┌─────────────────┐  ┌────────────────┐ │
│  │ AzureOpenAIService│  │ PlaygroundRAG   │  │ ProjectService │ │
│  │ (AI Integration)  │  │ Service (Search)│  │ (CRUD)         │ │
│  └───────────────────┘  └─────────────────┘  └────────────────┘ │
│  ┌─────────────────────────┐  ┌─────────────────────────────┐   │
│  │ DocumentParserService   │  │ AcademicTranslationService  │   │
│  │ (File Processing)       │  │ (AI Translation w/Personas) │   │
│  └─────────────────────────┘  └─────────────────────────────┘   │
│  ┌─────────────────────────┐  ┌─────────────────────────────┐   │
│  │ UsageMonitorService     │  │ PlaygroundSubscription      │   │
│  │ (Credits & Limits)      │  │ Service (Stripe Webhooks)   │   │
│  └─────────────────────────┘  └─────────────────────────────┘   │
├─────────────────────────────────────────────────────────────────┤
│                          Entities                                │
│  ┌────────────────┐ ┌──────────────┐ ┌──────────────────────┐   │
│  │PlaygroundProject│ │PlaygroundChat│ │PlaygroundNotebook    │   │
│  └────────────────┘ └──────────────┘ └──────────────────────┘   │
│  ┌──────────────────────┐  ┌────────────────────────────────┐   │
│  │PlaygroundTranslation │  │ PlaygroundSubscription         │   │
│  └──────────────────────┘  └────────────────────────────────┘   │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │                  PlaygroundUsageLog                         │ │
│  └────────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────┤
│                      External Services                           │
│  ┌───────────────┐  ┌──────────────┐  ┌─────────────────────┐   │
│  │ Azure OpenAI  │  │ ElasticSearch│  │ arXiv API           │   │
│  └───────────────┘  └──────────────┘  └─────────────────────┘   │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                        Stripe                              │  │
│  │          (Subscriptions, Webhooks, Payments)               │  │
│  └───────────────────────────────────────────────────────────┘  │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │                   SerpAPI (Web Search - Optional)          │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
```

#### Key Files and Directories

| Path | Description |
|------|-------------|
| `src/Controller/PlaygroundController.php` | Main UI controller for playground pages |
| `src/Controller/PlaygroundAPIController.php` | REST API endpoints for AI operations |
| `src/Controller/PlaygroundTranslationController.php` | Translation API endpoints |
| `src/Service/Playground/AzureOpenAIService.php` | Azure OpenAI integration service |
| `src/Service/Playground/PlaygroundRAGService.php` | Multi-source RAG search service |
| `src/Service/Playground/PlaygroundProjectService.php` | Project CRUD operations |
| `src/Service/Playground/DocumentParserService.php` | Document file parsing (TXT, DOC, DOCX) |
| `src/Service/Playground/AcademicTranslationService.php` | AI-powered translation with personas |
| `src/Service/Playground/UsageMonitorService.php` | Credit tracking & usage limits |
| `src/Service/Playground/PlaygroundSubscriptionService.php` | Stripe webhook handling |
| `src/Entity/PlaygroundProject.php` | Project entity (title, content, keywords) |
| `src/Entity/PlaygroundChat.php` | Chat message entity |
| `src/Entity/PlaygroundNotebook.php` | Notebook cells entity |
| `src/Entity/PlaygroundTranslation.php` | Translation session entity |
| `src/Entity/PlaygroundSubscription.php` | User subscription & credits entity |
| `src/Entity/PlaygroundUsageLog.php` | Usage tracking entity |
| `src/EventListener/PlaygroundCreditsLoginListener.php` | Reset credits on login |
| `src/Command/ResetPlaygroundCreditsCommand.php` | Cron command for credit reset |
| `templates/playground/` | Twig templates for UI |
| `playground_prompts/` | Markdown files with AI prompts |
| `config/routes/playground.yaml` | Route configuration |
| `translations/Playground.*.yml` | Translation files (ar/en) |

---

### Core Services & Libraries

#### 1. AzureOpenAIService
Main service for interacting with Azure OpenAI API.

```php
namespace App\Service\Playground;

class AzureOpenAIService
{
    // Chat with AI - supports streaming
    public function chat(array $messages, float $temperature = 0.7, int $maxTokens = 2000): string;
    public function streamChat(array $messages, float $temperature = 0.7): StreamedResponse;
    
    // Text processing actions
    public function processSelectedText(string $text, string $action, string $language, ?string $question, ?string $targetLanguage): Response;
    
    // Specialized features
    public function generateKeywords(string $title, string $language = 'ar', int $count = 5): array;
    public function applyTashkeel(string $text): string;
    public function generateCitation(array $source, string $format = 'APA'): string;
    public function summarizeResearch(string $content, string $language, int $maxLength): string;
    public function answerAboutSelection(string $selectedText, string $question, string $language): string;
}
```

#### 2. PlaygroundRAGService
Retrieval-Augmented Generation service for multi-source search.

```php
namespace App\Service\Playground;

class PlaygroundRAGService
{
    public const SOURCE_SHAMRA = 'shamra';
    public const SOURCE_ARXIV = 'arxiv';
    public const SOURCE_WEB = 'web';
    
    // Search across all enabled sources
    public function search(string $query, array $sources = [], array $keywords = [], int $limit = 10): array;
    
    // Source-specific search
    public function searchShamra(string $query, array $keywords = [], int $limit = 10): array;
    public function searchArxiv(string $query, array $keywords = [], int $limit = 10): array;
    public function searchWeb(string $query, int $limit = 10): array;
    
    // Format results for AI context
    public function formatResultsForContext(array $results): string;
}
```

#### 3. PlaygroundProjectService
CRUD operations for projects and related entities.

```php
namespace App\Service\Playground;

class PlaygroundProjectService
{
    // Project management
    public function createProject(User $user, string $title, string $language = 'ar'): PlaygroundProject;
    public function getProject(int $projectId, User $user): ?PlaygroundProject;
    public function getUserProjects(User $user, int $limit = 50, int $offset = 0): array;
    public function deleteProject(PlaygroundProject $project): void;
    
    // Content updates
    public function updateContent(PlaygroundProject $project, string $content): PlaygroundProject;
    public function updateTitle(PlaygroundProject $project, string $title): PlaygroundProject;
    public function updateKeywords(PlaygroundProject $project, array $keywords): PlaygroundProject;
    
    // Chat management
    public function addChatMessage(PlaygroundProject $project, string $role, string $message, ?string $source, array $citations): PlaygroundChat;
    public function getChatHistory(PlaygroundProject $project, int $limit = 20): array;
    public function clearChatHistory(PlaygroundProject $project): void;
}
```

#### 4. DocumentParserService
Parse uploaded documents and extract content.

```php
namespace App\Service\Playground;

class DocumentParserService
{
    // Supported formats: txt, doc, docx
    public function parseDocument(UploadedFile $file): array;
    // Returns: ['content' => 'HTML string', 'metadata' => ['originalFormat' => 'docx', ...]]
}
```

#### 5. AcademicTranslationService
AI-powered translation with specialized academic personas.

```php
namespace App\Service\Playground;

class AcademicTranslationService
{
    // Persona constants: general, scientific, medical, legal, technical
    
    // Translate text with persona
    public function translate(string $text, string $sourceLanguage, string $targetLanguage, string $persona = 'general'): string;
    
    // Translate large documents in chunks
    public function translateLargeText(string $text, string $sourceLanguage, string $targetLanguage, string $persona, int $chunkSize = 2000): string;
    
    // Parse files for translation
    public function parsePdf(string $filePath): array;
    public function parseDocx(string $filePath): array;
    
    // Language detection
    public function detectLanguage(string $text): string;
}
```

#### PHP Packages Used

| Package | Purpose |
|---------|---------|
| `symfony/http-client` | HTTP client for Azure OpenAI API calls |
| `phpoffice/phpword` | DOCX file reading and generation |
| `dompdf/dompdf` | PDF generation from HTML content |
| `smalot/pdfparser` | PDF text extraction for translation |
| `elasticsearch/elasticsearch` | ElasticSearch client for Shamra Library search |

---

### API Endpoints

All API endpoints are prefixed with `/api/playground` and require authentication.

#### Chat & AI Processing

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/playground/chat` | Chat with AI assistant (supports streaming) |
| POST | `/api/playground/process-text` | Process text (rephrase, proofread, criticize, ask, translate) |
| POST | `/api/playground/selection` | Answer question about selected text |
| POST | `/api/playground/keywords` | Generate keywords from title |
| POST | `/api/playground/tashkeel` | Apply Arabic vowelization |
| POST | `/api/playground/citation` | Generate formatted citation |
| POST | `/api/playground/summarize` | Summarize content |
| POST | `/api/playground/search` | Search across RAG sources |
| POST | `/api/playground/chat/save` | Save chat message after streaming |
| POST | `/api/playground/generate/search` | Search Shamra for related research (Write with AI) |
| POST | `/api/playground/generate` | Generate academic content (Write with AI) |

#### Translation API

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/playground/translate` | Translate text with academic persona |
| POST | `/api/playground/translate/upload` | Upload file for translation (PDF, DOCX, TXT) |
| POST | `/api/playground/translate/save` | Save translation session to project |
| GET | `/api/playground/translate/history/{projectId}` | Get translation history for project |
| GET | `/api/playground/translate/{id}` | Get specific translation |
| GET | `/api/playground/translate/personas` | Get available translation personas |

#### Project Management

| Method | Endpoint | Description |
|--------|----------|-------------|
| GET | `/playground` | Main playground page |
| POST | `/playground/project/new` | Create new project |
| POST | `/playground/project/upload` | Upload document and create project |
| GET | `/playground/project/{id}` | Load project |
| POST | `/playground/project/{id}/save` | Save project content |
| DELETE | `/playground/project/{id}/delete` | Delete project |
| GET | `/playground/projects` | List user projects |
| POST | `/playground/project/{id}/keywords` | Update project keywords |
| POST | `/playground/project/{id}/chat/clear` | Clear chat history |
| POST | `/playground/project/{id}/download/{format}` | Download as DOCX or PDF |

#### Request/Response Examples

**Chat Request:**
```json
{
    "projectId": 123,
    "message": "What is the main contribution of this research?",
    "sources": ["shamra", "arxiv"],
    "stream": true,
    "language": "ar"
}
```

**Process Text Request:**
```json
{
    "text": "The text to process...",
    "action": "rephrase",
    "language": "ar",
    "question": null,
    "targetLanguage": null
}
```

**Available Actions for `/process-text`:**
- `rephrase` - Improve text clarity and flow
- `proofread` - Fix grammar and style issues
- `criticize` - Provide academic criticism
- `ask` - Answer a question (requires `question` field)
- `translate` - Translate text (requires `targetLanguage` field)

**Translation Request:**
```json
{
    "text": "النص المراد ترجمته...",
    "sourceLanguage": "ar",
    "targetLanguage": "en",
    "persona": "scientific",
    "projectId": 123,
    "autoDetect": false
}
```

**Available Translation Personas:**
- `general` - Standard academic translation
- `scientific` - Scientific terminology
- `medical` - Medical terminology
- `legal` - Legal terminology
- `technical` - Technical/engineering terminology

**Write with AI - Search Request:**
```json
{
    "title": "تأثير الذكاء الاصطناعي على التعليم العالي",
    "keywords": ["الذكاء الاصطناعي", "التعليم العالي", "التعلم الآلي"],
    "context": "optional selected text from editor",
    "limit": 10
}
```

**Write with AI - Generate Request:**
```json
{
    "projectId": 123,
    "title": "تأثير الذكاء الاصطناعي على التعليم العالي",
    "keywords": ["الذكاء الاصطناعي", "التعليم العالي", "التعلم الآلي"],
    "context": "optional selected text from editor",
    "instructions": "اكتب مقدمة عن أهمية الذكاء الاصطناعي في التعليم",
    "selectedSources": [
        {
            "title": "Research Paper Title",
            "authors": "Author Name",
            "year": "2024",
            "abstract": "Paper abstract...",
            "url": "https://..."
        }
    ],
    "language": "ar",
    "stream": true
}
```

**Write with AI - Refinement Request:**
```json
{
    "projectId": 123,
    "messageHistory": [
        {"role": "user", "content": "Previous instruction..."},
        {"role": "assistant", "content": "Generated content..."}
    ],
    "refinementRequest": "اجعل النص أكثر تفصيلاً وأضف المزيد من الأمثلة",
    "language": "ar",
    "stream": true
}
```

**Write with AI Feature Workflow:**
1. User clicks "Write with AI" button (requires 3+ keywords and title)
2. User enters optional writing instructions
3. System searches Shamra index for relevant research papers
4. User selects relevant sources to use as grounding data
5. AI generates academic content with citations
6. User can refine the content with follow-up instructions
7. User inserts the final content into the editor

---

### Environment Configuration

The Playground service requires the following environment variables in your `.env` file:

```env
# Azure OpenAI Configuration (Required)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_API_KEY=your-api-key-here
AZURE_OPENAI_DEPLOYMENT=gpt-4
AZURE_OPENAI_API_VERSION=2024-02-15-preview

# Debug Mode (Optional)
PLAYGROUND_DEBUG=false

# Web Search via SerpAPI (Optional - for RAG web search)
SERP_API_KEY=your-serp-api-key

# Stripe Playground Product IDs (Required for subscriptions)
STRIPE_PRODUCT_BASIC_MONTH=prod_xxxxxxxxxxxx
STRIPE_PRODUCT_BASIC_YEAR=prod_xxxxxxxxxxxx
STRIPE_STARTER_MONTH=prod_xxxxxxxxxxxx
STRIPE_STARTER_YEAR=prod_xxxxxxxxxxxx
STRIPE_RESEARCHER_MONTH=prod_xxxxxxxxxxxx
STRIPE_RESEARCHER_YEAR=prod_xxxxxxxxxxxx
```

#### Environment Variables Reference

| Variable | Required | Description | Example |
|----------|----------|-------------|---------|
| `AZURE_OPENAI_ENDPOINT` | Yes | Azure OpenAI resource endpoint URL | `https://myresource.openai.azure.com` |
| `AZURE_OPENAI_API_KEY` | Yes | API key for Azure OpenAI service | `abc123...` |
| `AZURE_OPENAI_DEPLOYMENT` | Yes | Model deployment name | `gpt-4`, `gpt-4-turbo`, `gpt-35-turbo` |
| `AZURE_OPENAI_API_VERSION` | Yes | API version for Azure OpenAI | `2024-02-15-preview` |
| `PLAYGROUND_DEBUG` | No | Enable debug logging to `var/log/playground_debug/` | `true` or `false` |
| `SERP_API_KEY` | No | SerpAPI key for web search in RAG | `abc123...` |
| `STRIPE_PRODUCT_BASIC_MONTH` | Yes | Stripe product ID for Basic monthly | `prod_Twxxxxxx` |
| `STRIPE_PRODUCT_BASIC_YEAR` | Yes | Stripe product ID for Basic yearly | `prod_Twxxxxxx` |
| `STRIPE_STARTER_MONTH` | Yes | Stripe product ID for Starter monthly | `prod_Twxxxxxx` |
| `STRIPE_STARTER_YEAR` | Yes | Stripe product ID for Starter yearly | `prod_Twxxxxxx` |
| `STRIPE_RESEARCHER_MONTH` | Yes | Stripe product ID for Researcher monthly | `prod_Twxxxxxx` |
| `STRIPE_RESEARCHER_YEAR` | Yes | Stripe product ID for Researcher yearly | `prod_Twxxxxxx` |

#### Service Configuration (`config/services.yaml`)

```yaml
parameters:
    # Azure OpenAI parameters for Playground
    azure_openai_endpoint: "%env(AZURE_OPENAI_ENDPOINT)%"
    azure_openai_api_key: "%env(AZURE_OPENAI_API_KEY)%"
    azure_openai_deployment: "%env(AZURE_OPENAI_DEPLOYMENT)%"
    azure_openai_api_version: "%env(AZURE_OPENAI_API_VERSION)%"
    playground_debug: "%env(bool:PLAYGROUND_DEBUG)%"

services:
    App\Service\Playground\AzureOpenAIService:
        arguments:
            $projectDir: '%kernel.project_dir%'
            $debug: '%playground_debug%'
    
    App\Service\Playground\UsageMonitorService:
        arguments:
            $em: '@doctrine.orm.entity_manager'
            $usageLogRepo: '@App\Repository\PlaygroundUsageLogRepository'
            $subscriptionRepo: '@App\Repository\PlaygroundSubscriptionRepository'
            $logger: '@logger'

    App\Service\Playground\PlaygroundSubscriptionService:
        public: true
        arguments:
            $em: '@doctrine.orm.entity_manager'
            $repo: '@App\Repository\PlaygroundSubscriptionRepository'
            $logger: '@logger'
            $basicMonthProduct: '%stripe_basic_month%'
            $basicYearProduct: '%stripe_basic_year%'
            $starterMonthProduct: '%stripe_starter_month%'
            $starterYearProduct: '%stripe_starter_year%'
            $researcherMonthProduct: '%stripe_researcher_month%'
            $researcherYearProduct: '%stripe_researcher_year%'
```

#### Stripe Parameters (`config/packages/framework.yaml`)

```yaml
parameters:
    stripe_basic_month: '%env(STRIPE_PRODUCT_BASIC_MONTH)%'
    stripe_basic_year: '%env(STRIPE_PRODUCT_BASIC_YEAR)%'
    stripe_starter_month: '%env(STRIPE_STARTER_MONTH)%'
    stripe_starter_year: '%env(STRIPE_STARTER_YEAR)%'
    stripe_researcher_month: '%env(STRIPE_RESEARCHER_MONTH)%'
    stripe_researcher_year: '%env(STRIPE_RESEARCHER_YEAR)%'
```

---

### AI Prompt Templates

The Playground uses customizable prompt templates stored in `playground_prompts/` directory as Markdown files:

| File | Purpose |
|------|---------|
| `rephrase_prompt.md` | Improve text clarity while preserving meaning |
| `proofread_prompt.md` | Grammar, spelling, and style corrections |
| `criticize_prompt.md` | Constructive academic criticism |
| `ask_prompt.md` | Answer questions about text context |
| `translate_prompt.md` | Academic translation with terminology preservation |

#### Prompt File Structure

Each prompt file follows this structure:

```markdown
# Action Prompt

## Purpose
Brief description of what this prompt does.

## System Prompt
```
The system prompt that sets AI behavior and guidelines...
```

## User Prompt
```
The user prompt template with {{placeholders}}...
```

## Variables
| Variable | Description | Required |
|----------|-------------|----------|
| `{{text}}` | The text to process | Yes |
```

#### Available Placeholders

- `{{text}}` - The selected/input text
- `{{question}}` - User's question (for ask action)
- `{{target_language}}` - Target language name (for translate action)

---

### Database Schema

The Playground uses the following database entities:

#### PlaygroundProject

Main project entity storing user research documents.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `user_id` | int | FK to User |
| `title` | varchar(500) | Project title |
| `content` | text | HTML content |
| `keywords` | json | Array of keywords |
| `language` | varchar(10) | Project language (ar/en) |
| `status` | varchar(50) | draft, processing, complete, error |
| `is_public` | boolean | Public visibility |
| `settings` | json | User settings |
| `created_at` | datetime | Creation timestamp |
| `updated_at` | datetime | Last update timestamp |

#### PlaygroundChat

Chat messages for project AI conversations.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `project_id` | int | FK to PlaygroundProject |
| `role` | varchar(20) | user, assistant, system |
| `message` | text | Message content |
| `source` | varchar(50) | shamra, arxiv, web, general |
| `citations` | json | Array of source citations |
| `metadata` | json | Additional metadata |
| `created_at` | datetime | Creation timestamp |

#### PlaygroundNotebook

Notebook cells for structured content.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `project_id` | int | FK to PlaygroundProject |
| `content` | text | Cell content |
| `type` | varchar(50) | text, ai_response, citation, heading |
| `position` | int | Order position |
| `metadata` | json | Additional metadata |
| `created_at` | datetime | Creation timestamp |
| `updated_at` | datetime | Last update timestamp |

#### PlaygroundTranslation

Translation sessions linked to projects.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `project_id` | int | FK to PlaygroundProject |
| `source_text` | text | Original text |
| `target_text` | text | Translated text |
| `source_language` | varchar(10) | Source language code |
| `target_language` | varchar(10) | Target language code |
| `persona` | varchar(50) | Translation persona used |
| `status` | varchar(50) | Translation status |
| `created_at` | datetime | Creation timestamp |

#### PlaygroundSubscription

User subscription and credit tracking for Playground tiers.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `user_id` | int | FK to User |
| `tier` | varchar(50) | basic, starter, researcher |
| `stripe_subscription_id` | varchar(255) | Stripe subscription ID |
| `subscription_start` | datetime | When subscription began |
| `subscription_end` | datetime | When subscription expires |
| `is_active` | boolean | Active status |
| `monthly_credits` | int | Total credits per month |
| `used_credits` | int | Credits used this period |
| `daily_limit` | int | Max requests per day |
| `daily_requests` | int | Requests made today |
| `last_reset_date` | date | When credits were last reset |
| `created_at` | datetime | Record creation |
| `updated_at` | datetime | Last update |

#### PlaygroundUsageLog

Detailed usage tracking for analytics and auditing.

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `user_id` | int | FK to User |
| `project_id` | int | FK to PlaygroundProject (nullable) |
| `action` | varchar(50) | Action type (chat, rephrase, translate, etc.) |
| `input_tokens` | int | Tokens in the request |
| `output_tokens` | int | Tokens in the response |
| `status` | varchar(20) | success, error, rate_limited |
| `error_message` | text | Error details if failed |
| `metadata` | json | Additional context |
| `created_at` | datetime | Request timestamp |

---

### Subscription & Access Control

The Playground service has its own tiered subscription system separate from the main Shamra subscription. Access is controlled at both the controller and API levels.

#### Playground Subscription Tiers

| Tier | Price (Monthly) | Price (Yearly) | Monthly Credits | Daily Limit | Features |
|------|-----------------|----------------|-----------------|-------------|----------|
| **Basic** | $4.99 | $49.99 | 0 | 0 | PDF downloads only |
| **Starter** | $9.00 | $99.00 | 500 | 50 | AI Writing, Rephrase, Proofread |
| **Researcher** | $19.00 | $199.00 | 1,500 | 150 | All features + Translation |

**Note:** Annual subscriptions save ~17% compared to monthly billing.

#### Subscription Features Matrix

| Feature | Basic | Starter | Researcher |
|---------|-------|---------|------------|
| PDF Downloads | ✅ | ✅ | ✅ |
| AI-Powered Writing | ❌ | ✅ | ✅ |
| Rephrase & Proofread | ❌ | ✅ | ✅ |
| Research Chat | ❌ | ✅ | ✅ |
| Multi-Source RAG | ❌ | ✅ | ✅ |
| Academic Translation | ❌ | ❌ | ✅ |
| Priority Support | ❌ | ❌ | ✅ |

#### Stripe Integration

The Playground uses Stripe for payment processing. Configure your Stripe product IDs in `.env`:

```env
# Stripe Playground Product IDs
STRIPE_PRODUCT_BASIC_MONTH=prod_xxxxxxxxxxxx
STRIPE_PRODUCT_BASIC_YEAR=prod_xxxxxxxxxxxx
STRIPE_STARTER_MONTH=prod_xxxxxxxxxxxx
STRIPE_STARTER_YEAR=prod_xxxxxxxxxxxx
STRIPE_RESEARCHER_MONTH=prod_xxxxxxxxxxxx
STRIPE_RESEARCHER_YEAR=prod_xxxxxxxxxxxx
```

These are configured in `config/packages/framework.yaml`:

```yaml
parameters:
    stripe_basic_month: '%env(STRIPE_PRODUCT_BASIC_MONTH)%'
    stripe_basic_year: '%env(STRIPE_PRODUCT_BASIC_YEAR)%'
    stripe_starter_month: '%env(STRIPE_STARTER_MONTH)%'
    stripe_starter_year: '%env(STRIPE_STARTER_YEAR)%'
    stripe_researcher_month: '%env(STRIPE_RESEARCHER_MONTH)%'
    stripe_researcher_year: '%env(STRIPE_RESEARCHER_YEAR)%'
```

---

### Usage Monitoring & Credit System

The Playground implements a comprehensive usage tracking and credit management system to enforce subscription limits and provide analytics.

#### PlaygroundSubscription Entity

Tracks user subscription status and credit balance:

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `user_id` | int | FK to User |
| `tier` | varchar(50) | basic, starter, researcher |
| `stripe_subscription_id` | varchar(255) | Stripe subscription ID |
| `subscription_start` | datetime | When subscription started |
| `subscription_end` | datetime | When subscription expires |
| `is_active` | boolean | Subscription active status |
| `monthly_credits` | int | Total credits per month |
| `used_credits` | int | Credits used this period |
| `daily_limit` | int | Max requests per day |
| `daily_requests` | int | Requests made today |
| `last_reset_date` | date | When credits were last reset |
| `created_at` | datetime | Record creation |
| `updated_at` | datetime | Last update |

#### PlaygroundUsageLog Entity

Records every AI request for analytics and auditing:

| Column | Type | Description |
|--------|------|-------------|
| `id` | int | Primary key |
| `user_id` | int | FK to User |
| `project_id` | int | FK to PlaygroundProject (nullable) |
| `action` | varchar(50) | Action type (chat, rephrase, translate, etc.) |
| `input_tokens` | int | Tokens in the request |
| `output_tokens` | int | Tokens in the response |
| `status` | varchar(20) | success, error, rate_limited |
| `error_message` | text | Error details if failed |
| `metadata` | json | Additional context |
| `created_at` | datetime | Request timestamp |

#### UsageMonitorService

The `UsageMonitorService` handles all credit and limit enforcement:

```php
namespace App\Service\Playground;

class UsageMonitorService
{
    // Check if user can make a request
    public function canUserMakeRequest(User $user, string $action = 'chat'): bool;
    
    // Deduct credits after successful request
    public function deductCredits(User $user, string $action, int $tokens = 1): bool;
    
    // Log usage for analytics
    public function logUsage(User $user, string $action, int $inputTokens, int $outputTokens, string $status, ?int $projectId = null, ?string $error = null): PlaygroundUsageLog;
    
    // Get usage statistics
    public function getUserUsageStats(User $user): array;
    public function getUserDailyStats(User $user, int $days = 30): array;
    
    // Admin functions
    public function getGlobalUsageStats(\DateTime $from, \DateTime $to): array;
}
```

#### Credit Costs by Action

| Action | Credit Cost | Description |
|--------|-------------|-------------|
| `chat` | 1 | Chat with AI assistant |
| `rephrase` | 1 | Rephrase selected text |
| `proofread` | 1 | Proofread text |
| `criticize` | 1 | Academic criticism |
| `ask` | 1 | Answer about selection |
| `translate` | 2 | Translation (premium) |
| `generate` | 3 | Write with AI (content generation) |
| `tashkeel` | 1 | Arabic vowelization |
| `keywords` | 1 | Generate keywords |

#### API Credit Check Integration

All API endpoints check credits before processing:

```php
// In PlaygroundAPIController
public function chat(Request $request, UsageMonitorService $usageMonitor): Response
{
    $user = $this->getUser();
    
    // Check if user has credits
    if (!$usageMonitor->canUserMakeRequest($user, 'chat')) {
        return new JsonResponse([
            'error' => 'credit_exhausted',
            'message' => 'You have used all your monthly credits.',
            'usage' => $usageMonitor->getUserUsageStats($user)
        ], 429);
    }
    
    // Process request...
    
    // Deduct credits on success
    $usageMonitor->deductCredits($user, 'chat', $outputTokens);
    $usageMonitor->logUsage($user, 'chat', $inputTokens, $outputTokens, 'success', $projectId);
    
    return $response;
}
```

---

### Monthly Credit Reset System

The Playground implements a three-layer approach to ensure credits are properly reset at the start of each billing period.

#### 1. Lazy Reset on API Calls

When a user makes an API request, the system checks if credits need resetting:

```php
// In UsageMonitorService::canUserMakeRequest()
$subscription = $this->subscriptionRepo->findOneBy(['user' => $user, 'isActive' => true]);

if ($subscription && $subscription->needsCreditsReset()) {
    $subscription->checkAndResetCredits();
    $this->em->flush();
}
```

#### 2. Reset on Login

Credits are checked and reset when users log in:

```php
// src/EventListener/PlaygroundCreditsLoginListener.php
class PlaygroundCreditsLoginListener implements EventSubscriberInterface
{
    public static function getSubscribedEvents(): array
    {
        return [LoginSuccessEvent::class => 'onLoginSuccess'];
    }

    public function onLoginSuccess(LoginSuccessEvent $event): void
    {
        $user = $event->getUser();
        $subscription = $this->subscriptionRepo->findOneBy(['user' => $user, 'isActive' => true]);
        
        if ($subscription && $subscription->needsCreditsReset()) {
            $subscription->resetCredits();
            $this->em->flush();
            $this->logger->info('Reset playground credits on login', [
                'user_id' => $user->getId(),
                'tier' => $subscription->getTier()
            ]);
        }
    }
}
```

#### 3. Cron Job (Daily)

A scheduled command resets credits for users who haven't logged in:

```bash
# Add to crontab - runs daily at midnight
0 0 * * * cd /var/www/academia_v2 && php bin/console app:reset-playground-credits
```

**Command Options:**
```bash
# Normal run
php bin/console app:reset-playground-credits

# Dry run (preview only)
php bin/console app:reset-playground-credits --dry-run
```

#### Credit Reset Logic

The `PlaygroundSubscription` entity handles the reset logic:

```php
// src/Entity/PlaygroundSubscription.php

public function needsCreditsReset(): bool
{
    if (!$this->lastResetDate) {
        return true;
    }
    
    $now = new \DateTime();
    // Check if we're in a new month since last reset
    return $now->format('Y-m') !== $this->lastResetDate->format('Y-m');
}

public function checkAndResetCredits(): bool
{
    if ($this->needsCreditsReset()) {
        $this->resetCredits();
        return true;
    }
    return false;
}

public function resetCredits(): void
{
    $this->usedCredits = 0;
    $this->dailyRequests = 0;
    $this->lastResetDate = new \DateTime();
    $this->updatedAt = new \DateTime();
}
```

---

### Stripe Webhook Integration

The Playground integrates with Stripe webhooks to handle subscription lifecycle events automatically.

#### Webhook Events Handled

| Event | Action |
|-------|--------|
| `invoice.payment_succeeded` | Activate/renew subscription, reset credits if renewal |
| `invoice.payment_failed` | Deactivate subscription |
| `customer.subscription.deleted` | Cancel subscription |

#### PlaygroundSubscriptionService

Handles all Stripe webhook events for Playground subscriptions:

```php
// src/Service/Playground/PlaygroundSubscriptionService.php

class PlaygroundSubscriptionService
{
    // Check if product ID belongs to Playground
    public function isPlaygroundProduct(string $productId): bool;
    
    // Handle successful payment (new subscription or renewal)
    public function handlePaymentSucceeded(
        string $stripeSubscriptionId,
        string $customerEmail,
        string $productId,
        \DateTime $periodEnd,
        bool $isRenewal = false
    ): bool;
    
    // Handle failed payment
    public function handlePaymentFailed(string $stripeSubscriptionId): bool;
    
    // Handle subscription cancellation
    public function handleSubscriptionCancelled(string $stripeSubscriptionId): bool;
    
    // Utility methods
    public function isSubscriptionValid(User $user): bool;
    public function getUserTier(User $user): ?string;
}
```

#### Webhook Handler

The webhook endpoint at `/webhook` handles both legacy and Playground subscriptions:

```php
// src/syndex/AcademicBundle/Controller/HomepageController.php

public function webhookAction(Request $request): JsonResponse
{
    // Validate Stripe signature...
    
    switch ($event->type) {
        case 'invoice.payment_succeeded':
            $productId = $event->data->object->lines->data[0]->price->product;
            $billingReason = $event->data->object->billing_reason;
            $isRenewal = in_array($billingReason, ['subscription_cycle', 'subscription_update']);
            
            // Check if Playground subscription
            if ($playgroundService->isPlaygroundProduct($productId)) {
                $playgroundService->handlePaymentSucceeded(
                    $subscriptionId,
                    $customerEmail,
                    $productId,
                    $periodEnd,
                    $isRenewal  // Reset credits if renewal
                );
            }
            break;
            
        case 'invoice.payment_failed':
            $playgroundService->handlePaymentFailed($subscriptionId);
            break;
            
        case 'customer.subscription.deleted':
            $playgroundService->handleSubscriptionCancelled($subscriptionId);
            break;
    }
}
```

#### Stripe Webhook Configuration

Configure these events in your Stripe Dashboard → Developers → Webhooks:

1. `invoice.payment_succeeded`
2. `invoice.payment_failed`
3. `customer.subscription.deleted`

Webhook URL: `https://shamra-academia.com/webhook`

#### Renewal Detection

The system detects renewals via the `billing_reason` field:

| Billing Reason | Type | Credit Action |
|----------------|------|---------------|
| `subscription_create` | New subscription | Initialize credits |
| `subscription_cycle` | Monthly/yearly renewal | Reset credits |
| `subscription_update` | Plan change | Reset credits |
| `manual` | Manual invoice | No reset |

---

### Admin Dashboard for Playground

The Playground has a dedicated admin dashboard at `/jim19ud83/playground` for monitoring usage and managing subscriptions.

#### Dashboard Features

1. **Usage Statistics**
   - Total API calls (today, this month, all time)
   - Credits consumed by tier
   - Most active users
   - Error rates and common issues

2. **Subscription Management**
   - Active subscriptions by tier
   - Revenue tracking
   - Subscription lifecycle (new, renewed, cancelled)
   - Manual subscription override for support

3. **Cost Monitoring**
   - Azure OpenAI token consumption
   - Estimated costs per user/tier
   - Daily/monthly cost trends

#### Admin Routes

| Route | Description |
|-------|-------------|
| `/jim19ud83/playground` | Main dashboard |
| `/jim19ud83/playground/users` | User subscriptions list |
| `/jim19ud83/playground/usage` | Usage analytics |
| `/jim19ud83/playground/costs` | Cost monitoring |

---

### Subscription Check Logic

```php
// From PlaygroundController
private function isUserSubscribed(): bool
{
    $user = $this->getUser();
    if (!$user) {
        return false;
    }
    
    // Allow admins to access without subscription
    if ($this->isGranted('ROLE_ADMIN') || $this->isGranted('ROLE_SUPER_ADMIN')) {
        return true;
    }
    
    return $this->subscribeService->isSubscribed($user);
}
```

#### Access Flow

1. **Authentication Check**: User must be logged in
2. **Subscription Check**: User must have an active subscription
3. **Admin Override**: Users with `ROLE_ADMIN` or `ROLE_SUPER_ADMIN` bypass subscription requirement

#### Admin Override

Administrators automatically have access to the Playground service regardless of subscription status:

```php
// In PlaygroundAPIController::checkAccess()
private function checkAccess(): bool
{
    $user = $this->getUser();
    
    if (!$user) {
        return false;
    }
    
    // Allow admin users to bypass subscription check
    if (in_array('ROLE_ADMIN', $user->getRoles()) || in_array('ROLE_SUPER_ADMIN', $user->getRoles())) {
        return true;
    }
    
    return $this->subscribeService->isSubscribed($user);
}
```

#### Non-Subscribed User Flow

When a non-subscribed user accesses `/playground`:

1. User is redirected to `playground/subscribe_required.html.twig`
2. Template displays subscription requirement message
3. Link to subscription page is provided based on locale (Arabic/English)

```php
// From PlaygroundController::index()
if (!$this->isUserSubscribed()) {
    $locale = $request->getLocale();
    $subscribeRoute = $locale === 'ar' ? 'shamra_academia_subscription.ar' : 'shamra_academia_subscription.en';
    return $this->render('playground/subscribe_required.html.twig', [
        'subscribeUrl' => $this->generateUrl($subscribeRoute)
    ]);
}
```

#### Subscription Service Integration

The Playground uses `SubscribeService` from the AcademicBundle:

```php
use App\syndex\AcademicBundle\Service\SubscribeService;

// Check if user has active subscription
$subscribeService->isSubscribed($user);
```

#### Granting Admin Access

To grant a user admin access for Playground bypass:

```bash
# Using Symfony console
php bin/console fos:user:promote username ROLE_ADMIN

# Or for super admin
php bin/console fos:user:promote username ROLE_SUPER_ADMIN
```

#### Summary Table

| User Role | Subscription Status | Access Granted |
|-----------|---------------------|----------------|
| Anonymous | N/A | ❌ Redirected to login |
| User | No subscription | ❌ Shown subscription required page |
| User | Active subscription | ✅ Full access |
| ROLE_ADMIN | Any | ✅ Full access (bypass) |
| ROLE_SUPER_ADMIN | Any | ✅ Full access (bypass) |

---

### Grounding Data for AI Writing

The "Write with AI" feature uses Retrieval-Augmented Generation (RAG) to ground AI-generated content in actual research papers from the Shamra Academia index. This ensures the generated academic content is factually accurate and properly cited.

#### How It Works

1. **User Input**: User provides project title, keywords (minimum 3), and optional writing instructions
2. **Source Search**: System searches Shamra's ElasticSearch index for related research papers
3. **Source Selection**: User selects relevant papers to use as grounding data
4. **Content Generation**: AI generates academic content using selected sources with proper citations
5. **Refinement**: User can iteratively refine the generated content

#### Shamra Index Search Implementation

The `PlaygroundRAGService` handles searching the internal Shamra research index:

```php
// src/Service/Playground/PlaygroundRAGService.php

public function searchShamra(string $query, array $keywords = [], int $limit = 10): array
{
    // Build ElasticSearch query
    $searchQuery = [
        'query' => [
            'bool' => [
                'must' => [
                    ['term' => ['deleted' => false]]  // Exclude deleted research
                ],
                'should' => [
                    [
                        'multi_match' => [
                            'query' => $query,
                            'fields' => [
                                'arabic_full_title^3',    // Title weighted 3x
                                'english_full_title^3',
                                'arabic_abstract^2',      // Abstract weighted 2x
                                'english_abstract^2',
                                'authors',
                                'tag',
                                'ai_keywords'
                            ],
                            'type' => 'best_fields',
                            'fuzziness' => 'AUTO'
                        ]
                    ]
                ],
                'minimum_should_match' => 1
            ]
        ]
    ];

    // Add keyword matching for better relevance
    if (!empty($keywords)) {
        foreach ($keywords as $keyword) {
            $searchQuery['query']['bool']['should'][] = [
                'match_phrase' => ['tag' => $keyword]
            ];
            $searchQuery['query']['bool']['should'][] = [
                'match_phrase' => ['ai_keywords' => $keyword]
            ];
        }
    }

    // Search both Arabic and English indices
    $arabicResults = $this->elasticsearch->search($searchQuery, $arabicIndex);
    $englishResults = $this->elasticsearch->search($searchQuery, $englishIndex);
    
    // Combine, sort by score, and return top results
    return array_slice($combinedResults, 0, $limit);
}
```

#### ElasticSearch Indices Used

Configure the index names in your `.env` file:

```env
ELASTIC_ARABIC_RESEARCH_INDEX=arabic_research_test
ELASTIC_ENGLISH_RESEARCH_INDEX=english_research
```

| Index | Environment Variable | Symfony Parameter | Description |
|-------|---------------------|-------------------|-------------|
| Arabic Research | `ELASTIC_ARABIC_RESEARCH_INDEX` | `elastic_arabic_research_index` | Arabic language research papers |
| English Research | `ELASTIC_ENGLISH_RESEARCH_INDEX` | `elastic_english_research_index` | English language research papers |

#### Search Fields and Weights

| Field | Weight | Description |
|-------|--------|-------------|
| `arabic_full_title` | 3x | Arabic title (highest priority) |
| `english_full_title` | 3x | English title (highest priority) |
| `arabic_abstract` | 2x | Arabic abstract |
| `english_abstract` | 2x | English abstract |
| `authors` | 1x | Author names |
| `tag` | 1x | Research tags/keywords |
| `ai_keywords` | 1x | AI-extracted keywords |

#### Search Result Structure

Each search result contains:

```php
[
    'id' => 'elastic_doc_id',
    'title' => 'Research paper title',
    'abstract' => 'Paper abstract...',
    'authors' => 'Author Name(s)',
    'slug' => 'url-friendly-slug',
    'year' => 2024,
    'source' => 'shamra',
    'language' => 'ar',  // or 'en'
    'url' => '/show/url-friendly-slug',
    'score' => 15.234  // ElasticSearch relevance score
]
```

#### API Endpoint for Search

```
POST /api/playground/generate/search
```

**Request:**
```json
{
    "title": "Research project title",
    "keywords": ["keyword1", "keyword2", "keyword3"],
    "context": "Optional selected text from editor",
    "limit": 10,
    "searchShamra": true,
    "searchArxiv": false
}
```

**Response:**
```json
{
    "success": true,
    "results": [
        {
            "id": "123",
            "title": "Paper title",
            "abstract": "Paper abstract...",
            "authors": "Author Name",
            "year": 2024,
            "source": "shamra",
            "score": 15.234
        }
    ],
    "total": 10,
    "sources": {
        "shamra": true,
        "arxiv": false
    }
}
```

#### Integration with AI Content Generation

Once sources are selected, they are formatted and passed to the AI model:

```php
// Format sources for AI prompt
private function formatSourcesForPrompt(array $sources): string
{
    $formatted = [];
    foreach ($sources as $index => $source) {
        $num = $index + 1;
        $entry = "**Source {$num}:**\n";
        $entry .= "- Title: " . ($source['title'] ?? 'Unknown') . "\n";
        $entry .= "- Authors: " . ($source['authors'] ?? 'Unknown') . "\n";
        if (!empty($source['year'])) {
            $entry .= "- Year: " . $source['year'] . "\n";
        }
        $entry .= "- Abstract: " . ($source['abstract'] ?? 'No abstract') . "\n";
        $formatted[] = $entry;
    }
    return implode("\n", $formatted);
}
```

The AI is instructed to cite sources using `[Author, Year]` format when using information from the provided research papers.

---

### Security Considerations

The Playground service implements multiple security layers to protect against common attack vectors. This section documents the current security measures and recommendations.

#### 1. Authentication & Authorization

**Current Implementation:**
- ✅ All endpoints require user authentication via `isUserAuthenticated()`
- ✅ Subscription verification on every request via `isUserSubscribed()` and `checkAccess()`
- ✅ Project ownership checks using `isOwner()` before modifications
- ✅ Admin bypass properly checks `ROLE_ADMIN` and `ROLE_SUPER_ADMIN`
- ✅ CSRF token generated and passed to frontend via `csrf_token('playground')`

**Code Example:**
```php
// Project access with ownership check
$project = $this->projectService->getProject($id, $this->getUser());
if (!$project || !$this->projectService->isOwner($project, $this->getUser())) {
    return new JsonResponse(['error' => 'Project not found'], 404);
}
```

#### 2. Input Validation & Sanitization

**Current Implementation:**
- ✅ Required field validation on all API endpoints
- ✅ Action whitelist validation for process-text endpoint
- ✅ Language parameter validation
- ✅ Numeric parameter bounds checking (e.g., `min(10, max(1, $data['count'] ?? 5))`)
- ✅ HTML escaping in document parser via `htmlspecialchars()`

**Validated Parameters:**
| Endpoint | Validations |
|----------|-------------|
| `/chat` | message required, sources array, stream boolean |
| `/process-text` | text required, action whitelist validation |
| `/keywords` | title required, count bounded 1-10 |
| `/search` | query required, limit bounded 1-20 |
| `/translate` | text required, persona whitelist |

#### 3. File Upload Security

**Current Implementation:**
- ✅ File extension whitelist: `txt`, `doc`, `docx` (project upload), `pdf`, `docx`, `txt` (translation)
- ✅ MIME type validation
- ✅ File size limits: 25MB for projects, 10MB for translations
- ✅ Uploaded files deleted immediately after parsing
- ✅ Files processed in temp directory, never stored permanently

**File Validation Code:**
```php
// Validate file size (25MB max)
$maxSize = 25 * 1024 * 1024;
if ($file->getSize() > $maxSize) {
    return new JsonResponse(['error' => 'File too large'], 400);
}

// Validate file type
$allowedExtensions = ['txt', 'doc', 'docx'];
$extension = strtolower($file->getClientOriginalExtension());
if (!in_array($extension, $allowedExtensions)) {
    return new JsonResponse(['error' => 'Invalid file type'], 400);
}
```

#### 4. XSS Protection

**Current Implementation:**
- ✅ Server-side HTML escaping in document parser using `htmlspecialchars()`
- ✅ Twig auto-escaping enabled by default
- ✅ JSON encoding for data passed to JavaScript
- ⚠️ Client-side `innerHTML` usage with `formatMessage()` - uses basic regex replacement

**Recommendations:**
1. Add DOMPurify for client-side HTML sanitization
2. Implement Content Security Policy (CSP) headers
3. Review `formatMessage()` function for potential XSS vectors

**Suggested Fix for Client-Side:**
```javascript
// Add to playground.js
<script src="https://cdnjs.cloudflare.com/ajax/libs/dompurify/3.0.6/purify.min.js"></script>

const formatMessage = (message) => {
    if (!message) return '';
    let formatted = message
        .replace(/\*\*(.*?)\*\*/g, '<strong>$1</strong>')
        .replace(/\*(.*?)\*/g, '<em>$1</em>')
        .replace(/\n/g, '<br>');
    return DOMPurify.sanitize(formatted);
};
```

#### 5. CSRF Protection

**Current Implementation:**
- ✅ CSRF token generated in Twig template: `csrf_token('playground')`
- ✅ Token passed to Vue app for API requests
- ⚠️ CSRF token not currently validated on API endpoints

**Recommendation:** Add CSRF validation to API endpoints:
```php
use Symfony\Component\Security\Csrf\CsrfTokenManagerInterface;

// In controller
if (!$this->isCsrfTokenValid('playground', $request->headers->get('X-CSRF-Token'))) {
    return new JsonResponse(['error' => 'Invalid CSRF token'], 403);
}
```

#### 6. Rate Limiting

**Current Implementation:**
- ✅ Rate limiting configured for login attempts (4 attempts per 60 minutes)
- ⚠️ No rate limiting on Playground API endpoints

**Recommendation:** Add rate limiting for AI endpoints to prevent abuse:
```yaml
# config/packages/rate_limiter.yaml
framework:
    rate_limiter:
        playground_api:
            policy: 'sliding_window'
            limit: 100
            interval: '1 hour'
        playground_ai:
            policy: 'sliding_window'
            limit: 50
            interval: '1 hour'
```

```php
// In controller
public function __construct(
    RateLimiterFactory $playgroundLimiter
) {
    $this->limiter = $playgroundLimiter;
}

public function chat(Request $request): Response
{
    $limiter = $this->limiter->create($this->getUser()->getUserIdentifier());
    if (!$limiter->consume(1)->isAccepted()) {
        return new JsonResponse(['error' => 'Rate limit exceeded'], 429);
    }
    // ... rest of method
}
```

#### 7. API Key & Secrets Management

**Current Implementation:**
- ✅ All API keys stored in environment variables
- ✅ Keys loaded via Symfony ParameterBag
- ✅ No API keys exposed in templates or client-side code
- ✅ Azure OpenAI key sent via `api-key` header (not in URL)

**Environment Variables:**
- `AZURE_OPENAI_API_KEY` - Azure OpenAI authentication
- `SERP_API_KEY` - SerpAPI for web search (optional)
- Keys never logged or exposed in error messages

#### 8. AI Prompt Injection Mitigation

**Current Implementation:**
- ✅ System prompts defined in separate markdown files
- ✅ Clear separation between system instructions and user content
- ✅ Prompt templates use placeholder replacement, not string concatenation
- ⚠️ User input directly included in prompts without sanitization

**Potential Risks:**
- Users could craft inputs that attempt to override system prompts
- Malicious text could attempt to extract system instructions

**Recommendations:**
1. Add input length limits to prevent prompt overflow attacks
2. Implement output filtering for sensitive content
3. Consider adding a prompt injection detection layer

**Suggested Implementation:**
```php
// Add input length validation
private function validateTextLength(string $text, int $maxLength = 50000): void
{
    if (strlen($text) > $maxLength) {
        throw new \InvalidArgumentException("Text exceeds maximum length of {$maxLength} characters");
    }
}

// Add prompt injection pattern detection
private function detectPromptInjection(string $input): bool
{
    $patterns = [
        '/ignore\s+(previous|above|all)\s+instructions/i',
        '/system\s*:\s*/i',
        '/\[\[system\]\]/i',
        '/you\s+are\s+now\s+/i',
    ];
    foreach ($patterns as $pattern) {
        if (preg_match($pattern, $input)) {
            return true;
        }
    }
    return false;
}
```

#### 9. Error Handling & Information Disclosure

**Current Implementation:**
- ⚠️ Exception messages returned directly to client in many endpoints
- ⚠️ Debug logging with `error_log()` includes sensitive info (should be removed in production)
- ✅ Generic error messages for authentication failures

**Current Issue:**
```php
// This exposes internal error details
return new JsonResponse(['error' => $e->getMessage()], 500);
```

**Recommendations:**
1. Log detailed errors server-side, return generic messages to clients
2. Remove `error_log()` debug statements for production
3. Implement proper error logging service

**Suggested Fix:**
```php
try {
    // ... operation
} catch (\Exception $e) {
    $this->logger->error('Operation failed', [
        'exception' => $e,
        'user' => $this->getUser()?->getUserIdentifier(),
        'action' => 'chat'
    ]);
    return new JsonResponse([
        'error' => 'An error occurred. Please try again.',
        'code' => 'INTERNAL_ERROR'
    ], 500);
}
```

#### 10. Security Checklist

| Security Measure | Status | Priority |
|------------------|--------|----------|
| Authentication on all endpoints | ✅ Implemented | - |
| Authorization/ownership checks | ✅ Implemented | - |
| Input validation | ✅ Implemented | - |
| File upload restrictions | ✅ Implemented | - |
| CSRF token generation | ✅ Implemented | - |
| CSRF token validation | ⚠️ Partial | High |
| XSS protection (server) | ✅ Implemented | - |
| XSS protection (client) | ⚠️ Needs DOMPurify | Medium |
| Rate limiting on API | ❌ Not implemented | High |
| Prompt injection mitigation | ⚠️ Partial | Medium |
| Error message sanitization | ⚠️ Needs improvement | Medium |
| Debug logging removal | ⚠️ Needs cleanup | High |
| CSP headers | ❌ Not implemented | Medium |
| SQL injection protection | ✅ Doctrine ORM | - |

#### 11. Production Security Hardening

Before deploying to production, complete the following:

1. **Remove debug logging:**
   ```php
   // Remove these from PlaygroundAPIController
   error_log("PlaygroundAPI checkAccess - User: " ...);
   ```

2. **Set PLAYGROUND_DEBUG=false** in production `.env`

3. **Add rate limiting** to all AI endpoints

4. **Implement CSRF validation** on POST/DELETE endpoints

5. **Add DOMPurify** for client-side HTML sanitization

6. **Configure CSP headers** in `config/packages/framework.yaml`:
   ```yaml
   framework:
       headers:
           Content-Security-Policy: "default-src 'self'; script-src 'self' 'unsafe-inline' https://unpkg.com https://cdn.jsdelivr.net https://cdnjs.cloudflare.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com"
   ```

7. **Sanitize error messages** before returning to clients

---

## Admin Panel

The Shamra Academia admin panel is accessible via the `/jim19ud83` route and provides comprehensive management tools for the platform.

### Access and Security

The admin panel is protected by role-based authentication:

```php
<?php
// All admin controllers extend this security check
protected function checkUser(){
    $user = $this->getUser();
    
    if (!$this->container->get('security.authorization_checker')->isGranted('ROLE_ADMIN')) {
        throw new AccessDeniedException();
    }

    if (!is_object($user) || !$user instanceof UserInterface) {
        throw new AccessDeniedException('This user does not have access to this section.');
    }
}
```

### Main Admin Functionality

#### 1. User Management (`/jim19ud83/users`)
- **User Statistics**: View registered users, subscriptions, and analytics
- **User Roles**: Manage user permissions and roles
- **Subscription Management**: Create/cancel subscriptions, manage subscription types
- **User Filtering**: Filter users by country, subscription status, last login
- **User Actions**: Reset passwords, change privileges, view user details

#### 2. Research Management (`/jim19ud83/research`)
- **Research Panel**: Monitor research submissions and quality
- **Publisher Management**: Add/edit/remove research publishers
- **Field Management**: Manage academic fields and categories
- **Content Moderation**: Review and approve research content

#### 3. Course Management (`/jim19ud83/courses`)
- **Course Oversight**: Manage online courses and lessons
- **Course Statistics**: Track course enrollment and completion
- **Course Features**: Feature/unfeature courses
- **Course Approval**: Review and approve course submissions

#### 4. Communication Management
- **Email System** (`/jim19ud83/emails`): Manage newsletter and email campaigns
- **Contact Management**: Handle contact form submissions and replies
- **Mail Lists**: Create and manage mailing lists for different user segments

#### 5. System Configuration
- **Promo Codes** (`/jim19ud83/promo`): Create and manage promotional codes
- **Credit System** (`/jim19ud83/credit`): Manage user credit transactions
- **ElasticSearch Debug** (`/jim19ud83/es-debug`): Debug search functionality
- **General Settings**: Configure system-wide settings and partnerships

### Adding New Admin Tab

To add a new admin functionality tab, follow these steps:

#### 1. Add Route to Control Panel

Add your route in `config/routes/control_panel.yaml`:

```yaml
analytics_control_panel:
    path: /analytics
    defaults: { _controller: App\Controller\ControlPanelController::analyticsPanel}

analytics_export:
    path: /analytics/export/{type}
    defaults: { _controller: App\Controller\ControlPanelController::exportAnalytics}
```

#### 2. Create Controller Method

Add the method to `src/Controller/ControlPanelController.php`:

```php
<?php
// Add to ControlPanelController class

public function analyticsPanel(Request $request): Response
{
    $this->checkUser(); // Always include security check
    
    $analyticsRepo = $this->getDoctrine()->getRepository(Analytics::class);
    
    // Get analytics data
    $userStats = $analyticsRepo->getUserStatistics();
    $researchStats = $analyticsRepo->getResearchStatistics(); 
    $subscriptionStats = $analyticsRepo->getSubscriptionStatistics();
    
    return $this->render('ControlPanel/analytics.html.twig', [
        'userStats' => $userStats,
        'researchStats' => $researchStats,
        'subscriptionStats' => $subscriptionStats
    ]);
}

public function exportAnalytics(Request $request, string $type): Response
{
    $this->checkUser();
    
    // Handle export logic based on type (pdf, excel, csv)
    switch($type) {
        case 'pdf':
            return $this->generatePdfReport();
        case 'excel':
            return $this->generateExcelReport();
        case 'csv':
            return $this->generateCsvReport();
        default:
            throw new NotFoundHttpException('Invalid export type');
    }
}
```

#### 3. Create Template

Create `templates/ControlPanel/analytics.html.twig`:

```twig
<!DOCTYPE html>
<html lang="ar" dir="rtl">
<head>
    <meta charset="UTF-8">
    <title>لوحة التحليلات - شمرا اكاديميا</title>
    <link rel="stylesheet" href="{{ asset('bundles/user/css/controlpanel.css') }}">
</head>
<body>
    <div class="container">
        {% include "ControlPanelHeader.html.twig" %}
        
        <div class="analytics-panel">
            <h1>لوحة التحليلات والإحصائيات</h1>
            
            <div class="row">
                <div class="col-md-4">
                    <div class="stat-card">
                        <h3>إحصائيات المستخدمين</h3>
                        <p>إجمالي المستخدمين: {{ userStats.total }}</p>
                        <p>المشتركين النشطين: {{ userStats.activeSubscribers }}</p>
                        <p>تسجيلات اليوم: {{ userStats.todayRegistrations }}</p>
                    </div>
                </div>
                
                <div class="col-md-4">
                    <div class="stat-card">
                        <h3>إحصائيات الأبحاث</h3>
                        <p>إجمالي الأبحاث: {{ researchStats.total }}</p>
                        <p>أبحاث اليوم: {{ researchStats.today }}</p>
                        <p>التحميلات: {{ researchStats.downloads }}</p>
                    </div>
                </div>
                
                <div class="col-md-4">
                    <div class="stat-card">
                        <h3>إحصائيات الاشتراكات</h3>
                        <p>اشتراكات نشطة: {{ subscriptionStats.active }}</p>
                        <p>إجمالي الإيرادات: {{ subscriptionStats.revenue }}</p>
                        <p>معدل التجديد: {{ subscriptionStats.renewalRate }}%</p>
                    </div>
                </div>
            </div>
            
            <div class="export-section">
                <h3>تصدير التقارير</h3>
                <a href="{{ path('analytics_export', {'type': 'pdf'}) }}" class="btn btn-primary">تصدير PDF</a>
                <a href="{{ path('analytics_export', {'type': 'excel'}) }}" class="btn btn-success">تصدير Excel</a>
                <a href="{{ path('analytics_export', {'type': 'csv'}) }}" class="btn btn-info">تصدير CSV</a>
            </div>
        </div>
    </div>
</body>
</html>
```

#### 4. Add Navigation Tab

Update `templates/ControlPanelHeader.html.twig`:

```twig
<!-- Add after existing tabs -->
<a href="{{ path('analytics_control_panel') }}">
    <button class="nav-link {% if route =='analytics_control_panel' %}{{ 'active' }}{% endif %}" 
            id="nav-analytics-tab" 
            data-bs-toggle="tab" 
            data-bs-target="#nav-analytics" 
            type="button" 
            role="tab" 
            aria-controls="nav-analytics" 
            aria-selected="false">
        إدارة التحليلات
    </button>
</a>
```

#### 5. Add Required Permissions

Ensure your user has `ROLE_ADMIN` in the database or add the role programmatically:

```bash
# Using Symfony console
php bin/console fos:user:promote username ROLE_ADMIN
```

### Existing Admin Routes Summary

| Tab | Route | Functionality |
|-----|-------|---------------|
| User Management | `/jim19ud83/users` | User statistics, roles, subscriptions |
| Course Management | `/jim19ud83/courses` | Course oversight and statistics |
| Questions | `/jim19ud83/question` | Q&A moderation and management |
| Contact Us | `/jim19ud83/contact_list` | Handle contact form submissions |
| Publishers | `/jim19ud83/publisher/list` | Manage research publishers |
| Promo Codes | `/jim19ud83/promo` | Create/manage promotional codes |
| General | `/jim19ud83/general` | General platform settings |
| Research | `/jim19ud83/research` | Research content management |
| Partners | `/jim19ud83/partner` | Partnership management |
| Mail System | `/jim19ud83/mail` | Email campaign management |
| Store | `/jim19ud83/store` | Content store management |
| Newsletters | `/jim19ud83/emails` | Newsletter management |
| Credit System | `/jim19ud83/credit` | User credit management |
| ES Debug | `/jim19ud83/es-debug` | ElasticSearch debugging |

### Admin Panel Security Best Practices

1. **Always use `checkUser()`** in admin controller methods
2. **Validate all input data** from admin forms
3. **Log admin actions** for audit trails
4. **Use CSRF protection** for forms
5. **Implement proper error handling** for admin operations

---

## Google Analytics (gtag.js)

### Setup

Google Analytics 4 is integrated via the global `gtag.js` snippet in `templates/base.html.twig`, loaded on every page.

- **Measurement ID**: `G-17J5HLCR9P`
- **Dashboard**: [analytics.google.com](https://analytics.google.com/)

### Custom Events

The following custom events are tracked across the platform:

| Event Name | Category | Location | Description |
|---|---|---|---|
| `ocr_submit` | OCR | `templates/ocr/index.html.twig` | User submits a file for OCR processing. Value = credits charged. |
| `ocr_download` | OCR | `templates/ocr/index.html.twig` | User downloads an OCR result (label: `docx`, `pdf`, or `md`). |
| `ai_tool_use` | Playground | `public/js/playground.js` | User triggers an AI text action (label: `rephrase`, `proofread`, `translate`, `criticize`, `ask`). |
| `ai_chat_message` | Playground | `public/js/playground.js` | User sends a message to the research assistant chat. |
| `ai_selection_action` | Playground | `public/js/playground.js` | User uses the text selection popup (label: `explain`, `summarize`, `translate`, `expand`). |
| `research_plan_initiate` | Playground | `public/js/playground.js` | User initiates a new research plan. |
| `research_download` | Research | `show.html.twig`, `homepage.js` | User downloads a research paper (label: slug). |
| `sign_up` | User | `register_content.html.twig` | User submits the registration form. |
| `begin_checkout` | Subscription | `subscription_new.html.twig` | User clicks a subscribe button (label: plan type, e.g. `starter-monthly`). |

### Adding New Events

All `gtag()` calls are guarded so they fail silently when analytics is blocked:

```javascript
if (typeof gtag === 'function') {
    gtag('event', 'event_name', {
        event_category: 'Category',
        event_label: 'label_value'
    });
}
```

Events appear in GA4 under **Reports > Engagement > Events** (may take a few hours for new events to show).

---

This documentation covers the core development patterns used in your Symfony academia application. Each section provides practical examples that follow the existing codebase structure and conventions.