dluvaha/maragoli_llm_dataset_app

# Maragoli LLM Dataset Collection System A production-ready Django web application for collecting, managing, and exporting Maragoli-English parallel corpus data for training a Maragoli Large Language Model (LLM). ## Features - **Side-by-side Display**: Maragoli text and English translations displayed side-by-side - **Excel Bulk Import**: Upload `.xlsx` files with preview and validation before import - **Duplicate Detection**: SHA-256 hash-based duplicate prevention - **JSON Export**: Three export formats optimized for different LLM training pipelines - **Validation Workflow**: Mark entries as validated for quality control - **Category Organization**: Organize datasets by topic (Greetings, Proverbs, etc.) - **Search & Filter**: Full-text search, category filter, validation status filter - **Import History**: Full audit trail of all import operations ## Quick Start ### Prerequisites - Python 3.10+ - pip ### Installation ```bash # 1. Clone the repository git clone github.com cd maragoli_llm_dataset_app # 2. Create a virtual environment python -m venv venv source venv/bin/activate # Linux/Mac # OR: venv\Scripts\activate # Windows # 3. Install dependencies pip install -r requirements.txt # 4. Run migrations (creates all database tables) python manage.py migrate # 5. Load sample data (41 Maragoli translations + admin user) # Admin login: admin / admin123 python manage.py loaddata datasets/fixtures/initial_data.json python manage.py loaddata datasets/fixtures/admin_user.json # 6. Run the development server python manage.py runserver ``` Visit `127.0.0.1` and log in with your superuser credentials. ## Project Structure ``` maragoli_dataset/ ├── maragoli_llm/ # Django project settings │ ├── settings.py # Configuration (database, static files, etc.) │ ├── urls.py # Root URL configuration │ └── wsgi.py # WSGI entry point ├── datasets/ …