Eddek1/llm-data-recipe-pipeline

Recipe for Mabaan-English Translation # LLM Data Recipe Pipeline for Multilingual Translation A small, runnable demonstration of how raw multilingual text becomes a clean, evaluated, human-reviewed dataset -- the "data recipe" workflow used in most LLM projects. This is just a personal project for mabaan language. ## A note on the data The sample sentence pairs are **synthetic placeholders**, not real Mabaan translations. A real, usable English Mabaan parallel corpus doesn't exist publicly -- Mabaan has roughly 400,000 speakers and very little digitized bilingual text. The pipeline logic below is written so the placeholder target-language column could be swapped for a real sourced corpus without changing any of the cleaning, splitting, or evaluation logic. ## Structure - `data/raw/` -- Messy synthetic input data - `data/processed/` -- Cleaned data + SQLite database - `data/splits/` -- train/validation/test CSVs - `scripts/clean_pairs.py` -- Recipe 1: cleaning (drop missing, dedupe, flag near-dupes) - `scripts/create_splits.py` -- Recipe 2: train/val/test split (reproducible via random seed) - `scripts/analyze_with_sql.py` -- Recipe 3: SQL analytics (joins, window functions) - `scripts/score_evaluations.py` -- Recipe 4: rubric-based evaluation gate - `scripts/run_pipeline.py` -- Runs all of the above end to end - `evaluation/translation_rubric.md` -- Scoring rubric (accuracy/fluency/cultural fit) - `evaluation/sample_scored_pairs.csv` -- Human-reviewed example scores ## How to run it ```bash pip install -r requirements.txt cd scripts python run_pipeline.py ``` ## The recipe, end to end Raw bilingual pairs (messy CSV) -> Clean & validate (drop missing, remove exact dupes, flag near-dupes) -> Human review against rubric -> Train / validation / test split (reproducible, fixed seed) -> SQL analytics on pipeline health (joins, window functions) -> Evaluation report (pass/fail gate applied consistently) ## Design decisions worth knowing - **Near-duplicates are flagged, not deleted.** Two sentences di …