Amine-Gharout/tamazight-nlp-toolkit

End-to-end NLP pipeline for Tamazight : OCR, ASR, translation, RAT, and evaluation tools for low-resource languages. # Low Resource Language Toolkit 🌍 This repository contains reusable tools, notebooks, and pipelines designed for NLP research involving Low-Resource Languages (LRLs). Originally formulated for Tamazight, the structural pipelines herein (OCR, translation, Retrieval-Augmented Translation) can be seamlessly adapted to any low-resource language pairs with minimal modifications. ## Project Structure - `0_data/`: Raw datasets, cleaned files, and indexed corpora (parallel text `.txt` pairs) - `01_data_collection/`: Scraping podcasts + downloading parallel data from MADLAD-400 / Tatoeba - `02_language_id/`: Language identification filtering (GlotLID, Meta MMS) - `03_ocr/`: Optical Character Recognition via Gemini & Qwen VL (prompts, scripts, notebooks, outputs) - `04_asr/`: Automatic Speech Recognition — batch transcription + live radio pipeline - `05_data_cleaning/`: Cleaning and curating parallel corpora (tab cleanup, dedup, filtering) - `06_translation/`: Machine translation using NLLB-200-3.3B with CTranslate2 fast inference - `07_rat/`: Retrieval-Augmented Translation (FAISS + LangChain + Gemini) - `08_evaluation/`: Translation quality benchmarks (FLORES+, chrF++) - `09_deployment/`: Azure cloud infrastructure for 24/7 radio recording ## Setup Requirements Ensure you meet the required dependencies: ```bash pip install -r requirements.txt ``` ### Download Data & Models Large files (datasets and NLLB model) are **not stored in git**. Download them: ```bash # 1. Download raw datasets (~619 MB) from Google Drive bash scripts/download_data.sh # 2. Download NLLB-200-3.3B CTranslate2 model (~6.5 GB) from HuggingFace python scripts/download_nllb_model.py # float16 (best quality) python scripts/download_nllb_model.py --quant int8_float16 # int8 (less VRAM) ``` > **First-time setup:** If you're the repo owner, run `bash scripts/pack_data.sh` to create `raw_data.tar.gz`, upload it to Google Drive, then paste the file ID into `scripts/download_dat …