lemneya/hassania-data-pipeline

Hassania Arabic dialect data collection and processing pipeline for LLM fine-tuning (Project #2) # Hassania Data Pipeline A data collection and processing pipeline for authentic Hassaniya dialect content, designed for fine-tuning language models. ## Overview This repository contains: - **Raw data** collected from various Mauritanian sources - **Processed data** in HDRP (Hassaniya Dialect Resource Package) format - **Conversion scripts** to transform raw data into training-ready formats ## Dataset Statistics | Bucket | Episodes | |--------|----------| | everyday_chat | 229 | | marketplace_qa | 100 | | public_comments | 4 | | **Total** | **333** | ### Data Sources | Source | Episodes | Description | |--------|----------|-------------| | Peace Corps Mauritania | 138 | Official language learning materials | | YouTube Language Beat | 51 | Video lesson transcriptions | | Voursa Marketplace | 56 | Mauritanian marketplace listings | | Facebook Marketplace Nouakchott | 44 | Local marketplace content | | mo3jam Dictionary | 26 | User-contributed vocabulary | | Omniglot | 18 | Linguistic reference materials | ## Data Format ### HDRP Format (JSONL) Each line contains a JSON object with: ```json { "english": "English translation or description", "hassaniya_ar": "الحسانية بالعربية", "hassaniya_en": "Hassaniya in Latin transliteration", "bucket": "everyday_chat|marketplace_qa|public_comments", "source": "data_source_identifier", "context": "situational_context" } ``` ### Buckets - **everyday_chat**: Greetings, phrases, vocabulary for daily conversation - **marketplace_qa**: Product descriptions, real estate listings, marketplace vocabulary - **public_comments**: Social media posts, proverbs, public discussions ## Directory Structure ``` hassania-data-pipeline/ ├── data/ │ ├── raw/ # Original collected data │ │ ├── dictionary/ # Dictionary sources │ │ ├── facebook/ # Facebook page data │ │ ├── marketplace/ # Voursa and FB Marketplace │ │ ├── reference/ # Peace Corps, Omniglot │ │ ├── soci …