nasro-bay/Darija-data_collec

attempt to build algerian Darija corpus # Algerian Darija Text Corpus A collection pipeline and toolset for Algerian Darija (Algerian Arabic dialect) — a low-resource language variety written in both Arabic script and Arabizi (Latin script with numerals standing in for Arabic sounds, e.g. `3=ع`, `7=ح`, `9=ق`), and frequently mixed with French. Darija is spoken by tens of millions of people but remains comparatively under-resourced next to MSA and other major languages — little annotated data, few tokenizers or embeddings trained specifically for it, and almost no dialect-aware tooling. This project builds toward closing that gap: a clean corpus, a tokenizer, word embeddings, a dialect classifier, and benchmark evaluations, each in its own subdirectory (see each one's own `README.md` for detail — this file stays brief on purpose). ## Data collection | Source | Directory | Method | |---|---|---| | YouTube comments (Algerian channels: vlogs, comedy, sports, news, podcasts) | `Youtube_scrap/` | YouTube Data API v3, curated seed channels/videos | | Djelfa.info forum (منتديات الجلفة) | `Mountada_djelfa_scrap/` | Direct scraping, robots.txt-compliant | Both sources go through the same shape of pipeline: scrape → clean → schema → JSONL. Cleaning is regex-based text normalization, derived empirically against real samples rather than assumed: collapsing elongated letters, punctuation runs, and repeated emoji; stripping Arabic diacritics (tachkil); anonymizing URLs and @mentions into placeholders; and (forum-specific) stripping BBCode, quote-wrappers, and tatweel. Near-duplicate detection (MinHash/LSH) exists as a reusable module for later use. ## Tokenization (`Tokenization/`) Multiple tokenizer variants trained and compared head-to-head at matched vocabulary sizes (1K–30K): SentencePiece Unigram (with and without subword regularization), WordPiece, and byte-level BPE. Evaluated on held-out data via fertility, compression factor, vocabulary utilization, OOV/byte-fallback rate, known-word fragmentation under …