translatorswb/wca-nlp-landscape

WIP Mapping of NLP resources for West and Central Africa # UNICEF WCARO NLP Landscape > **Note:** This repository is no longer the primary location. The official repo and live site have moved to UNICEF-Ventures/wca-nlp-landscape. Please refer to that repository for the latest updates. Mapping of NLP/language technology resources for West and Central Africa. **Status:** Work in progress. Not yet open for external contributions. --- ## About This project maps available language technology (ASR, TTS, MT, LLM) for languages spoken in UNICEF's West and Central Africa region. It aggregates data from HuggingFace, Common Voice, Wikipedia, and other sources to provide an overview of available models, datasets, and actors working on these languages. Developed by CLEAR Global for UNICEF WCARO. ## Setup ```bash pip install -r scripts/requirements.txt ``` ## Populating Language Data `populate_research.py` fetches data from multiple sources and saves it per language under `Research/Languages/{iso}/`. Languages to process are listed in `Research/focused_languages.yaml` (ISO 639-3 codes). For each language, the script: 1. **African Language Grid** (Kamusi Project) — Looks up the language in `Source data/African-language-grid/` and writes basic metadata (countries, population, endangerment, alternate names) to `info.yaml` 2. **Wikipedia** — Scrapes the language's Wikipedia page for family, speaker counts (L1/L2), writing system. Appended to `info.yaml` under `wikipedia:` 3. **MMS** (Meta) — Checks `Source data/mms_language_coverage.yaml` for ASR/TTS/LID support. Added to `info.yaml` under `tech_resources:` 4. **HuggingFace** models — Queries the API for models tagged with the language code, grouped by task (ASR, TTS, translation, LLM). Top models by downloads saved to `models.yaml` with total counts 5. **HuggingFace** datasets — Same approach, saved to `datasets.yaml` 6. **Mozilla Common Voice** — Extracts corpus statistics (hours, clips, gender breakdown) from the local Common Voice JSON file …