MurariAmbati/yoruba-distilmbert

Yoruba-specialized DistilmBERT: Continued MLM pretraining + NER/POS evaluation # Yoruba-specialized DistilmBERT A minimal, reproducible pipeline for adapting DistilmBERT to Yoruba through continued masked language modeling (MLM) pretraining, with evaluation on NER and POS tagging. ## Research Question > Does a small amount of Yoruba-only continued MLM pretraining improve token-level performance (NER + POS) compared to vanilla DistilmBERT? ## Quick Start ```bash # Setup environment make setup # Run full reproducible pipeline make reproduce ``` Or step by step: ```bash # 1. Setup python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt # 2. Prepare data python download_yoruba_c4.py # C4 Yoruba python download_masakhaner2_yoruba.py # MasakhaNER2 python scripts/download_ud_yoruba.py # UD Yoruba YTB python scripts/01_build_mlm_corpus.py # Build MLM corpus # 3. Train python scripts/02_run_mlm.py # MLM pretraining python scripts/03_finetune_ner.py # NER fine-tuning python scripts/04_finetune_upos.py # UPOS fine-tuning # 4. Results python scripts/05_collect_results.py # Collect results ``` ## Project Structure ``` eacl/ ├── configs/ │ ├── mlm.yaml # MLM pretraining config │ ├── ner.yaml # NER fine-tuning config │ └── upos.yaml # UPOS fine-tuning config ├── scripts/ │ ├── 01_build_mlm_corpus.py │ ├── 02_run_mlm.py │ ├── 03_finetune_ner.py │ ├── 04_finetune_upos.py │ ├── 05_collect_results.py │ └── download_ud_yoruba.py ├── data/ │ ├── yoruba_c4/ # C4 Yoruba raw data │ ├── masakhaner2_yoruba/ │ ├── ud_yoruba_ytb/ │ ├── mlm_train.jsonl # Processed MLM train │ └── mlm_dev.jsonl # Processed MLM dev ├── checkpoints/ │ ├── distilmbert-yo-mlm/ # Adapted backbone │ ├── distilmbert-yo-mlm-ner/ # NER head (adapted) │ ├── distilmbert-yo-mlm-upos/ # UPOS head (adapted) │ ├── distilmbert-baseline-ner/ │ └── distilmbert-baseline-upos/ ├── results/ │ ├── results.csv │ ├── ner_test.json │ └── upos_test.json ├── downloa …