Jeff-2000/KreyolAfricBench

Unified benchmark scaffold for Haitian Creole and African-language NLP, covering NER, QA, classification, retrieval, summarization, translation, code-switching, annotation guidelines, schemas, sample data, and evaluation utilities. # KreyolAfricBench Unified benchmark scaffold for Haitian Creole and African-language NLP. KreyolAfricBench is designed as a publishable, reproducible benchmark suite covering: - Named entity recognition - Question answering - Topic/sentiment/intent classification - Retrieval - Summarization - Translation - Code-switching and language identification The benchmark uses one shared instance envelope across tasks: ```json { "id": "hat_ner_news_0001", "task": "ner", "language": "hat", "script": "Latn", "country_region": "HT", "domain": "news", "source": {"name": "sample", "url": null, "license": "CC-BY-4.0"}, "split": "test_public", "input": {}, "target": {}, "metadata": {} } ``` ## Repository Layout - `docs/benchmark_blueprint.md`: publishable research blueprint and benchmark architecture. - `docs/annotation_guidelines.md`: task-specific annotation protocol. - `docs/data_card_template.md`: Hugging Face data card template. - `schemas/instance.schema.json`: JSON Schema for the shared task envelope. - `schemas/labels.json`: canonical label sets. - `data/sample/*.jsonl`: small valid examples for each task. - `src/kreyolafricbench/evaluation.py`: metric implementations. - `src/kreyolafricbench/validate.py`: JSONL schema validator. - `tests/test_evaluation.py`: smoke tests for metrics. ## Quick Start Validate sample rows: ```bash python3 -m kreyolafricbench.validate data/sample/ner.jsonl --schema schemas/instance.schema.json ``` Run tests: ```bash python3 -m unittest discover -s tests ``` Evaluate predictions programmatically: ```python from kreyolafricbench.evaluation import classification_scores scores = classification_scores( predictions=["health", "sports"], references=["health", "politics"], ) print(scores["macro_f1"]) ``` ## Hugging Face Dataset Configs Planned dataset configs: - `ner` - `qa` - `classification_topic` - `classification_sentiment` - `classification_intent` - `retrieval` - `summarization` - `translation` - `codeswitch` - `robustness` Spl …