Logo Lanfrica

SpeakNigeriaOrg/kaikki-yoruba

Domain:

natural language processing

Record type:

dataset
Creator:
Spe
Host:
# kaikki-yoruba Shared Kaikki/Wiktionary normalization for Yoruba - one canonical artifact feeding two sibling projects, `yorubadict` (a static dictionary website) and `yoruba_student_dict_platform` (a curation platform for a student dictionary/curriculum, cross-checked against Kaikki as a reference source). ## What this is, in plain terms Kaikki is a project that extracts Wiktionary's dictionary content into clean, machine-readable data - instead of the raw wiki-markup Wiktionary pages are actually written in, Kaikki publishes one JSON object per dictionary entry, as a plain text file where each line is one entry (this format is called **JSONL** - "JSON Lines"). We use the slice of that data covering Yoruba words, currently 6,273 entries. But Kaikki's raw JSON is still messy and inconsistent in places - a word's "real" spelling isn't always tagged the same way twice, cross-references point at other words by plain text rather than a stable link, and useful data is scattered across differently-shaped fields depending on the entry. **"Normalizing" here means taking each of those raw, inconsistent records and rewriting it into one predictable, consistent shape** - same fields, same structure, every time - that other code can rely on without needing to know Kaikki's own quirks. Concretely, for one real (abbreviated) example, the raw Kaikki record for the letter "A": ```json { "word": "A", "pos": "character", "lang_code": "yo", "forms": [{"form": "a", "tags": ["lowercase"]}], "sounds": [{"ipa": "/a/", "tags": ["phoneme"]}], "senses": [{"glosses": ["The first letter of the Yoruba alphabet..."], "tags": ["letter", "uppercase"]}] } ``` becomes this repo's normalized entry: ```json { "id": "en-A-yo-character-9n~aNY1j", "headword": "A", "lang": "Yoruba", "langCode": "yo", "pos": "character", "canonicalForm": { "value": "A", "inferenceMethod": "fallback_headword", "confidence": 0.5, "originalValue": "A" }, "altForms": [{ "form": "a", "tags": ["lowercase"] }], "ipa": [{ …

Languages