temitope/yoruba-dictionary-data

Curated Yoruba-English lexicon with tone marks, digraph-aware tokenization, and English glosses. 937 entries from Wiktionary via kaikki.org. # Yoruba Dictionary Data A curated, machine-readable Yoruba-English lexicon with proper **tone marks** and **diacritics**, extracted and cleaned from Wiktionary via kaikki.org. ## What's Included | File | Description | |------|-------------| | `data/words.json` | **937 curated entries** — tokenized, normalized, with English glosses and part-of-speech tags | | `scripts/ingest-kaikki.js` | The pipeline script that downloads, filters, tokenizes, and deduplicates the raw kaikki.org data | ### Entry Schema (`WordEntry`) ```jsonc { "id": "1", "word": "Àgbọ̀n", // Display form (NFC, capitalized) "normalized": "agbon", // Tone-stripped, lowercase "tokens": ["À", "gbọ̀", "n"], // Digraph-aware tokenization (Gb, Kp treated as single tokens) "difficulty": "easy", // easy (2-3 tokens), medium (4), hard (5-6) "meanings": { "en": "coconut" // Concise English gloss }, "partOfSpeech": "n" // Optional: n, v, adj, adv, pron, num, excl } ``` ## Filter Pipeline The ingest script applies these filters sequentially to the raw kaikki.org JSONL (6,269 entries): 1. **Proper nouns** removed (names) 2. **Multi-word entries** removed (spaces, hyphens) 3. **Ajami script** removed (Arabic-script Yoruba) 4. **No usable gloss** removed (includes entries where all glosses are cross-references) 5. **No Latin characters** removed 6. **No Yoruba characters** removed — requires at least one tone mark (acute/grave) or special character (ẹ/ọ/ṣ) 7. **Non-Yoruba diacritics** removed — tilde (ẽ), macron, etc. via NFD decomposition check 8. **Token length** filtered — only 2–6 token words kept 9. **Deduplication** — by normalized form and exact spelling ### Gloss Quality - `firstGloss()` skips cross-reference senses ("alternative form of…", "archaic spelling of…", etc.) and picks real English definitions - 15 cross-reference patterns are detected and filtered - Long glosses have been AI-summarized for conciseness ## Usage ### Use the pre-built data ``` …