temitope/yoruba-dictionary-data
Domain:
natural language processingRecord type:
dataset
Creator:
Tem
Host:
Curated Yoruba-English lexicon with tone marks, digraph-aware tokenization, and English glosses. 937 entries from Wiktionary via kaikki.org.
# Yoruba Dictionary Data
A curated, machine-readable Yoruba-English lexicon with proper **tone marks** and **diacritics**, extracted and cleaned from Wiktionary via kaikki.org.
## What's Included
| File | Description |
|------|-------------|
| `data/words.json` | **937 curated entries** — tokenized, normalized, with English glosses and part-of-speech tags |
| `scripts/ingest-kaikki.js` | The pipeline script that downloads, filters, tokenizes, and deduplicates the raw kaikki.org data |
### Entry Schema (`WordEntry`)
```jsonc
{
"id": "1",
"word": "Àgbọ̀n", // Display form (NFC, capitalized)
"normalized": "agbon", // Tone-stripped, lowercase
"tokens": ["À", "gbọ̀", "n"], // Digraph-aware tokenization (Gb, Kp treated as single tokens)
"difficulty": "easy", // easy (2-3 tokens), medium (4), hard (5-6)
"meanings": {
"en": "coconut" // Concise English gloss
},
"partOfSpeech": "n" // Optional: n, v, adj, adv, pron, num, excl
}
```
## Filter Pipeline
The ingest script applies these filters sequentially to the raw kaikki.org JSONL (6,269 entries):
1. **Proper nouns** removed (names)
2. **Multi-word entries** removed (spaces, hyphens)
3. **Ajami script** removed (Arabic-script Yoruba)
4. **No usable gloss** removed (includes entries where all glosses are cross-references)
5. **No Latin characters** removed
6. **No Yoruba characters** removed — requires at least one tone mark (acute/grave) or special character (ẹ/ọ/ṣ)
7. **Non-Yoruba diacritics** removed — tilde (ẽ), macron, etc. via NFD decomposition check
8. **Token length** filtered — only 2–6 token words kept
9. **Deduplication** — by normalized form and exact spelling
### Gloss Quality
- `firstGloss()` skips cross-reference senses ("alternative form of…", "archaic spelling of…", etc.) and picks real English definitions
- 15 cross-reference patterns are detected and filtered
- Long glosses have been AI-summarized for conciseness
## Usage
### Use the pre-built data
``` …