Biruk-gebru/fidelkey

Recover Unicode Ethiopic text from PDFs built with legacy pre-Unicode Ge'ez fonts # FidelKey Recover real Unicode Ethiopic (Ge'ez/Amharic) text from PDFs built with legacy, pre-Unicode fonts. "Fidel" is the traditional name for the Ethiopic script, and this is basically the key that unlocks it from these old, broken fonts. ## What is this Before Unicode support for Ethiopic script matured, roughly pre-2010, most Amharic documents (church bulletins, poetry, newsletters, government forms) were typeset using custom "hack" fonts. Open the file in the app that made it and it looks like Amharic. But the underlying text isn't Amharic at all. It's Latin letters or arbitrary codes remapped, glyph by glyph, to draw Ethiopic shapes on screen. The PDF itself has no idea it's Amharic. So copy-paste, search, or any text-extraction tool (`pypdf`, `pdftotext`, etc.) run on these files just produces garbage. Sometimes that's literal internal font tokens like `/ETHIOPIC_SYLLABLE_GE` leaking into the output. Sometimes it's meaningless byte soup with no structure at all. I ran into this while building an Amharic poetry dataset from public-domain sources. A large chunk of the documents I had, 43 real church-archive PDFs, turned out to be unusable for exactly this reason. This tool is what came out of solving that problem, cleaned up so anyone else hitting the same wall doesn't have to start from zero. **Before:** ``` /ETHIOPIC_SYLLABLE_GLOTTAL_E/ETHIOPIC_SYLLABLE_GE/ETHIOPIC_SYLLABLE_ZI/ETHIOPIC_SYLLABLE_GLOTTAL_A ``` **After:** ``` እግዚአ ``` There are two distinct failure modes in real documents, and they need different approaches with different levels of confidence. See `ARCHITECTURE.md` for the full technical breakdown, including glyph-level images of how the harder case actually gets solved. ## 1. `differences_converter.py`, production ready Some legacy fonts embed a PDF `/Differences` encoding array that names each glyph after its semantic identity, things like `/ETHIOPIC_SYLLABLE_GE` or `/uni1265`. Text extraction libraries that don't recognize these n …