bakiizese/local-language-nlp-pipeline

A first-principles NLP preprocessing and classification pipeline built in PyTorch for low-resource language(Amharic), demonstrating explicit geometric and high-dimensional tensor transformations. # Local-Language NLP Pipeline ### A First-Principles PyTorch Text Classification System for Low-Resource Languages --- This project is a production-structured, end-to-end NLP classification pipeline built from the ground up using raw PyTorch tensor mechanics — no high-level NLP frameworks, no pre-trained tokenizers, no black-box abstractions. It uses **Amharic (Ge'ez script)** as the reference corpus, demonstrating that rigorous ML engineering discipline is language-agnostic and transfers directly to any low-resource language setting. Every design decision prioritises **transparency of data transformation** over convenience. The goal is a system where any engineer can trace exactly how a raw string becomes a class prediction, at the level of individual tensor operations. --- ## Core Philosophy & Architecture ### Systems-Thinking, Not Library-Stacking Most NLP pipelines delegate critical decisions — tokenization strategy, vocabulary construction, sequence representation — to pre-built libraries. This project makes every one of those decisions explicit and auditable: - **Tokenization** is implemented from scratch with a deterministic vocabulary builder. The choice of character-level tokenization for Ge'ez is linguistically grounded: each Ethiopic Unicode code point encodes one consonant-vowel syllable (*fidel*), making characters the natural atomic unit of meaning — not an arbitrary engineering convenience. - **Text normalization** encodes domain knowledge directly. Modern Amharic has three glyph series (ሐ-series, ኀ-series, ዐ-series) that are phonologically identical to their canonical counterparts. Conflating them is not optional cleanup — it is a correctness requirement that any serious NLP system for this language must enforce. - **Sequence padding** is handled dynamically per batch, not globally. This is the difference between a system designed for production throughput and one designed to pass a notebook. ### First-Principles Data Handling The pipeline …