OmondiKevin/SesoFix

This project fine-tunes a byte-level sequence-to-sequence model (ByT5) to harmonize South African and Lesotho Sesotho orthographies. The model automatically converts South African Sesotho text into its standardized Lesotho variant, bridging regional spelling splits to enable downstream NLP interoperability. # SesoFix: Orthographic Harmonization in Sesotho SesoFix is an orthographic harmonization framework designed to convert South African Sesotho text into the standardized Lesotho Sesotho variant. By resolving regional spelling and morphological discrepancies, SesoFix bridges the orthographic split that fragments low-resource Sesotho corpora, improving downstream natural language processing (NLP) performance. SesoFix fine-tunes a byte-level pre-trained transformer (**ByT5**) in a sequence-to-sequence framework, outperforming traditional rule-based converters and subword-level multilingual models (**mT5**). --- ## Key Features * **Byte-Level Sequence-to-Sequence Modeling**: Utilizes Google's ByT5 model, which operates directly on UTF-8 bytes to prevent subword tokenization fragmentation on regional spelling variations. * **Leakage-Prevention Splitting**: Automatically isolates rule-based synthetic data strictly to the training set, keeping validation and test splits 100% clean and composed of real-world text. * **Baselines Included**: Contains implementations for Identity Copy, regular expression Rule-Based systems, character-level LSTMs with attention, and Edit-Distance retrieval. * **Downstream Integration**: Verified to improve downstream Sesotho-to-English translation quality. --- ## Project Structure ``` SesoFix/ ├── data/ │ ├── processed/ # Processed data splits (sample_data.csv) │ ├── input/ # Raw source/target files │ └── output/ # Generated model predictions ├── models/ # Saved model weights ├── logs/ # Tensorboard training logs ├── checkpoints/ # Training checkpoint saves ├── scripts/ │ ├── __init__.py │ ├── data_preprocessing.py # Data loading, tokenization, and leakage-prevention splitting │ ├── model_config.py # ByT5/mT5 configurations and parameter counters │ ├── train_model.py # Seq2Seq fine-tuning script │ └── evaluate.py # Automatic metrics evaluation (BL …