Kola9INE/yo_auto_diacritizer

A simple attempt to achieve automatic diacritization and tone marking of Yoruba texts. # Automatic Yoruba Diacritizer This repository contains a machine learning project designed to automatically restore diacritics (tone marks) to Yoruba text. The system takes undiacritized Yoruba text as input and outputs the correctly diacritized version using a Bi-Directional LSTM (BiLSTM) model. ## 📂 Project Structure ### Data & Preprocessing * **`pre_build.ipynb`**: The data collection notebook. It uses **Selenium** to scrape Yoruba articles from Global Voices (Yoruba). It handles dynamic content, extracts article bodies, and compiles them into raw text files. * **`data.parquet`**: The processed dataset stored in a highly efficient binary format. It contains paired examples for training: * `feature`: The input text (undiacritized). * `label`: The target text (diacritized). * **`yo_corpus.txt`, `news_sites.txt`, `owe.txt**`: Raw text files containing Yoruba corpora, proverbs, and news content used as source material for the dataset. * **`char2idx.pkl`**: A pickle file containing the character-to-index mapping, essential for encoding text for the character-level model. ### Models & Training * **`model_build.ipynb`**: The primary notebook for building and training the character-level BiLSTM diacritization model. * **`fasttext-project.ipynb`**: An alternative experimental notebook that utilizes **FastText** embeddings combined with PyTorch to train the model, exploring word/subword level representations. * **`char_bilstm_diacritizer.pt`**: The saved PyTorch model checkpoint (weights) for the Character BiLSTM Diacritizer. ### Utilities * **`char-level.ipynb`**: A utility notebook for feature engineering and model development. ## 🛠 Installation & Requirements To run the notebooks and use the model, you will need **Python 3.x** and the following libraries: ```bash pip install torch pandas numpy selenium fasttext scikit-learn tqdm pyarrow ``` *Note: For `pre_build.ipynb`, you will also need a compatible WebDriver (e.g., ChromeDriver) installed for Selenium …