mewaeltsegay/tigrinya_text_preprocessing

A pipeline for cleaning Tigrinya text. # Haddas Eritra Preprocessing Pipeline A comprehensive text preprocessing pipeline for the Haddas Eritra Tigrinya dataset. ## Overview This project provides a suite of preprocessing scripts to clean and normalize Tigrinya text data from the Haddas Eritra dataset. ## Scripts - **preprocess.py** - Main preprocessing orchestrator - **remove_english.py** - Remove English text from Tigrinya content - **normalize_spacing.py** - Normalize whitespace and spacing - **punctuation.py** - Handle and correct punctuation marks - **replace_dots.py** - Replace or normalize dot characters - **find_quotes.py** - Identify and process quotation marks - **handle_abrivations.py** - Handle abbreviations using the abbreviations list ## Pipeline Flow ``` raw/haddas_eritra.jsonl ↓ [remove_english] → english_removed/haddas_eritra.jsonl ↓ [normalize_spacing] → normalized/haddas_eritra_normalized.jsonl ↓ [find_quotes] → quotes_corrected/haddas_eritra.jsonl ↓ [punctuation] → punc_corrected/haddas_eritra_even_replaced.jsonl ↓ [preprocess] → preprocessed/haddas_eritra_even.jsonl ``` ## Data Directories - **raw/** - Original unprocessed JSONL data - **english_removed/** - Data after English text removal - **normalized/** - Data after spacing normalization - **quotes_corrected/** - Data after quote mark processing - **punc_corrected/** - Data after punctuation correction - **preprocessed/** - Final preprocessed data - **abbreviations/** - Abbreviation lists for reference ## View - **view/** - Web interface for viewing and formatting processed text - server.py - Flask/HTTP server - text_formatting.html - Frontend visualization ## Usage Run individual preprocessing scripts or use the main preprocess.py to execute the complete pipeline: ```bash python preprocess.py ```