dantesfai/tigrinya-normalizer

# Tigrinya Normalizer A Python package and command-line tool for normalizing Tigrinya text. It applies linguistic rules, dictionary-based replacements, and cleaning steps to standardize Tigrinya text data. --- ## Features - Normalize clitics and abbreviations - Handle hyphenated and slash-separated words - Remove unwanted punctuation and extra spaces - Customizable punctuation preservation - Load dictionaries for enhanced normalization - Command-line interface (CLI) for easy use --- ## Installation ### Requirements - Python 3.7+ - See `requirements.txt` for dependencies ### Install from source ```bash git clone github.com cd tigrinya-normalizer pip install . ``` ### Or install dependencies manually: ```bash pip install -r requirements.txt ``` ## Usage ### As a Python package ```python from tigrinya_normalizer.normalizer import TigrinyaNormalizer normalizer = TigrinyaNormalizer(dataset_file="path/to/input.txt") normalized_text = normalizer.normalize("ቛንቋ ትግርኛ") print(normalized_text) ``` ### As a CLI Tool ```bash python cli.py -i path/to/input.txt -o normalized_output.txt ``` CLI Options: | Argument | Description | Default | | ---------------------- | ---------------------------------------- | ----------------------- | | `-i` / `--input` | Path to input dataset file (required) | - | | `-o` / `--output` | Output filename | `normalized_output.txt` | | `-p` / `--punctuation` | Punctuation marks to preserve (optional) | None | ### Example: Preserve Punctuation ```bash python cli.py -i input.txt -o output.txt -p "።፧?" ``` ### Project Structure ``` tigrinya-normalizer/ ├── tigrinya_normalizer/ │ ├── __init__.py │ ├── normalizer.py # Main normalization logic │ ├── utils.py # Utility functions | ├──dictionary_generator.py # Core TiDictionary log …