diarray-hub/bambara-normalizer

A python package for normalizing Bambara text for NLP # bambara-normalizer `bambara-normalizer` is a Python package for normalizing Bambara text, tailored for Natural Language Processing (NLP) tasks. The package provides tools to preprocess text by removing symbols, diacritics, and performing additional transformations required for various NLP applications such as ***number normalization***. ## Features - **BasicTextNormalizer**: A generic text normalization class that removes symbols, diacritics, and optionally splits letters. - **BasicBambaraNormalizer**: Extends `BasicTextNormalizer` with specific rules for Bambara text, such as preserving hyphens in compound words and handling apostrophes. - **BambaraASRNormalizer**: A specialized normalizer for Automatic Speech Recognition (ASR) tasks in Bambara, designed to retain parenthetical and bracketed text that might appear in spoken transcriptions. - **BambaraNumberNormalizer**: Add number normalization capability to the package, both number2bam and bam2number including money amounts (Bambara 'dɔrɔmɛ' counting system where 5 CFA equals 1 dɔrɔmɛ) (up to millions) ## Installation ### Install from PyPI To install the package, run: ```bash pip install bambara-normalizer ``` ### Install from Source To install the package from source, clone the repository and build the package: ```bash git clone github.com cd bambara-normalizer python -m build --wheel pip install dist/bambara_normalizer-1.1.0-py3-none-any.whl ``` ## Usage ### BasicTextNormalizer ```python from bambara_normalizer import BasicTextNormalizer normalizer = BasicTextNormalizer(remove_diacritics=True, split_letters=False) text = "Cliché text with symbols & diacritics!" normalized_text = normalizer(text) print(normalized_text) # Output: "cliche text with symbols diacritics" ``` ### BasicBambaraNormalizer ```python from bambara_normalizer import BasicBambaraNormalizer normalizer = BasicBambaraNormalizer() text = "à tɔ́gɔ kó : sìrajɛ." normalized_text = norm …