zirak-ai/nlpashto

NLP Toolkit for Low-resource Pashto Language # NLPashto – NLP Toolkit for Pashto NLPashto is a Python suite for Pashto Natural Language Processing. It provides tools for fundamental text processing tasks, such as text cleaning, tokenization, and chunking (word segmentation). Additionally, it includes state-of-the-art models for POS tagging and sentiment analysis (specifically offensive language detection). ## Prerequisites To use NLPashto, you will need: - Python 3.8+ ## Installing NLPashto Install NLPashto via PyPi: ```bash pip install nlpashto ``` ## Basic Usage ### Text Cleaning This module contains basic text cleaning utilities: ```python from nlpashto import Cleaner cleaner = Cleaner() noisy_txt = "په ژوند کی علم 📚🖖 , 🖊 او پيسي 💵. 💸💲 دواړه حاصل کړه پوهان به دی علم ته درناوی ولري اوناپوهان به دي پیسو ته... t.co" cleaned_text = cleaner.clean(noisy_txt) print(cleaned_text) # Output: په ژوند کی علم , او پيسي دواړه حاصل کړه پوهان به دی علم ته درناوی ولري او ناپوهان به دي پیسو ته ``` Parameters of the `clean` method: - `text` (str or list): Input noisy text to clean. - `split_into_sentences` (bool): Split text into sentences. - `remove_emojis` (bool): Remove emojis. - `normalize_nums` (bool): Normalize Arabic numerals (1, 2, 3, ...) to Pashto numerals (۱، ۲، ۳، ...). - `remove_puncs` (bool): Remove punctuations. - `remove_special_chars` (bool): Remove special characters. - `special_chars` (list): List of special characters to keep. ### Tokenization (Space Correction) This module corrects space omission and insertion errors. It removes extra spaces and inserts necessary ones: ```python from nlpashto import Tokenizer tokenizer = Tokenizer() noisy_txt = 'جلال اباد ښار کې هره ورځ لس ګونه کسانپهډلهییزهتوګهدنشهيي توکو کارولو ته ا د ا م ه و رک وي' tokenized_text = tokenizer.tokenize(noisy_txt) print(tokenized_text) # Output: [['جلال', 'اباد', 'ښار', 'کې', 'هره', 'ورځ', 'لسګونه', 'کسان', 'په', 'ډله', 'ییزه', 'توګه', 'د', 'نشه', 'يي', 'توکو', 'کارولو', 'ته', 'ادامه', 'ورکوي']] …