liateg/Text-analyzer-project

TextAnalyzer is a Python project for text manipulation and analysis of Amharic and Tigrigna texts, including data cleaning, tokenization, stopword removal, and grapheme-to-phoneme conversion using IPA. # Text-analyzer-project ## Project Overview TextAnalyzer is a Python project designed to manipulate text and analyze Amharic and Tigrigna texts. It includes functionalities for loading data, cleaning text, tokenizing, removing stopwords, counting word frequencies, extracting unique and overlapping words, and converting graphemes to phonemes using IPA (International Phonetic Alphabet). ## Installation Instructions To install and set up the project, follow these steps: ### Clone the repository: ``` git clone github.com cd Text-analyzer-project ``` ### Create a virtual environment (optional but recommended): ``` python -m venv venv source venv/bin/activate # On Windows use venv\Scripts\activate ``` ### Install the required libraries: `pip install pandas nltk` ### Download necessary NLTK data: ``` import nltk nltk.download('punkt') nltk.download('stopwords') ``` Add Amharic and Tigrigna stopwords files: NLTK does not natively support Amharic and Tigrigna stopwords. To use these languages, you need to add custom stopword files to the NLTK data path. Save the Files: `amharic` and `tigregna` in this repository in the` nltk_data/corpora/stopwords` directory. If this directory does not exist, you can create it. ## Usage ### Here’s how you can use the various functions in this project: ### Loading Data ### To load data from a fixed-width formatted file: ``` from text_analyzer import load_data file_path = 'path/to/your/file.txt' data = load_data(file_path) ``` ### Note: Ensure that the first line of the Amharic text file is “Amharic” and the first line of the Tigrigna text file is “Tigregna”. ## Cleaning Data ### To clean the text data by removing everything except for spaces and word characters: ``` from text_analyzer import clean_data cleaned_data = clean_data(data, 'column_name') ``` ## Tokenizing Data ### To tokenize the cleaned text data: ``` from text_analyzer import tokenize_data tokens = tokenize_data(cleaned_data, 'col …