dantesfai/tig_stopwords

Stopwords Generator for Tigrinya # Stopwords Generator for Tigrinya ## Purpose **Stopword Generator** is a Python tool for generating stopwords in **Tigrinya** text using an **Inverse Document Frequency (IDF)**-based approach. This tool can handle both **CSV** and **TXT** file formats and supports preprocessing text, removing non-informative words, and visualizing word importance through IDF scores. ### Features - **Multi-file support**: Process both **TXT** and **CSV** files. - **Customizable settings**: Control how text is processed via a **config.json** file. - **IDF-based stopword generation**: Identifies and removes common words based on their IDF scores. - **Visualization**: Generate IDF plots to visualize the distribution of word importance. - **Configurable IDF threshold**: Control which words are considered stopwords using the IDF threshold. - **Command-Line Interface (CLI)**: A simple CLI for ease of use. ## Installation ### Prerequisites Ensure that you have **Python 3.x** installed. You'll also need to install the required dependencies via **pip**. ### Steps to Install 1. Clone the repository to your local machine: ```bash git clone github.com ``` 2. Navigate to the project directory: ```bash cd stopword-generator ``` 3. Install required dependencies: ```bash pip install -r requirements.txt ``` 4. (Optional) If you want to run tests, you can install pytest: ```bash pip install pytest ``` ## Usage ### Command-Line Interface (CLI) To run the stopword generator on your text files: ```bash python stopword_generator.py data/input.txt -c config.json -o output/stopwords.txt -t 2.5 ``` **Arguments**: - data/input.txt: Path to the input file (either .txt or .csv file). - -c config.json: Path to the configuration file (default is config.json). - -o output/stopwords.txt: Path where the stopwords file will be saved (default is stopwords.txt). - -t 2.5: Threshold for IDF to identify stopwords (default is 2.5). ### Programmatic Usage ` …