Nathanielight/NLP-for-amharic

Amharic Languages Text Analysis System ## Technology Stack - **Backend:** Django (Python) - **Frontend:** Django Templates, Chart.js (for visualization) - **NLP Libraries:** NLTK, langdetect, scikit-learn, BeautifulSoup4, pandas, numpy - **Database:** SQLite (default with Django) ## Features ### Text Collection and Preprocessing - Support for multiple file formats (HTML, XML, plain text) - Automatic markup removal - Language detection - Text normalization ### Text Analysis 1. **Tokenization** - Word-level tokenization - Special handling for Ethiopian language characters - Punctuation handling 2. **Statistical Analysis** - Word frequency calculation - Word ranking by frequency - Frequency vs. Rank plotting - Zipf's law correlation analysis - Product of rank and frequency calculation 3. **Text Normalization** - Stop word removal - Custom stemmer for Ethiopian languages - Prefix and suffix handling ### Luhn's Idea Implementation - Upper and lower cut-off points for word selection - Automatic index term selection - Word frequency distribution analysis ### Visualization Dashboard - Interactive dashboard for preprocessing steps (markup removal, tokenization, normalization, stopword removal) - Donut/progress charts using Chart.js ## Installation & Running the Project ### 1. Clone the repository ```bash git clone github.com cd IR-system-for-Amharic ``` ### 2. Create and activate a virtual environment ```bash python -m venv venv # On Windows: venv\Scripts\activate # On Mac/Linux: source venv/bin/activate ``` ### 3. Install dependencies ```bash pip install -r requirements.txt ``` ### 4. Run Django migrations ```bash python manage.py migrate ``` ### 5. Start the Django development server ```bash python manage.py runserver ``` ### 6. Access the Web Interface Open your browser and go to: 127.0.0.1 - Upload your text file (HTML, XML, or plain text) - View the analysis results and interactive preprocessing …