chelone936/felagi-isr

A full-stack Amharic news retrieval system that scrapes, indexes, and enables efficient search and analysis of Amharic-language articles. # Felagi – Amharic Information Retrieval System Felagi is a **full-stack Amharic news retrieval and analysis system**. It integrates: * A **Django backend** (API + database) * A **Next.js frontend** (search UI) * **Python web scrapers** for Amharic news sources * A custom **Information Storage and Retrieval (ISR) system** with Amharic-specific NLP preprocessing, inverted indexing, and search. Built on **11,000+ scraped articles**, Felagi tackles the challenges of **low-resource languages** by enabling efficient and accurate retrieval of Amharic text. --- ## 📑 Table of Contents * Project Structure * Dataset * ISR System (NLP + Search) * Backend (Django) * Frontend (Next.js) * Scrapers * Data Flow * Setup & Installation * Usage * License --- ## 🗂 Project Structure ``` felagi/ # Django backend frontend/ # Next.js frontend isr_system/ # Amharic NLP + indexing + search scraper/ # Web scrapers outputt/ # Processed article outputs ``` --- ## 📊 Dataset Felagi is built on a dataset of **11,000+ scraped Amharic news articles**. To support **open research on low-resource languages**, I have made the dataset **publicly available on Hugging Face**: 👉 Amharic News Dataset on Hugging Face The dataset includes raw and processed news articles collected from major Amharic news outlets (VOA Amharic, Ethiopian Reporter, etc.) and can be directly used for: * Amharic **NLP research** (tokenization, stopword lists, stemming, etc.) * **Information Retrieval tasks** * **Corpus linguistics** and **statistical analysis** --- ## 🧠 ISR System (NLP + Search) Located in `isr_system/`, this module is **custom-built for Amharic text**. It applies **linguistically informed NLP** techniques for preprocessing, indexing, and retrieval. ### Amharic-Specific Features * **Abbreviation Expansion** → expands Amharic abbreviations using a curated dataset. * **Orthographic Normalization** → maps interchangeable letters (e.g., ሐ/ኀ/ሀ, ፀ/ጸ) t …