yisMeb/amharic_IR_system

Amharic Information Retrieval (IR) System Overview This project aims to develop an Amharic Information Retrieval (IR) system, focusing on methodologies and strategies to enhance search capabilities in the Amharic language. The system addresses challenges specific to Amharic text processing, such as morphological complexity and the need for efficient stemming techniques. By leveraging current research and findings from various studies, this project provides a comprehensive foundation for creating an effective Amharic IR system, enabling better access to information in this underrepresented language. System Architecture The system architecture is composed of several interconnected modules designed for both document and query processing: Tokenization Module: Splits the raw text of documents and queries into individual tokens (words), preparing them for further processing. Stemming Module: Applies a longest match stemming approach to reduce words to their root forms, removing prefixes and suffixes while handling infixes and plural forms. Stopword Removal Module: Removes common stopwords that carry little semantic weight (e.g., "እና", "ስለዚህ") from documents and queries. Indexing Module: Maps terms to their corresponding document identifiers, facilitating efficient retrieval. The indexed data is stored in indexed_doc.json . Term Weighting Module: Calculates the importance of each term relative to the entire document corpus using term frequency (TF) and inverse document frequency (IDF). The term weights are saved in termWeights.json . Term Matching (Searching) and Retrieval Module: Matches processed query tokens against indexed terms in the document corpus, retrieving the most relevant documents based on the computed term weights. Implementation Technologies Used Programming Languages Python: The primary language for development due to its simplicity, extensive libraries, and strong community support. Libraries and Tools BeautifulSoup: Use …