Miki-b/IR-system-for-Amharic

information retrieval system that retrieves searched queries from Amharic documents and ranks them according to their relevance # **Amharic Information Retrieval System** ## **Introduction and Overview** We are currently enhancing the retrieval process of Amharic search engines. This project focuses on developing efficient text preprocessing, indexing, and creating a posting file. After weighing every term in the corpus using **TF-IDF**, the IR system compares these weights with a given query using **Cosine Similarity** and ranks the documents for retrieval. --- ## **Statement of the Problem and Justification** Amharic is the official working language of the Federal Democratic Republic of Ethiopia and is spoken by over 20 million people. Despite its wide use in text processing activities in governmental, non-governmental, and private institutions, there are no standardized tools available for Amharic text processing, including: - No affixes dictionary for Amharic. - Lack of a general stemmer for Amharic text. - Absence of a standard stop-word list. We aim to address these gaps using the **Hornmorphology Stemmer** for extracting the roots of words, despite the algorithm's inefficiency at times. We will enhance it to handle large text more efficiently. --- ## **Methodology** This project is divided into four main parts, each addressing a specific problem in the IR system, from text preprocessing to query optimization and retrieval: ### **Part 1: Text Processing Pipeline** - **Objective**: Process Amharic language documents using various Python libraries to extract, normalize, and tokenize text. - **Key Actions**: - Extract text from **PDF files**. - Tokenize and normalize text, handling Amharic punctuation and apostrophes. - Analyze word frequencies and remove high-frequency index terms. - Save filtered tokens to a file for further use in indexing. ### **Part 2: Stop Word Removal and Index Generation** - **Objective**: Remove stop words to improve text analysis and create an index. - **Key Actions**: - Read stop words from **"stopwords.txt"**. - Remove stop words from tokenized t …