ab-dek/Amharic-IRS

amharic search engine # Amharic Information Retrieval System an amharic search engine written in python, , designed to crawl, index, and search Amharic language documents from the web. ## Libraries Used - **Scrapy**(for crawling/scraping) - **SQLite**(metadata storage) - **Flask**(creating the API endpoint) ## Components The search engine consists of three main components: 1. **Crawler:** Discovers and fetches Amharic web documents. 2. **Indexer:** Processes these documents, creates an inverted index, calculates TF-IDF related scores, and stores metadata. 3. **Query Engine:** Accepts user queries, processes them, and retrieves relevant documents using the index, serving results via a Flask API. A **Preprocessor** library is developed to handle Amharic text processing and improve the performance of the IR system. the preprocessing pipeline includes cleaner(removing puncutations, stopwords...), transliteration, abbreviation expansion and stemming. #### the Stemmer Due to the highly inflected nature and complex morphology of Amharic, a hybrid stemming approach was developed. the approaches include: 1. Lookup Table 2. Rule-Based Stemming 3. Iterative Affix Stripping ## Running the System ### Crawler The crawling process was initiated using a set of seed URLs known to contain Amharic content. The selected seed URLs were: - Addis Standard: addisstandard.com - Ethiopian News Agency (ENA): ena.et - Ethiopian Reporter: ethiopianreporter.com - Voice of America (VOA) Amharic: amharic.voanews.com - Walta Information Center: waltainfo.com Individual spiders were developed for each target website. to run a crawler spider: ```bash scrapy crawl -o repository/ ``` ### Indexer Raw document files (JSON) crawled are processed and transformed into searchable datastructure including term weights(inverted_index.json and idf.json) and SQLite database (metadata.db) is created to store metadata. to run the indexer: ```bash python indexer.py ``` ### …