nalonix/Amharic-Stemmer-and-Search-Engine

# Amharic Document Search Engine A TF-IDF based search and retrieval system for Amharic documents. This project demonstrates the complete pipeline: document preprocessing, stemming, indexing, and ranking using vector space model. ## Table of Contents - Overview - Project Structure - Pipeline Architecture - Setup & Installation - Usage - Data Processing - Search & Retrieval - How It Works --- ## Overview The system processes a corpus of ~14 Amharic news documents and enables full-text search using: - **Stemming**: Morphological analysis to extract root forms - **Tokenization**: Break documents into stemmed terms - **Indexing**: Build inverted index and document frequency tables - **Ranking**: TF-IDF scoring with cosine similarity - **Retrieval**: Return ranked results matching user queries ### Key Features ✓ Stemmed document tokenization ✓ Inverted index construction ✓ TF-IDF vector representation ✓ Cosine similarity ranking ✓ Query preprocessing and retrieval --- ## Project Structure ``` stemmer-amh/ ├── documents/ # Input corpus (14 Amharic news documents) │ ├── news1.txt │ ├── news2.txt │ └── ... (through news14.txt) │ ├── preprocess.js # Stage 1: Document tokenization & stemming ├── processed.json # Output: Stemmed tokens per document │ ├── build_inv_idx.js # Stage 2: Build indexes ├── out/ │ ├── index.json # Inverted index (term → {docId: tf}) │ └── df.json # Document frequency (term → count) │ ├── searchEngine.js # Stage 3: Search implementation (TF-IDF + cosine) ├── testSearch.js # Stage 4: Test queries │ └── [Stemming modules] ├── stemmer.js ├── fidel.js ├── phonetic.js ├── affixes.js └── lexicon.js ``` --- ## Pipeline Architecture ``` ┌─────────────────────────────────────────────────────────────────┐ │ 1. PREPROCESS (preprocess.js) │ ├─────────────────────────────────────────────────────────── …