kidist-amde/amharic-ir-benchmarks

Official codebase for the ACL 2025 Findings paper: Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval. # Amharic-IR-Benchmarks This repository accompanies our ACL 2025 Findings paper: **"Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval"** ✨ We provide a reproducible benchmark suite for Amharic information retrieval, including: * BM25 sparse baseline * Dense embedding models (RoBERTa / BERT variants fine-tuned for Amharic) * ColBERT-AM (late interaction retriever) --- ## 👐 Features - **Pretrained Amharic Retrieval Models** Includes ( RoBERTa-Base-Amharic-Embd, RoBERTa-Medium-Amharic-Embd, BERT-Medium-Amharic-Embd, and ColBERT-AM for dense retrieval.) - **Hugging Face model & dataset links for easy access** - **Training, evaluation, and inference scripts for reproducibility** - **Benchmarks BM25 (sparse retrieval), bi-encoder dense retrieval, and ColBERT (late interaction retrieval) for Amharic.** - **MS MARCO-style dataset conversion script & direct dataset links** --- ## 📁 Repository Structure ``` amharic-ir-benchmarks/ ├── baselines/ # BM25, ColBERT, and dense Amharic retrievers │ ├── bm25_retriever/ │ ├── ColBERT_AM/ │ ├── colbert-amharic-pylate/ │ └── embedding_models/ ├── data/ # Scripts to download, preprocess, and prepare datasets ├── scripts/ # Shell scripts for training, indexing, evaluation ├── utils/ # Utility functions ├── amharic_environment.yml # Conda environment ├── requirements.txt └── README.md ``` --- ## ⚙️ Installation ### Conda (Recommended) ```bash conda env create -f amharic_environment.yml conda activate amharic_ir ``` Or using pip: ```bash pip install -r requirements.txt ``` --- ## 📚 Datasets We use two publicly available Amharic datasets: | Dataset | Description | Link | | ---------------- | ----------------------------------- | ---------------------------------------------------- …