Akramtaha98/faiss-arabic-qa

FAISS index benchmarking (Flat, IVF-PQ, HNSW) vs. BM25 for low-resource Arabic QA - accuracy, latency, and index size tradeoffs. # FAISS-Based Retrieval Optimization for Low-Resource Arabic QA Benchmarking FAISS index types (Flat, IVF-PQ, HNSW) against a BM25 baseline for Arabic question-answering retrieval, evaluated on accuracy, latency, and index size. ## Key finding BM25 beats a general-purpose multilingual dense retriever by 10–15 recall points at every k, on every corpus tested — the opposite of what's usually assumed about Arabic retrieval. Within FAISS itself, HNSW is effectively lossless versus exact search, while IVF-PQ trades a small accuracy loss for the smallest index footprint. | Corpus | Method | R@1 | R@5 | R@10 | R@20 | MRR | |---|---|---|---|---|---|---| | ARCD (N=465) | Flat | 30.6 | 51.0 | 58.3 | 65.1 | .402 | | ARCD (N=465) | **BM25** | **45.9** | **70.5** | **75.9** | **79.7** | **.569** | | TyDi-AR (N=842) | Flat | 44.5 | 62.1 | 68.9 | 73.1 | .522 | | TyDi-AR (N=842) | **BM25** | **54.8** | **72.2** | **78.1** | **82.0** | **.629** | | Combined (N=1307) | Flat | 32.3 | 50.4 | 56.4 | 61.9 | .404 | | Combined (N=1307) | **BM25** | **44.0** | **66.5** | **72.4** | **77.3** | **.540** | Full comparison across all four methods (Flat, IVF-PQ, HNSW, BM25) and the accuracy-latency sweeps are in `results/results.json`. Results were independently reproduced end-to-end on two architecturally different machines (x86/aarch64 Linux and Apple Silicon macOS), with identical recall/MRR values on both. ## Repo structure ``` code/ Reproduction scripts data/ Small, redistributable dataset files (ARCD + TyDi QA Arabic subset) results/ results.json — the numbers behind every table in the paper ``` ## Reproducing the results **1. Install dependencies** ```bash python3 -m venv venv && source venv/bin/activate pip install torch --index-url download.pytorch.org pip install -r requirements.txt ``` **2. Get the embedding model** Download `sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2` into `model_minilm/` at the repo root: ```bash huggingf …