Ephrem0001/Amharic-Rag-Engine

# Amharic RAG Engine A production-oriented **Retrieval-Augmented Generation (RAG)** API for Amharic-language documents. Upload PDFs, ask questions in Amharic, and receive answers grounded in your documents with source citations. --- ## Overview This service provides: - **Document ingestion:** PDF upload with configurable chunking and embedding (RoBERTa-based Amharic text embeddings). - **Vector search:** FAISS indices per document and embedding model for fast similarity retrieval. - **Answer generation:** Context-conditioned answers in Amharic via a LLaMA-based generator (e.g. `rasyosef/Llama-3.2-400M-Amharic`), with citations tied to retrieved chunks. Typical flow: **Upload PDF → Chunk → Embed → Index (FAISS)**; then **Ask question → Embed query → Retrieve top‑k chunks → Generate answer with citations.** --- ## Architecture | Layer | Technology | |-------|------------| | **API** | FastAPI (Python 3.10+) | | **Database** | PostgreSQL (documents, pages, chunks, metadata) | | **Embeddings** | RoBERTa Amharic text embedding models (Hugging Face); sentence-transformers | | **Vector store** | FAISS (per-document, per–embedding-model indices on disk) | | **Generator** | LLaMA-based causal LM (e.g. Llama-3.2-400M-Amharic) via Hugging Face Transformers | - **Documents:** Stored on disk under `UPLOAD_DIR`; metadata and chunk text in PostgreSQL. - **Chunks:** Configurable target length (default 1000 chars, range 700–1200 recommended) with 10–15% overlap; each chunk gets a vector and is written to the document’s FAISS index. - **FAISS indices:** Stored as `{document_id}__{model_type}.faiss` and `{document_id}__{model_type}.mapping.json` under `INDEX_DIR` (e.g. `indexes/abc123__base.faiss`). Some specs use a single index per document (`{document_id}.faiss`); this project uses the `{document_id}__{model_type}.faiss` naming so the same document can have separate indices for base, medium, and finetuned embeddings, enabling direct comparison without reindexing. - **RAG:** …