FarahHammamii/Tanzania-Conservation-RAG-System

An end-to-end Retrieval-Augmented Generation system for querying multi-source Tanzania conservation data, featuring a full Explainable AI (XAI) panel. # Tanzania Conservation RAG System An end-to-end **Retrieval-Augmented Generation** system for querying multi-source Tanzania conservation data, featuring a full **Explainable AI (XAI)** panel. --- ## Data Sources | Source | Type | Records | |--------|------|---------| | **IUCN Red List** | Species assessments | ~23 k chunks | | **CRS** | Policy reports (PDF) | 18 documents | | **World Bank** | Development indicators | ~12 indicators | | **Protected Planet** | Protected area statistics | 1 summary | All sources are pre-processed into `data/processed/documents.jsonl` (24 k chunks). --- ## Architecture ``` conservation-rag/ ├── .env # API keys (copy from .env.template) ├── requirements.txt ├── data/ │ ├── processed/documents.jsonl # 24 k document chunks (existing) │ └── indices/ # built by build_indices.py ├── src/ │ ├── indexing/build_indices.py # FAISS + BM25 indexing pipeline │ ├── retrieval/ │ │ ├── ensemble.py # Weighted BM25 + FAISS fusion │ │ ├── multi_query.py # LLM query expansion (3 variations) │ │ ├── reranker.py # Cross-encoder reranking (BGE) │ │ └── self_query.py # Auto metadata filter extraction │ ├── xai/ │ │ ├── attribution.py # Token-level LOO attribution │ │ ├── confidence.py # 4-factor confidence scoring │ │ ├── citations.py # Answer->source mapping │ │ ├── counterfactual.py # Evidence scenario comparisons │ │ └── shap_explainer.py # SHAP Kernel Explainer │ ├── generation/groq_llm.py # Groq Mixtral streaming generation │ ├── evaluation/metrics.py # Hit@k, MRR, NDCG@k, latency │ └── utils/ │ ├── config.py │ ├── logger.py │ └── cache.py └── app/streamlit_app.py # Full chat UI with XAI sidebar ``` --- ## Quick Start ### 1. Install dependencies ```bash pip install -r requirements.txt ``` ### 2. Configure API keys ```bash cp .env.templat …