Odelolasolomon/RAG-Based-Semantic-Search-System-Agro-Food-Domain-in-Africa-

Developed a semantic search system using RAG and Gemini embeddings to improve access to agro-food research. Achieved a 30% boost in relevance by enhancing query understanding, reranking with Gemini, and combining TF-IDF with semantic filters. Scaled retrieval with ChromaDB and LangChain, enabling precise answers from 1,000+ research chunks. # AgroFoodAfrica RAG System ## Overview The **AgroFoodAfrica RAG (Retrieval-Augmented Generation) System** is a sophisticated AI-powered platform designed to provide comprehensive, data-driven insights about agriculture and food security across Africa. By combining advanced document retrieval from a specialized knowledge base with Google Gemini's cutting-edge generation capabilities, the system delivers accurate, contextual responses to complex agricultural queries with proper source attribution. This system addresses the critical need for accessible, reliable agricultural information in Africa by leveraging state-of-the-art natural language processing and machine learning technologies. ## 🌟 Key Features ### Core Capabilities - **📄 Advanced Document Processing**: Seamlessly converts PDFs from arXiv and other academic sources into searchable Markdown format - **🗄️ Vector Database Integration**: Utilizes ChromaDB with Gemini embeddings for high-performance semantic search - **🔍 Hybrid Search Technology**: Combines vector similarity and keyword matching for optimal information retrieval - **🤖 Intelligent Agent Orchestration**: Coordinates multiple specialized components based on query intent analysis - **📊 Comprehensive Evaluation Framework**: Features automated metrics and human-aligned performance ratings ### Advanced RAG Pipeline - **Query Preprocessing**: Intelligent stopword removal and keyword expansion - **Context-Aware Generation**: Produces responses with proper source attribution and citations - **Real-Time Information**: Integrates web search for up-to-date agricultural data - **Structured Data Extraction**: Automated visualization and data processing capabilities ## 🏗️ System Architecture ### Core Modules #### 1. Document Ingestion Pipeline - **PDF Extraction**: Powered by Docling for high-fidelity document conversion - **Text Segmentation**: Uses LangChain's RecursiveCharacterTextSplitter for optimal chunk sizing - **Embedding Generation**: Cu …