maimouna-mbacke/wolof-RAG

Local RAG system for Wolof linguistics - ChromaDB + LLM pipeline. # wolof-RAG **Retrieval-Augmented Generation for Wolof linguistics** *Semantic search · Multi-LLM generation · Confidence scoring · Evaluation dashboard* --- ## What is this? `wolof-RAG` is a local RAG system built for **Wolof**, a Niger-Congo language. The system indexes a curated corpus of primary Wolof linguistic sources into a semantic vector store, enabling grounded question-answering with full source traceability and confidence scoring. A companion to `wolof-nlp`, which handles Wolof text processing (tokenisation, POS tagging, NER, morphological glossing). --- ## Why this project matters Most mainstream NLP tooling has little to no support for Wolof. This project explores how Retrieval-Augmented Generation can be adapted to low-resource languages using curated linguistic corpora, semantic retrieval, and transparent confidence scoring. --- ## Features - **End-to-end RAG pipeline** — PDF extraction → linguistic chunking → vector indexing → retrieval → LLM generation - **Confidence scoring** — every chunk is scored; the system flags low-confidence responses rather than hallucinating - **Multi-LLM support** — RAG only (no API), Groq/Llama 3 (free), or Claude/Anthropic (paid) - **Structured knowledge base** — 5,585 segments across 13 linguistic categories (morphology, syntax, phonology, TAM, noun classes, clitics, etc.) - **Evaluation dashboard** — real-time latency, confidence tracking, and query history in the Streamlit interface - **Bilingual** — responds in the language of the question (French or English) --- ## Architecture ``` Wolof_Resources/ ← Source PDFs (grammar books, papers, dissertations) │ ▼ extract_text.py ← PDF text extraction (PyPDF2) │ ▼ segment_texts.py ← Chunking: clean → classify (5 types) → split │ ▼ extract_knowledge.py ← Keyword extraction → 13 linguistic categories │ ▼ build_rag.py ← ChromaDB vector indexing (5,585 segments) │ ▼ wolof_query.py ← Semantic sear …