yasmine-sassi/RAG-pipeline-for-TounsiLM-8b

A Retrieval-Augmented Generation (RAG) system built on a Tunisian Arabic knowledge base, designed to improve # RAG Pipeline for TounsiLM-8b A Retrieval-Augmented Generation (RAG) system built on a structured Tunisian Arabic knowledge base, designed to ground `alabenayed/TounsiLM-8b` in verified dialectal knowledge. ## Features - **Hybrid retrieval** — BM25 (40%) + semantic embeddings (60%) merged via Reciprocal Rank Fusion - **Query rewriting** — generates up to 3 query variants with Arabizi digit normalization (`7→h`, `5→kh`, `3→a` …) - **Automatic query routing** — detects entry type from the query and restricts search to the relevant category - **Confidence scoring** — high / medium / low signal per query based on mean retrieval score - **Token-based context truncation** — uses TounsiLM's tokenizer to stay within the 4 096-token context window - **Typed knowledge base** — 1 647 entries across 11 validated types, each with a Pydantic schema ## Project Structure ``` RAG/ ├── run_rag.py # CLI entry point ├── requirements.txt ├── tounsilm_rag_kaggle.ipynb # Kaggle test notebook └── rag_kb/ ├── data/ # JSON knowledge base files │ ├── expressions.json # Tunisian expressions │ ├── expressions2.json # Number slang │ ├── expressions3.json # Additional expressions │ ├── proverbs.json # Tunisian proverbs (1 276 entries) │ ├── food.json # Dishes & ingredients │ ├── rituals.json # Social rituals & greetings │ ├── code-switching.json # French-Tunisian code-switching │ ├── series_movies.json # Tunisian TV & films │ └── colors.json # Colors in Tunisian dialect │ ├── schemas/ # Pydantic validation schemas │ ├── base_schema.py │ ├── expression_schema.py │ ├── number_slang_schema.py │ ├── proverb_schema.py │ ├── food_schema.py │ ├── ritual_schema.py │ ├── code_switch_schema.py │ ├── media_schema.py │ └── color_schema.py │ ├── pipeline/ # Core RAG logic │ ├── qu …