Tanzania-AI-Community/twiga-indabax-challenge

# 🚀 Twiga AI Challenge: Document Processing & RAG System ## 🎯 Overview This repository contains two interconnected challenges for building a complete document processing and RAG (Retrieval-Augmented Generation) system for academic research papers. ## 📝 What You'll Build This project is split into two main challenges: - **Document Parsing**: Convert academic PDFs into clean, structured markdown, preserving academic sections, citations, and logical hierarchy. This prepares your data for downstream AI tasks. - **RAG System**: Build a Retrieval-Augmented Generation (RAG) pipeline that uses your parsed documents to answer academic questions with accurate, cited responses. ## 📁 Project Structure ``` twiga-challenge-1/ ├── README.md # This file - Main overview ├── parsing-challenge/ # Challenge 1: Document Parsing │ ├── README.md # Parsing challenge documentation │ └── strategy1_llamaparse_direct.ipynb ├── rag-challenge/ # Challenge 2: RAG Implementation │ ├── README.md # RAG challenge documentation │ └── strategy1_chromadb_basic.ipynb RAG ├── data/ │ ├── papers/ # Original PDF files │ ├── input_papers/ # Parsed markdown files │ └── vector_store/ # Vector database storage └── LICENSE ``` ## 🏆 Challenge Progression ### Phase 1: Document Parsing - **Notebook**: `parsing-challenge/strategy1_llamaparse_direct.ipynb` - **Goal**: Parse academic PDFs into structured markdown for RAG. ### Phase 2: RAG System - **Notebook**: `rag-challenge/strategy1_chromadb_basic.ipynb` - **Goal**: Build a RAG pipeline for question answering over your parsed documents. ## 🛠️ Setup & Installation ### Prerequisites ```bash # Core dependencies pip3 install llama_parse pypdf together pydantic # RAG dependencies pip3 install chromadb sentence-transformers langchain openai pip3 install faiss-cpu numpy pandas matplotlib ``` ### API Keys Required Create a `.en …