Omer-alt/Tundah-RAG

Building a Cultural Retrieval-Augmented Generation (RAG) System to Combat Misinformation About Customary Marriages in Africa. # Advenced RAG Local RAG pipeline we're going to build: All using open-source tools. In our specific example, we will create TündahChat, a RAG workflow that allows a person to query knowledge bases about customary marriage practices in Africa in general. Note: Tündah is a web platform where information is published on how marriages are organized in Cameroon in particular and in Africa in general. You can also run notebook `Tundah.ipynb` directly in locally ### Project Structure - `Tundah.ipynb`: This notebook outlines the sequential workflow of Tundah-RAG, providing a step-by-step process. - `assets`: A repository for supplementary data that supports the project. - `Pdf_path`:This directory holds all the source PDFs, which serve as the information backbone for our RAG system. - `static`: Contains supplementary assets related to the Streamlit interface, including logos and other media. - `Structured_file`:Stores the processed results of the PDFs, refined through the Marker model for structured output. - `Transcript_path`:Includes the video.json file, which contains links and titles of YouTube videos relevant to customary marriages in Africa. - `Tundah/Classes`: This folder organizes classes designed to structure our codebase, making it reusable, maintainable, and compliant with best practices in software engineering. - `main.py`: Provides a command-line interface for testing the code. - `streamlit.py`:The Streamlit interface facilitates a seamless user-RAG interaction, enhancing usability. ## Dataset The dataset used to build the RAG system focuses on aspects related to customary marriages in 10 African countries: Cameroon, Kenya, Nigeria, South Africa, Zimbabwe, Tanzania, Uganda, Botswana, and Mali. Currently, the data is sourced from two main channels: cultural articles/books and YouTube videos. In the future, we plan to perform web scraping on websites and blogs that are rich in reliable and accessible information. ## Architecture #### Global archit …