Tanzania-AI-Community/twiga-warehouse

# Twiga Warehouse This repository provides tools for extracting text and important information from textbooks, preparing them for ingestion into a vector database for use with Large Language Models (LLMs) in its parent project, Twiga. Currently, the extraction pipeline only supports text from PDF files, chunking and splitting content using different strategies, and outputting structured data suitable for downstream LLM applications. ## Features - Extracts text from PDF textbooks. - Supports multiple chunking strategies (e.g., LangChain, LLM-based). - Outputs structured JSON for easy ingestion into vector databases. - Modular design for easy extension and integration. ## Installation 1. Clone the repository: ```sh git clone github.com cd twiga-warehouse ``` 2. Install dependencies: ```sh pip install -r requirements.txt ``` You can also install the dependencies via the `pyptoject.toml` file, if preferred. 3. Set up your environment variables: ```sh cp .env.template .env # Edit .env as needed ``` The two Unstructured env variables are not mandatory for running the pipeline. Make sure `INPUT_BOOKS_PATH` and `OUTPUT_BOOKS_PATH` point to your input/output directories. ## Before running the pipeline Parsing a PDF and splitting its contents in chunks is not an easy task. For this pipeline to work, we are making some assumptions on how we must store the book so that we can parse it. 1. Where should the books placed for their parsing? You can place the books wherever you want, as long as the root directory is stored in `INPUT_BOOKS_PATH`. Inside that directory, create a folder per book. We recommend organizing by form and subject, for example: ``` ${INPUT_BOOKS_PATH}/form_4/biology/ biology_form_4.pdf info.yaml ``` For CLI runs, `--input_dir` is the relative path (e.g. `form_4/biology/`) and `--input_file_name` is the PDF filename. For Dagster runs, the `subject_name` and `form` config values are used to build t …