Swahili-System/hesabu-ai

# Tanzanian Primary Mathematics Dataset Tools This workspace contains a small scraper for FlipHTML5 books and a dataset builder for Swahili primary mathematics fine-tuning data. ## Quick Start Scrape the configured book text: ```bash python3 scripts/scrape_fliphtml5.py --manifest books.json --out-dir data ``` By default, the fine-tune JSONL skips front matter such as title pages, copyright, contents, acknowledgements, and introduction pages. Keep those pages in the fine-tune file if needed: ```bash python3 scripts/scrape_fliphtml5.py --manifest books.json --out-dir data --include-front-matter ``` Also download page images listed in the FlipHTML5 config: ```bash python3 scripts/scrape_fliphtml5.py --manifest books.json --out-dir data --download-images ``` Build computed question-answer examples for fine-tuning: ```bash python3 scripts/build_qa_dataset.py --pages data/pages/all_pages.jsonl --out-dir data/qa ``` ## Outputs For each book id in `books.json`, the script writes: - `data/raw/ .html` - downloaded FlipHTML5 HTML snapshot. - `data/pages/ .jsonl` - one JSON object per extracted page. - `data/finetune/ .jsonl` - chat-style training examples. - `data/images/ /...` - optional page images when `--download-images` is used. - `data/qa/ .jsonl` - computed Q&A examples from parsed exercises. It also writes combined datasets: - `data/pages/all_pages.jsonl` - `data/finetune/all_finetune.jsonl` - `data/qa/all_qa.jsonl` For question-answering fine-tuning, prefer `data/qa/all_qa.jsonl`. The `data/finetune/all_finetune.jsonl` file is better for style and textbook-like teaching content. The QA builder includes computed arithmetic, multiplication, division (with remainders), place-value, money, fractions, percentage, average, and word-problem examples spanning Darasa la Kwanza through Saba. These examples are tagged with `metadata.skill` (for example `multiplication_word_problem` or `division_remainder`) and `metadata.grade`, so they can be inspected or filter …