The-African-Research-Collective/itewe

This code contains all the processing and scripting for processing and running OCR on documents using consumer hardware # Ìtèwé DataTrove-native OCR pipeline for processing PDF documents with a server-backed vision model. ## Overview This repository focuses on a working local OCR pipeline and a monitoring UI, with scaffolding for Azure ingestion and postprocessing stages. **Working pieces:** - PDF reader - OCR inference via DataTrove `InferenceRunner` (server-backed endpoint) - JSONL output writer - Progress + record monitor UI (`OCRInferenceProgressMonitor`) **Scaffolded / placeholder (not implemented yet):** - Azure ingestion (`AzureManifestReader`, `AzureFetchPDF`) - Page regrouping (`GroupPages`) - Postprocess blocks (`LanguageTag`, `BoilerplateRemover`) - Quality scoring (`BertQualityScore`) - Stage scripts in `scripts/` (config loading and pipeline builders are TODO) ## Quick Start (Local OCR) 1. Install dependencies: ```bash uv pip install -e . ``` 2. Start your OCR server (OpenAI-compatible chat/completions endpoint). 3. Run the local pipeline with a config file: ```bash python tewe.py reader.backend=azure \ reader.azure.container_path=az://mycontainer \ reader.input_dir="mypdfdir" \ ocr.server_url=127.0.0.1 \ ocr.model_name=mradermacher/KarantaOCR-GGUF \ output.output_dir=outputs ``` ### Sample Config A ready-to-edit sample lives here: - `configs/tewe.yaml` Required keys for `tewe.py`: - `limit` - `monitor` - `job_name` - `ocr.server_url` - `ocr.model_name` - `output.output_dir` - `output.output_filename` - `reader.input_dir` ## Monitoring UI Enable monitoring in the config: ```yaml monitor: true ``` The monitor runs by default on: - `127.0.0.1` It exposes: - `/` list view - `/record?record_id=...` detail view The UI is served from: - `itewe/monitoring/ui/index.html` - `itewe/monitoring/ui/record.html` ## Project Structure ```yaml configs/ tewe.yaml itewe/ blocks/ assemble/ group_pages.py # placeholder postprocess/ boilerplate.py # placeholder language_tag.py # placeholder quality/ bert_score. …