tsegaimerhawi/tigrinya-agent

Scaling Tigrinya NLP with LangGraph agents and Qdrant. Automated data acquisition from Haddas Ertra and self-healing linguistic verification. # Tigrinya Agent - Complete NLP Pipeline A comprehensive tool for scraping, processing, and analyzing Haddas Ertra Tigrinya newspapers from shabait.com. Features a complete NLP pipeline from raw PDF extraction to advanced linguistic analysis. ## Features ### 🖥️ **Script Runner UI** - **Single dashboard**: Run all pipeline scripts (Scraper, PDF Processor, Llama Ingest, Pipeline, Store Data, etc.) from one web interface - **Live output**: Stream script output in real time - **Configuration**: Central popup to set scraper limit, Qdrant host/port, collection names, batch sizes ### 📥 **Data Acquisition** - 🕷️ **Automated Scraping**: Download a configurable number of Haddas Ertra newspaper PDFs (default 20) - 📄 **Multi-page Navigation**: Handles pagination to access older articles - 🔍 **Smart PDF Detection**: Locates download links using image-based navigation ### 🧹 **Text Processing** - 📖 **PDF Extraction**: Advanced text extraction from newspaper PDFs - 🌍 **Ge'ez Script Focus**: Preserves only Tigrinya characters, numbers, and punctuation - 🧹 **Noise Removal**: Eliminates English words, navigation elements, and artifacts - 📊 **Quality Validation**: Word count verification and content checks ### 🤖 **NLP Pipeline** - 🏷️ **Named Entity Recognition**: 3-agent NER pipeline (extract → validate → refine) with Gemini 2.5 Flash - 🌍 **Geopolitical Focus**: PERSON, LOCATION, GPE, ORGANIZATION, DATE, etc. ## Installation ### Prerequisites - Python 3.8+ - Git ### Step 1: Clone the Repository ```bash git clone github.com cd tigrinya-agent ``` ### Step 2: Create Virtual Environment ```bash python -m venv .env source .env/bin/activate # On Windows: .env\Scripts\activate ``` ### Step 3: Install Dependencies ```bash pip install -r requirements.txt ``` ### Step 4: Install Playwright Browsers ```bash playwright install chromium ``` ### Step 5: Set up Google Gemini API Key The NER pipeline requires a Google Gemini API key (for ent …