tsegaimerhawi/tigrinya-web-scraper

# 📰 Tigrinya News **Browse Haddas Ertra and ask questions in Tigrinya or English—powered by RAG and the Ge'ez script.** A full pipeline for **Tigrinya news**: scrape PDFs from Haddas Ertra, extract and clean Ge'ez text, embed with **LlamaIndex** + **Gemini**, store in **Qdrant**, and query via a **React + FastAPI** app. Built for researchers, linguists, and anyone working with Tigrinya NLP and low-resource language tech. --- ## ✨ What You Can Do | **In the app** | **In Script Runner** | |----------------|----------------------| | 📖 Browse scraped articles with full text and metadata | 🕷️ Download Haddas Ertra PDFs (configurable limit) | | 📋 Copy text, run word frequency, stats, sentence extraction | 📄 Extract Ge'ez text, NER, image descriptions → `raw_data.json` | | 🤖 **Ask questions in Tigrinya or English**—answers from the ingested corpus (RAG) | 📦 Ingest into Qdrant (LlamaIndex + Gemini embeddings) | | 🔍 Semantic search over the news corpus | ✅ Check Qdrant, validate metadata and raw data | The **app** (React + FastAPI) is for **reading and asking**. The **Script Runner** (or CLI) is for **scraping, processing, and ingesting**. Run the pipeline once (or periodically), then use the app to explore and query. --- ## 🏗️ How It Fits Together ``` ┌─────────────────────────────────────────────────────────────────┐ │ Pipeline UI: localhost (or script_runner │ │ on :8765) → Scraper, PDF Processor, Llama Ingest │ └───────────────────────────┬─────────────────────────────────────┘ │ ▼ pdf_metadata.json raw_data.json Qdrant │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ App: localhost (backend :8000) │ │ Articles (browse, copy, NLP tools) + Ask (RAG) │ └─────────────────────────────────────────────────────────────────┘ ``` --- ## 🛠️ Tech Stack - **Frontend:** React, TypeScript, Vite - **Backend:** FastAPI, Python 3.8+ - **Scraping:** Playwri …