obash123/African-Language-Sentiment-and-Topic-Dashboard

# African Language Sentiment & Topic Model Dashboard A multilingual sentiment/topic dashboard for Yoruba, Hausa, Igbo, Nigerian Pidgin, and Swahili social comments — YouTube-comment scraping, a fine-tuned DistilBERT sentiment classifier, per-language topic modeling, and a Streamlit dashboard, containerized for Google Cloud Run. | Piece | Status | |---|---| | YouTube comment scraper (`scraper/`) | **Built and unit-tested**, not run against live YouTube at scale — see below | | Sentiment model (`nlp/train_sentiment.py`) | **Actually fine-tuned** on the real `masakhane/afrisenti` dataset — real accuracy in `models/sentiment-distilbert/metrics.json` | | Topic modeling (`nlp/topics.py`) | **Actually run** (BERTopic, multilingual sentence-transformer) to produce the demo data | | Dashboard (`dashboard/app.py`) | **Built and run locally**, verified in-browser | | Cloud Run deployment (`deploy/`) | **Prepared, not deployed** — Dockerfile/cloudbuild/deploy.sh are ready to run; deploying needs your own GCP project + billing + the `gcloud` CLI | ## Architecture ``` YouTube video IDs │ ▼ scraper/pipeline.py ──(youtube-comment-downloader, Selenium fallback)──▶ data/raw/*.csv │ ▼ nlp/lang_filter.py (tags each comment: hau / yor / ibo / pcm / swa) │ ▼ nlp/infer_sentiment.py (fine-tuned DistilBERT: negative / neutral / positive) │ ▼ nlp/topics.py (BERTopic per language, shared multilingual embedding space) │ ▼ dashboard/app.py (Streamlit + Plotly: heatmap, topic trends, comment explorer) │ ▼ deploy/ (Dockerfile → Cloud Run) ``` ## Setup ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` ## Train the sentiment model ```bash python -m nlp.train_sentiment --epochs 3 --max-train-per-lang 1600 ``` Fine-tunes `distilbert-base-multilingual-cased` into a 3-class classifier across the five AfriSenti language configs, evaluates on the full held-out test split per language, and writes the checkpoint + `metrics.json` to `models/sentimen …