Balisa50/gambia-political-risk

Sentiment + topic NLP pipeline producing a weekly Political Risk Index for The Gambia # Gambia News Sentiment Analyzer & Political Risk Index End-to-end NLP pipeline that scrapes Gambian news, runs sentiment + topic analysis, and produces a weekly Political Risk Index (PRI) for The Gambia. Every step is documented, every technical choice is justified. **Live demo**: Dashboard on Vercel · API on Render **Data sources**: The Point Newspaper, Foroyaa, Gainako, Standard Newspaper ## Why this exists The Gambia has no public-facing political risk index. International indices (Moody's, Economist Intelligence Unit, Fitch) cover the country annually with proprietary methodology and a paywall. Citizens, diaspora investors, and small businesses making decisions about risk in The Gambia have nothing they can read in real time. This project builds a transparent, weekly, public PRI grounded in actual Gambian news coverage. Anyone can audit the methodology. ## Pipeline ``` news scrape → preprocessing → features → sentiment + topics → weekly PRI → API + dashboard ``` ### 1. Data collection (`src/scraper.py`) - BeautifulSoup + requests - 4 sources, polite rate limiting (1-3s random delays), graceful failure per page - Target 5,000+ articles across all sources - Output: `data/raw/gambia_news_raw.csv` ### 2. Preprocessing (`src/preprocessor.py`) - Lowercase, strip HTML / URLs / emails / non-text characters - NLTK tokenisation, English stopword removal, WordNet lemmatisation - Drop empty / short articles, de-duplicate by URL and headline - Output: `data/processed/gambia_news_clean.csv` ### 3. Feature engineering (`src/features.py`) - **TF-IDF** (10k features, bigrams, min_df=3, max_df=0.85) - **Sentence embeddings** via `sentence-transformers/all-MiniLM-L6-v2` - **Article-level**: word/sentence count, lexical diversity, NER count - **Topic flags**: political / economic / crime via curated keyword sets - **Date features**: day/month/quarter/year, days from nearest election, COVID period flag - Output: `data/features/article_features.pkl` ### 4. Sentiment (`src/ …