oselumeseagbonrofo/Naija-Gov-Election-Scraper

This is a repo that scrapes and analyses social media data on the Enugu 2023 and Anambra 2025 Governorship Elections # Naija Gov Election Scraper Public-data election sentiment pipeline for governorship candidates in Enugu (2023) and Anambra (2025). ## Implemented Scope - Multi-source collection adapters: Nairaland (primary public collection), X (xpoz-sm MCP primary path), Facebook (public-search fallback, partial coverage). - Reddit collector exists in code but is currently disabled in the full pipeline orchestration path as relevant data couldn't be found from searches done on it is not a major hub for Nigerian political discourse . - Candidate mention tagging from configured canonical names and aliases - Sentiment scoring using VADER - Candidate-level summaries, profiles, and summary report generation - Configurable runtime via YAML settings ## Project Structure - main.py: full pipeline CLI entrypoint. - config/settings.yaml: states, candidates, limits, timeframe windows, X queries, HTTP/MCP options. - scripts/update_x_records.py: refresh only X records and recompute downstream outputs. - scripts/apify_x_refresh.py: normalize Apify X export schema variants and rebuild outputs. - src/naija_gov_election_scraper/pipeline.py: orchestration, append+deduplicate behavior, and artifact writing. - src/naija_gov_election_scraper/collectors: source collectors. - src/naija_gov_election_scraper/sentiment/analyzer.py: sentiment labeling and summary construction. - src/naija_gov_election_scraper/profiling/reporting.py: profile/report/csv writers. ## Setup 1. Create and activate a Python environment. 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. For X MCP collection, ensure Copilot CLI is installed/authenticated and xpoz-sm is configured in ~/.copilot/mcp-config.json. ## Run Full pipeline (all enabled collectors + analysis outputs): ```bash python main.py --config config/settings.yaml --verbose ``` X-only refresh (MCP-only in script path): ```bash python scripts/update_x_records.py --config config/settings.yaml --verbose ``` Apify export refresh (exampl …