gabrielntwari/rwanda-jobs-scraper

Comprehensive job market dataset for Rwanda built by scraping multiple online platforms including Mifotra's civil service portal. Includes historical data from 2017 archived via Wayback Machine. # 🇷🇼 Rwanda Jobs Scraper Automated job scraping system for Rwanda with multi-page dashboard. ## 📊 Features - **10 Job Sources**: JobInRwanda, NewTimes, GreatRwandaJobs, Mucuruzi, MIFOTRA - **Automated Scraping**: Runs daily at 6 AM EAT via GitHub Actions - **Auto-Deduplication**: Removes duplicate jobs automatically - **Beautiful Dashboard**: Multi-page Plotly Dash interface - **Database**: Supabase PostgreSQL ## 🚀 Live Dashboard 🌐 **Dashboard URL**: [rwanda-jobs-scraper.onrende…] ## 📦 Project Structure ``` rwanda_jobs/ ├── scrapers/ │ ├── jobinrwanda_scraper.py │ ├── newtimesjobs_scraper.py │ ├── greatrwandajobs_scraper.py │ ├── mucuruzi_scraper.py │ └── mifotra_scraper.py ├── db_adapter.py ├── run_all_5_scrapers.py # Master scraper with auto-dedup ├── multi_page_dashboard.py # Dashboard ├── requirements.txt # Dependencies └── .github/workflows/ └── scrape_jobs.yml # Automated scheduling ``` ## ⚙️ Setup ### 1. Clone Repository ```bash git clone github.com cd rwanda-jobs-scraper ``` ### 2. Install Dependencies ```bash pip install -r requirements.txt ``` ### 3. Set Database URL ```bash # Windows set DATABASE_URL=your_postgresql_url # Linux/Mac export DATABASE_URL=your_postgresql_url ``` ### 4. Run Scrapers ```bash python run_all_5_scrapers.py ``` ### 5. Run Dashboard ```bash python multi_page_dashboard.py # Open localhost ``` ## 🤖 Automated Scraping GitHub Actions runs the scrapers **daily at 6:00 AM EAT** automatically. To enable: 1. Add `DATABASE_URL` secret in GitHub Settings → Secrets 2. Push code to GitHub 3. GitHub Actions will run automatically ## 📊 Database Schema **Table**: `jobs` Key columns: - `title`, `company`, `source`, `sector` - `district`, `employment_type`, `job_level` - `experience_years`, `education_level` - `description`, `posted_date`, `deadline` - `source_url`, `scraped_at`, `is_active` ## 🎨 Dashboard P …