Divyansh-Artixio/Ghana-Scraper-GatesFoundation

# Ghana FDA Regulatory Events Scraper A comprehensive Python scraper and ETL pipeline that extracts **Product Recalls, Alerts, and Public Notices** from FDA Ghana's newsroom and stores them in a normalized PostgreSQL database structure. ## 🎯 Features - **Multi-content scraping**: Product Recalls, Alerts, and Public Notices - **Robust PDF handling**: Direct PDFs, HTML fallbacks, OCR extraction - **Error resilience**: Handles 404s, broken PDFs, missing content - **Normalized database**: Separate tables for events, companies, countries - **AI enrichment**: Automatic company metadata enhancement (optional) - **Comprehensive logging**: Full audit trail of scraping activities - **Duplicate prevention**: Smart company deduplication and reason extraction - **Multi-product support**: Handles complex multi-product recalls ## 📊 Latest Results - ✅ **77 Product Recalls** with detailed reasons - ✅ **29 Product Alerts** - ✅ **67 Public Notices** - ✅ **173 Total Records** processed in ~20 minutes - ✅ **46 Unique Companies** (no duplicates) - ✅ **All PDF files** downloaded and organized ## 📂 Database Schema The scraper uses a normalized PostgreSQL database with three main tables: ### `regulatory_events` Stores all regulatory events (recalls, alerts, notices) with event-specific fields including: - Event type, dates, product information - Company relationships (manufacturer_id, recalling_firm_id) - Detailed recall reasons and actions - PDF paths and source URLs ### `companies` Stores company information with optional AI-enriched metadata: - Company name, country, type (Manufacturer/Reselling Firm) - Founding dates, founders, business descriptions (AI-enhanced) ### `countries` Reference table with country codes and WHO maturity levels. ## 🚀 Quick Start ### 1. Prerequisites - Python 3.8+ - PostgreSQL 12+ - Chrome/Chromium browser (for Playwright) ### 2. Installation ```bash # Clone the repository git clone github.com cd Ghana # Create vir …