YoPerry007/gse-corporate-actions-crawler

FastAPI crawler for Ghana Stock Exchange dividend and AGM announcements # GSE Corporate Actions Crawler This FastAPI service crawls the Ghana Stock Exchange press-release archive and extracts: - dividend amount, currency, type, record date, ex-dividend date, payment date, and status; - AGM/EGM date, time, venue, meeting format, company, ticker, and fiscal year. One GSE announcement may create both a dividend record and a meeting record. Records are upserted by the original GSE announcement URL, so scheduled runs do not create duplicates. ## 1. Secure the database account If a MongoDB password has ever appeared in source code, rotate it in MongoDB Atlas before continuing. Add the replacement URI only to `.env`; `.env` is ignored by Git. ## 2. Create the environment ```powershell python -m venv .venv .\.venv\Scripts\Activate.ps1 python -m pip install --upgrade pip python -m pip install -r requirements.txt Copy-Item .env.example .env ``` Edit `.env` and insert the rotated `MONGO_URI`. ## 3. Verify without changing the database ```powershell python -m unittest discover -s tests -v python crawler.py --dry-run --max-pages 1 ``` Use `--max-pages 0` only when you intentionally want to backfill the entire archive. Normal scheduled runs use `GSE_MAX_PAGES` from `.env` to avoid repeatedly crawling hundreds of historical pages. ## 4. Run the API ```powershell uvicorn main:app --reload ``` Open for the interactive FastAPI documentation. Key endpoints: - `GET /api/v1/dividends` - `GET /api/v1/meetings` - `POST /api/v1/crawler/trigger?max_pages=1` - `GET /api/v1/crawler/status` The scheduler runs at 10:00 GMT on weekdays. Ghana uses GMT throughout the year. ## Data-quality behavior - Missing amounts or meeting dates remain `null`; the crawler does not invent zero values or today's date. - Incomplete or inferred records are marked `needs_review: true`. - PDFs without selectable text are reported in the crawl summary because they require an OCR extension. - The crawler identifies itself with a user agent, reads `robots.txt`, retries …