amaboh/african_financial_markets

This a repository implementing a data pipeline of African Financial Market Data # African Financial Markets This repository implements a data pipeline for African Financial Market data. ## 1. Overview The objective of this project is to provide data to equity analysts and investment professionals focusing on African Capital Markets. This project implements an ELT pipeline by scraping data using Scrapy and uploading it to Google Cloud Platform. Analysts can then perform high-level transformations and analysis in BigQuery using dbt and visualize the data using Tableau. The data is scraped from african-markets. ### 1.1 Motivation This project is motivated by my experience as a treasury analyst, which involved analyzing investment instruments on the African markets where treasury reserves could be invested for high ROI with low risk. Additionally, as a freelancer on Upwork, I was assigned to a gig as a Research/Investment Analyst, which necessitated access to financial data behind paywalls. ## 2. Project Overview This project demonstrates the implementation of a highly scalable and containerized web crawler using Scrapy to scrape data at regular intervals and upload the data to Google Cloud. The scraped data is first loaded into PostgreSQL, then queried and uploaded to Google Cloud Storage, and subsequently to BigQuery. ## Architecture ### 2.1 Tech Stack 1. **Terraform**: Used to provision the cloud infrastructure, specifically Google Cloud Storage and BigQuery. Cloud Composer could also be provisioned instead of using Airflow, but Airflow is more cost-effective. 2. **Scrapy**: Used to scrape the website and manage the scraping process. 3. **Airflow**: Orchestrates the workflow from data collection to loading data into BigQuery. The Google alternative is Cloud Composer. 4. **Docker**: Containerizes the data pipeline to ensure portability and consistency. 5. **PGAdmin**: Manages the PostgreSQL database. 6. **Google Cloud Resources**: - **Google Cloud Storage**: Google's immutable object storage for unstructured data. - **BigQuery**: Google's …