txs7n/NG_energy_generation_etl_pipeline

An ETL data pipeline that extracts energy generation data from the TCN (Transmission Company of Nigeria) website and is automated using Apache Airflow with the synchronization of several AWS services such as an EC2 instance, AWS Lambda, and Eventbridge. # NG_energy_generation_etl_pipeline An ETL data pipeline that extracts energy generation data from the TCN (Transmission Company of Nigeria) website and is automated using Apache Airflow with the synchronization of several AWS services such as an EC2 instance, AWS Lambda, and Eventbridge. ## Data Hourly Generation by GenCos ## Tools - Amazon EC2: To host and run ETL operations. - AWS Lambda: To manage the start and stop actions of your EC2 instance based on specific triggers. - Amazon EventBridge: To schedule and trigger AWS Lambda functions for starting and stopping ec2 instance at designated times. - Airflow: To orchestrate and manage data collection workflows. - Amazon S3: For storing the collected energy data securely in the cloud. - Python: For data processing and modeling. - Pandas: Library used for data manipulation. - Requests: Library used to make HTTP requests to external websites. - BeautifulSoup: Library used for parsing HTML documents. ## Task The goal here is to first extract historic energy generation data from the TCN website since the time they started archiving data. While the site showed they started reporting generation data since 2004, I decided to start my data collection from 2021. This is due to storage reasons and the fact that consistent daily data collection did not begin till 2021. ## Extract After importing the necessary libraries, the next step is the extract the data from the website. In contrast to static websites, this particular one was dynamically loaded with JavaScript, requiring a user to submit form requests to query the database and return a result. Hence, for the extract process to work, I needed to simulate the interactions that a user would typically make, such as selecting dates or clicking buttons. So, I made an initial GET request to load the page and capture important data. I then used the data captured in the initial request to make a POST request that simulates selecting a date and clicking the "Get Generation" b …