DixieKorley/bcc_timeline_scraper

Scraped BBC country profile pages of all countries from the African continent. # Continent Timeline Exercise ## PowerBI, Excel and Python Practiced web scraping by collecting and preparing all African BBC country profiles via BBC Country Profiles using Python, then importing them into PowerBI to clean and turn into a timeline visual. ### Links: For country profiles page `news.bbc.co.uk` Each country profile link is stored in `data/tl_links.csv` ## Final datasets generated `final_timeline.csv` is a table of all the events that occurred from each countries: Columns | Column | Example | Data Type | --------|:-------:| :-------:| | EventID | 1 | Number | | EventYear | c. 600BC | Text | | EventDescription | Carthaginians expand settlements along the North African coast.| Text | CountryName | Algeria | Text | | EventLink | news.bbc.co.uk | Text | CountryID | DZ | Text | | ConvertedYear | -600 | Number | `cids.csv` is a table mapping each country to their unique ids: | Column | Example | Data Type | --------|:-------:| :-------:| | CountryID | DZ | Text | CountryName | Algeria | Text `tl_links.csv` contains all country profile links and their corresponding countries: | Column | Example | Data Type | --------|:-------:| :-------:| | CountryName | DZ | Text | Links | news.bbc.co.uk | Text ## Year table in Power BI I resolved the 19 remaining errors manually in Excel, then created a year table based on the corrected dataset from `final_timeline.csv` in Power BI. | Column | Example | Data Type | --------|:-------:| :-------:| | ConvertedYear | -399.0 | Number ## Scripts used to collect and clean data `fill_cids.py` for fixing missing country ids in the timeline table. The output from this is `updated_timeline_v2.csv` `fix_century.py` for fixing any wrong year integer with 'EventYear' that contains the text 'century'. The output from this is `final_timeline.csv` `fix_missing_links.py` fo …