annikamcginnis/uganda-migrant-jobs-middleeast

An auto-updating database of the companies recruiting Ugandans for domestic worker jobs in the Middle East # Developing an Auto-Updating Database and Map of Job Openings for Ugandans in the Middle East and the Recruiting Companies In this repo, I scrape, analyze and visualize data from Uganda's Ministry of Gender, Labour & Social Development External Employment Management System (EEMIS). I develop a running database of all jobs and companies that auto-updates daily, noting which jobs and companies are currently active on the recruiting site. I also develop an interactive map that shows the recruiting companies and currently open jobs. ## Step 1: Scraping the Companies Data In the Jupyter notebook ("1. Scraping EEMIS Companies.ipynb"), I scrape the companies information on the EEMIS system. 1. First, I scrape the list of companies on the EEMIS system (eemis.mglsd.go.ug). In the Jupyter notebook, I use Playwright to press the ‘load more’ button until the list of companies is finished. 2. I scrape the list of companies with BeautifulSoup. 3. Through a loop, I generate a list of dictionaries with each element representing a single company, with the links to each individual company page included within each list element. 4. Then, I conduct a multipage scrape to simultaneously scrape all the information on the individual company pages. I use the page links, saved in the companies list dictionary, the requests library, and BeautifulSoup to go to each company’s page and extract information. 5. I add the information from each company page to the same list of dictionaries containing the information from the overall companies page. This is especially important since different addresses for the same companies are often included on the individual company pages as compared to the overall companies list. I save repeated variables along with their source (company page, companies page, etc) to remember where the information came from. 6. I save the companies information to "companies.csv". ## Step 2: Scraping the Jobs Data In the Jupyter notebook ("2. Scraping EEM …