GBADsInformatics/pdfScrapingETHCSA

Python3 and bash scripts for scraping data tables from reports of the Ethiopian Central Statistical Agency # pdfScrapingETHCSA This repository contains scripts on extracting and cleaning up livestock data, including the raw and clean data from the Ethiopia CSA Report PDFs. ## Installation `pip3 install -r requirements.txt` ## Documentation Main directory contains: * `pdfETHCSApipeline.sh` runs the following scripts: `extractConvertPDFTables.py`, `cleanTable.py`, `addRegionsZones.py` `and addHeader.py` (more documentation in files) * `analyzeRegionsZones.py` is used for name variants and changing number of zones in regions study * `compareTables.py` is used for finding absolute error differences in cattle data Each year folder contains: * Data for cattle, poultry, sheep, goats and camels * TAR files containing meta data and cleaned-up tables * Flags for cattle data (error of > 5%) * Region-zone csv files * Notes on cleaning up tables ## Usage * Make sure ` Pipeline.sh`, `pdfETHCSApipeline.sh`, ` -Livestock.pdf` and `ethiopia RegionsZones.csv` files are all in main directory * Run pipeline file: `./ Pipeline.sh` * Script will generate `table .csv` meta files and ` _Table_ .csv` table files * Look to year folder's notes to clean up tables