michaelnigusse2/Ethiopian-Business-License-and-Company-Data-Retrieval-Automation

# Ethiopian Business License and Company Data Retrieval Automation This is a fast and robust Python tool designed to fetch business information from the etrade.gov.et portal using a list of business license numbers. ## Features - **Fast and Concurrent**: Uses asynchronous requests (`httpx` + `asyncio`) with a tuned connection pool. - **Smart Retries and Rate Limiting**: Automatically handles 429 Too Many Requests errors with exponential backoff and jitter. - **Adaptive Variant Learning**: Tries multiple parameter variations (e.g., with or without Tin, different languages) and learns the most successful patterns. - **Caching**: Saves responses to `cache_etrade.jsonl` to skip redundant network calls on subsequent runs. - **Preserves Ordering**: Outputs exactly match the input order and duplicate rows are maintained without extra network calls. - **Outputs**: Generates both CSV and Parquet files for easy data analysis. ## Prerequisites - Python 3.8+ ## Installation 1. Clone the repository (once you push it to your remote): ```bash git clone cd "Ethiopian Business-License and Company-Data Retrieval Automation" ``` 2. Install the required dependencies: ```bash pip install -r requirements.txt ``` ## Usage Run the script from your terminal. By default, it looks for an input file named `queries.xlsx` and reads the column `license_number` from the first sheet. ```bash python new_scraper.py [sheet=0] [column=license_number] [out_csv=etrade_profiles.csv] [out_parquet=etrade_profiles.parquet] ``` ### Examples **Basic usage:** ```bash python new_scraper.py input_licenses.csv ``` **Specifying a column name and output file:** ```bash python new_scraper.py data.xlsx 0 "License_No" results.csv results.parquet ``` ## Output The script outputs two files by default: - `etrade_profiles.csv` - `etrade_profiles.parquet` It will also create a local `cache_etrade.jsonl` file to cache responses and speed up subsequent runs.