nhlamulokhoza/AGMP

African Genomic Data Hub AGMP project # GWAS Data Processing Pipeline ## Overview This Metaflow-based pipeline processes GWAS (Genome-Wide Association Studies) data by integrating information from multiple sources, including BioMart queries for gene annotations, UniProt data, and geographic classifications. The pipeline handles data cleaning, transformation, and enrichment to produce a comprehensive dataset suitable for further analysis. ## Features - Automated processing of GWAS catalog data - BioMart integration for gene information retrieval - UniProt data integration - Geographic region classification based on African Union standards - Concurrent processing for API queries - Automatic retry mechanism for failed API calls - Progress tracking and logging - Data validation and cleaning - Export to multiple formats (CSV and Excel) ## Prerequisites - Python 3.7 or higher - pip (Python package installer) ### Required Python Packages ```bash pip install metaflow==2.9.1 pip install pandas==2.0.3 pip install requests==2.31.0 pip install tqdm==4.65.0 pip install openpyxl # for Excel file handling ``` ## Input Requirements The pipeline expects the following input file: - `gwas_africa_assoc_MERGED.xls`: Initial GWAS data file containing association data ## Installation 1. Clone this repository: ```bash git clone github.com ``` 2. Install required packages: ```bash pip install -r requirements.txt ``` ## Usage Run the pipeline using the following command: ```bash python gwas-metaflow.py run ``` ### Pipeline Steps 1. **Start**: Loads and initializes GWAS data 2. **Prepare rsIDs**: Cleans and formats rsID data 3. **Query BioMart Genes**: Retrieves gene information from BioMart 4. **Query BioMart UniProt**: Retrieves UniProt and function information 5. **Process Results**: Combines and processes query results 6. **Process Geographic Data**: Handles regional classifications 7. **Finalise Data**: Performs final data cleaning and saves results ### Output Files The pipeline gene …