Tandah-Marcelle/Scraping_analysis_African_tech_job

This project aims to bring transparency to tech compensation in Africa (Nigeria, Ghana). The objective is to transform a raw stream of job postings into a tool capable of predicting a fair salary for a tech talent based on their profile. # Predictive Analysis of the Tech Job Market in Africa > **Mini-Project | Ethical Scraping • Statistics • Machine Learning** ## Project Overview This project aims to bring transparency to tech compensation in Africa (Nigeria, Ghana). The objective is to transform a raw stream of job postings into a tool capable of predicting a fair salary for a tech talent based on their profile. ## Technical Pipeline ### 1. Data Collection (Ethical Scraping) - **Target:** Major platforms (Jobberman, MyJobMag, BrighterMonday). - **Ethical Rules:** Strict compliance with `robots.txt`, identification via `User-Agent`, and implementation of `Rate Limiting` (3s pause). - **Volume:** +250 real job offers collected and unified. ### 2. Statistical Analysis & Inference - **Distribution Analysis:** Shapiro-Wilk test (p < 0.05) revealing a bimodal distribution (Juniors vs. Seniors). - **Correlations:** Spearman coefficient of **0.98** between experience and salary. - **Hypothesis Testing:** Mann-Whitney U test to compare purchasing power between the Nigerian and Ghanaian markets. ### 3. Machine Learning (Modeling) - **Feature Engineering:** One-Hot Encoding, Scaling, and missing value handling. - **Tested Models:** Linear Regression, Random Forest, XGBoost. - **Performance:** The final model (**Random Forest**) achieves an **R² of 0.99** with a Mean Absolute Error (MAE) of only 12,000 FCFA. ## Strategic Insights (Storytelling) 1. **Experience is King:** 90% of salary weight is determined by seniority, far more than by programming language. 2. **Market Segmentation:** Absence of a middle class; the market forces a rapid transition to "Senior" status. 3. **Decisive P-Value:** Geographic salary differences are statistically proven and not due to chance. ## Installation and Usage ```bash pip install -r requirements.txt jupyter notebook notebooks/Analyse_Marche_Tech.ipynb