colinpriest/AI-Driven-Climate-Policy-Scenario-Generation-for-Sub-Saharan-Africa
Domain:
climateRecord type:
project
Creator:
col
Host:
# CPSG Replication — AI-Driven Climate Policy Scenario Generation for Sub-Saharan Africa
Replication of Badekale & Akinfaderin (2025), *AI-Driven Climate Policy Scenario
Generation for Sub-Saharan Africa* (arXiv:2505.18694, DSA 2025), using OpenAI models
instead of local ollama models and automated LLM validation instead of human experts.
See specs/replication-specs.md for the full specification
and results/analysis_report.md for the final report.
## Deviations from the paper
| Role | Paper | This replication |
|---|---|---|
| Scenario + follow-up generator | llama3.2-3B / llama3-8B (ollama) | OpenAI `gpt-5-mini` |
| Scenario validation | 2 human authors | OpenAI `gpt-5.5` LLM-as-validator |
| Evaluator panel | Human expert, gemma2-2B, mistral-7B | `gpt-5-mini` and `gpt-4o-mini` |
| Embeddings | ollama `nomic-embed-text` | HF `nomic-ai/nomic-embed-text-v1.5` (same weights, local) |
## Results (vs paper Table 1)
30 prompts → 27 validated by gpt-5.5 (90%; paper: 30/34, 88%) → RAGAs evaluation.
15/16 success criteria passed (the miss: gpt-4o-mini faithfulness 0.695 vs the 0.70
threshold, within judge run-to-run variance; see report).
| Evaluator | Faithfulness | Answer Relevancy | Context Utilization |
|---|---|---|---|
| gpt-5-mini | 0.902 ± 0.106 | 0.826 ± 0.037 | 1.000 ± 0.000 |
| gpt-4o-mini | 0.695 ± 0.299 | 0.795 ± 0.042 | 0.993 ± 0.037 |
| *Paper: Human* | *0.760 ± 0.081* | *0.993 ± 0.037* | *0.713 ± 0.101* |
| *Paper: gemma2-2B* | *0.966 ± 0.102* | *0.888 ± 0.057* | *0.939 ± 0.184* |
| *Paper: mistral-7B* | *0.848 ± 0.271* | *0.893 ± 0.050* | *0.980 ± 0.108* |
Spearman between judges (paper Table 2 analogue): answer relevancy 0.635 (p<0.001),
faithfulness 0.138, context utilization undefined (gpt-5-mini constant at the 1.0
ceiling).
### Calibrations required by the model swap (documented in spec §3.2/§3.5)
1. **Generation format constraint** — unconstrained gpt-5-mini writes ~12k-char
knowledge-rich essays that score ~0.2 RAGAs faithfulness; the temp …