triveshsivnarain004/signalshield-africa

Explainable, human-in-the-loop scam triage for AI-amplified messaging fraud in South Africa. # SignalShield Africa — Final Reproducibility Package ## Contents - `SignalShield_Africa_Final_Benchmark.csv` — 160 fictional, redacted scenarios - `evaluate_signalshield_final.py` — evaluation script - `requirements.txt` — dependencies - `results/` — created/updated when the script runs ## Benchmark design | Split | Harmful | Benign | Total | |---|---:|---:|---:| | Development | 64 | 16 | 80 | | Held-out | 64 | 16 | 80 | | **Total** | **128** | **32** | **160** | The held-out split contains wording variations and benign near-misses. It should be used for the main final report results **after the rules have been frozen**. ## Run on Windows Open PowerShell or Command Prompt in the extracted package folder: ```bash py -m pip install -r requirements.txt py evaluate_signalshield_final.py ``` If `py` is unavailable: ```bash python -m pip install -r requirements.txt python evaluate_signalshield_final.py ``` ## Outputs The script writes these files to `results/`: - `metrics_by_split.csv` - `heldout_metrics_for_report.csv` - `per_message_predictions.csv` - `confusion_matrices.txt` - `development_performance_chart.png` - `heldout_performance_chart.png` - `combined_performance_chart.png` - `dataset_sha256.txt` - `reporting_guidance.txt` ## Honest reporting language > We evaluated a deterministic proof of concept on a controlled, fictional, > redacted and author-created benchmark. Rules were finalised on a development > split and evaluated on a held-out split designed to introduce wording variation > and benign near-misses. The held-out split provides controlled evidence of > feasibility, not independent external validation or an estimate of real-world > deployment performance. ## Safety note The benchmark contains no real victims, live links, real institution credentials, phone numbers, malicious payloads or operational scam infrastructure. The prototype does not prove AI authorship or fraud, and it is not intended to make automatic customer-account decisions …