assoudi-typica-ai/darija-toxicity-benchmark

Reproducibility materials and evaluation notebook for the paper: A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs > **This repository accompanies the paper:** # A Comparative Benchmark of a Moroccan Darija Toxicity Detection Model (Typica.ai) and Major LLM-Based Moderation APIs (OpenAI, Mistral, Anthropic) > Hicham Assoudi > Typica.ai, Montreal, Canada --- ## 📄 Abstract This paper presents a comparative benchmark evaluating the performance of Typica.ai’s custom Moroccan Darija toxicity detection model against major LLM-based moderation APIs: OpenAI (omni-moderation-latest), Mistral (mistral-moderation-latest), and Anthropic Claude (claude-3-haiku-20240307). We focus on culturally grounded toxic content, including implicit insults, sarcasm, and culturally specific aggression often overlooked by general-purpose systems. Using a balanced test set derived from the **OMCD_Typica.ai_Mix** dataset, we report precision, recall, F1-score, and accuracy, offering insights into challenges and opportunities for moderation in underrepresented languages. Our results highlight Typica.ai’s superior performance, underlining the importance of culturally adapted models for reliable content moderation. --- ## 📊 Evaluation Results Here’s the final weighted F1-score comparison graph: --- ## 📦 Repository Contents ``` /data eval_baseline.csv final_evaluation_report.csv /pred claud_results.csv mistral_results.csv openai_results.csv typica_ai_results.csv eval_reproducibility.ipynb LICENSE README.md ``` --- ## 🚀 How to Run You can run this notebook using the Colab badge, but note that you will need to provide the required `data/` and `pred/` files. 1️⃣ **Download the Repository Files** - Download this repository as a ZIP and extract it, or - Clone it using: ```bash git clone github.com ``` --- 2️⃣ **Launch Notebook in Colab** - Use the Colab badge below: --- 3️⃣ **Upload Required Files in Colab** - Once inside Colab, manually upload the contents of: - `/data` → CSV files - `/pred` → CSV files You can do this via the **Colab …