0khacha/darija-toxicity-detection
Domain:
natural language processingRecord type:
modelsoftware
Creator:
0kh
Host:
NLP pipeline for toxicity detection in Moroccan Darija and Arabizi. Powered by a fine-tuned ArabERT model, it robustly detects detects bad words, insults, hate speech, and evasion attempts.
# Darija Toxicity Detection π‘οΈ
A specialized NLP pipeline for detecting toxic content (bad words, insults, hate speech, spam) in **Moroccan Darija** and **Arabizi** using transformer-based models.
This project tackles the unique challenge of "Arabizi" (Arabic chat alphabet latin), where numbers are used as letters (e.g., `3` for `ΨΉ`, `7` for `Ψ`) and informal spelling is common.
## π Features
- **π€ Pre-trained Model**: Zero-setup inference using models hosted on Hugging Face Hub.
- **π Multi-Lingual Support**: Robust handling of Darija mixed with Arabic, English, French, and Tamazight.
- **π Rich Dataset**: Trained on over **16,000 Darija/Arabizi sentences** covering diverse social contexts.
- **π§Ή Advanced Cleaning**: Specialized preprocessing for Arabizi patterns (e.g., `in3al` detection, repetitive characters).
- **β‘ Fast Inference**: GPU-accelerated predictions with granular confidence scores.
- **π― High Accuracy**: Fine-tuned specifically on a curated Moroccan Darija dataset.
## π Quick Start
### Installation
```bash
# Clone the repository
git clone github.com
cd darija-toxicity-detection
# Install dependencies
pip install -r requirements.txt
```
### Basic Usage
```python
from src.predict import DarijaPredictor
# Initialize predictor (downloads model from Hugging Face)
predictor = DarijaPredictor()
# Predict toxicity
label, probs = predictor.predict_toxicity("salam khouya")
print(f"Label: {label}")
print(f"Confidence: {probs[label]:.2%}")
```
### Interactive Mode
```bash
python test_pipeline.py
```
### Demo Script
```bash
python run_project.py
```
## π Usage Examples
### Single Prediction
```python
from src.predict import DarijaPredictor
predictor = DarijaPredictor()
label, probs = predictor.predict_toxicity("bghit nakol")
print(f"Prediction: {label}")
print(f"Probabilities: {probs}")
# Output: Prediction: Safe
# Probabilities: {'Safe': 0.9845, 'Toxic': 0.0155}
```
### Batch Processing
` β¦