rouatorjmen1/tunisian-hate-speech-pipeline

# Tunisian Dialect Hate Speech and Abuse Detection Using TunBERT This repository provides a comprehensive pipeline for detecting hate speech, abusive language, and neutral content in the Tunisian dialect. The approach leverages the pre-trained BERT-based language models, specifically adapted for the nuances of Tunisian dialect. This work replicates the experimental methodology detailed in the peer-reviewed publication: "Tunisian Dialect Hate Speech and Abuse Detection with BERT-Based Models" Roua Torjmen and Kais Haddar — Accepted Subject to Revision at KES 2025 ## Overview Overview The repository offers tools and scripts to preprocess dialectal Arabic text, fine-tune the TunBERT model on the publicly available T-HSAB dataset, and evaluate classification performance on hate speech and abusive language detection tasks. ## Key Features Utilizes the publicly available T-HSAB dataset annotated for hate speech, abuse, and neutral content. Implements preprocessing and normalization techniques tailored to dialectal Arabic. Employs Hugging Face’s AutoTokenizer for tokenization aligned with the AraBERT, ARBERT, MARBERT, and TunBERT models. Fine-tunes the transformer model using TensorFlow. Provides evaluation metrics: F1-score, Precision, Recall, and Confusion Matrix. Compatible with Google Colab and local execution environments. ## Repository Structure: . ├── pipeline.py # Main notebook implementing the full pipeline ├── convert_tunbert_tf.py # Script to convert TunBERT checkpoint to TensorFlow SavedModel format ├── requirements.txt # Python dependencies ├── README.md # Project documentation ├── data/ │ └── sample_thsab.csv # Extract of the T-HSAB dataset ├── models/ │ └── tunbert_sa_tf/ # TunBERT-TF directory for Sentiment Analysis │ ├── config.json │ ├── vocab.txt │ ├── tuned_model.ckpt.data-00000-of-00001 │ ├── tuned_model.ckpt.index │ …