Nizar-Charrada/Tunisian-Dialect-Toxicity-Detection

Tunisian Dialect-Specific Toxicity Detection # Tunisian Dialect Toxicity Detection **Note: This AI model is part of a larger project, which is a browser extension aimed at filtering social media content.** The Tunisian Dialect Toxicity Detection project aims to develop a reliable and efficient system for detecting toxic or offensive language in Tunisian dialect text written with arabic or latin characters. The goal is to provide a comprehensive solution that can identify and flag harmful content, promoting safer online communication in the Tunisian dialect. ### Description This project focuses on building a machine learning model capable of analyzing text data in the Tunisian dialect and determining if it contains toxic or offensive content. The model is trained on a labeled dataset comprising various types of toxic language commonly found in online platforms. The model supports Tunisian dialect written using Arabic characters (Arabic) and/or Latin characters (Arabizi) through a transliteration mechanism. This allows the model to handle text inputs in either writing system and provide accurate toxicity predictions. To optimize the model for deployment and resource efficiency, we applied knowledge distillation techniques. Knowledge distillation involves training a smaller model (the student model) to mimic the behavior of a larger, more complex model (the teacher model). By distilling the knowledge from the teacher model, we were able to reduce the size of the model without sacrificing its performance significantly. This makes the model easier to deploy on various platforms with limited computational resources. By leveraging knowledge distillation, we achieved a compact and efficient Tunisian dialect toxicity detection model that can be seamlessly deployed in real-world applications. ## Prediction Pipeline: ## Project Structure: . ├── transliteration # This directory contains multiple scripts used to train the transliteration model ├── toxic_detection # This directory contains multiple s …