ahmed-zine-eddine/Multilingual-Toxic-Content-Detection-AI-System-

A lightweight,AI project to detect toxic words, hate speech, insults, offensive language, and harmful content from Text, PDFs, and Images using advanced Natural Language Processing (NLP) and Optical Character Recognition (OCR). Multilingual Support: English, French, Arabic, and Algerian Darija. # Multilingual Toxic Content Detection System >>>>>> c70dac81ec03fd7337026dbc251503b6fab17cc1 ## Features - **Multilingual Support**: English, French, Arabic, and Algerian Darija. - **Three Analysis Modes**: 1. **Text**: Direct text input analysis. 2. **PDF**: Extracts text from PDFs using `pdfplumber` and analyzes it. 3. **Image (OCR)**: Extracts text from images using `EasyOCR` and analyzes it. - **Lightweight AI Pipeline**: Built to run on standard hardware (CPU) using DistilBERT/XLM-RoBERTa architecture. - **Synthetic Dataset Generation**: Includes a script to generate a customized multilingual toxicity dataset for fine-tuning. --- ## 🏗 Architecture Explanation The system uses a hybrid AI pipeline decoupled into distinct services inside a modular Django architecture: 1. **Text Extraction Layer** - **PDF**: Uses `pdfplumber` for fast, lightweight text extraction. - **Image**: Uses `EasyOCR` initialized with English, French, and Arabic language models to extract text from images. 2. **Language Detection & Preprocessing Layer** - Automatically detects the language using `langdetect`. - Cleans the text (removes URLs, redundant whitespace). - Normalizes Arabic/Darija specific characters (removing diacritics, normalizing Alef, etc.) in `preprocessing.py`. 3. **Transformer Toxicity Detection Layer** - The core engine is a Transformer model. By default, it looks for a fine-tuned model in the `trained_model` directory. - If not found, it falls back to a highly optimized pre-trained multilingual model (`cardiffnlp/twitter-xlm-roberta-base-offensive`) to ensure the application works out-of-the-box without requiring immediate GPU training. - _Why this approach?_ Full BERT or LLMs are too heavy for CPU inference. DistilBERT and XLM-RoBERTa-base provide the best tradeoff between multilingual accuracy and inference speed on standard hardware. 4. **Highlighting Layer** - Due to CPU limitations, calculating exact token attributions (like Integrated Gradients) on every …