mahdertesf/Transformer-Classifier

This repository contains code for pretraining a Transformer model on a large Amharic corpus using Masked Language Modeling (MLM) and subsequently fine-tuning it for Amharic hate speech recognition. The resulting model is deployed on Mahder AI for real-time detection. # Amharic Transformer: Pre-training and Hate Speech Recognition This repository contains the code and resources for pre-training a Transformer model from scratch on an Amharic dataset and fine-tuning it for Amharic hate speech recognition. The goal is to build Transformer models from scratch, pre-train them using Masked Language Modeling (MLM), and fine-tune them on labeled data to develop a model capable of understanding the nuances of the Amharic language and accurately identifying hate speech content. ## Table of Contents - Introduction - Project Structure - Dataset - Model Architecture - Pre-training - Fine-tuning - Evaluation - Deployment - Conclusion - Future Work ## Introduction This project explores the application of Transformer networks to the Amharic language, focusing on pre-training and hate speech recognition. Amharic, being a low-resource language, benefits significantly from pre-training techniques that allow models to learn contextualized word representations from large amounts of unlabeled data. This project leverages a Masked Language Model (MLM) approach for pre-training, followed by fine-tuning on a labeled dataset for hate speech detection. The trained model is then deployed on the **Mahder AI** web application. ### Key Steps: - Data collection and preprocessing - Transformer model implementation - Pre-training using MLM - Fine-tuning for hate speech recognition - Evaluation and performance metrics - Deployment on **Mahder AI** ## Project Structure ``` . |--images # images for the train.ipynb notes | ├── data/ │ ├── totaldata.json # Unlabeled Amharic text for pre-training │ ├── original_hate_speech_data/ │ │ ├── labels.txt # Hate speech labels (Hate/Free) │ │ ├── posts.txt # Hate speech posts │ │ ├── labels_binary.txt # Hate speech labels (0/1) │ ├── cleaned_data.txt # Cleaned pre-training …