Adeeem2/Tunisian-dialect-classification

A machine learning model that classifies Tunisian dialect sentences as either positive or negative in sentiment. # Tunisian Dialect Classification A machine learning project for classifying Tunisian dialect tweets using transformer-based models and MARBERT tokenization. ## 📋 Project Overview This project analyzes and classifies Tunisian dialect text from social media (tweets) using state-of-the-art Arabic NLP models. The project includes comprehensive text preprocessing, tokenization analysis, and prepares the foundation for dialect classification tasks. ## 🚀 Features - **Dataset Processing**: Automated loading and processing of the Tunisian Dialect Corpus - **MARBERT Tokenization**: Advanced Arabic text tokenization using UBC-NLP's MARBERT model - **Statistical Analysis**: Token length distribution analysis with percentile calculations - **Data Export**: CSV export functionality for further analysis and model training - **Tweet Classification**: Binary classification setup for Tunisian dialect detection ## 📊 Dataset - **Source**: arbml/Tunisian_Dialect_Corpus - **Content**: Tunisian dialect tweets with binary labels - **Columns**: - `Tweet`: Raw tweet text in Tunisian dialect - `label`: Classification labels for dialect detection - **Language**: Tunisian Arabic dialect ## 🛠️ Technologies Used - **Python 3.x** - **Transformers**: HuggingFace transformers library - **MARBERT**: Multi-dialectal Arabic BERT model (UBC-NLP/MARBERT) - **Datasets**: HuggingFace datasets library - **Pandas**: Data manipulation and analysis - **NumPy**: Numerical computations and statistical analysis ## 📋 Requirements Install dependencies using: ```bash pip install -r requirements.txt ``` Or manually install: ```bash pip install transformers datasets pandas numpy torch ``` ## 🚀 Getting Started 1. **Clone the repository**: ```bash git clone github.com cd Tunisian-dialect-classification ``` 2. **Install dependencies**: ```bash pip install -r requirements.txt ``` 3. **Run the analysis**: ```bash jupyter notebook main.ipynb ``` Or o …