omarkami007/SLM-Fine-tuning-for-Darija-summarization

Fine-tuning a Small Language Model for Darija (Moroccan Arabic) summarization # README ## Project: Fine-Tuning a Small Language Model for Arabic Summarization ### Overview This project focuses on fine-tuning a **Small Language Model (SLM)** for the task of **Arabic text summarization**. Using Google Colab, we leverage a combination of unannotated Arabic datasets, synthetic annotations generated by Large Language Models (LLMs), and state-of-the-art evaluation metrics to create a robust summarization model. The project is structured into five key steps: 1. **Dataset Selection**: A dataset of 5,000 Arabic documents (e.g., from Arabic Wikipedia or news articles) is curated. 2. **Dataset Annotation**: High-quality summaries are generated for the documents using powerful LLMs like Atlas-Chat-27B or Qwen2.5-32B-Instruct, with quantization techniques applied to fit within Colab's hardware constraints. 3. **Data Splitting**: The annotated dataset is split into training, validation, and test sets to ensure proper model evaluation. 4. **Model Fine-tuning**: A lightweight SLM (e.g., Qwen2.5-0.5B-Instruct, Atlas-Chat-2B, or mT5-Base) is fine-tuned on the training set. 5. **Model Evaluation**: The fine-tuned model is evaluated on the test set using metrics such as **ROUGE**, **BERTScore**, and **LLM-as-a-Judge** to measure performance.