Mahdyy02/llm-tunisian-language

# Tunisian Arabic LLM Evaluation (TUNIZI) This repository contains scripts, datasets, and helper code used to evaluate several large language models (LLMs) on tasks involving Tunisian Arabic: transliteration/normalization evaluation, translation similarity, and sentiment classification. The project collects model outputs (stored in the `task 1/`, `task 2/`, and `task 3/` folders), computes standard metrics, and exports results as CSV and LaTeX tables. ## Repository layout - `dataset.csv` - main dataset (semicolon-separated) used as ground truth for the tasks. - `dataset.py` - orchestrates evaluation across Task 1 (transliteration), Task 2 (translation), and Task 3 (sentiment classification). Computes metrics and writes per-task CSV summaries. - `distribution.py` - small script to visualize sentiment class distribution using Matplotlib/Seaborn. - `tunizi_to_tn_ar_similarity.py` - utility functions used for string similarity metrics on Arabic/Tunisian text (CER, Levenshtein, LCS). - `csv_to_latex.py` - converts the Task 3 classification metrics CSV into a LaTeX table and writes `sentiment_table.tex`. - `sentiment_table.tex` - generated LaTeX table (committed here as an example / output). - `review.txt`, `tunizi.txt`, `dataset.txt` - auxiliary text files used for reference. - `task 1/`, `task 2/`, `task 3/` - folders containing prompts, raw model outputs (one file per LLM), and generated metrics CSVs. Example output CSVs: - `task 1/task1_metrics.csv` - `task 2/task2_translation_metrics.csv` - `task 3/task3_classification_metrics.csv` ## Python environment & dependencies The scripts are written for Python 3.8+ and rely on common scientific and NLP packages. Key dependencies observed in the code: - pandas - numpy - scikit-learn - matplotlib - seaborn - nltk - python-Levenshtein - bert-score (package name `bert-score`) Install dependencies (recommended in a virtual environment). Example (Windows / cmd.exe): ```cmd python -m venv .venv .venv\Scripts\activate pip …