Souha-BH/TounsiBench-Benchmarking-Large-Language-Models-for-Tunisian-Arabic

# TounsiBench-Benchmarking-Large-Language-Models-for-Tunisian-Arabic Thank you for using the TounsiBench evaluation framework! This repository allows you to evaluate a new language model on TounsiBench and compare its performance against the models included in our leaderboard. ## What's Included The `data/` folder contains: * **Instructions** used in TounsiBench * **Gold responses** written by native Tunisian Arabic speakers * **Topic labels** associated with each prompt * Responses generated by the models evaluated in our paper These files can be used for analysis, benchmarking, and reproducing our results. --- ## Evaluating Your Own Model ### Step 1: Generate Responses Run your model on all prompts provided in the dataset and save the outputs in: ```text data/new_model_responses.csv ``` A template file is already provided. The file should have the following format: ```csv id,Prompt,Your Model Name 1,Prompt text...,Model response... 2,Prompt text...,Model response... ... ``` Please: * Keep the `id` column unchanged. * Keep the `Prompt` column unchanged. * Replace the empty response column with your model's responses. * Rename the column from `New model name` to your actual model name. For example: ```csv id,Prompt,Llama-4-Scout 1,...,... 2,...,... ``` --- ### Step 2: Update the Notebook In `TounsiBench_Evaluation.ipynb`, locate the following variable: ```python NEW_MODEL_NAME = "NEW_MODEL_NAME_HERE" ``` Replace it with the exact name of the column you used in `new_model_responses.csv`. For example: ```python NEW_MODEL_NAME = "Llama-4-Scout" ``` The column name in the CSV and the variable in the notebook must match exactly. --- ### Step 3: Provide Your OpenAI API Key The evaluation framework uses GPT-4o as the judge model. Before running the notebook, set your OpenAI API key: ```bash export OPENAI_API_KEY="your-api-key" ``` or ```python os.environ["OPENAI_API_KEY"] = "your-api-key" ``` Please note that evaluation costs are the respo …