EthioNLP/EthioProverbEval

EthioProverbEval: LLM Evaluation for Low-Resource Language Understanding # ProverbEval: LLM Evaluation for Low-Resource Language Understanding This repository contains the YAML configuration files for the **ProverbEval** benchmark, designed for evaluating Large Language Models (LLMs) on their understanding of proverbs in low-resource Ethiopian languages (Amharic, Afaan Oromo, Tigrinya, Ge'ez) and English. The evaluations are intended to be run using the EleutherAI Language Model Evaluation Harness. The benchmark and its findings are detailed in the paper: "ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understanding." ## Tasks Overview The YAML files in this repository define three main evaluation tasks: 1. **Meaning Multiple Choice**: Models select the correct meaning of a proverb from four options. 2. **Fill in the Blank**: Models complete a proverb by selecting the correct missing word. 3. **Proverb Generation**: Models generate a proverb based on a given description. These tasks are further broken down by language, prompt language (native or English), and other experimental conditions as described in the paper. ## How to Use To use the evaluation tasks in this repository: 1. **Clone this repository:** ```bash git clone github.com cd EthioProverbEval ``` 2. **Ensure you have EleutherAI's `lm-evaluation-harness` installed and configured.** If not, please follow their installation guide. 3. **Run evaluations using `lm-evaluation-harness`.** You will point the harness to the YAML configuration files located within the `choice/`, `fill_blank/`, and `generation/` directories of this repository. For example, to run the Amharic multiple-choice task with an English prompt and native language choices: ```bash # Make sure lm-eval is in your PATH or call it via python # The exact task name will correspond to the 'task' field in the YAML file. # You might need to register these tasks with the harness or specify the path to the YAML. # Example assuming tasks are re …