francois-meyer/nglueni

Repository for the NGLUEni benchmark, a standardised evaluation suite for evaluating Nguni PLMs. Presented in the 2024 LREC-COLING paper "NGLUEni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages". # NGLUEni: Benchmarking PLMs for Nguni Languages Paper: *NGLUEni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages*, Francois Meyer, Haiyue Song, Abhisek Chakrabarty, Jan Buys, Raj Dabre and Hideki Tanaka, LREC-COLING 2024 NGLUEni is a benchmark for evaluating pretrained language models (PLMs) for the Nguni languages - isiXhosa (**xh**), isiZulu (**zu**), isiNdebele (**nr**), and Siswati (**ss**). It covers natural language understanding and generation, spanning 6 tasks and 11 datasets. This is a centralised repository to access all the datasets in the NGLUEni evaluation suite.   # Natural Language Understanding (NLU) NGLUEni is a collection of existing, publicly available Nguni datasets. Most of these datasets are already preprocessed and split into train/valid/test sets for finetuning. However, some of the datasets (SADiLaR NER, NCHLT Genre, NCHLT PC) are publicly available as raw annotated datasets (not separated into training and evaluation sets), so we have split these into standardised train/valid/test sets (80%/10%/10%) and released these splits for reproducibility (see standardised-data). The following table summarises the NLU tasks in NGUEni. | Task | Dataset | xh | zu | nr | ss | NGLUEni version | |---------------------|---------------|----|----|----|----|------ | | NER | MasakhaNER | ✓ | ✓ | | | Original dataset | | | SADiLaR NER | ✓ | ✓ | ✓ | ✓ | NGLUEni standardised split | | POS tagging | MasakhaPOS | ✓ | ✓ | | | Original dataset | | | NLAPOST | ✓ | ✓ | ✓ | ✓ | Original dataset | | Classification | MasakhaNEWS | ✓ | | | | Original dataset | | | ANTC | | ✓ | | | Original dataset | | | NCHLT Genre | ✓ | ✓ | ✓ | ✓ | NGLUEni standardised split | | Phrase chunk | NCHLT PC | ✓ | ✓ | ✓ | ✓ | NGLUEni s …