ReML-AI/Irish-BLiMP

[LREC 2026] Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting # Irish-BLiMP Dataset Evaluation Harness This repository accompanies the Irish-BLiMP dataset paper, which introduces minimal-pair grammatical acceptability data for Irish (Gaeilge) and evaluates baseline language models exclusively on this benchmark using the LM Evaluation Harness. Paper: arxiv.org ```bibtex @article{mcgiff2025irish, title={Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting}, author={McGiff, Josh and Tran, Khanh-Tung and Mulcahy, William and Luin{\'\i}n, D{\'a}ibhidh {\'O} and Dalzell, Jake and Bhroin, R{\'o}is{\'\i}n N{\'\i} and Burke, Adam and O'Sullivan, Barry and Nguyen, Hoang D and Nikolov, Nikola S}, journal={arXiv preprint arXiv:2510.20957}, year={2025} } ``` ## Note This remains the original LM Evaluation Harness codebase; the sole modification is the addition of Irish-BLiMP data, tasks, and documentation layered atop the upstream release at: irish_blimp --- ## Latest News 📣 - [2025/07] Added `think_end_token` arg to `hf` (token/str), `vllm` and `sglang` (str) for stripping CoT reasoning traces from models that support it. - [2025/03] Added support for steering HF models! - [2025/02] Added SGLang support! - [2024/09] We are prototyping allowing users of LM Evaluation Harness to create and evaluate on text+image multimodal input, text output tasks, and have just added the `hf-multimodal` and `vllm-vlm` model types and `mmmu` task as a prototype feature. We welcome users to try out this in-progress feature and stress-test it for themselves, and suggest they check out `lmms-eval`, a wonderful project originally forking off of the lm-evaluation-harness, for a broader range of multimodal tasks, models, and features. - [2024/07] API model support has been updated and refactored, introducing support for batched and async requests, and making it significantly easier to customize and use for your own purposes. **To run Llama 405B, we recommend using VLLM's Op …