E-Nyamsuren/qLMM-Lua-Eval-Pipeline

Script and data for the manuscript titled "Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks" # qLMM-Lua-Eval-Pipeline Script and data for the manuscript titled "Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks". ## Files - ".env" contains variables used by the python scripts. Make sure to set the "MODEL_REP_PATH" variable with a path to a directory that contains the locally stored code LLMs. - "config.json" provides the list of used code LLMs and benchmarks. - "hfDatasetDownloader.py" downloads and formats the MultiPL-HumanEval, MultiPL-MBPP, and MCEVAL benchmarks from HuggingFace. The download benchmarks are stored inside the "benchmarks" directory. - "genPipe.py" script that loads code LLMs one by one and applies code generation tasks to them. All generated code is stored inside the "genOutput" directory. - "evalPipe.py" evaluates the Lua and Python code generated by the code LLMs using several metrics mentioned in the manuscript. The evaluation results are stored inside the "evalOutput" directory. - "analysis.R" to analyze the content of the "evalOutput" directory. - "genOutput.zip" contains the "genOutput" directory with the data from code generation. - "evalOutput.zip" contains the "evalOutput" directory with the data from evaluation. ## config.json structure { "name": "Unique name for the model.", "family": "Model family name. Same at all quantization precisions.", "id": "HuggingFace URI of the model.", "max_tokens": "Maximum number of tokens to generate.", "temp": "Temperature at which the model is run.", "top_k": "Next token sampling rate.", "lua_eos": "End-Of-Sequence tokens for Lua tasks", "py_eos": "End-Of-Sequence tokens for Python tasks", "qBits": "Precision. 2, 4, 8 for integer quantization precision and 16 for half-precision floating point.", "passK": "Value of k in pass@k, For example, 10 for pass@10.", "skip": "0 or 1. If 1 the model will be ignored by genPipe.py." } { "name": "benchmark name", "id": "jsonl file with the benchmark", "sample": "None or integer number. If integer number N …