raya-mez/Lost-in-compression

Effect of model compression (quantization) of LLMs on machine translation performance for high vs. low resource languages # Lost-in-compression Welcome to the official repository for our DL4NLP project "**Lost-in-compression**", a project to determine the impact of quantization on the performance of pretrained translation models on different languages (grouped by resource availablity). The code in this repository contains all the scripts (creating pseudo-quantization models, getting translations from models, computing scores), results and the analysis we performed for the project. ## Table of Content - Installation Guide - Datasets - How to Run - Creating Pseudo Quantization models - Generating Model Translations and Scores ## Installation Guide Download the repository as a .zip or clone the repository using: git clone git@github.com:raya-mez/Lost-in-compression.git Run the remaining steps only if running the code on local machine: Install the correct version of the used packages from the .yml file using the following command: conda env create -f dl4nlp.yaml Upon installation of the environment, it can be (de)activated using: conda activate dl4nlp_env conda deactivate dl4nlp_env The environment can be deleted using: conda remove -n dl4nlp_env --all Additional packages can be installed using pip: pip install [package_name] If running on Snellius create and run slurm file using the dl4nlp_env.yaml file provided. ## Datasets The WMT24++ dataset is used for evaluating the models and is downloaded dynamically via the code (based on the lang_pairs parameter passed when running the scripts) and do not need to be separately downloaded. ## How to Run Now that the environment has been correctly installed, it is time to run the code. ### Creating Pseudo Quantization models You can perform pseudo-quantization on the Llama or Tower-Instruct model by running the create_pseudo_quant_model.py file with appropriate command line arguments. The following command line arguments are available for use: model_id_key - This argument is used to specify the pretrained model to …