epfl-nlp/ConLID

ConLID: Supervised Contrastive Learning for Low-Resource Language Identification [EACL - 2026] --> # ConLID: Supervised Contrastive Learning for Low-Resource Language Identification Code and model for paper: "ConLID: Supervised Contrastive Learning for Low-Resource Language Identification" arXiv - 2025 **TL;DR:** We introduce **ConLID**, a model trained on GlotLID-C dataset using Supervised Contrastive Learning. It supports **2,099 languages** and is, especially, effective for **low-resource languages**. ### 🛠️ Setup ```bash git clone github.com cd ConLID # set the evironment variables as in `.env_example` source setup.sh ``` **Download the models** ```python from huggingface_hub import snapshot_download, hf_hub_download # Download the GlotLID and ConLID models snapshot_download( repo_id="epfl-nlp/ConLID", local_dir="checkpoints/conlid" ) hf_hub_download( repo_id="cis-lmu/glotlid", filename="model.bin", local_dir="checkpoints/glotlid", local_dir_use_symlinks=False ) ``` ### 🤖 Usage **Use the ConLID model as:** ```python from model import ConLID model = ConLID.from_pretrained(dir='checkpoints/conlid') # print the supported labels print(model.get_labels()) ## ['aai_Latn', 'aak_Latn', 'aau_Latn', 'aaz_Latn', 'aba_Latn', ...] # prediction model.predict("The cat climbed onto the roof to enjoy the warm sunlight peacefully!") # (['eng_Latn'], [0.970989465713501]) model.predict("The cat climbed onto the roof to enjoy the warm sunlight peacefully!", k=3) ## (['eng_Latn', 'sco_Latn', 'jam_Latn'], [0.970989465713501, 0.006496887654066086, 0.00487488554790616]) ``` ### 📊 Replicating UDHR results Run the following command to replicate the results for the UDHR dataset. The results will be stored under results directory. ```bash python evaluate_udhr.py ``` ### 💪🏻 Training **Download the train dataset under `data/glotlid/`** ```bash huggingface-cli download cis-lmu/glotlid-corpus --repo-type dataset --local-dir data/glotlid ``` **Run data preprocessing pipeline** ```bash bash scripts/preprocess_dataset.sh ``` **Run trainings** …