UBC-NLP/afrolid

AfroLID, a powerful neural toolkit for African languages identification which covers 517 African languages. AfroLID, a neural LID toolkit for 517 African languages and varieties. AfroLID exploits a multi-domain web dataset manually curated from across 14 language families utilizing five orthographic systems. AfroLID is described in this paper: **AfroLID: A Neural Language Identification Tool for African Languages**. ## What's New in AfroLID v1.5? - **Fine-tuned on SERENGETI**, a massively multilingual language model covering 517 African languages and language varieties. - **Enhanced model performance**, improving macro-F1 from 95.95 to 97.41. - **Built on Hugging Face Transformers** for seamless integration. - **Optimized for easy use** with the Hugging Face pipeline. - **Better efficiency and accuracy**, making it more robust for African langauges identification. ## How to use AfroLID v1.5? ``` python from transformers import pipeline afrolid = pipeline("text-classification", model='UBC-NLP/afrolid_1.5', use_fast=False) input_text = "6Acï looi aya në wuöt dït kɔ̈k yiic ku lɔ wuöt tɔ̈u tëmec piny de Manatha ku Eparaim ku Thimion , ku ɣään mec tɔ̈u të lɔ rut cï Naptali" result = afrolid(input_text) # Extract the label and score from the first result language = result[0]['label'] score = result[0]['score'] print(f"detected language: {language}\tscore: {round(score*100, 2)}") ``` **Output**: ``` detected langauge: dip score: 99.99 ``` ## Requirements - Download AfroLID model: ```shell wget demos.dlnlp.ai tar -xf afrolid_model.tar.gz ``` ## Installation - To install AfroLID and develop directly using pip: ```shell pip install -U afrolid ``` - To install AfroLID and develop directly GitHub repo using pip: ```shell pip install -U git+github.com ``` - To install AfroLID and develop locally: ```shell git clone github.com cd afrolid pip install . ``` ## Getting Started The full documentation contains instructions for getting started, translation using diffrent methods, interg …