AlokChedambath64/EfficientCPT-of-LLMs-for-LRLs

Domain:

natural language processing

Record type:

software

Creator:

Alo

Host:

Implementation of the two core algorithms in the paper "Efficient Continual Pre-training of LLMs for Low-resource Languages. # EfficientCPT-of-LLMs-for-LRLs Implementation of the two core algorithms in the paper "Efficient Continual Pre-training of LLMs for Low-resource Languages. 1. Vocabulary.ipynb: Algorithm to choose the best tokens to extend your LLMs vocabulary with 2. Corpus.ipynb: Algorithm to choose the best sentence to pick from your corpus Adding tokens to your vocabulary/Adding sentences to your training data increase compute. These algorithms help you choose the most efficient additions of both. # Paper: arxiv.org # Corpus Selection Algorithm # Vocabulary Selection Algorithm

Visit

github.com

Tasks

language modeling

chinmayjainnnn/LLMs-for-Translation-of-Low-Resource-Languages

Machine translation from assamese to english and vice versa using state of the art LLM's # Hindi-En

SpokenNativQA: Multilingual Everyday Spoken Queries for LLMs

Large Language Models (LLMs) have demonstrated remarkable performance across various disciplines and

kalisto263/LLMs

A hands-on demo showing how AI and UlizaLlama can power multilingual health chatbots for low-resourc

Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English

Although LLMs have been extremely effective in a large number of complex tasks, their understanding

Toward Robust Multilingual Adaptation of LLMs for Low-Resource Languages

Large language models (LLMs) continue to struggle with low-resource languages, primarily due to limi

Nsanku: Evaluating zero-shot performance of LLMs for Ghanaian languages

Nsanku

A project by Ghana NLP to test the machine translation performance of Large Language Models (LLMs) on Ghanaian languages.

Project Overview

Nsanku is an initiative that evaluates how well various open-source language models perform