Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss
Domaine:
natural language processingType de record:
paper
Créateur:
Tur
Hôte:
Neural language models often struggle with low-resource languages due to the limited availability of training data, making tokens from these languages rare in the training set. This paper addresses a specific challenge during training: rare tokens are disproportionately affected by marginalization, which prevents them from learning effectively. We propose a thresholding technique that reduces the impact of this marginalization, allowing rare tokens to benefit from more meaningful alignment. Through experiments with a character-level language model, we demonstrate that this method significantly improves performance on low-resource language validation data. This work is the first to show how negative sampling can be applied to improve the representation of rare tokens by limiting the harmful influence of excessive marginalization, offering a new approach to enhancing language model performance for underrepresented languages. Accepted at LoResLM 2025 (COLING 2025 workshop). Oral presentation
Similaires
Discrete Audio Tokens Enhance Cross-Lingual Speech Recognition in Low-Resource LanguagesLow-Resource Cross-Lingual Adaptive Training for Nigerian PidginLow Resource Cross Lingual Adaptive Training for Nigerien PidginThe Linguistic Marginalization of Low-Resource Languages: Confronting AI's Limits in Digital Cultural HeritageAdaptive and Efficient Large Language Models for Low-Resource African LanguagesDialect recognition for low resource language using an adaptive filter bank