CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

Domain:

natural language processing

Record type:

paper

Creator:

ZhaBasEll

Host:

Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, both tuned models win against their respective base across 11 evaluated languages. Overall, we show promising directions for multilingual preference tuning.

Visit

arxiv.org

Contrastive Learning Effects on Zero-Shot Cross-Lingual Euphemism Detection in XLM-R Fine-Tuning

Euphemisms are culturally variable and often ambiguous, posing challenges for language models, espec

Contrastive-Adversarial Fine-Tuning for Zero-Shot Cross-Lingual Performance in XLM-R

Pre-trained multilingual language encoders, such as multilingual BERT and XLM-R, show great potentia

S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method-Tuning: A Simple Cross-lingual Sub-network Tuning Method S4 - Tuning: طريقة ضبط بسيطة عبر الشبكة الفرعية اللغوية: طريقة ضبط بسيطة عبر الشبكة الفرعية اللغوية S4-Tuning : A Simple Cross-lingual Sub-network Tuning Method-Tuning : A Simple Cross-lingual Sub-network Tuning Method S4-Tuning: A Simple Cross-lingual Sub-network Tuning Method-Tuning: A Simple Cross-lingual Sub-network Tuning Method

The emergence of multilingual pre-trained language models makes it possible to adapt to target langu

Cross-lingual Contrastive Loss Effects on Multilingual Embedding Alignment Quality

Existing zero-shot cross-lingual transfer methods rely on parallel corpora or bilingual dictionaries

MuCoT contrastive training vs. standard fine-tuning for zero-shot cross-lingual transfer in low-resource languages