Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

Domain:

natural language processing

Record type:

papermodelsoftware
Creator:
MinPaiRek
Host:avatar
Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method -- called WECHSEL -- to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available. NAACL 2022

Visit

arxiv.org

Tasks

transfer learning

Languages

Swahili

Tags

Computation and Language

Similar

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual ModelsComparative Analysis of Cross-Lingual Transfer in Multilingual Versus Monolingual Models on Domain-Specific BenchmarksEffects of Language Relatedness for Cross-lingual Transfer Learning in Character-Based Language ModelsSubword Tokenization Strategies in Cross-Lingual Retrieval for Kinyarwanda: mBERT Versus Monolingual RoBERTa on XTREMEBUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual TransferCode-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models

FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models

Using model weights pretrained on a high-resource language as a warm start can reduce the need for d

Comparative Analysis of Cross-Lingual Transfer in Multilingual Versus Monolingual Models on Domain-Specific Benchmarks

This paper shows that pretraining multilingual language models at scale leads to significant perform

Effects of Language Relatedness for Cross-lingual Transfer Learning in Character-Based Language Models

Character-based Neural Network Language Models (NNLM) have the advantage of smaller vocabulary and t

Subword Tokenization Strategies in Cross-Lingual Retrieval for Kinyarwanda: mBERT Versus Monolingual RoBERTa on XTREME

Pre-trained multilingual language models (e.g., mBERT, XLM-RoBERTa) have significantly advanced the

BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer

Despite remarkable advancements in few-shot generalization in natural language processing, most mode

Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models

While large language models (LLMs) exhibit strong multilingual abilities, their reliance on English