hailaykidu/LGSE-Project-

Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation # LGSE: Lexically Grounded Subword Embedding Initialization ### Official implementation — LREC 2026 This is the official repository for **LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation** (Teklehaymanot, Fazlija & Nejdl, LREC 2026). Paper: aclanthology.org It implements the method described in the paper for **Amharic** and **Tigrinya**, two morphologically rich Ethio-Semitic languages. --- ## Overview Adapting pretrained multilingual language models to low-resource, morphologically rich languages is limited by how new vocabulary is initialized. Standard vocabulary expansion relies on arbitrary subword units, which fragment morphological structure and degrade semantic alignment. LGSE initializes new token embeddings from linguistic structure rather than from random vectors: 1. **Morphological decomposition** — words are segmented into meaningful morphemes using a supervised lexicon. 2. **Morpheme-averaged embeddings** — each new token's embedding is the average of its morphemes' FastText representations, aligned into the model's embedding space by a linear projection `W ∈ R^(d×d)` (Sec 4.1). 3. **Character n-gram fallback** — tokens with no usable segmentation fall back to character n-gram representations. 4. **Regularized LAPT** — during language-adaptive pretraining the encoder is frozen and only the new embeddings are updated, with ``` L_total = L_MLM + λ · ‖e_new − μ‖² ``` penalizing drift from the initialized values (Sec 4.2). ### Pipeline | Stage | Module | |---|---| | Token selection | `src/lgse/token_selection.py` | | Morphological segmentation | `src/lgse/segmentation.py` | | Morpheme embeddings + projection W | `src/lgse/morpheme_embeddings.py`, `src/lgse/projection.py` | | Character n-gram fallback | `src/lgse/char_ngrams.py` | | Embedding initialization | `src/lgse/initializer.py` | | Regularization | `src/lgse/regularization.py` | | Language-adaptive pretraining | `src/lg …