White2360/vocab-scaling-low-resource

Official code and data for ACL 2026 paper: "Scaling Laws or Threshold Effects". We identify a 9k token threshold and a 79.5k Pareto-optimal vocabulary size for Mongolian, Tibetan, and Uyghur (BBPE/BPE). # Scaling Laws or Threshold Effects: Exploring the Optimal Vocabulary Size for Balancing Performance and Efficiency in Low-Resource Languages This is the official repository for the paper: **"Scaling Laws or Threshold Effects: Exploring the Optimal Vocabulary Size for Balancing Performance and Efficiency in Low-Resource Languages"**, accepted by **ACL 2026 (Findings)**. ## 🌟 Key Contributions We systematically investigate vocabulary scaling for three non-Latin-script, low-resource languages: **Mongolian**, **Tibetan**, and **Uyghur**. Our findings challenge the conventional monotonic scaling laws in Byte-level BPE (BBPE) architectures: 1. **The BBPE Threshold Effect**: We identify a critical initiation threshold of **~9,000 total tokens** (3,000 per language). Below this, performance actually degrades due to representation instability. 2. **Pareto-Optimal Configuration**: Through Pareto Frontier Analysis, we pinpoint **79,500 tokens** as the universal "sweet spot" for BBPE, reducing continual pre-training duration by **>71%** while enhancing performance. 3. **Efficiency Paradox**: We reveal how oversized vocabularies can lead to an "efficiency backlash" in generative tasks due to embedding/Softmax layer overhead. --- ## 🚀 Model Zoo (Comprehensive Collection) We have released all model checkpoints on **ModelScope**, covering different scaling levels (L1–L10), architectures, and training stages. ### 🔗 Click here to access the Full Model Collection ### 📂 Collection Structure & Naming Convention The collection is organized by architecture and task. Each model entry contains a complete series of checkpoints spanning all scaling levels (**L1 to L10**). **Naming Pattern**: `{Architecture}-{Stage}-{Task}` **Example**: `Qwen3-8B-SFT-MT` refers to the Qwen3-8B model series fine-tuned for Machine Translation (including all vocabulary scales from **L1 to L10**). 1. **Architectures**: `Qwen3-8B`, `Qwen2.5-7B`, `Qwen2.5-1.5B`, `Llama2-7B`. 2. **Stages & Task …