Gradient-Based Importance Smoothing for Dynamic Rank Allocation in Parameter-Efficient Fine-Tuning for Low-Resource Languages

Abstract Large pre-trained language models (PLMs) achieve state-of-the-art performance across natural language processing (NLP) tasks but remain computationally expensive to fine-tune, particularly in resource-constrained settings. Parameter-efficient fine-tuning (PEFT) mitigates this challenge by updating only a small subset of model parameters; however, existing adaptive rank allocation methods, such as AdaLoRA and SaLoRA, introduce additional optimization overhead and have not been systematically evaluated for morphologically rich low-resource languages such as Tigrinya. In this work, we propose Progressive PEFT, a lightweight family of adaptive rank allocation methods that dynamically allocate adaptation capacity using a gradient-based importance estimator with two-level exponential smoothing. We introduce two complementary strategies: (i) a budget-based progressive allocation scheme that incrementally expands rank capacity and (ii) Structure-Adaptive Progressive PEFT (SAP-PEFT), which redistributes a fixed parameter budget using a temperature-controlled softmax mechanism. To improve optimization stability during dynamic rank transitions, we further employ rank-change thresholding with optimizer resetting. We evaluate the proposed methods on two Tigrinya NLP tasks, sentiment analysis and news classification, using three pre-trained models (TiroBERTa, Afro-XLM-R, and GPT-2) against eight baselines across five random seeds with 95\% confidence intervals and Holm-corrected statistical significance tests. Experimental results show that SAP-PEFT achieves accuracy comparable to full fine-tuning, typically within 0.6 percentage points, while reducing trainable parameters by up to 336× and peak GPU memory usage by up to 29\%. The proposed methods consistently outperform LoRA and AdaLoRA in parameter–accuracy trade-offs while maintaining practical inference efficiency. Extensive ablation studies further confirm the effectiveness of the proposed design choices. Overall, our findings demonstrate that Progressive PEFT provides an efficient and scalable solution for adapting PLMs in low-resource and resource-constrained environments while contributing one of the first systematic investigations of adaptive PEFT for the Tigrinya language. All code, configurations, adapters and analysis scripts will be released publicly.