Expanding the Frontier of Speaker-Adaptive TTS: Stronger Generalisation, Less Storage, Wider Languages

Speaker-adaptive text-to-speech (TTS) aims to synthesise speech in a specified target voice by conditioning on one or a few reference utterances. This technology is widely used in personalised voice assistants and audiobooks. Despite recent progress, current systems exhibit limited voice generalisation, limited adaptation reliability, and limited language coverage. For voice generalisation, current zero-shot-based systems often struggle with out-of-domain voices, particularly second-language (L2) speakers whose acoustic characteristics differ substantially from training data that are dominated by native speakers, leading to reduced speaker similarity in synthesised speech. For adaptation reliability, few-shot methods still exhibit catastrophic forgetting, whereby fine-tuning overwrites previously learnt capabilities, and entail substantial per-speaker storage overhead. For language coverage, most cross-language methods support only one or a few high-resource languages and provide no support for low-resource languages, which constitute the majority of languages worldwide, thereby limiting real-world applicability. This thesis addresses these three limitations. Firstly, for the voice generalisation limitation, this thesis addresses it from both the modelling and data perspectives. On the modelling side, this thesis presents a novel zero-shot speaker-adaptive model that introduces disentangled representation learning together with adversarial discriminators to strengthen the model’s voice generalisation. On the data side, this thesis introduces a new TTS corpus with diverse accents to further improve voice robustness during pre-training. Following that, for the adaptation reliability limitation, this thesis proposes a unified speaker-adaptive model that can perform both zero-shot and few-shot speaker adaptation. During few-shot adaptation, it introduces lightweight adapters that comprise fewer than 1% of the full model’s parameters, mitigating catastrophic forgetting and reducing storage requirements. Finally, for the language coverage limitation, this thesis presents, to our knowledge, the first speaker-adaptive TTS system capable of synthesising intelligible speech for lowresource languages. It proposes a modular multilingual speaker-adaptive TTS architecture that separates language-specific experts from a language-agnostic speaker representation. New languages are supported by adding the corresponding language expert after pre-training and by using only one hour of paired text–speech data, without retraining the entire model. In summary, the proposed methods in this thesis address the three significant limitations of speaker-adaptive TTS across zero-shot, few-shot, and cross-language settings. The experimental results demonstrate their effectiveness and show that they substantially broaden the practical applicability of speaker-adaptive TTS.