Building text-to-speech systems with limited linguistic resources

Recent approaches to neural text-to-speech synthesis (TTS) often rely on considerable linguistic resources. For high-quality commercial TTS in large languages of Europe, this means years of collective effort by linguists building text processing frontends and pronunciation lexicons, and tens or hundreds of hours of professionally recorded transcribed speech data, a costly investment unlikely to be replicated for the vast majority of the world’s 7,000 languages. In this thesis, we investigate ways to move away from this heavy reliance on linguistic expertise and speech data, with the aim of making it easier to build a TTS system for a new language from start to finish. The first strand of work concerns reducing audio data requirements. We argue that assumptions about the amount of data needed to successfully train a neural TTS system arise from robustness issues in the attention-based autoregressive architectures which brought about the first successes in end-to-end TTS, which are especially troublesome in low-resource scenarios. Such assumptions have in turn discouraged some working on TTS for their own languages from even attempting to train a neural model, instead turning to older methods with more modest data requirements. Recent developments in non-autoregressive architectures with explicit duration predictors provide a straightforward solution to the text–speech alignment problem, and we show that such models can produce intelligible voices when trained with less than an hour of data. We then move on to methods for removing linguistic knowledge requirements, particularly with regard to lexical resources. We begin with an investigation of self-supervised (SSL) speech representations, which have recently been shown to produce powerful general-purpose features for a wide variety of speech tasks. Our aim is to provide an explicitly phonetic account of discretised acoustic units derived from such models, and give assurance that they may sensibly be used as an alternative to phones in pronunciation modelling for TTS. We apply a particular SSL speech model (HuBERT) pre-trained on English to encode speech from four different languages (English, French, Mandarin and Scottish Gaelic), and find that the model generalises well in most cases, and does indeed represent speech in ways that are phonetically plausible. We demonstrate the effectiveness of our approach throughout the thesis for several low-resource languages, including some Indigenous languages in Canada and Scottish Gaelic. For Gaelic in particular, we prove the entire voice building pipeline by segmenting and aligning found audio recordings to construct a new speech corpus, without making use of any pre-existing speech recognition system or phonetic knowledge in the target language, then build and compare a number of TTS systems using phone, character or discrete acoustic unit inputs. We also experiment with a text-to-acoustic unit sequence modelling approach which shows promise as an alternative to pronunciation modelling without phones, and which can effectively make use of lower-quality recordings in the target language as well. Subjective evaluation shows promising results for character- and discrete acoustic unit-based voices compared to phones; many other low-resource languages have similarly regular orthographies as Gaelic and so might avoid the upfront cost of developing lexical resources, while acoustic units appear to perform especially well when clean audio recordings are particularly scarce. Our work is informed in part by discussions with Indigenous language organisations in Canada who see a place for TTS in language classrooms as part of their revitalisation efforts. For minoritised language communities who might be interested in developing their own voices, the approaches we develop here can help to maintain data sovereignty by enabling more efficient model training with smaller amounts of in-house data. Overall, we hope that the methods and results presented in this thesis serve to challenge previous assumptions around training data requirements for high-quality neural TTS and enable others to build voices for their own languages.