Takaaki-Saeki/zm-text-tts

[IJCAI'23] Learning to Speak from Text for Low-Resource TTS # Learning to Speak from Text for Low-Resource TTS Implementation for our paper "Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining" to appear in IJCAI 2023. This repository is standalone but highly dependent on ESPnet. >**Abstract:** While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language. All experiments were conducted using public datasets and the implementation will be made available for reproducibility. ## Environment setup ```shell $ cd tools $ ./setup_anaconda.sh ${output-dir-name|default=venv} ${conda-env-name|default=root} ${python-version|default=none} # e.g. $ ./setup_anaconda.sh miniconda zmtts 3.8 ``` Then install espent. ```shell $ make TH_VERSION={pytorch-version} CUDA_VERSION=${cuda-version} # e.g. $ make TH_VERSION=1.10.1 CUDA_VERSION=11.3 ``` You can also setup system python environment. For other options, refer to the ESPnet installation. ## Data preparation 1. Prepare a root directory (referred to as `db_root`) for severa …