Takaaki-Saeki/zm-text-tts
Domaine:
natural language processingType de record:
paper
Créateur:
Tak
Hôte:
[IJCAI'23] Learning to Speak from Text for Low-Resource TTS
# Learning to Speak from Text for Low-Resource TTS
Implementation for our paper "Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining" to appear in IJCAI 2023.
This repository is standalone but highly dependent on ESPnet.
>**Abstract:**
While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language. All experiments were conducted using public datasets and the implementation will be made available for reproducibility.
## Environment setup
```shell
$ cd tools
$ ./setup_anaconda.sh ${output-dir-name|default=venv} ${conda-env-name|default=root} ${python-version|default=none}
# e.g.
$ ./setup_anaconda.sh miniconda zmtts 3.8
```
Then install espent.
```shell
$ make TH_VERSION={pytorch-version} CUDA_VERSION=${cuda-version}
# e.g.
$ make TH_VERSION=1.10.1 CUDA_VERSION=11.3
```
You can also setup system python environment.
For other options, refer to the ESPnet installation.
## Data preparation
1. Prepare a root directory (referred to as `db_root`) for severa …