Improving Accented Speech Recognition using Data Augmentation based on Unsupervised Text-to-Speech Synthesis

Accented speech recognition is an important research topic of automatic speech recognition (ASR). Because of its importance, this research topic has been receiving attention and being addressed with various research approaches. In general, these approaches can be classified as accent-agnostic approaches, in which the modeling of accents inside the ASR systems is not made specific, and accent-aware approaches in which additional information about the accents of the input speech are used [1]. Among accent-agnostic approaches, adversarial learning was used to establish accent classifier and accent relabeling which led to performance improvement [2, 3, 4]. In addition, similarity losses such as cosine losses or contrastive losses were used to build accent neutral models [5]. In accent-aware approaches, multi-domain training [6], accent embeddings [7], or accent information fusion [8] are among the approaches which have been investigated.

Text-to-speech synthesis (TTS) is an useful technology which can be used to improve ASR in a number of ways, for instance to improve the pre-training of self-supervised learning (SSL) models [9] or to improve the recognition of out-of-vocabulary words in end-to-end ASR [10]. TTS was also used as a data augmentation method to improve speech recognition for Librispeech task [11] and low-resource speech recognition [12, 13, 14]. More specifically, synthetic data were used for data augmentation in the context of low-resource ASR using conventional hybrid structure [13] and to augment the training of RNN-T (recurrent neural network - transducer) ASR model [15]. In [14], cross-lingual multi-speaker speech synthesis and cross-lingual voice conversion were applied to data augmentation for ASR. The authors showed that it is possible to achieve promising results for ASR model training with just a single speaker dataset in a target language, making it viable for low-resource scenarios [14].

While having been widely used in various ASR tasks, TTS, especially unsupervised TTS which is trained with unsupervised audio data [16], has not been extensively studied as a data augmentation method in accented speech recognition. In a recent study on using TTS as data augmentation for accented speech recognition [17], accented speech were generated by passing English text prompts through TTS system for a language corresponding to the target accent. For example, English text prompts passing through Spanish TTS will approximate Spanish-accented English. The study in [17] used commercial TTS systems whose training data were not accessible by users.

In this paper, we investigate the use of unsupervised TTS as a data augmentation method to improve accented speech recognition. In our approach, we make use of a small amount of accented speech data which do not have manual transcriptions to train TTS systems. This approach enables the use of accented speech data without manual transcriptions to perform data augmentation for accented speech recognition. Indeed, from a small amount of unsupervised accented speech data used to train the TTS systems, we can generate larger amount of synthetic accented speech data once the TTS systems are trained. In this paper, from 58 hours of accented speech data, selected from two speech corpora of read speech: L2-ARCTIC [18] and British Isles [19], we train unsupervised TTS and generate 250 hours more of synthetic accented speech data which help to achieve better gains on the evaluation data of spontaneous conversational speech from the Edinburgh international accents of English corpus (EdAcc) [20].

The paper is organized as follows. In section II, details of the data augmentation for accented speech recognition based on unsupervised TTS are introduced. The training and inference of TTS systems are presented in section III. Section IV introduces the data used in the experiments, experimental results, and discussion. Finally, section V concludes the paper.

We use Wav2vec2.0 SSL framework [21] for our experiments with accented speech recognition. Assume that a Wav2vec2.0 model was pre-trained via SSL on large amount of unsupervised speech data to cover various English accents and speakers, we can fine-tune this pre-trained model to downstream ASR task using available non-accented speech data. The non-accented speech data could be any available data which can be used to train ASR systems, for instance Librispeech training data [22]. When using publicly available Wav2vec2.0 pre-trained models, we assume that only the models are available and their training data are not available.

In addition to the non-accented speech data, we assume that a small amount of accented speech training data, named AccD, is available. These accented speech training data will be used to train the TTS systems. In accented speech recognition, it is not practical to find accented speech training data spoken in the same speaking styles and by the same speakers in the evaluation data. It is actually more viable to find accented speech training data which are spoken by speakers whose first languages are similar to those of the speakers in the evaluation data. Using these speech data to train TTS systems and generate more accented speech data for ASR training should create more accent variability, and hence, improve accented speech recognition performance.

VITS is an end-to-end multi-speaker TTS system which can generate high-quality waveforms [23]. During the training of VITS (see Fig. 3), a Posterior Encoder encodes linear spectrogram from natural speech into a latent variable $z$ [24] which is then used in a Decoder to restore waveform. HiFi-GAN (Generative Adversarial Network) [25], a GAN-based neural vocoder [26], is used in the decoder to synthesize high-fidelity speech. The latent variable $z$ is also fed into the Flow $f$ which computes the Kullback-Leibler divergence with the Text Encoder outputs. The Flow $f$ is trained to remove speaker information and reduce posterior complexity [27]. During training, speakers identities (IDs) are used to extract speaker embeddings for training multi-speaker TTS.

During TTS inference (see Fig. 4), an inverse transform $f^{-1}$ of the Flow $f$ is used to synthesize speech. The output of the Text Encoder is stretched by the Length Regulator based on the predicted duration, and then the sampled latent variable $z^{\prime}\sim\mathcal{N}(z^{\prime};\mu_{\theta}(text),\sigma_{\theta}(text))$ is transformed by the inverse Flow $f^{-1}$ together with speaker information. Speech is subsequently generated by the decoder. All the VITS systems in this paper use the same architecture and are trained with the same number of training iterations, i.e. 300K. We observe that after 300K training iterations, the quality of the synthesized waveforms saturates. Greater details of the VITS models and their implementation can be found in [23].

The Wav2vec2.0 model is pre-trained with more than 60K hours of unsupervised speech data from Libri-Light, CommonVoice, Switchboard, and Fisher corpora. These speech training data were spoken in various English accents. The non-accented speech data consist of 960 hours of training data from Librispeech corpus [22] which include 2200 speakers. Although these speakers spoke US-English, we consider Librispeech as non-accented in the context of this study because US-English is only one of the English accents in the evaluation data. Subsequent experimental results confirm our assumption.

Unsupervised TTS, trained on unsupervised accented speech training data, was used to generate synthetic accented speech data for data augmentation in accented speech recognition. Experiments showed that the Wav2vec2.0 models which used the synthetic accented speech data yielded up to 6.1% relative WER reductions compared to a large Wav2vec2.0 baseline. These gains are close to those obtained in the supervised scenario. The results demonstrate that unsupervised accented speech data, even when available in limited quantities and are spoken in different styles by speakers who differ from those in the evaluation data, can be effectively used to train TTS systems for data augmentation. This approach improves accented speech recognition, particularly when the speakers in the unsupervised accented speech data and those in the evaluation data have some overlaps on speakers’ first languages.