Enhancing Out-of-Vocabulary Performance of Indian TTS Systems for Practical Applications through Low-Effort Data Strategies

While creating recording scripts we focus on maximizing the coverage of OOV words along with missing OOV bigrams present in them. Instead of recording semantically meaningful sentences which are complete, we prepare a list of OOV words and randomly join sets of five words separated by commas to create unique utterances. We ensure no word repeats across utterances. To select the OOV words, we iterate through the text corpora collated in Section 2 and use a greedy algorithm to find words that maximize the frequency of missing OOV bigrams in the selected set. Owing to limitations in recording capacity and budget, we restrict our recordings to approximately 2000 words per language. In light of this restriction, we adjust the greedy algorithm to exclude a character bi-gram from consideration once its frequency in the chosen set surpasses $k$. We empirically choose $k=6$, and this parameter can be raised when recording additional data is necessary. Note that none of the OOV words present in IndicOOV, are allowed to be a part of our recording scripts (although OOV words in our recording scripts may share bigrams with OOV words in IndicOOV.)

We found interested volunteers by circulating a form within our institution. Prior to recording, all volunteers were clearly informed regarding the intended use of their voice data and the compensation they would receive for the same. Following this explanation, speakers were presented with consent forms, which they reviewed and signed. These forms explicitly documented their informed consent for the utilization of their recordings in the training of speech synthesis systems. To safeguard speaker privacy and prevent potential misuse of the speech data, we have opted to withhold the recorded audio samples from public release. However, we release the recording scripts employed during data collection to facilitate reproducibility and transparency in research methodology. The entire process of data collection was approved by our Institute Ethics Committee with the compensation in line with recommended norms.

Since renting a professional studio can be expensive, we instead rely on recording data in acoustic pods present in our workspace. We record audio using a professional condenser microphone with a pop filter to record the volunteers’ voices. Getting the words to be spoken out clearly was crucial for our experiments. To ensure clarity in voice, we ascertained that participants were well-hydrated before recording sessions. A few volunteers indicated that the OOV words were difficult to read. We thus requested all volunteers to practice with the recording script before recording sessions. During the recording, an expert proficient in the language listened to all recordings live and pointed out any pronunciation mistakes made by the speaker. During the recording process, this expert also filtered out words that were deemed inappropriate due to spelling mistakes, archaic nuances, profanity, or toxicity. In this manner, we record a total of 6 speakers, consisting of two male speakers and one female speaker for Tamil and Hindi each. The detailed statistics of the recordings are present in Table 1. Note that the use of volunteers as opposed to professional voice artists made, ensured that the data collection was relatively inexpensive with an 85% reduction in costs and the entire process was completed in 2 working days (counting studio time and post-processing time).

We use the IndicTTS dataset [1] for all experiments. Specifically, we train models on the Hindi and Tamil subsets. The Hindi subset contains approximately 10 hours and 4 minutes of female speech and 10 hours and 5 minutes of male speech. Similarly, the Tamil subset contains approximately 10 hours and 2 minutes of female speech and 10 hours and 33 minutes of male speech. Additionally, we finetune models on the recorded data described in Section 3.

We train and evaluate with two state-of-the-art text-to-speech (TTS) models: FastPitch (FP)[13] and VITS [14]. We fine-tune a pre-trained FastPitch [15], a non-autoregressive transformer-based spectrogram prediction model, starting from the open-sourced pre-trained checkpoint on IndicTTS. To learn durations the model employs an unsupervised alignment learning framework [16] that aligns textual features with acoustic representations. Mel-spectrogram outputs are then converted to audio waveforms using the HiFiGAN V1 vocoder [17], pre-trained on the IndicTTS corpus from a publicly available checkpoint [15]. We fine-tune this vocoder on our internal dataset for better speaker generalization. We also train VITS, an end-to-end speech synthesis model, with the same hyperparameter settings as prior work [15]. Before training, all the audio samples were downsampled to $22\,050\text{\,}\mathrm{Hz}$ and converted to a mono-channel configuration. All the models were trained on an NVIDIA A100 40GB GPU, with a batch size of 16, upto 2500 epochs or until convergence.

We measure both the intelligibility and perceptual quality of the speech generated by the models. We rely on human intelligibility tests to assess the intelligibility of TTS systems on OOV and IV words. In this test, a rater proficient in the language is tasked to listen to an audio sample along with the complete corresponding text and the benchmark word-of-interest highlighted. The rater is then asked to provide a binary rating of whether the word-of-interest was intelligible or not. Raters are instructed to penalize partially intelligible words too, and mark them as “not intelligible”. Raters are provided options for slowing down the audio and selecting and playing segments in repeat if required. Furthermore, raters were encouraged to discuss with each other in case of confusion. All raters who participated in the test were expert listeners with prior experience in evaluating TTS systems and who also aid in quality assurance of high-quality TTS data collection efforts. In our evaluations, we rely on 8 such expert listeners and report the percentage of unintelligible words as the Intelligibility Error Rate (%). Next, to evaluate the perceptual quality of generated speech we rely on two objective metrics - (i) VISQOL [18, 19] and (ii) S-SIM or speaker similarity. VISQOL is a perceptual speech quality estimator that uses a spectro-temporal measure to measure the similarity between ground-truth and reference speech. We use this measure to evaluate the retention of speech quality when training with additional OOV data recorded with low expenses. Since we record data in multiple voices from volunteers and use it to attempt to improve the OOV performance of the original TTS speaker, we measure the speaker similarity of the synthesized model outputs with that of the original speaker. To compute speaker similarity we report the cosine-similarity between the ground-truth and synthesized samples using embeddings extracted from Titanet [20].

In Figure 2, we visualize the intelligibility error rates of the four baseline models - FastPitch and VITS, each trained on Hindi and Tamil data. Clearly, all models perform worse on OOV words in comparison to IV words across categories. The most noticeable difference is in the Hindi FastPitch, showing high intelligibility error rates of 24% on OOV words which is twice the IV error rate at 12%. Surprisingly, all models show relatively high intelligibility error rates on IV words, too, but this possibly reflects on the low-resource setups on which these systems have been trained.

In Table 2, we compare the intelligibility of baseline models and models fine-tuned on OOV recordings. On average, across categories, the intelligibility error rates of the fine-tuned models reduce in comparison to the baselines for both OOV and IV words. More specifically, the overall relative reduction in OOV errors is 40.59%, 36.33 %, 22.50 %, and 30.94% for Hindi FP, Tamil FP, Hindi VITS, and Tamil VITS respectively. This clearly shows the utility of low-cost recordings in significantly improving model performance on OOV (as well as IV) words.

To assess whether the quality of the TTS output in the original speaker’s voice degrades when adding training data from alternative amateur speakers, we compare the voice quality of the fine-tuned models and baselines with two metrics, viz., ViSQOL for perceptual quality estimation and S-SIM for speaker similarity. Both S-SIM and VISQOL are full-reference metrics and require ground truth samples to be provided as reference audios. To assess the relative quality of audios with respect to the original IndicTTS speakers, we use the test set utterances as references. We first compute VISQOL and S-SIM of the baseline (Base) with respect to the reference. We then compute VISQOL and S-SIM of the fine-tuned model (I+O) with respect to the reference. We observe that the speaker similarity for baseline, S-SIM (Base) and the speaker similarity for the fine-tuned model, S-SIM (I+O) are comparable across languages, models, and TTS voice, indicating that training on OOV recordings of alternate speakers does not degrade the voice of the original TTS speaker. Likewise, the VISQOL scores of the baseline method, VISQOL (Base) and the scores for the fine-tuned model VISQOL (I + O) are comparable indicating there is no degradation in the perceptually quality of speech too.

We finetune the FastPitch model with single-gender recordings - (i) Using only one female speaker - F1 and (ii) Using only two male speakers - M1 and M2 . We aim to check if improvements in the intelligibility are agnostic to the gender of the recording volunteer. Table 4 summarizes the intelligibility scores averaged across seven categories for the base FastPitch model and models fine-tuned on male (Base + M1 + M2) and female (Base + F1) speakers respectively. We observe from the scores that fine-tuning the model with data from one gender improves the OOV performance across all TTS voices (M & F) except for the Tamil male voice when fine-tuning only on Male gender data. Furthermore, fine-tuning the baseline model on only Female gender data reduces the intelligibility error rates from 0.28 to 0.15 for the female speaker and 0.20 to 0.11 for the male speaker in Hindi. Similiarly, fine-tuning the baseline model on only Male gender data reduces the intelligibility error rates from 0.28 to 0.12 for the female Hindi speaker and 0.30 to 0.13 for the female Tamil speaker. This shows that one may train a model on single gender OOV recordings and expect to get OOV performance improvements across both male and female TTS output voices.

In the baseline models, we find some interesting trends in the pronunciation errors for both FastPitch and VITS, across languages. The pronunciations were not sharp for consecutive vowels in Abbreviations like AISEC (\textipa/a:ie:si:si/)and IAEA\textipa(/a:ie:i:e:/). We find that consecutive vowel combinations are not common in the IndicTTS dataset, with only 278 words and 12 words with such combinations from 188K and 100K word corpus for Hindi and Tamil respectively. In contrast, the English TTS dataset LJSpeech [21] has 36.5K combinations of consecutive vowels in a corpus of 212K words. Such combinations are prevalent in categories like Abbreviations, Government Schemes, and Navigation that borrow English words.