Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models

Inspired by the success of LM-based TTS models, we adopt VALL-E [14] as the model backbone, as shown in Figure. 1. It consists of a text encoder, an audio embedding, an autoregressive (AR) transformer decoder, a non-autoregressive (NAR) transformer decoder, and a well-trained neural audio codec. The data flow and training method of the backbone model are the same as VALL-E.

We conduct pre-training of the backbone on a large-scale dataset and subsequently fine-tune it on a spontaneous style corpus. This process results in a decoder’s feature space that more closely aligns with human performance. Since the training process of the AR transformer decoder does not use explicit prompts, to ensure consistency between training and inference, we concatenate each audio clip with another audio segment during the fine-tuning stage for data augmentation.

Spontaneous behaviors refer to the unusual durations or pitch excursions produced by paralinguistic phenomena in speech that are not fully predictable from syntax. Based on previous researches, we classify the diverse spontaneous behaviors in Mandarin into three major categories: disfluency, interjections, and non-speech sounds [21]:

(1) Disfluency: Disfluency occurs when speech becomes incoherent due to hesitation, nervousness, thinking, or emphasis. It includes several types, such as filled pause, repetitions, stuttering, and prolongation [22, 23, 24].

(2) Injections: An interjection is a word or expression that expresses a spontaneous feeling or reaction [25]. We categorize them into doubt, response, surprise, positive feedback, reminder, realization, sigh, coquetry and snort based on pragmatic functions.

(3) Non-speech sound: Non-speech sounds include all recognizable verbal but non-speech sounds, for instance laughter [21]. We further subdivide laughter into the smile, cachinnation, wry smile, awkward laughter, scoff and involuntary laughter.

Our model focuses on the modeling of the 19 spontaneous behaviors mentioned above, which are explicitly annotated in the transcripts of the spontaneous style corpus¹¹1More details on the definitions, lexical features and acoustic characteristics of spontaneous behaviors can be found on: https://thuhcsi.github.io/interspeech2024-SponLMTTS.

To model the above diverse spontaneous behaviors, we propose a syntactic-aware spontaneous behavior encoder consisting of a label embedding and a syntactic encoder, as shown in Figure. 2.

The character-level spontaneous label sequence is expanded to the phoneme-level by simple repetition, and the representation of spontaneous behavior is obtained through a label embedding. The syntactic positions of spontaneous behaviors correspond to their various pragmatic functions, which leads us to extract the syntactic information of spontaneous behavior. We obtain the syntactic structure for each spontaneous behavior label at the character level, while parts without spontaneous behavior are represented by zero.

After extending the syntactic information to the phoneme level and processing it with a syntactic encoder consisting of two linear layers and ReLU, we combine the resulting syntactic representation with the label embedding output to obtain the syntactic-aware spontaneous behavior embeddings (L-embeddings). The L-embeddings are added to the text embeddings, ensuring that each text token is accurately added with the corresponding spontaneous behavior, and making better projections of spontaneous representation into the text feature space during the fine-tuning process.

In addition, a NAR label predictor consisting of transformer decoders is used to predict the spontaneous behavior sequence from text embeddings, which can enhance the stability of the prediction results. The predicted spontaneous labels enable the model to synthesize semantically coherent spontaneous behavior without explicit labels.

To simulate the varied prosody features in spontaneous speech, we introduce a spontaneous prosody extractor to extract fine-grained prosody representations associated with spontaneous behavior from mel-spectrograms, and it guides the prediction explicitly. In addition, a LM-based prosody predictor is employed to generate spontaneous prosody representations (P-embeddings), which are added to the text embeddings as conditions for the acoustic decoder.

Firstly, we pre-train the backbone model described in section 2.1 on a large-scale dataset, enabling it to acquire strong semantic understanding and the ability to generate expressive speech. During the fine-tuning stage, we use the pre-trained acoustic model to guide the training of the other modules, allowing the extracted spontaneous behavior and prosody representations to be better projected into the feature space.

Specifically, we divide the fine-tuning stage into three steps: 1) We jointly train the backbone model, the spontaneous behavior encoder, the label predictor, and the spontaneous prosody extractor, excluding the audio codec. The loss function is shown in equation 1, where ${C}$ represents the acoustic tokens, ${L}$ represents the spontaneous labels, and ${\lambda}$ is ${0.1}$. 2) We use the trained spontaneous prosody extractor as a teaching model and train the prosody predictor separately. The MSE loss between the predicted and extracted P-embeddings is the loss function. 3) Considering the losses in prosody prediction, we jointly train the prosody predictor and the acoustic model with a lower learning rate while freezing the spontaneous prosody extractor, in order to further improve the naturalness of the synthesized speech.

We conduct our experiments on two Mandarin corpora. For pre-training, we use the open-source corpus WenetSpeech [26] that contains 10k hours of multi-domain speeches. For fine-tuning, we use a high-quality internal dataset, which includes 5.4 hours of spontaneous speech consisting of 4968 records from a native Mandarin female speaker naturally pronouncing within the context of specific conversation texts. It contains 19 behavior annotations as described in section 2.2, for a total of 5907 spontaneous labels. We randomly selected 200 spontaneous records as test set. A pre-trained neural audio codec model, EnCodec²²2Implemented based on: https://github.com/facebookresearch/encodec[27], was used to encode the original waveform at 24kHz sampling rate and reconstruct the waveform based on the predicted acoustic tokens.

In our implementation, the acoustic decoder follows the setting of VALL-E. The label predictor and prosody predictor consist of 3 layers of transformer blocks. The spontaneous prosody extractor consists of 8 convolution layers with a kernel size of 5. We pre-train the backbone model for 40 epochs on 8 NVIDIA A100 GPUs. During the fine-tuning stage, we train 30k iterations for the first train step, then 20k for the second train step and 16k for the final train step. The ScaledAdam [28] optimizer is adopted with $\beta_{1}=0.9$, $\beta_{2}=0.95$.

We compare four models for spontaneous speech synthesis.

FastSpeech 2 A vanilla FastSpeech 2 which is trained on the spontaneous corpus, dose no explicitly behavior modeling.

VALL-E An open-source implementation of VALL-E [14]. We trained the model in two Mandarin corpora and used it as our baseline model, as described in section 2.1.

Base-L The VALL-E with the syntactic-aware spontaneous behavior modeling, excludes the prosody representations.

Proposed The proposed model that considers both syntactic-aware spontaneous behavior modeling and spontaneous prosody modeling based on VALL-E.

To evaluate the ability of the model to generate natural spontaneous speech, we conducted two mean opinion score (MOS) tests. The first test assesses the prosody naturalness of the overall speech (PN-MOS) and the second test focuses only on the naturalness of the spontaneous behavior (LN-MOS), and we both provide explicit labels for control. We randomly selected 20 sentences from the test set and invited 25 native Mandarin speakers to rate the speeches from 1 to 5. As shown in Table 1, the proposed method significantly outperforms other models in both prosody naturalness and spontaneous behavior naturalness. The performance of VALL-E significantly surpasses that of FastSpeech 2, demonstrating the capability of large-scale language models in synthesizing natural speech. Compared to the baseline, Base-L with the introduction of spontaneous behavior modeling significantly improves speech naturalness, particularly for the naturalness of spontaneous behavior. Our proposed method achieves 4.09 in PN-MOS and 4.046 in LN-MOS, indicating that our model is capable of synthesizing spontaneous speech that is closer to human performance.

In addition, a subjective preference test is administered to ask subjects for their preferences regarding the spontaneous style of a pair of sentences. We conducted a comparison between the speech generated using manual labels and the speech generated from predicted labels. As shown in Figure 4, the result for manual labels is only 6.8% higher than that for predicted labels, and there is 39.2% no preference result, suggesting that the use of a label predictor allows the model to predict reasonable spontaneous behaviors from the text information.

In order to objectively measure the naturalness of the synthesised speech, we computed Mel Cepstral Distortion (MCD) [29]. To account for potential differences in the lengths of predicted and ground-truth speech, we utilized dynamic time warping (DTW) to establish the alignment between the two mel-spectrograms. Next, we determined the minimum MCD by aligning the two mel-spectrograms. The evaluation results of different models on the test set is shown in Table 1. The results show that our proposed model outperforms other models, indicating its ability to synthesize spontaneous speech.

Two ablation studies are conducted to demonstrate the efficacy of several techniques utilized in our proposed model, including the use of spontaneous prosody modeling and spontaneous behavior modeling. The Comparative Mean Opinion Score (CMOS) was used to compare synthetic speech in terms of the prosody naturalness of the overall speech (PN) and the naturalness of spontaneous behavior (LN). The results are shown in Table 2. Eliminating the use of spontaneous prosody results in $-0.320$ CMOS on the prosody naturalness of the speech. This demonstrates that the addition of spontaneous prosody representations greatly improved the naturalness of the synthesised speech. Moreover, we find that not explicitly model spontaneous behavior resulted in $-0.408$ CMOS. This indicates that explicit label control enhances the model’s ability to simulate spontaneous behaviour.

In order to explore the ability of the model to control spontaneous phenomena, a case study is conducted to synthesize the same utterance with different labels. As shown in Figure 5, the mel-spectrograms, pitch contours, and durations of these speeches are significantly different. The duration of positive feedback is the shortest, the pitch of coquetry is higher, while the tone of filled pause is lower, and the following speech is also relatively low-pitched. The results of the case study demonstrate that our model can effectively control spontaneous behavior in spontaneous speech, and affect the stress, pitch, and duration of the synthesized speech. Some audio samples are provided for listening³³3Sample: https://thuhcsi.github.io/interspeech2024-SponLMTTS.