[]Weiqin Li1,∗, Peiji Yang2, Yicheng Zhong2, Yixuan Zhou1,
Zhisheng Wang2, Zhiyong Wu1,3,†, Xixin Wu3, Helen Meng3
Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models
Abstract
Spontaneous style speech synthesis, which aims to generate human-like speech, often encounters challenges due to the scarcity of high-quality data and limitations in model capabilities. Recent language model-based TTS systems can be trained on large, diverse, and low-quality speech datasets, resulting in highly natural synthesized speech. However, they are limited by the difficulty of simulating various spontaneous behaviors and capturing prosody variations in spontaneous speech. In this paper, we propose a novel spontaneous speech synthesis system based on language models. We systematically categorize and uniformly model diverse spontaneous behaviors. Moreover, fine-grained prosody modeling is introduced to enhance the model’s ability to capture subtle prosody variations in spontaneous speech. Experimental results show that our proposed method significantly outperforms the baseline methods in terms of prosody naturalness and spontaneous behavior naturalness.
keywords:
text-to-speech, language model, expressive speech synthesis, spontaneous style1 Introduction
Text-to-speech (TTS) aims to synthesize intelligible and natural speech from text [1]. With the development of deep learning, existing TTS models are able to generate highly expressive speech [2, 3, 4]. Nonetheless, the synthesis of speech with a spontaneous style, which usually occurs in daily conversations, talk shows, and podcasts, has not been well studied.
Several researchers have explored spontaneous speech synthesis by conducting high-quality and well-annotated spontaneous speech corpus to train neural TTS models [5]. Incorporating explicit labels to model and control spontaneous behaviors in speech have been proven to be effective in producing more human-like speech [6]. However, they typically focused on only a few specific types of spontaneous behaviors, such as filled pause [7, 8], breathing [9], interjections [10], laughter [11], and creaky phonation [12], neglecting the wide diversity of spontaneous behaviors, which limits the expressiveness of generated speech. Some studies have extracted global and local speaking style representations as latent spontaneous prosody features, and predicted them from text in an implicit manner [11, 13]. But capturing the diverse and complex prosody characteristics of spontaneous speech poses a significant challenge. Additionally, relying solely on text information, without considering the cues from spontaneous behaviors, also hinders the accurate prediction of prosody features.
Previous works on spontaneous speech synthesis are trained on limited data, leading to a discernible disparity in the naturalness of the synthesized speech compared to real human speech. A data augmentation method based on semi-supervised pre-training has been proposed to utilize low-quality spontaneous style corpus, neglecting the impact of noise from low-quality data on the synthesized speech [8]. Inspired by the advancements in text language models, recent TTS systems [14, 15, 16, 17] encode speech waveforms into discrete tokens with neural audio codecs [18, 19] as an intermediate representation, and model them with the paradigm of prompt-based language models. These language model (LM)-based TTS models can be trained on large, diverse, and low-quality speech datasets. Besides, benefiting from the in-context learning ability of LM, these models are able to capture local and long-range dependencies in speech while also acquiring a strong semantic understanding, resulting in a substantial enhancement in the diversity and naturalness of the generated speech.
However, current LM-based TTS models primarily focus on voice cloning and have not been well explored for generating human-like, authentic spontaneous speech. Although these models can learn some spontaneous expressions from a large amount of speech data, there are still two main challenges in achieving high-quality spontaneous speech synthesis: 1) The diverse spontaneous behaviors that distinguish spontaneous speech from read speech [9, 20] necessitate effective modeling and control; 2) The intrinsically diverse prosody variations in spontaneous speech are difficult for TTS models to adequately capture, which limits the performance of prosody details in generated speech.
In this paper, we propose a novel spontaneous speech synthesis system based on language models that can learn powerful semantic understanding and diverse speech expressions effectively from a massive amount of data. To categorize and control various spontaneous speech patterns in human speech, we have identified 19 spontaneous behaviors based on linguistic characteristics and explicitly modeled them by providing behavior labels and syntactic information, which is the most comprehensive work to the best of our knowledge. In addition, we introduce fine-grained spontaneous prosody representations and utilize context, spontaneous behavior labels and linguistic information to predict them. Experimental results show that our proposed method enables the LM-based TTS model to synthesize more natural spontaneous style speech. Explicit label control enhances the model’s ability to simulate spontaneous behaviors, while the addition of spontaneous prosody modeling significantly improves the naturalness of synthesized speech.
2 Methodology
The architecture of our proposed model is illustrated in Figure. 1. We use an acoustic decoder based on VALL-E [14] as the backbone, with text and acoustic prompts as conditions. To model the spontaneous behaviors, we introduce a syntactic-aware spontaneous behavior encoder, and use a label predictor to predict spontaneous labels from text embeddings. Besides, a spontaneous prosody extractor and a LM-based prosody predictor are used to guide the fine-grained spontaneous prosody modeling, for enhancing the expressiveness of the synthesised speech. In the following subsections, we describe the major parts of the model, and the methods of pre-training and fine-tuning.
![Refer to caption](x1.png)
2.1 Backbone Framework
Inspired by the success of LM-based TTS models, we adopt VALL-E [14] as the model backbone, as shown in Figure. 1. It consists of a text encoder, an audio embedding, an autoregressive (AR) transformer decoder, a non-autoregressive (NAR) transformer decoder, and a well-trained neural audio codec. The data flow and training method of the backbone model are the same as VALL-E.
We conduct pre-training of the backbone on a large-scale dataset and subsequently fine-tune it on a spontaneous style corpus. This process results in a decoder’s feature space that more closely aligns with human performance. Since the training process of the AR transformer decoder does not use explicit prompts, to ensure consistency between training and inference, we concatenate each audio clip with another audio segment during the fine-tuning stage for data augmentation.
2.2 Analysis and Taxonomy of Spontaneous Behaviors
Spontaneous behaviors refer to the unusual durations or pitch excursions produced by paralinguistic phenomena in speech that are not fully predictable from syntax. Based on previous researches, we classify the diverse spontaneous behaviors in Mandarin into three major categories: disfluency, interjections, and non-speech sounds [21]:
(1) Disfluency: Disfluency occurs when speech becomes incoherent due to hesitation, nervousness, thinking, or emphasis. It includes several types, such as filled pause, repetitions, stuttering, and prolongation [22, 23, 24].
(2) Injections: An interjection is a word or expression that expresses a spontaneous feeling or reaction [25]. We categorize them into doubt, response, surprise, positive feedback, reminder, realization, sigh, coquetry and snort based on pragmatic functions.
(3) Non-speech sound: Non-speech sounds include all recognizable verbal but non-speech sounds, for instance laughter [21]. We further subdivide laughter into the smile, cachinnation, wry smile, awkward laughter, scoff and involuntary laughter.
Our model focuses on the modeling of the 19 spontaneous behaviors mentioned above, which are explicitly annotated in the transcripts of the spontaneous style corpus111More details on the definitions, lexical features and acoustic characteristics of spontaneous behaviors can be found on: https://thuhcsi.github.io/interspeech2024-SponLMTTS.
![Refer to caption](x2.png)
2.3 Syntactic-aware Spontaneous Behavior Modeling
To model the above diverse spontaneous behaviors, we propose a syntactic-aware spontaneous behavior encoder consisting of a label embedding and a syntactic encoder, as shown in Figure. 2.
The character-level spontaneous label sequence is expanded to the phoneme-level by simple repetition, and the representation of spontaneous behavior is obtained through a label embedding. The syntactic positions of spontaneous behaviors correspond to their various pragmatic functions, which leads us to extract the syntactic information of spontaneous behavior. We obtain the syntactic structure for each spontaneous behavior label at the character level, while parts without spontaneous behavior are represented by zero.
After extending the syntactic information to the phoneme level and processing it with a syntactic encoder consisting of two linear layers and ReLU, we combine the resulting syntactic representation with the label embedding output to obtain the syntactic-aware spontaneous behavior embeddings (L-embeddings). The L-embeddings are added to the text embeddings, ensuring that each text token is accurately added with the corresponding spontaneous behavior, and making better projections of spontaneous representation into the text feature space during the fine-tuning process.
In addition, a NAR label predictor consisting of transformer decoders is used to predict the spontaneous behavior sequence from text embeddings, which can enhance the stability of the prediction results. The predicted spontaneous labels enable the model to synthesize semantically coherent spontaneous behavior without explicit labels.
2.4 Spontaneous Prosody Modeling
To simulate the varied prosody features in spontaneous speech, we introduce a spontaneous prosody extractor to extract fine-grained prosody representations associated with spontaneous behavior from mel-spectrograms, and it guides the prediction explicitly. In addition, a LM-based prosody predictor is employed to generate spontaneous prosody representations (P-embeddings), which are added to the text embeddings as conditions for the acoustic decoder.
2.4.1 Spontaneous Prosody Extractor
The spontaneous prosody extractor has three convolution layers and a multi-head attention mechanism, as described in Figure. 3. The first convolution layer condenses mel-spectrograms into frame-level hidden states. Then, three middle layers, including two convolution stacks and an average pooling layer, generate intermediate phoneme-level representations based on phoneme boundaries. The final convolution layer combines and processes them to produce fine-grained prosody representation.
Spontaneous behavior significantly influences the prosody of spontaneous speech, so we obtain spontaneous prosody embeddings by merging prosody and spontaneous behaviors information. We use a multi-head attention mechanism, with L-embeddings as the query and prosody representation as the key and value, to understand the prosody selection of each spontaneous label, generating P-embeddings.
![Refer to caption](x3.png)
2.4.2 LM-based Prosody Predictor
Since spontaneous prosody is related to text, spontaneous behaviors, and syntactic structure information, during the inference process, we propose an autoregressive prosody predictor based on a language model consisting of transformer decoders that takes in these information and predicts P-embeddings. The AR predictor can capture both local and global dependencies for prosody modeling, helping the acoustic decoder generate expressive spontaneous speech.
2.5 Pre-training and Fine-tuning
Firstly, we pre-train the backbone model described in section 2.1 on a large-scale dataset, enabling it to acquire strong semantic understanding and the ability to generate expressive speech. During the fine-tuning stage, we use the pre-trained acoustic model to guide the training of the other modules, allowing the extracted spontaneous behavior and prosody representations to be better projected into the feature space.
Specifically, we divide the fine-tuning stage into three steps: 1) We jointly train the backbone model, the spontaneous behavior encoder, the label predictor, and the spontaneous prosody extractor, excluding the audio codec. The loss function is shown in equation 1, where represents the acoustic tokens, represents the spontaneous labels, and is . 2) We use the trained spontaneous prosody extractor as a teaching model and train the prosody predictor separately. The MSE loss between the predicted and extracted P-embeddings is the loss function. 3) Considering the losses in prosody prediction, we jointly train the prosody predictor and the acoustic model with a lower learning rate while freezing the spontaneous prosody extractor, in order to further improve the naturalness of the synthesized speech.
(1) |
3 Experiments
3.1 Basic Setup
We conduct our experiments on two Mandarin corpora. For pre-training, we use the open-source corpus WenetSpeech [26] that contains 10k hours of multi-domain speeches. For fine-tuning, we use a high-quality internal dataset, which includes 5.4 hours of spontaneous speech consisting of 4968 records from a native Mandarin female speaker naturally pronouncing within the context of specific conversation texts. It contains 19 behavior annotations as described in section 2.2, for a total of 5907 spontaneous labels. We randomly selected 200 spontaneous records as test set. A pre-trained neural audio codec model, EnCodec222Implemented based on: https://github.com/facebookresearch/encodec[27], was used to encode the original waveform at 24kHz sampling rate and reconstruct the waveform based on the predicted acoustic tokens.
In our implementation, the acoustic decoder follows the setting of VALL-E. The label predictor and prosody predictor consist of 3 layers of transformer blocks. The spontaneous prosody extractor consists of 8 convolution layers with a kernel size of 5. We pre-train the backbone model for 40 epochs on 8 NVIDIA A100 GPUs. During the fine-tuning stage, we train 30k iterations for the first train step, then 20k for the second train step and 16k for the final train step. The ScaledAdam [28] optimizer is adopted with , .
3.2 Compared Methods
We compare four models for spontaneous speech synthesis.
FastSpeech 2 A vanilla FastSpeech 2 which is trained on the spontaneous corpus, dose no explicitly behavior modeling.
VALL-E An open-source implementation of VALL-E [14]. We trained the model in two Mandarin corpora and used it as our baseline model, as described in section 2.1.
Base-L The VALL-E with the syntactic-aware spontaneous behavior modeling, excludes the prosody representations.
Proposed The proposed model that considers both syntactic-aware spontaneous behavior modeling and spontaneous prosody modeling based on VALL-E.
3.3 Subjective Evaluation
To evaluate the ability of the model to generate natural spontaneous speech, we conducted two mean opinion score (MOS) tests. The first test assesses the prosody naturalness of the overall speech (PN-MOS) and the second test focuses only on the naturalness of the spontaneous behavior (LN-MOS), and we both provide explicit labels for control. We randomly selected 20 sentences from the test set and invited 25 native Mandarin speakers to rate the speeches from 1 to 5. As shown in Table 1, the proposed method significantly outperforms other models in both prosody naturalness and spontaneous behavior naturalness. The performance of VALL-E significantly surpasses that of FastSpeech 2, demonstrating the capability of large-scale language models in synthesizing natural speech. Compared to the baseline, Base-L with the introduction of spontaneous behavior modeling significantly improves speech naturalness, particularly for the naturalness of spontaneous behavior. Our proposed method achieves 4.09 in PN-MOS and 4.046 in LN-MOS, indicating that our model is capable of synthesizing spontaneous speech that is closer to human performance.
In addition, a subjective preference test is administered to ask subjects for their preferences regarding the spontaneous style of a pair of sentences. We conducted a comparison between the speech generated using manual labels and the speech generated from predicted labels. As shown in Figure 4, the result for manual labels is only 6.8% higher than that for predicted labels, and there is 39.2% no preference result, suggesting that the use of a label predictor allows the model to predict reasonable spontaneous behaviors from the text information.
3.4 Objective Evaluation
In order to objectively measure the naturalness of the synthesised speech, we computed Mel Cepstral Distortion (MCD) [29]. To account for potential differences in the lengths of predicted and ground-truth speech, we utilized dynamic time warping (DTW) to establish the alignment between the two mel-spectrograms. Next, we determined the minimum MCD by aligning the two mel-spectrograms. The evaluation results of different models on the test set is shown in Table 1. The results show that our proposed model outperforms other models, indicating its ability to synthesize spontaneous speech.
Models | PN-MOS | LN-MOS | MCD |
---|---|---|---|
FastSpeech 2 | |||
VALL-E | |||
Base-L | |||
Proposed |
3.5 Ablation Study
Two ablation studies are conducted to demonstrate the efficacy of several techniques utilized in our proposed model, including the use of spontaneous prosody modeling and spontaneous behavior modeling. The Comparative Mean Opinion Score (CMOS) was used to compare synthetic speech in terms of the prosody naturalness of the overall speech (PN) and the naturalness of spontaneous behavior (LN). The results are shown in Table 2. Eliminating the use of spontaneous prosody results in CMOS on the prosody naturalness of the speech. This demonstrates that the addition of spontaneous prosody representations greatly improved the naturalness of the synthesised speech. Moreover, we find that not explicitly model spontaneous behavior resulted in CMOS. This indicates that explicit label control enhances the model’s ability to simulate spontaneous behaviour.
![Refer to caption](x4.png)
Modes | CMOS |
---|---|
Proposed | |
-spontaneous prosody modeling (PN) | |
-spontaneous behavior modeling (LN) |
![Refer to caption](x5.png)
3.6 Case Study
In order to explore the ability of the model to control spontaneous phenomena, a case study is conducted to synthesize the same utterance with different labels. As shown in Figure 5, the mel-spectrograms, pitch contours, and durations of these speeches are significantly different. The duration of positive feedback is the shortest, the pitch of coquetry is higher, while the tone of filled pause is lower, and the following speech is also relatively low-pitched. The results of the case study demonstrate that our model can effectively control spontaneous behavior in spontaneous speech, and affect the stress, pitch, and duration of the synthesized speech. Some audio samples are provided for listening333Sample: https://thuhcsi.github.io/interspeech2024-SponLMTTS.
4 Conclusions
In this paper, we propose a novel spontaneous speech synthesis system based on language models that can learn powerful semantic understanding and diverse speech expressions from a massive amount of data. We incorporate explicit spontaneous behavior modeling and use fine-grained spontaneous prosody representations to enhance the model’s ability to synthesize spontaneous speech. Experimental results show our proposed method significantly outperforms the baseline method in terms of prosody naturalness and spontaneous behavior naturalness.
5 Acknowledgement
This work is supported by National Natural Science Foundation of China (62076144), National Social Science Foundation of China (13&ZD189), Shenzhen Science and Technology Program (WDZC20220816140515001) and Tencent Rhino-Bird Focused Research Program (RBFR2023015).
References
- [1] X. Tan, T. Qin, F. Soong, and T.-Y. Liu, “A survey on neural speech synthesis,” arXiv preprint arXiv:2106.15561, 2021.
- [2] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” in International Conference on Learning Representations, 2020.
- [3] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan et al., “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4779–4783.
- [4] J. Kim, J. Kong, and J. Son, “Vits:variational inference with adversarial learning for end-to-end text-to-speech,” Cornell University - arXiv,Cornell University - arXiv, Jun 2021.
- [5] H. Guo, S. Zhang, F. K. Soong, L. He, and L. Xie, “Conversational end-to-end tts for voice agents,” in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 403–409.
- [6] J. Cong, S. Yang, N. Hu, G. Li, L. Xie, and D. Su, “Controllable context-aware conversational speech synthesis,” arXiv preprint arXiv:2106.10828, 2021.
- [7] Y. Yan, X. Tan, B. Li, G. Zhang, T. Qin, S. Zhao, Y. Shen, W.-Q. Zhang, and T.-Y. Liu, “Adaspeech 3: Adaptive text to speech for spontaneous style,” 2021.
- [8] W. Li, S. Lei, Q. Huang, Y. Zhou, Z. Wu, S. Kang, and H. Meng, “Towards spontaneous style modeling with semi-supervised pre-training for conversational text-to-speech synthesis,” arXiv preprint arXiv:2308.16593, 2023.
- [9] É. Székely, G. E. Henter, J. Beskow, and J. Gustafson, “Breathing and speech planning in spontaneous speech synthesis,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7649–7653.
- [10] R. Fernandez, D. Haws, G. Lorberbom, S. Shechtman, and A. Sorin, “Transplantation of conversational speaking style with interjections in sequence-to-sequence speech synthesis,” arXiv preprint arXiv:2207.12262, 2022.
- [11] K. Mitsui, T. Zhao, K. Sawada, Y. Hono, Y. Nankaku, and K. Tokuda, “End-to-end text-to-speech based on latent representation of speaking styles using spontaneous dialogue,” arXiv preprint arXiv:2206.12040, 2022.
- [12] H. Lameris, S. Mehta, G. E. Henter, J. Gustafson, and É. Székely, “Prosody-controllable spontaneous tts with neural hmms,” arXiv preprint arXiv:2211.13533, 2022.
- [13] H. Li, X. Zhu, L. Xue, Y. Song, Y. Chen, and L. Xie, “Spontts: modeling and transferring spontaneous style for tts,” arXiv preprint arXiv:2311.07179, 2023.
- [14] C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023.
- [15] Z. Zhang, L. Zhou, C. Wang, S. Chen, Y. Wu, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,” arXiv preprint arXiv:2303.03926, 2023.
- [16] E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, read and prompt: High-fidelity text-to-speech with minimal supervision,” arXiv preprint arXiv:2302.03540, 2023.
- [17] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “Audiolm: a language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
- [18] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022.
- [19] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “Soundstream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021.
- [20] E. Székely, J. Mendelson, and J. Gustafson, “Synthesising uncertainty: The interplay of vocal effort and hesitation disfluencies.” in INTERSPEECH, 2017, pp. 804–808.
- [21] S.-C. Tseng, “Taxonomy of spontaneous speech phenomena in mandarin conversation,” in ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003.
- [22] Y. Zhao and D. Jurafsky, “A preliminary study of mandarin filled pauses,” in Disfluency in Spontaneous Speech, 2005.
- [23] S.-C. Tseng, “Repairs and repetitions in spontaneous mandarin,” in ISCA Tutorial and Research Workshop on Disfluency in Spontaneous Speech, 2003.
- [24] T.-L. Lee, Y.-F. He, Y.-J. Huang, S.-C. Tseng, and R. Eklund, “Prolongation in mandari,” in Interspeech ICSLP 2004, Jeju Island, Korea, 4–8 October 2004, vol. 3, 2004, pp. 2181–2184.
- [25] N. R. Norrick, “Interjections as pragmatic markers,” Journal of pragmatics, vol. 41, no. 5, pp. 866–891, 2009.
- [26] B. Zhang, H. Lv, P. Guo, Q. Shao, C. Yang, L. Xie, X. Xu, H. Bu, X. Chen, C. Zeng, D. Wu, and Z. Peng, “Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2022. [Online]. Available: http://dx.doi.org/10.1109/icassp43922.2022.9746682
- [27] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022.
- [28] Z. Yao, L. Guo, X. Yang, W. Kang, F. Kuang, Y. Yang, Z. Jin, L. Lin, and D. Povey, “Zipformer: A faster and better encoder for automatic speech recognition,” Oct 2023.
- [29] J. Kominek, T. Schultz, and A. Black, “Synthesizer voice quality of new languages calibrated with mean mel cepstral distortion,” Workshop Spoken Language Technologies for Under-resourced Languages,Workshop Spoken Language Technologies for Under-resourced Languages, Jan 2008.