PPPR: Portable Plug-in Prompt Refiner for Text to Audio Generation

Each audio clip can have multiple different text descriptions, representing the fine-grained diversity of text descriptions for the audio. By increasing this diversity, the exposure of audio clip to different text descriptions during training can be enhanced. Through increased exposure, the model can better learn the mapping between text descriptions and audio events.

The annotation and comprehension capabilities of LLM have reached a level comparable to that of humans. PPPR actively increases the diversity of audio descriptions at a fine-grained level by leveraging LLM. We aim to increase the diversity of text descriptions by performing multiple rewriting iterations. Specifically, to generate rewritten text-audio pairs, it is necessary to first randomly select text-audio pairs from the AudioCaps [19] dataset. Then, the designed prompt is inputted into LLM Llama. Llama rewrites the text from the randomly selected text-audio pairs based on the prompt. It is crucial to ensure that the semantics of the text descriptions remain unchanged before and after rewriting; otherwise, rewriting will introduce noise into the training set. We have designed a prompt to assist Llama in rewriting while preserving the semantic meaning of the text descriptions. The specific prompt is as follows:

Rewrite the following text description using different
wording while preserving the same meaning.

As shown in the Figure 1, the phrase ”Multiple people speak” is rewritten as ”Several people engage in conversation”, carrying the same meaning. Therefore, the rewritten sentence retains the granularity of information from the original description. By repeatedly executing the above steps, we can increase the diversity of text descriptions.

A TTA system should be able to accurately generate target audio based on input text descriptions. However, inaccuracies or incomplete descriptions in the text can lead to a decrease in accuracy of the generated audio. Trained on massive text data, LLM possesses excellent text processing capabilities under prompt guidance. The CoT enhances LLM’s reasoning abilities, enabling it to better handle complex tasks.

PPPR utilizes CoT to gradually guide Llama in reasoning. Llama progressively adjusts the input text descriptions into more accurate and error-free expressions, thus enhancing the accuracy of the generation model. Specifically, by prompting ”Reasoning with the following prompts step by step.”, Llama is instructed to reason step by step. Firstly, a spelling check is required for the input text. Incorrect spelling may lead to semantic errors, causing the generation model to misunderstand the semantics of the text. Llama is fully capable of identifying and correcting errors. Next, extract the existing events from the text description and review each event description for completeness and accuracy. If any description is incomplete or inaccurate, supplement it to make the content more easily understood by the audio generation model. To implement these functionalities, we design the following prompt:

Reasoning with the following prompts step by step.
  1.First, check for spelling errors. Correct any found.
  2.Then, extract sound events from the input text.
  3.Review each event description for completeness and
  accuracy. Supplement inaccurate or incomplete
  description.

As shown in the Figure 1, PPPR corrects spelling errors in the input using CoT, replacing the misspelled ”cot” with ”cat”. Additionally, PPPR supplements the description that is not easily understandable. For instance, ”A toilet flushing” is supplemented with ”A toilet flushing like the sound of water rushing down a narrow channel, followed by a hollow gurgling as it refills”. This enables the generation model to generate based on the supplemented description, even if it has not seen ”the sound of toilet flushing” during training. Through these steps, the model’s generation performance has been greatly improved.

After being enhanced by LLM Llama, the diversity of text descriptions in the original training set has increased. The distribution of the enhanced text descriptions has become more complex, so using small pre-trained language models like BERT [20] is no longer sufficient to meet the demands. The FLAN-T5 model [21] is pre-trained on a large-scale chain-of-thought-based and instruction-based dataset, making it suitable for complex text processing tasks. Therefore, we use the pre-trained LLM FLAN-T5-LARGE [21] as the text encoder to obtain text embedding. Specifically, given input text T, Flan-T5-LARGE extracts its feature embedding $E^{T}$. $E^{T}$ serves as conditional information to guide the training of the generation model.

The LDM [3] is used to generate intermediate latent features, with the feature embedding $\bm{E^{T}}$ extracted by Flan-T5-Large from the textual descriptions as conditions. During training, the LDM involves two processes: 1) A forward process in which the latent variable $z_{0}$, obtained by compressing mel-spectrogram $m$ using the VAE Encoder, gradually transforms into a standard Gaussian distribution $z_{N}$ over $N$ steps, with noise $\epsilon$ added at each step. 2) A reverse process for the model to predict the transition probabilities $\bm{\epsilon_{\theta}}$ of each step $\bm{n}$, for reconstruction the data $\bm{z_{0}}$ by removing the noise $\bm{z_{N}}$. The loss function [3] as:

$$L_{n}(\theta)={E}_{\bm{z}_{0},\bm{\epsilon},n}||\bm{\epsilon}-\bm{\epsilon}_{\theta}(\bm{z}_{n},n,\bm{E^{T}})||^{2}_{2}$$

(1)

where $\bm{\epsilon_{\theta}}$ is the Gaussian distribution predicted by LDM with current state $\bm{z_{n}}$, current step $\bm{n}$, and current condition $\bm{E_{T}}$.

We use VAE to compress spectrograms into latent features or to reconstruct spectrograms from latent features, and we use HiFi-GAN to reconstruct spectrograms into audio signals.

This work conducts experiments using the AudioCaps dataset, which comprises 38,679 audio clips in the training set, each paired with manually annotated captions. The validation set consists of 2,240 instances. Utilizing Llama, the audio caption corresponding to an audio clip in the training set is rewritten into four different sentences, resulting in five semantically equivalent but syntactically different captions per audio clip after rewriting.

During training, one caption is randomly selected from the five captions corresponding to each audio clip to form a (text, audio) pair for training the LDM. Therefore, during training, the training set comprises a total of 193,395 unique instances, including 38,679 original manual annotations and 38,679 * 4 (PPPR Augmentaion) instances. For testing, the original and the PPPR optimized AudioCaps test set will be used to evaluate the performance of the model respectively.

All datasets are resampled to 16kHz sampling rate and mono format, with samples padded to 10.24 seconds. We then extract mel-spectrograms from audios using parameters of 64 mel filter bands, 1024 window length, 1024 FFT, and 160 hop size, resulting in (1,64,1024) mel-spectrograms, akin to grayscale images with 64 height and 1024 width in 1 channels. The FLAN-T5-LARGE text encoder is frozen in our setting and we only train the parameters of the latent diffusion model. The diffusion model is based on U-Net [24] architecture and has a total of 866M parameters. We use 8 channels and a cross-attention [25] dimension of 1024 in the U-Net model. We use the Adafactor optimizer [26] and AdafactorSchedule for training. We train the model for 16 epochs on the training dataset and report results for the checkpoint with the best validation loss. We use four 4090 GPUs for training, where it takes a total of 62 hours to train 16 epochs, with validation at the end of every epoch. We use a per GPU batch size of 4 with 4 gradient accumulation steps. The effective batch size for training is 4 (instance) * 4 (accumulation) * 4 (GPU)= 64.

We compare our method with several mainstream methods in TTA tasks that employ mixed concatenation strategies for data augmentation: AudioGen [8], AudioLDM [10], and Tango [11]. AudioGen and Tango utilize a strategy that mixes audio clips and concatenates their corresponding audio captions. AudioLDM, trained solely on audio clips, only mixes audio clips.

In this work, we perform both objective evaluation and human subjective evaluation. The main metrics are used for objective evaluation include frechet distance (FD), inception score (IS), and kullback–leibler (KL) divergence. Analogous to the Frechet Inception Distance (FID) used in image synthesis, the FD score in audio domain quantifies the global similarity between created audio samples and the target samples without the need of using paired reference audio samples. The IS score is effective in evaluating both sample quality and diversity. The KL score is calculated using paired samples and it measures the divergence between two probability distributions. All of these three metrics are built upon a state-of-the-art audio classifier PANNs [27]. A higher IS score indicates a larger variety in the generated audio, while lower KL and FD scores indicate better audio quality. Following previous subjective evaluation method [10, 11] in TTA field, we ask five human evaluators to assess two aspects of the generated audio, including overall audio quality (OVL) and relevance to the text caption (REL). We randomly select 20 audio samples from each of baseline and proposed method generated, and ask participants to rate them on a scale from 1 to 100.

Table 1 displays the main evaluation results on the original AudioCaps test set. It is worth noting that without using the PPPR to enhance the dataset during training, i.e., training on the original AudioCaps dataset, the audio generation model based on LDM does not demonstrate any advantage. Model trained using the proposed PPPR method achieves relatively good results in both objective and subjective evaluations. It obtains the highest score of 8.72 on the IS metric, indicating high diversity in the generated audio among all methods. Additionally, the scores on the FD and KL metrics are 24.59 and 1.39, respectively, which are not significantly different from the scores of 23.86 for AudioGen and 1.38 for Tango. Considering the scores of FD, KL, and IS, our method can produce diverse and high-quality audio. The proposed method also demonstrates very promising results in subjective evaluation, with an overall audio quality score of 79.51 and a relevance score of 75.86. We speculate that increasing the diversity of text descriptions at a fine-grained level has improved the robustness of acoustic models, thereby enhancing the performance of generation.

Table 2 presents the test results after using PPPR to process the text descriptions. Subjective evaluation scores indicate that the generation performance of all models improves when the test text descriptions are regularized. Our method achieved the highest scores in the OVL and REF metrics, with scores of 82.53 and 77.39, respectively. We believe that after the text descriptions are regularized by PPPR, their accuracy is improved, thereby enhancing the accuracy of model in generating audio.

Furthermore, we analyzed how the PPPR method based on CoT performs when the test text descriptions contain multiple events. Consider the following examples: “Leaves rustling followed by a small bell chiming as birds chirp in the background” contains three separate sequential events, whereas “A duck quacks continuously” contains only one. We segregate the AudioCaps test set using the presence of temporal identifiers –when, while, before, after, then, follow, during – into two subsets, one with multiple events and the other with single event. We show the subjective evaluation results for audio generation on these subsets in Table 3. Our method achieves the best OVL and REF scores for both multiple events and single event instances. We infer that PPPR utilizes CoT to extract and process events from the descriptions one by one, enabling the information within the descriptions to be effectively decomposed and optimized, thus enhancing the accuracy of the entire generation system.