Preset-Voice Matching for Privacy
Regulated Speech-to-Speech Translation Systems

The Similarity Feature Extraction module extracts meaningful features from the input voice. These features are used to determine the most similar consented preset-voice in the target-language from our preset-voice library. This module takes in speech signals as input and outputs similarity feature encodings (gender-emotion pair combinations in the case of GEMO-Match) to match a consented similar preset-voice.

Module 2, the Preset-Voice Library, contains a collection of preset-voices in desired target-languages. The Preset-Voice Library acts as a feature codebook, informing the mapping between feature encodings and target-language preset-voice samples. This module takes in a feature code as input, and outputs a matched consenting speaker preset-voice sample.

As input, the TTS Module takes in the matched consented preset-voice and target-language text (from an auxiliary TTT module). The TTS Module outputs a clone of the most similar preset-voice in a desired language relative to the features extracted in the Similarity Feature Extraction module. Any voice cloning TTS model supporting the desired target-languages can be used in the TTS Module. Therefore, PVM is a general framework and is easily modifiable for many industry settings.

The GEMO-Match Similarity Feature Extraction module contains 3 classifiers in two stages. The first stage contains the gender classifier, and the second stage includes both the male-emotion classifier, and the female-emotion classifier. The Similarity Feature Extraction classifiers are trained in the source language (English).

In GEMO-Match, the Preset-Voice Library contains previously consenting speakers in desired target-languages for a given S2ST task. The Preset-Voice Library partitions target-language preset-voices by language, gender, and emotion. The number of target-languages supported by GEMO-Match depends on the ability to gather preset-voices in each desired target-language. The Preset-Voice Library in our provided implementation includes two target-languages, French and German. Therefore, the GEMO-Match implementation can translate from English to either French or German.

The GEMO-Match TTS Module performs TTS. The TTS Module is straightforward and performs TTS given a matched preset-voice and a text prompt in the desired target-language. We implement GEMO-Match with two distinct TTS models, discussed in 6-B and 6-C.

First, source language speech is input to the Similarity Feature Extraction module. The gender classifier then classifies the input voice as male or female. Next, given the gender classification result, the source speech is input to the corresponding gender-dependent emotion classifier. The appropriate gender-dependent emotion classifier will then classify the source language speech as happy, angry, sad, disgust, or neutral. The two-stage classifier output pair is then concatenated (i.e., Female - Sad).

The resulting concatenation is used alongside the intended target-language to query the most similar preset-voice in the Preset-Voice Library. Finally, the feature-matched preset-voice is passed alongside a text prompt to the voice cloning TTS model. This algorithm assumes that the intended target-language will be an input to the system. The performance of GEMO-Match depends primarily on the robustness of the Similarity Feature Extraction classifiers.

RAVDESS is a benchmark emotional speech dataset containing 1440 audio files of 24 professional actors (12 female and 12 male) with the emotions calm, happy, sad, angry, fearful, surprise, and disgust [33]. As GEMO-Match requires consistent labeling across source and target-language data, we focus on a subset of 5 common emotions: happy, angry, sad, disgust, and calm (neutral). Each speech sample was originally provided with two intensities, normal and strong. We filtered the speech files to include only strong intensities as the emotion is more apparent in those samples. After filtering, the RAVDESS subset contains a total of 5 speech recordings per actor per emotion.

Training a robust gender-emotion classifier requires numerous samples of speakers from various demographics, speaking a variety of utterances with different emotional intensities. We found that many available speech-emotion datasets have limited variance in regards to at least one of these features. To help facilitate gender-dependent training research, we provide a Combined Gender-Dependent Dataset (CGDD), made from combining four benchmark emotional speech datasets: RAVDESS, CREMA-D, SAVEE, and TESS [33, 34, 35, 36].

The RAVDESS dataset is explained in section 5-A. CREMA-D is comprised of 7,442 audio recordings of 91 actors. These clips include 48 male actors and 43 female actors, with ages ranging from 20 to 74. SAVEE database includes four English male speakers aged between 27 and 31, totaling 480 files. The TESS database contains two female speakers, one aged 26 and the other aged 64, with a total of 2800 files.

The CGDD dataset is processed for gender-dependent training, useful for hierarchical emotion detection algorithms like GEMO-Match. We further processed the audio based on pitch frequency and loudness to obtain a higher-quality dataset. As pitch and loudness are crucial attributes of speech, we filter the data to ensure the files are within a suitable range for speech recognition [37]. Additionally, we use RMS loudness to eliminate excessively quiet or loud files. The best quality was found with a pitch frequency range of 75 Hz to 3000 Hz. We removed audio samples with RMS loudness less than -23 dBFS and greater than -20 dBFS.

We processed the RAVDESS and CGDD datasets to be compatible with the hierarchical gender-dependent emotion classification architecture of the GEMO-Match Similarity Feature Extraction module. For both datasets, we partitioned the speech signal files on gender and further organized them into five gender-emotion directories. We then converted the speech signals to mel-spectograms using the Fast Fourier Transform. Next, the mel-spectrograms were converted to image representation (PNG format) to be processed by a pre-trained ResNet50 model initialized with ImageNet weights [38]. Our data pre-processing methodology is similar to the procedures outlined in [39]. The Python library Librosa was used to convert speech signal files to mel-spectrogram signals.

For this test, we assess the robustness of GEMO-Match. The performance of GEMO-Match depends on the three Similarity Feature Extraction classifiers. We fine-tuned and evaluated these classifiers on the RAVDESS and our CGDD dataset in terms of accuracy and precision. Each classifier was implemented as a ResNet50 previously pre-trained on ImageNet. The results of the six classifiers are shown in tables I and II.

The same approach was used to train each ResNet50. The gender classifiers were trained for 20 epochs, while the male-emotion and female-emotion classifiers required 30 epochs to converge. Each emotion classifier was trained using a dynamic learning rate schedule: 0.01 for the first 20 epochs, reduced to 0.001 for the remaining 10.

We used the Adam optimizer, and the Pytorch ImageDataGenerator function for data augmentation [40]. The classifiers were trained using a batch size of 32 and a train-test-validation split of 60-20-20. The models were optimized using categorical cross entropy as the loss function, incorporating batch normalization and dropout layers for regularization. The activation functions used were ReLU for internal layers and softmax for the output layer.

We test GEMO-Match in terms of speaker naturalness on the task of translating English speech into French and German speech. GEMO-Match is implemented within a cascaded S2ST system using SeamlessM4T for TTT, and XTTS as the TTS module [41, 27]. XTTS is a state-of-the-art TTS model which supports zero-shot voice cloning across 17 languages. Instead of performing STT, we provide ground truth source-language (English) text directly to the TTT model (SeamlessM4T) to measure the isolated performance of GEMO-Match across multiple languages. We measured speaker naturalness using the standard metric Non-intrusive Objective Speech Quality Assessment (NISQA) [42, 43].

We show PVM algorithms lead to higher naturalness in S2ST outputted speech by alleviating the need to perform cross-lingual voice cloning. We compare two cases of S2ST. The first case is when XTTS performs cross-lingual cloning from an English voice input to the target-languages German and French. In the second case, GEMO-Match performs the cross-lingual matching, allowing XTTS to run monolingual TTS given the matched target-language voice as input.

The French and German preset-voices used in this experiment are sourced from the CAFE, and EmoDB datasets respectively [44, 45]. For each target-language in both experimental pipelines, we used 150 English text transcriptions from the CREMA-D dataset alongside emotive English audios from RAVDESS as input [34]. We ensured that our RAVDESS audios had an average NISQA (3.54) similar to the preset-voices in our target-languages. For additional context, we included the average preset-voice NISQA scores for both target-languages in Table III.

We compared the run-time of GEMO-Match to state-of-the-art TTS models VALL-E X, XTTS, SeamlessM4T, and OpenVoice, as shown in Figure 1 [46]. The gender, male-emotion, and female-emotion classifiers were implemented using the same lightweight ResNet50 models as in 6-A. Each model was given 10 identical utterances with their respective transcriptions, and average inference run-times were calculated. The inputs were each 15 seconds and varied in tone, emotion, pacing, and vocabulary.

We compared PVM (using GEMO-Match) with OpenVoice as they are both cascaded TTS frameworks that decouple voice-cloning from voice synthesis. OpenVoice uses a variation of VITS for TTS in its open-source implementation [47]. For consistent comparisons with OpenVoice, we use StyleTTS2 for TTS with GEMO-Match [48]. StyleTTS2 and VITS are both styling-based models and display similar run-times. StyleTTS2 is a monolingual TTS model, and we use it to show the run-time benefits of PVM removing cross-lingual voice cloning in cascaded S2ST systems.

Figure 2 compares GEMO-Match with the OpenVoice framework in terms of run-time scaling in multi-speaker scenarios. We plotted the number of times each system must re-run auxiliary modules while performing TTS over time in multi-speaker instances. The plots were generated using Python.

Tables I and II show the precision and accuracy of the Similarity Feature Extraction module classifiers. Testing GEMO-Match on RAVDESS across emotions, the Male-Emotion Classifier performs best on happy, angry, and neutral, which have precision scores of 78%, 78%, and 80%, respectively. The Female-Emotion Classifier performs well on angry and neutral, achieving 100% and 90% precision, respectively. We find GEMO-Match overfits to certain gender-emotion classes when trained on RAVDESS. This is prevalent in the Female-Emotion Classifier performance, as it classifies angry emotions with perfect precision, but classifies sad and disgust with 40% precision.

As illustrated in Table I, GEMO-Match generalizes more consistently across emotions when trained on CGDD compared to RAVDESS. In the cases of both datasets shown in Table I, GEMO-Match tends to classify angry and neutral effectively. The improvements in generalization described in Table I when using CGDD instead of RAVDESS showcases that some benchmarks are currently lacking variation. CGDD can remedy this, as it has higher variance compared to RAVDESS, comprising of multiple benchmark datasets as described in section 5-B.

Table II shows the accuracy of GEMO-Match on RAVDESS and CGDD. The GEMO-Match gender classifier scored 94% accuracy on the RAVDESS dataset, and 97% on CGDD. The best GEMO-Match emotion classifier results are found when training and testing on CGDD, which results in 63% accuracy for the Male-Emotion Classifier and 71% for the Female-Emotion Classifier. Therefore, our proposed CGDD dataset can improve model generalization compared to benchmark datasets like RAVDESS.

The results in Table III show PVM implementations can significantly improve the output naturalness of S2ST systems by enabling monolingual TTS within S2ST. We find this trend holds across the two tested languages, French and German. When XTTS performs cross-lingual TTS from English to German, NISQA values decrease from 3.54 (English) to 3.41 (German). Similarly, when XTTS cross-lingually clones from English to French, the input-output NISQA values are 3.54 and 3.54, respectively. Overall, XTTS either maintained or degraded the input naturalness when performing cross-lingual cloning in our experiments.

We find XTTS performs much better in a monolingual setting, which can significantly enhance S2ST quality. The average NISQA score when XTTS cloned from German preset-voices to German outputs increased from 3.47 to 3.69. The same increase is seen with French, though to a lesser degree. For our tested language pairs, GEMO-Match consistently improves output naturalness by allowing S2ST pipelines to clone in a monolingual context while maintaining cross-lingual behavior.

The run-time results of different TTS approaches are shown in Figure 1. VALL-E X and XTTS, deep multilingual voice cloning models, are slowest on average. SeamlessM4T offers multilingualism in multiple modalities, but does not clone voices, and has significantly lower runtime than the aforementioned models. This underscores additional complexities inherent to achieving speech translation and voice cloning in a single embedding space.

In our experiments, the lowest run-times were achieved by our PVM implementation (GEMO-Match with StyleTTS2) and OpenVoice. Both of these frameworks are not strictly limited to a specific TTS module for processing. As such, the runtime of their auxiliary, decoupled systems are noted separately in Figure 1. OpenVoice uses the post-processing tone extractor described in [46], and PVM uses GEMO-Match. For these isolated auxiliary modules, we achieved an average runtime of 0.52 for OpenVoice and 0.61 seconds for GEMO-Match.

Figure 2 compares these auxiliary modules under sequential inference on long multi-speaker inputs. For this comparison, we focus on the run-time of the entire S2ST system. Figure 2 shows that GEMO-Match need only run when a new speaker is presented in the input, while OpenVoice must always post-process the TTS output to achieve the desired result. Therefore, PVM offers favourable scaling properties, making it desirable for many commercial use-cases.

In this section, we include some examples of cases where PVM can be applied to industry settings.

APIs are a common avenue for controlled public access to ML models and pipelines. These access points are commonly subjected to adversarial attacks, where imperceptible artefacts are injected into inputs to produce undesirable results. In the PVM framework, the audio input by our user is not directly passed to the TTS model, and is only matched to a consented speaker using feature similarity. This limits the scope of poor results that could be triggered by an adversarial user by negating direct access to the TTS model. Additionally, propagating audio input data from a genuine user through fewer modules in the pipeline limits opportunities for sensitive bio-metric data to be extracted by malicious third parties. Ultimately, removing direct control over synthesis of the input voice prevents bad actors from cloning a non-consenting speaker for nefarious goals.

We also consider how PVM can be extended to help regulate open-source models. As mentioned in Section 8, an autoencoder could be applied to derive robust latent space representations of the preset-voices. Matching based on similarity would still occur on the raw preset-voice audios, but their corresponding preset encodings would be passed as input to the voice cloning TTS model. The encoder/decoder models would not be published alongside the rest of the system. As the TTS model would have only been trained on the latent embeddings, the published system could not be hijacked to clone non-consenting voices.

In content localization systems, media content is leased by distributing platforms, while rights to the reproduction of the likenesses of individuals present in the content is not readily available. Not only can PVM secure these systems in the manners mentioned above, but its regulated application can help bring this budding market to life by efficiently producing translated content in only the voices of consenting speakers. We believe PVM provides feasibility to the commercialization of such systems while being robust against future industry regulations.

We hope these examples give insight into the vast extensibility of the PVM framework.