Prosody-Driven Privacy-Preserving Dementia Detection

Our threat model focuses on medical speech-processing systems developed for detecting dementia and their handling. These systems use audio data to generate embeddings that can help identify signs of dementia. This raises significant privacy concerns as speaker embeddings can potentially reveal individuals’ identities. We consider an adversary that has access to the anonymized embeddings and aims to re-identify the user by using the embeddings not for their primary purpose (dementia classification) but for speaker recognition. This information could be exploited for discriminatory purposes or targeted advertising. Effective anonymization techniques should prevent such linkage attacks while preserving speech naturalness, intelligibility, and performance in dementia detection.

We aim to safeguard the privacy of user identity in different scenarios that involve voice embeddings or medical speech processing. This involves obfuscating any identifying information that a user may not want to share, without compromising the functionality of the system. We also emphasize the need to offer various privacy settings to balance the trade-off between privacy and utility, and to encourage transparent privacy management practices.

We devise a prosody-based privacy-preserving extraction for speaker representations, trained on a larger auxiliary dataset that does not need to have the target attribute label. We leverage domain knowledge for the task of dementia detection and propose a method that performs disentanglement that focuses on prosody features relevant to dementia. Indeed, previous studies have shown that features such as speech rate, mean energy levels, number of pauses and lengths [19, 20] are informative for dementia classification. We explore two approaches: adversarial learning and mutual information-guided shuffling.

The proposed adversarial model is based on domain adversarial training [21] which aims to create domain-invariant latent spaces by maximising the domain discriminator’s confusion while minimizing the task-specific loss. In our case, we train our model to extract dementia-relevant prosody features while maximising the loss of speaker-relevant features. To train this model, we can take advantage of a larger dataset and extract prosodic features. At inference time, we run the model on any dataset we wish to anonymize. The model consists of several components: the feature extractor that extracts speaker representations, and prosody regressors for each prosodic feature. The loss of our model is defined as:

where $\mathcal{L}_{\text{ADpros}_{i}}$ denotes the $i^{th}$ loss associated with the $i^{th}$ dementia-relevant prosody feature extraction, and $\mathcal{L}_{\text{SPKpros}_{j}}$ the $j^{th}$ loss associated with the $j^{th}$ speaker-relevant prosody feature regressor. $\lambda_{j}$ is a hyperparameter that controls the balance between the different objectives and $n$ is the total number of prosody regressors.

We propose a feature selection approach based on mutual information to identify important features relevant to dementia while perturbing less relevant features to lower speaker recognition accuracy. Our method is similar to [16] where they extract Shapley values from a classifier to select key dimensions, however, unlike theirs, our approach is classifier-free. The intuition behind this approach is that critical features associated with dementia are likely to have higher mutual information with the target variable (dementia), while shuffling the remainder to preserve dementia-related features but decrease speaker recognition accuracy. We also explore using prosody features instead of the dementia label as the target variable.

Mutual information is a measure of mutual dependence between two random variables. In our context, we compute the mutual information between the distribution of dimension of the speaker embeddings and the distribution of the dementia label across the dataset. Formally, the mutual information $I(X;Y)$ between random variables $X$ and $Y$ is defined as:

where $p(x,y)$ is the joint probability mass function of $X$ and $Y$, and $p(x)$ and $p(y)$ are the marginal probability mass functions of $X$ and $Y$, respectively. To estimate the mutual information, we use the k-Nearest-Neighbour-based MI estimator [22] which can be applied to both continuous and discrete variables. We design the feature selection strategy as follows: 1) Compute the mutual information between each embedding dimension and the dementia/prosody variable across the whole corpus. 2) Select top $n$ dimensions as important dementia-related features. 3) Shuffle the remaining features. When combining several features, compute the top $n$ dimensions for each and compute the union as the set of important features.

We train our disentanglement model on the LibriSpeech dataset [23], a corpus of read English speech. We selected the train-clean-100 subset which consists of 100 hours of speech from 251 speakers. We split the data into 25685, and 2850 samples for train, and dev sets respectively. We extract prosody features for each sample.

For testing dementia detection, we use two publicly available datasets widely studied in dementia classification: ADReSS [24] and ADReSSo [24]. The ADReSS (ADR) dataset is a subset of the DementiaBank dataset, a collection of recordings from control (CD) and dementia (AD) patients describing the Cookie Theft Picture [25]. Manual transcripts are provided and we split each sample into segments using the sentence-level timestamps from the transcripts. We group the original test and train sets and split the data to contain one sample per speaker in the test set and keep the remaining samples as training data. We end up with 1723 samples (868 CC $|$ 855 AD) in the training set and 156 (78 CC $|$ 78 AD) in the test and validation sets. The ADReSSo (ADRo) dataset is another subset of DementiaBank designed for detecting dementia from spontaneous speech only, without access to manual transcriptions. The original set consists of 151 train samples (87 CC $|$ 74 AD) and 71 test samples (35 CC $|$ 35 AD). We use the provided segmentation timestamps to isolate segments spoken by the patients and get 2705 samples in the train set (920 CC $|$ 1026 AD) and 231 samples (74 CC $|$ 87 AD) in the test and validation sets, one per speaker.

We select a pre-trained ECAPA-TDNN [27] embedding extractor, regarded as the state-of-the-art (SOTA) for speaker embeddings. We use the SpeechBrain [28] implementation and pretrained model which was trained on the VoxCeleb dataset. We augment the ECAPA-TDNN embedding extractor with classifiers for each prosodic feature, each consisting of two layers with 126 hidden dimensions. We use the Mean-Square-Error Loss for the prosody regressors and investigate $\lambda$ values in {1,5,10,30} and set $\lambda$ to 1.0. We train the model with a batch size of 8 with cyclical learning rate policy (CLR) and set the base and maximum learning rates to 1$\text{e-}7$ and 1$\text{e-}5$ respectively. We finetune the models for 15 epochs with early stopping. We compare our model to several models: a model trained on a dementia dataset to classify dementia while fooling the speaker classifier (ADV SPK_AD + AD), a model trained on the LibriSpeech dataset to fool the speaker classifier (ADV SPK_LS), a random shuffling method (Shuffle_Random) and a shuffling from [16] where we train an XGBoost model [29] on dementia and select top $n$ Shapley values (Shuffle_Shap). For the mutual information shuffling method (Shuffle_MI), we compute the mutual information using the sklearn library  [30]. The dementia classifier is a two-layer model with a hidden space size of 8 and a ReLu activation between layers, trained with Binary Cross-Entropy. The speaker classifier is a two-layer model with a hidden space size of 96 trained with Cross-Entropy Loss. We optimise regressors with bayesian search.

We choose prosody features based on mutual information (MI) with dementia labels in the ADReSS dataset, selecting the top 35 quantiles. Then, we assess their importance in speaker recognition by computing MI with speaker labels. Figure 1 shows MI scores for features w.r.t. class and speaker identity. Notably, mean f0 and energy are significant for both dementia and speakers. However, mean energy is strongly linked with speaker identity, while f0 is a known speaker characteristic. To remove speaker information, we decide to separate features into prosodic and disfluency features to disentangle speakers and dementia. We select speech rate (spr), number of pauses (pnum), length of pauses (plength) and number of syllables as dementia features and mean f0 (f0) and energy (nrg) as speaker-relevant for the ADV PROS and Shuffle_{MI_pros} systems.

Table 1 shows the performance of the anonymization systems. We see that ADV SPK_AD + AD drops the speaker F1-score to 0% for both ADR and ADRo and increases the EER by 12%, 10% respectively, giving the best anonymization performance. We note that the EER on the original samples is quite high (35%), which we attribute to the noisy nature of the dataset. However, in this work, we are interested in the relative increase. We see the system reduces the AD detection F1-score to 65%, on par with the shuffling approaches. Through experimentations, we observe that the energy (nrg) gave us stronger privacy guarantees across subsets, as shown in Table 3. We report ADV_nrg PROS as our best system. ADV_nrg PROS, drops the accuracy to 0.01% and adds 8% to the EER while preserving 73% F1-score on ADR, a drop of 7% from the original performance, making it the best-performing approach. On ADRo it achieves comparable performance to ADV SPK_AD + AD and shuffling approaches. We compare it to the model trained adversarially against a speaker classifier ADV SPK_LS. Surprisingly, the models improved the speaker recognition and dementia classifiers on both datasets. We evaluated it in combination with other features, and a similar pattern was seen, hence we report only ADV SPK_LS. Finally, we evaluate different shuffling approaches. We report the systems selecting top 50 features as it gave us the best privacy/utility tradeoff for all systems. We note that the selection process with MI based on the label (Shuffle_{MI_AD}) and prosody (Shuffle_{MI_pros}) seems to add little to the accuracy when compared to Shuffle_Random. Nevertheless, the shuffling approaches achieve similar results to the strong baseline ADV SPK_AD + AD, while being simpler.

The Voice Privacy Challenge (VPC) revealed that all methods, including x-vector embeddings and signal processing techniques, could lead to a degradation in speech naturalness and intelligibility [7, 34]. We thus conducted an objective evaluation on the ADReSS dataset using speech synthesis models using two text-to-speech (TTS) systems: SpeechT5 and YourTTS (zero-shot TTS systems) to assess the effectiveness of anonymized embeddings in text-to-speech (TTS) tasks. We utilize sentence-level transcriptions as input data and condition the TTS system on speaker embeddings. For a fair comparison, we regenerate the raw recordings using the raw embedding extracted from the original recordings. We then use the anonymized embeddings of different settings to generate different test sets. Rankings were based on average performance across all metrics, with lower average ranks indicating better overall speech quality. MOSNet represents the Mean Opinion Score predicted by a neural network model for the quality of speech signals. Higher MOSNet, SI-SDR, and STOI scores indicate better quality and speech intelligibility. Table 2 reports the results of our proposed method under different settings and the baseline (synthesized raw recordings) using two TTS systems. We find that the anonymization results in almost the same voice distinctiveness as the data originally had, and, according to speech recognition, produces intelligible speech recordings. ADV SPK_AD + AD seems to strike the best balance in maintaining voice distinctiveness while ensuring anonymity, as shown by its lower Avg. Rank, even though all methods introduce some level of degradation in SI-SDR and WER, which is a trade-off for achieving speaker anonymity.

We investigate the impact of dementia-relevant and speaker-relevant prosodic features and report the results in Table 3. We find that the addition of prosody features improves dementia detection with a negligible impact on privacy metrics. Nearly all features are useful for preserving dementia, while the speaking rate did not seem to affect the detection.