Pre-Trained Foundation Model representations to uncover Breathing patterns in Speech

The lungs play a central role in speech vocalization, where they act as the source of air that is pumped through the vocal tract, which acts as a filter (Stevens, 2000) to generate acoustic speech. Breathing is the source of most sounds that humans vocalize and speech production requires control and coordination of breathing and speech articulation, also known as speech breathing (Fuchs and Rochet-Capellan, 2021). Speech breathing demands more effort than regular breathing, where speech breathing is characterized by short inhalations to minimize interruptions during speech production, whereas regular breathing consists of equal phases of inhalation and exhalation (Hixon, 1987). Due to short inhalations, the velocity of air-inflow is higher compared to regular breathing (Conrad and Schoenle, 1979), hence, breath sound is normally audible in speech (Arafath K. and Routray, 2019). The volume of air exhaled during speech is influenced by the length and loudness of the intended utterance, and the exhale-duration is dependent upon the linguistic intent and sounds produced during speech production (Winkworth et al., 1994; Klatt et al., 1968). Speech production and breathing are inherently coupled and (Nallanthighal et al., 2020) aimed at sensing speech breathing patterns from the linguistic content and prosodic factors of speech.

Respiratory rate ($RR$) is a vital metric, where studies have shown that $RR$ is the most valid marker of exertion (Nicolò et al., 2014, 2017) and a reduction in $RR$ is an indicator of a person’s relaxation response (Grant and Rainville, 2009; Wielgosz et al., 2016; Kral et al., 2022, 2023) and self-reported well-being (Kral et al., 2023). Speech breathing parameters have been used for clinical applications (Solomon and Hixon, 1993) as well as for affective analysis (Goldman-Eisler, 1955; Heim et al., 1968). Prior work on breath-sound detection from audio has focused on the detection and categorization of particular breath sounds to distinguish between healthy and abnormal breath sounds (Li et al., 2017; Castro and Marti-Puig, 2014). $RR$ estimation has been investigated from both contact-based sensors and non-contact-based sensors (Sierra et al., 2006, 2004; Ren et al., 2015; Kumar et al., 2021; Ahmed et al., 2023; Rahman et al., 2022), to acquire nasal breath recordings and wearable microphones. In this work, we investigate estimating respiratory parameters from speech recorded using close-talking microphones, that is more likely to sense respiratory sounds in speech, compared to distant-microphones, due to their proximity to the mouth.

Prior work on speech-breathing focused mostly on using traditional acoustic features such as log-mel spectrograms (Nallanthighal et al., 2020), or their discrete cosine transformed counterparts (a.k.a, mel-frequency cepstral coefficients or MFCCs) (Arafath K. and Routray, 2019; Ruinskiy and Lavner, 2007; MacIntyre et al., 2020). However, in case of limited-size datasets, such representations make the downstream machine learning models prone to over-fitting, and as a consequence restrict the generalization capacity and robustness of the machine learning (ML) model. Recent advances in foundation models (Bommasani et al., 2021) have resulted in significant performance boost of speech technologies, where pre-trained model representations (Baevski et al., 2020; Hsu et al., 2021) have shown state-of-the-art performance for speech recognition (Zuluaga-Gomez et al., 2023), speaker recognition (Zuluaga-Gomez et al., 2023), and emotion recognition (Mitra et al., 2022). Representations from pre-trained foundation models have demonstrated better generalization capacity and robustness across different speech tasks, under various acoustic conditions and for multiple languages, hence we hypothesize that such representations will be quite useful for the task of speech based respiration parameter estimation.

Self-supervised learned (SSL) models such as Wav2Vec2 (Baevski et al., 2020) or HuBERT (Hsu et al., 2021) are trained on large volumes of unlabeled data and are anticipated to learn acoustic units from the training data. The learned acoustic units should be discriminable in their spectro-temporal representations, and represent distinct acoustic phonetic units (such as vowels, voiced/unvoiced consonants, pauses, aspirated noise etc.) or their sub-states.

We hypothesize that pre-trained representations should have information that can help with the above tasks and demonstrate better performance compared to standard mel-filterbank (MFB) based acoustic features given that they are pre-trained with large speech datasets.

Note that unlike prior works (Nallanthighal et al., 2020; Arafath K. and Routray, 2019) that have used standard acoustic features, we demonstrate that pre-trained model representations can be used for speech breath detection, and can demonstrate superior performance. In addition, we present a metric (breath-event error rate: $BER$) that indicates how closely the detected breath-events align with the groundtruth data. Finally, we present a convolution LSTM (Conv-LSTM) model and show that the network-depth and fusion of pre-trained representations and MFB helps to better estimate the breath time-series data from speech.

The rest of the paper is organized as follows: Section (2) presents the dataset used in our study, Section (3) introduces feature representations investigated and details on the acoustic model and its parameters, Section (4) presents the results, followed by conclusions in Section (5).

Publicly available speech datasets containing respiration time-series reference do not exist, hence we collected data internally. The 2020 speech paralinguistic challenge (Schuller et al., 2020) explored speech based respiration event detection, however the dataset used in that challenge is not publicly available. Data were collected from 26 adult speakers under realistic background acoustic environments (consisting of background noise) in an indoor setting. American English speakers, between the age 25 to 60, balanced by gender, were employed for the data collection. Data were recorded using microphone-enabled, wearable headphones. Speech data collected using wearable microphones and chest-belt measurements were collected across multiple sessions. During the data collection, participants were prompted to read a paragraph, where the reading session varied from 45 to 90 seconds. Note that conversational speech is not considered in this study, however we expect that findings from this work should generalize to such speech.

A strain-gauge chest-belt sensor (Vernier Go Direct Respiration Belt) was used during the data collection to obtain groundtruth reference chest contraction and relaxation (corresponding to inhalation and exhalation) measurements. Figure 1 shows a plot of a sample respiration signal spectrogram and its corresponding chest belt measurement. Due to calibration and subject variability, chest-belt measurements were observed to have variations, hence a quality check of the chest-belt measurement was performed manually and any data with erroneous measurement were removed. Chest-belt data were z-score normalized and dynamic range compressed before being used for model training. For some sessions the participant did not speak, hence they did not contain any recorded speech; such data were excluded from our experiments.

Data augmentation was performed to simulate faster and slower breathing by altering the speed of the entire audio signal. We have used 25 hours of speech data from 26 speakers in this study, where data from 3 and 4 speakers (roughly one hour of speech data/speaker) were set aside for validation and test sets, and the remaining 19 speakers were used for model training. Note that the validation and test split speakers were balanced by gender. Speech data were segmented into chunks of 30 seconds for model training, to ensure it contains at least one full breath cycle.

We trained baseline acoustic models using (i) MFB and (ii) Wav2Vec2 embeddings obtained from the $2^{nd}$ through $12^{th}$ transformer layers of the model. The performance of the respiration time-series estimation model is shown in Table 1. We present the results using metrics focusing on the time-series respiration signal estimation, where we have used $CCC$ (Lawrence and Lin, 1989) and root-mean-squared error ($RMSE$). Table 1 shows the baseline time-series estimation performance obtained from MFB and Wav2Vec2 representations. We also evaluated the segment-level $RR$ estimation performance, where for segment-level $RR$ estimation, we have used the following metrics: mean-absolute error ($MAE$) and Accuracy at 2 br/min error tolerance ($Acc@2bpm$). $MAE$ is computed by comparing the number of breath-events detected from the estimated time-signal obtained from the model, with that observed in the chest-belt groundtruth signal.

Accuracy for a segment is measured at a tolerance bound of +/-2 breaths/min (bpm) (we made this selection to have a conservative error-bound), where an estimate outside the tolerance-bound is treated as an error. Table 1 shows that the pre-trained representations from Wav2Vec2 perform better than the MFB features for the test-set, and the relative improvement was at-least 2.4% increase in $CCC$ and 6.8% relative reduction in $RMSE$.

Interestingly, Table 1 also shows that representations from different transformer layers of the Wav2Vec2 features had different impact on the performance, where the representations from layers 4 to 9 were more effective than the final layers 10 through 12. The best performance was obtained from layers 4 and 7, which gave 12.3% relative improvement in $CCC$, and 14.3% relative reduction in $RMSE$ compared to the MFB features. Even though we have used the SSL trained Wav2Vec2 (which is not fine-tuned on any specific task), the final layers may contain more phonetic-discriminatory information which may not be essential for breath-signal estimation (see section 3.2). The middle layers may contain more broad acoustic-level information that helps to detect the breathing patterns in speech, speech-activity and silent pauses, hence, they helped to generate better performance than the final layers. Note that given the findings in Table 1, we will be using the representations from layers 4 and 7 in the remaining of this paper to train (Conv-LSTM) models with 2 LSTM layers.

Next, we investigated the depth of the LSTM layers and Figure 5 shows that a 2-layered LSTM model overall performed the best providing higher $Acc@2bpm$ and lower $MAE$ for all the features. Table 2 show the validation set performance, when MFB feature and representations from different transformer layers of Wav2Vec2 was used.

We also investigated if saliency-driven feature selection can help to reduce the model size, while retaining the model performance. Using the approach outlined in section 3.4 we investigated pruning input representations, by keeping only 90%, 75%, 50% and 25% of the input representations, which in turn resulted in reducing the model parameter size by 9%, 22%, 44% and 66% respectively. Table 3 shows the result obtained from selecting salient representations from Wav2Vec2 layers 4 and 7. We introduce a metric: breath error rate ($BER$) to measure the accuracy of detecting breath events. $BER$ is computed by comparing the inhalation events in the groundtruth and estimated time-series signals, where we have only deletion of inhale-events (deletion errors, $D$) and inserted inhale-events (insertion errors, $I$), and use the total number of inhale events $N$ in the groundtruth data, to measure $BER$:

Table 3 shows that the representations from layer 4 performed better than those from layer 7, especially for the segment-level $RR$ metrics ($MAE$, $Acc@2bpm$ and $BER$). Selecting the top 50% representation based on saliency resulted in the best $BER$ with some regression in $MAE$ and $Acc@2bpm$ compared to the model trained with the full layer 4 representations. Note that the 50% representation based model is smaller than the full-representation based model by 44% (Figure 6 show the time-series estimate from the model). The above findings indicate that: (1) the earlier layers of Wav2Vec2 contain more respiration-relevant representation that resulted in better performance across multiple metrics, (2) $RR$ estimation $MAE$ as low as 1.6 bpm can be achieved using speech as input data, where an $RR$ estimation accuracy as high as 84% can be obtained for a tolerance of +/-2 bpm, and (3) saliency-based representation can help to reduce the model size by 44% that can provide better $BER$ but some regression in $RR$ estimation performance. Note that for segment-level $RR$ estimation the $MAE$ and $Acc@2bpm$ obtained from MFB are 2.38 and 69.3% respectively, indicating that Wav2Vec2 representations performed better than MFBs for the segment-level metrics as well. We investigated fusion of 50% salient layer-4 representation with MFB features ($Wav2Vec2_{4,50}$+MFB), result shown in the last row of table 3, where we observed that fusion of information helped to achieve the best BER, with comparable MAE and $Acc@2bpm$ from the best single-feature system (Wav2Vec2 layer 4), with 27% reduction in model parameter size. The fusion results indicate that the Wav2Vec2 and MFB representations may have complementary information, hence their fusion resulted in improved performance.

In this work we demonstrated that respiration signal can be estimated from speech data collected through close-talking microphones. Results from our work has shown a time-series estimation performance with $CCC$ as-high-as 0.77 and an $RMSE$ as-low-as 0.11, where the groundtruth respiration signal was z-score normalized. At the segment-level, we observed that $RR$ can be estimated with a $MAE$ of 1.6 bpm. We also observed that pre-trained model representations from Wav2Vec2 SSL model performed better than standard MFB feature, providing a relative $MAE$ reduction of 33.6% and relative improvement in estimation $CCC$ by 10%. Additionally, we observed that fusion of Wav2Vec2 and MFB features provided the best overall performance.

Future studies should explore the use of representations from fine-tuned foundation models with speech data containing respiration-relevant information. Additionally, the impact of subjective variance and the models’ generalization capacity should be investigated using a dataset containing larger number of subjects than what was available in the dataset used in this study. A limitation of this study is that it uses a dataset containing read speech, future work should investigate spontaneous speech for estimating respiration signal.