MR-RawNet: Speaker verification system with multiple temporal resolutions for variable duration utterances using raw waveforms

Speaker verification (SV) is the task of verifying whether an anonymous speaker is the target speaker registered in the system. With the development of deep neural networks (DNNs), traditional machine learning-based SV systems have been largely replaced by DNN-based SV systems [1, 2, 3]. Despite the exceptional performance achieved by DNN-based SV systems, most systems are typically evaluated using long utterances [4, 5, 6]. However, in real environments, shorter utterances of 1 to 2 seconds are often encountered, and SV systems experience a performance degradation as the length of the utterance decreases. This degradation occurs because short utterances may not contain sufficient speaker-specific phonetic characteristics that can be obtained from speech [7, 8, 9]. Thus, SV systems should be constructed to yield similar performance across utterances of various lengths, particularly shorter ones.

To enhance robustness against variable lengths, we initially focused on the input features of the system. The inputs for SV systems can be traditional handcrafted features such as MFCCs or Mel-filter banks [2, 3, 4, 5], or raw waveforms, which have recently begun to be utilized [10, 11, 12, 13]. While handcraft features can provide refined information through processing based on human knowledge, they may also offer information loss compared to the unprocessed raw waveforms [14, 15, 16]. Especially for SV systems that consider limited information, as much information as possible should be available from the input. Therefore, we use raw waveforms as input to ensure robustness to utterances of various lengths, which have high potential due to the absence of information loss.

In traditional raw waveform-based systems, a one-dimensional input is converted into a two-dimensional feature map through a special module in the first layer, such as [17, 18]. The first-layer module extracts meaningful representations from the input, typically based on a fixed frame size. However, representations based on a constant frame size are unable to offer superior time and spectrum resolution simultaneously [19]. Consequently, these systems often prioritize spectral resolution, potentially degrading performance for various temporal utterances. At this point, we introduced a multi-resolution encoder (MRE) designed to capture information at different temporal resolutions. The MRE, a module proposed in Han $et$ $al.$ [19], extracts multiple features at various temporal resolutions. We specifically tailored this module for use in the first-layer module of a raw waveform-based system, naming it multi-resolution feature extractor (MRFE). Thus, the MRFE is able to consider both temporal and spectral resolution simultaneously,deriving features that maintain robustness across various lengths.

In addition, we propose a new bottleneck named the multi-resolution attention (MRA) block. The MRA combines the advantages of Res2Dilated block [4] and an extended dynamic scaling policy (EDSP) [9]. The Res2Dilated block gradually builds up the temporal context using dilated convolutional layers and hierarchical residual connection [20]. The EDSP, an extension of the Elastic [21], trains the network to operate dynamically based on the scale of the data. These two modules each focus on different aspects of the temporal context: the Res2Dilated block focuses on broader contexts, while the EDSP considers variety of contexts. Meanwhile, various studies have focused on exploring different temporal resolutions to effectively maximize speaker information from utterances for robustness against utterance length [22, 23]. By merging these two modules, the MRA block can consider a wider and more diverse temporal context, thereby enhancing robustness to changes in utterance length.

We finally propose a novel structure called MR-RawNet, which is robust to variations in utterance length by utilizing both the MRFE and the MRA. Experimental results on VoxCeleb1&2 datasets [24, 25] demonstrated that MR-RawNet showed superior performance for variable duration utterances in raw waveform-based systems.

In a recent study on SV that utilized raw waveforms as input, RawNet3 [13] achieved remarkable performance. Nevertheless, the authors posit that there is still potential for enhancement in RawNet3. This section delineates the architecture of RawNet3 and outlines potential areas for improvement.

RawNet3 is a modified structure of the ECAPA-TDNN [4], which is widely used in SV research. As shown in Figure 1-(a), RawNet3 is composed of a parameterized analysis filterbank (ParamFbank) [18] layer and a Res2Dilated block with $\alpha$-feature map scaling and max pooling (AFMS-Res2MP). When the one-dimensional raw waveform is fed into RawNet3, it is transformed into a two-dimensional feature map by the ParamFbank layer. The ParamFbank learns real-valued parameterised filterbanks as an extension of the SincNet layer [17]. The feature map, processed through the ParamFbank layer, is then hierarchically passed through three AFMS-Res2MP blocks to capture speaker information at various levels. The AFMS-Res2MP is a block that replaces the squeeze-excitation layer in the squeeze-excitation block [26] with the $\alpha$-feature map scaling (AFMS) [12]. The extracted speaker information is finally processed into speaker embeddings using convolution, attentive statistics pooling (ASP) [27] with channel- and context-dependent attention values, and linear hierarchy.

We suggest that there is room for improvement in both the ParamFbank module and the AFMS-Res2MP block within RawNet3. Utilizing information from different time scales could allow them to be robust to different lengths of input speech.

Our proposed architecture enhances the existing RawNet3 to be robust against utterances of various lengths by applying both MRFE and MRA methods. Figure 1-(b) illustrates the overall structure of our proposed system. The MR-RawNet is a structure that replaces the first convolutional module and bottleneck blocks in the baseline with The MRFE and MRA. The raw waveform of length $T$ is fed to the MRFE module to generate a time-frequency representation. The MRFE consists of $N$ feature extractors (FEs) and combines the outputs of the FEs into a feature map $o_{1}\in\mathbb{R}^{F\times\frac{T}{S}}$, where $F$ and $S$ refer to the channel and stride size of the MRFE, respectively. The feature map $o_{1}$ is then fed into a convolutional layer for input into the bottleneck stages, and its output feature $o_{2}\in\mathbb{R}^{C\times\frac{T}{S}}$, where $C$ refer to the channel size of the MRA block, is digested to the bottleneck stage. Each bottleneck stage consists of $B$ MRA blocks, its outputs $o_{3},o_{4},o_{5}\in\mathbb{R}^{C\times\frac{T}{S}}$ are concatenated into a feature $o_{6}\in\mathbb{R}^{3C\times\frac{T}{S}}$, similarly to the ECAPA-TDNN. The concatenated feature $o_{6}$ is input into a convolutional layer with a kernel size of 1, followed by a ReLU function. Then, the output $o_{7}\in\mathbb{R}^{1536\times\frac{T}{S}}$ is aggregated into an utterance-level feature, and the speaker embedding is an output obtained by inputting the utterance-level features $o_{8}\in\mathbb{R}^{3072\times 1}$ into a linear layer.

Table 1 compares the performance of our proposed framework with recently proposed SV systems for short utterances across various lengths (1s, 2s, and 5s). Although RawNet3 did not provide the performance across various utterance lengths, we measured the performance of short utterances using the official model parameters of the RawNet3. RawNet3 demonstrated comparable performance to other SV systems for short utterances, even though it exhibited superior performance for full-length utterances. The proposed MR-RawNet showed outstanding performance in various utterances, not only against raw waveform models but also against other models. Compared to the RawNet3, MR-RawNet displayed a relative error reduction (RER) rate of 20.2% in 1-second utterances. These experimental results suggest that a framework capable of exploring information across various time scales can enhance robustness against variable lengths.

We conducted ablation experiments to validate the effects of the proposed methods as shown in Table 2. These experiments were performed based on the RawNet3 structure, which served as our baseline system. Experiments #1, #2, #3, #4 applied only MRFE to the baseline, showing varying performance based on the number of MRFEs ($N$) applied. Systems that utilized only one or two resolutions fell short when compared to the baseline. Yet, the application of three or more MRFEs surpassed the baseline performance. These outcomes suggest that the incorporation of information from various resolutions is beneficial for improving the system’s effectiveness with short utterances. Through the results of #4, we confirmed that the model could become more resilient to variable lengths by appropriately processing and utilizing information at various resolutions through different blocks. Experiments #5 and #6 apply MRA to the baseline, reflecting the alterations in MRA channels $C$ and blocks $B$. For the consistency of parameter quantities, we decreased the number of channels according to the increase in the number of MRA blocks. Both experiments achieved improved performance compared to the baseline, notably experiment #6, which set $C$ and $B$ to 256 and 3, respectively. Experiments #7 and #8 represent the combination of experiments #4 with #5 and #6, respectively, and experiment #8 outperforms other experiments, recorded an RER of 15.6% for 1-second utterances against the baseline. From these results, we believe that MR-RawNet was able to improve the performance for variable duration utterances further by encouraging the model to focus on complementary temporal context through the MRFE and MRA modules.

Table 3 provides a comparison of our system and the baseline model for out-of-domain utterances of various lengths. MR-RawNet evaluation was carried out employing System #8, which showed the best performance under the VoxCeleb1 test condition. Despite the reduction in the number of parameters compared to RawNet3, MR-RawNet demonstrated improved performance across various utterance durations in the out-of-domain dataset. Specifically, MR-RawNet presented an 13.4% performance improvement over RawNet3 for 1-second utterances, thus proving its effectiveness for short utterances and the generalizability of the system.

We proposed MR-RawNet, which is a novel speaker verification system that is robust to various duration utterances. Our system uses MRFE and MRA to improve the performance of variable duration utterances by focusing more on complementary temporal context. The results on VoxCeleb show that MR-RawNet outperformed other raw waveform-based systems, notably improving performance by 20.2% in 1-second test compared to RawNet3. Although this paper focused on comparison with the raw waveform-based models, future research could conduct comparison with models using a variety of input features.

This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No.RS-2023-00263037, Robust deepfake audio detection development against adversarial attacks)