Systematic Evaluation of Online Speaker Diarization Systems Regarding their Latency

The classic speaker diarization pipeline usually consists of the three sub tasks: Speech Activity Detection (SAD), Segmentation and Clustering. SAD checks whether the incoming audio segment contains speech. During segmentation, the audio segments are cut so that they only contain one speaker. In clustering, the audio segments are then assigned to known or new speakers [1].

In the past, each sub task was solved by a separate model. By using deep learning methods, several sub tasks or even the entire pipeline can now be automated by a single neural network. Systems that automate the entire pipeline with one model are referred to as end-to-end systems [1].

In 2021, Coria et al. [3] developed an online speaker diarization system that combines an end-to-end approach with a modular approach. The advantage of end-to-end systems is that they can automatically deal with speaker overlaps through supervised training. The disadvantage of end-to-end approaches is that the maximum number of speakers must be known in advance. This is difficult with online speaker diarization, as new speakers can always be added to an infinite input stream. Coria et al. therefore apply the end-to-end approach to individual audio chunks with a limited number of speakers, thereby generating local speaker labels. Subsequently, an incremental clustering algorithm maps the local speaker labels to cross-chunk global speaker labels.

The module for the local speaker labels receives a 5s rolling audio buffer as input. Diarization is performed at intervals of the parameter step_size (e.g. 250ms). The outputs are active speaker propabilities for each frame. These are passed on to the subsequent clustering module.

The clustering module generates a speaker embedding from the active frames for each speaker in the rolling buffer. Constrained incremental clustering then assigns the local speaker embeddings to the global speaker embeddings. If the distance between a local speaker embedding and the existing global speaker embeddings is too great, a new global speaker embedding is created from the local speaker embedding. The global speaker embeddings are then updated with the local speaker embeddings.

The online diarization system by Fini et. al. [6] is an adapted version of the original UIS-RNN [7]. UIS-RNN is a supervised approach for the clustering component of a speaker diarization pipeline. Fini et. al. have developed an adapted loss function and a formula for calculating the parameter $\alpha$ for the original UIS-RNN algorithm. In the original loss function of the UIS-RNN algorithm, the error is calculated using the Mean Square Error (MSE) from the following components:

• •

  Average of the outputs of the gated recurrent unit (GRU) instance

• •

  Embedding of the next audio sequence

In this case, the network only learns a connection between the current observation and the next audio sequence. With the adapted loss function, on the other hand, the MSE is composed as follows:

• •

  Average of the outputs of the gated recurrent unit (GRU) instance

• •

  Average of an unseen collection of embeddings of the current speaker

The adapted loss function allows the network to establish a connection between the current observation and the current speaker. This leads to faster convergence, better minima and better generalization. This is also reflected in the measurement results of Fini et. al [6].

The $\alpha$ parameter of the UIS-RNN algorithm represents the probability that a new speaker will enter the conversation. If the $\alpha$ is too high, the algorithm overestimates the number of speakers. If $\alpha$ is too low, the number of speakers is underestimated. In the original UIS-RNN, a constant value is used that is specified by the user. In the adapted version of the UIS-RNN, Fini et. al. have developed a formula that determines $\alpha$ using the ratio of the count of speaker labels and the count of speaker changes. However, the calculation of $\alpha$ only leads to a slight improvement in performance [6].

Frame-wise streaming end-to-end diarization (FS-EEND) is an end-to-end online diarization system introduced by Liang et. al. in 2023 [4]. The system receives log-mel audio features as input, which consist of one or more audio frames. These are converted into audio embeddings by an embedding encoder. The attractor decoder generates attractors from the audio embeddings. An attractor is the representation of a speaker. Each speaker embedding is then assigned the corresponding attractor and therefore the corresponding speaker label via a similarity comparison. Both the embedding encoder and the attractor decoder use self-attention in order to achieve better diarization results and to avoid losing information about past frames.

The aim of this work is to analyze which online diarization system has the lowest latency values from audio input to speaker label output on a uniform hardware. Therefore a structured experiment is carried out. For a uniform dataset, the latency is measured for different combinations of segmentation and embedding models of the DIART framework. In addition, the measurement is carried out for the online diarization systems UIS-RNN-SML and FS-EEND. The results are then classified and discussed.

There are several different pretrained segmentation and embedding models for the DIART framework. However, there are no pretrained models for the online diarization systems UIS-RNN-SML and FS-EEND. Corresponding models must therefore be trained for these systems. In this paper the TIMIT dataset [8] is used for this purpose. For comparable accuracy results as in the original papers, there is not enough training data available in this research work. Therefore, only the latency is reported in this paper. For the accuracy and Diarziation Errors (DER) of the systems, this work refers to the corresponding original paper.

A subset of the Voxconverse testset [9] is used in this work as a testset for measuring latency. The subset consists of the first four audio files aepyx.wav, aggyz.wav, aiqwk.wav and auzuru.wav. These contain a total of around 20 minutes of audio recordings.

The DIART framework can directly process the audio files in wav format and therefore requires no further preprocessing. The speaker labels can also be provided in rttm format to measure the DER.

The implementation of the UIS-RNN-SML, on the other hand, only contains the clustering component and requires audio embeddings for clustering as input [6]. Suitable audio embeddings in the form of d-vectors [10] can be generated with the audio embedding system from Harry Volek [11].

As described above, FS-EEND is an end-to-end diarization system that can work directly with log-mel audio features. However, the corresponding implementation by Liang et al. [12] is currently structured in such a way that it expects the input data in the Kaldi style. Kaldi is a toolkit for speech recognition [13], which also implements its own format for data preparation. For the conversion of the TIMIT dataset into the Kaldi style, the toolkit provides corresponding scripts that can be used with some adaptations. For the conversion of the test data from the Voxconverse dataset to Kaldi style separate scripts had to be created.

All combinations of the following embedding and segmentation models are tested for the DIART framework.
Embedding models:

• •

  pyannote/embedding

• •

  pyannote/wespeaker-voxceleb-resnet34-LM

• •

  speechbrain/spkrec-ecapa-voxceleb

• •

  speechbrain/spkrec-xvect-voxceleb

Segmentation models:

• •

  pyannote/segmentation

• •

  pyannote/segmentation@Interspeech2021

• •

  pyannote/segmentation-3.0

These are the models with the lowest latency in the measurement by Coria et al. [3].

The UIS-RNN-SML implementation [6] only contains the online clustering algorithm based on an recurrent neural network (RNN) architecture. For a complete speaker diarization pipeline, the clustering module must be preceded by a SAD and segmentation module. The implementation by Harry Volek [11] is used for this purpose. This implementation is based on a Long Short-Term Memory (LSTM) model for generating d-vectors as audio embeddings.

The last machine learning algorithm used in this work is the end-to-end diarization system FS-EEND [4]. The system is based on an encoder-decoder architecture with self-attention modules.

As described above, there are no corresponding pretrained models for the UIS-RNN-SML and FS-EEND systems. Therefore, models are trained with the TIMIT dataset for this work.

For the UIS-RNN-SML system, both a model for generating the d-vectors and a model the UIS-RNN-SML algorithm must be trained. The implementation by Harry Volek [11] provides corresponding scripts for preprocessing and training with the TIMIT dataset. The training is carried out with the provided default training configuration. The configuration suggests 950 epochs and a learning rate of 0.01 for an LSTM with 768 hidden layers. The resulting model can then be used to create d-vectors for the TIMIT dataset. The d-vectors are then used to train the UIS-RNN-SML. There are also suggested default values for the training of the UIS-RNN-SML. These are 10 epochs, 20,000 train iterations and a learning rate of 0.003 for an RNN with a hidden size of 512.

FS-EEND expects the input data in Kaldi style. The Kaldi toolkit provides scripts to convert the TIMIT dataset into the Kaldi style. Only functions for creating the reco2dur file and the segements file need to be added here. The training is carried out for 100 epochs, 100,000 warm-up steps and a learning rate of 1 for a Transformer model with 4 encoding and 2 decoding layers.

The trained models do not have the same quality as the models of the original paper. However, the quality of the models is irrelevant for a pure latency measurement.

To enable a comparable evaluation of the different systems, 250ms chunks are transferred to each system. The evaluation is carried out on an Intel(R) Xeon(R) Gold 5215 CPU @ 2.50GHz. The Python library time is then used to measure the following time span:

The function time.perf_counter is used for the measurement. The function uses a performance counter with the highest possible resolution on the test system for the time measurement [14].

The evaluation results of the DIART framework models are shown in table I. The table shows the measurement results for the best 5 model combinations. It can be seen that the lowest latency is achieved with the combination pyannote/segmentation and pyannote/embedding. The mean latency for this model combination is 0.057065s. It is also noticeable that the embedding models are primarily decisive for the latency.

The results of the evaluation of the UIS-RNN-SML are shown in table II. The table shows the measurement results for the first three and the last three chunks as well as the corresponding mean values. Some steps, e.g. 0.25s, are missing. This is due to the fact that the SAD component sorts out all non-speech segments directly [7]. The latency for creating the d-vectors is a constant value. However, the latency for clustering a chunk shows a constantly increases proportional to the length of the audio stream.

The results for the latency evaluation of the FS-EEND system can be seen in table III. The latency per chunk is independent of the total length of the audio file. The mean latency is 0.058293s, which is similar to the best system of the DIART framework.

Table I shows that pipelines with the embedding model pyannote/embedding have the lowest latency. Pipelines with the embedding model pyannote/wespeaker-voxceleb-resnet34-LM are in second place and those with the model speechbrain/spkrec-ecapa-voxceleb are in third place. The reason for this is the size and depth of the embedding models. The pyannote/embedding model is based on a canonical x-vector TDNN-based architecture with approximately 4.3 million parameters [15]. Pyannote/wespeaker-voxceleb-resnet34-LM is based on ResNet and already has 6.6 million parameters [16]. Speechbrain/spkrec-ecapa-voxceleb has 22.1 million parameters and is based on an ECAPA-TDNN model [17].

The segmentation models have a much smaller impact on the overall latency, as they are smaller models. The segmentation models are built from a combination of convolutional and LSTM layers with approximately 1.4 million parameters [18]. Since the DIART framework works with a rolling buffer, the latency per chunk remains constant regardless of the length of the audio stream. However, as described in II-B, incremental clustering is performed for the assignment of local and global speaker lables. Thus, in theory, the latency must increase with the number of speakers. However, this correlation could not be observed with the used test setup.

In the results of the UIS-RNN-SML system, it is noticeable that the latency per chunk increases with the length of the audio stream. The reason for this is that UIS-RNN performs a beam search on the label space for internal decoding. For each new chunk a further beam state is added to the beam search. This increases the decoding time per chunk linearly $O(t)$ (figure 1). However, the beam search must be performed once for each chunk. This results in a total runtime of $O(t^{2})$ (figure 2). Due to the increasing latency, UIS-RNN-SML is not suitable for longer audio recordings or audio streams.

With an average latency of 0.058s, FS-EEND achieved similarly good results in the evaluation as the best pipeline of the DIART framework. The latency remains constant regardless of the length of the audio stream. The self-attention modules of the FS-EEND are used for the correct assignment of already known speakers. Although these modules have a longer memory than for example a LSTM, self-attention modules do not memorize already known speakers forever [4]. FS-EEND can therefore not be recommended without restriction for very long audio streams. In addition, FS-EEND is a pure end-to-end diarization system. For such systems, the maximum number of speakers must be known in advance in order to produce good results [1]. This is only practicable to a limited extent for an online diarization scenario as described in II-B.