TokenVerse: Unifying Speech and NLP Tasks via Transducer-based ASR

Automatic Speech Recognition  We train our XLSR-Transducer model after removing all task tokens from the multitask dataset. This serves as a baseline for comparison with the multitask models on the ASR task. Evaluation  It is evaluated with WER. For TokenVerse models, we remove task tokens from both the reference and hypothesis to compute WER for a fair comparison. We also report WER including task tokens, which reflects its prediction errors.

Named-Entity Recognition  We finetune pretrained BERT²²2https://huggingface.co/google-bert/bert-base-uncased [32] model on our datasets for subword-level NER classification. We evaluate the models on both reference and hypothesis from the ASR model. Evaluation  NER systems are usually evaluated by comparing their outputs against human annotations, either using an exact-match or soft-match approach [6]. We adapted these metrics to a scenario where the text comes from an ASR system. Exact-Match: Let $P=\{P_{1},P_{2},\ldots,P_{n}\}$ be the set of predicted entities, and $A=\{A_{1},A_{2},\ldots,A_{n}\}$ be the set of actual entities, where each $P_{i}$ and $A_{i}$ is accompanied by its corresponding [NE]-[/NE] tokens (See Fig.1). Thus, an entity $P_{i}$ is considered correctly identified if and only if: $\forall i\in\{1,2,\ldots,n\},P_{i}=A_{i}$, including the tokens. Soft-Match: in this case we only count for the paired sets of [NE]-[/NE] tokens without considering if the predicted entity value $P_{i}$ was correctly transcribed. After obtaining each pair, we evaluate NER with F1-score.

Speaker Change Detection  For the SCD baseline, we utilize the diarization pipeline³³3huggingface.co/pyannote/speaker-diarization-3.1 from PyAnnote [33] to extract speaker change timestamps from the audio. In literature, the SCD is predominantly regarded as a task within the audio domain [26], we opt not to establish an independent text-based baseline for this task. Evaluation  We evaluate SCD in two ways: text-based (only valid for TokenVerse) and time-based. In text-based evaluation, we align the reference and hypothesis using edit-distance. For each occurrence of the [SCD] token in the reference, matching with the same token in the hypothesis counts as True Positive; else, False Negative. Unmatched tokens in the hypothesis are considered False Positive. F1 score is calculated by standard definitions. In time-based evaluation, we obtain the timestamps where [SCD] tokens are predicted in the hypothesis. We calculate F1 score [13], using a collar of 250ms during timestamp matching, following common practice in speaker diarization literature [5]. Additionally, segment coverage, purity [26], and their F1 score are also reported. We use pyannote.metrics [34] to compute all time-based metrics.

Endpointing  Considering semantic endpointing, we fine-tune BERT [32] for [ENDP] token classification on the multitask training text, termed as BERT-ENDP. Results are reported on both reference text and hypothesis text obtained from TokenVerse. From the audio perspective, we use segmentation pipeline⁴⁴4huggingface.co/pyannote/segmentation-3.0 from PyAnnote to obtain endpoint timestamps. Evaluation  Endpointing is also evaluated in two ways: text-based and time-based. The text-based evaluation follows the same approach as described previously for SCD. In the time-based evaluation, the F1 score computation also follows the same approach as for SCD. Additionally, we also report false alarms (FA), missed speech (MS), and detection error rate (DER), which are common metrics in endpointing literature [3].

Automatic Speech Recognition  For the DefinedAI (Tab. 2) set, WERs are reported both with and without task tokens in the reference and hypothesis for multitask models. However, the baseline ASR model is trained without task tokens in transcripts, so there is no distinction between them. Including all tasks in TokenVerse (exp 2) leads to a 4% relative improvement in WER compared to the baseline ASR model (exp 1). For models trained on a single task (exp 3a-c), ASR results remain similar except for SCD. When comparing WERs before and after token removal, we observe a relatively large gap between all-tasks and single-task models, potentially due to higher token insertion or deletion as compared to non-token words in the hypothesis. In single-task models, a larger gap is observed for [NE] as the model must accurately predict both tokens, introducing additional error sources. On the CallHome dataset (Tab. 5), the multitask model with all tokens yields a 7.7% relative improvement. Overall, the results on both datasets indicate that the all-tasks TokenVerse improves ASR performance.

Named-Entity Recognition  As expected, compared to evaluating BERT-NER on reference text, a significant degradation is observed when evaluated on hypothesis (Tab. 3) due to ASR errors [9]. In exact-match, on both the DefinedAI (Tab. 3) and CallHome (Tab. 5) test sets, the all-tasks TokenVerse outperforms the baseline BERT-NER models trained on their respective datasets and evaluated on hypothesis in F1 score. This is not the case for soft-match evaluation on the DefinedAI test set, where the F1 score is similar. This degradation is mostly attributed to the incorrect prediction of [/NE] tag by the baseline, resulting in only a partial match of the named entity words. The absolute F1 score is low on the CallHome dataset due to higher ASR errors on named entities, attributed to their low repetition in the training text (see Tab. 1).

Speaker Change Detection  On the DefinedAI (Tab. 4), including all tasks in TokenVerse outperforms the baseline PyAnnote model in time-based evaluations. Interestingly, models trained for single-task SCD perform better than the all-tasks model in terms of F1, but show similar results for Coverage-Purity based F1. Upon closer scrutiny, we found that including [ENDP] delays the prediction for [SCD] tokens, causing the hypothesis timestamps of these tokens to fall outside the tolerance window (250ms). Increasing the tolerance window further improves the F1 for both models, with a much higher rate of increase for the all-tasks model. This observation is reinforced in the text-based F1 score, where the all-tasks model achieves an F1 score of 90.3% compared to 88.5% from the single-[SCD] model. On the CallHome test set (Tab. 5), the all-tasks model outperforms the PyAnnote baseline. These evaluations suggest that excluding [SCD] from TokenVerse is preferable for precise speaker change timestamps, while including all tasks improves speaker-attributed text segmentation.

Endpointing  In text-based evaluation on the DefinedAI (Tab. 3) and CallHome (Tab. 5) test sets, the all-tasks TokenVerse outperforms the BERT-ENDP models trained on respective datasets. Additionally, on the DefinedAI dataset, we evaluate the BERT-ENDP model on both reference and hypothesis to understand the effect of ASR errors on [ENDP] token prediction. Interestingly, we do not observe a significant degradation when evaluating on the hypothesis compared to the reference. This suggests that errors introduced by ASR may not drastically affect the semantic meaning of the sentences. In time-based evaluation on the DefinedAI test set (Tab 4), the all-tasks model outperforms the baseline PyAnnote segmentation model. However, single-task ENDP is better than including all tasks in DER due to lower false alarms.