Written Term Detection
Improves Spoken Term Detection

BeKWS—depicted in Figure 1(a)—is a model trained to predict the probabilities of a query occurring in each frame of a spoken document. For a (possibly multi-word) query $\boldsymbol{\mathbf{q}}=\left(q_{1},q_{2},\dots,q_{K}\right)$ comprising a sequence of $K$ letters and a document $\boldsymbol{\mathbf{X}}=\left(\boldsymbol{\mathbf{x}}_{1},\boldsymbol{\mathbf{x}}_{2},\dots,\boldsymbol{\mathbf{x}}_{N}\right)$ comprising a sequence of $N$ acoustic frames, the model is used to predict the sequence $\boldsymbol{\mathbf{y}}\left(\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}})=(y_{1},\dots,y_{N}\right)$, where each $y_{n}\in\left\{0,1\right\}$ is a binary random variable indicating the existence of the query, i.e:

The model comprises a document encoder and a query with parameters $\boldsymbol{\mathbf{\Delta}}$ and $\boldsymbol{\mathbf{\psi}}$ respectively. Given the query $\boldsymbol{\mathbf{q}}$ and the document $\boldsymbol{\mathbf{X}}$, the model outputs the sequence $\boldsymbol{\mathbf{z}}\left(\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}};\boldsymbol{\mathbf{\theta}}\right)=\left(z_{1},\dots,z_{{N}}\right)$ of query occurrence probabilities where:

where $\boldsymbol{\mathbf{\theta}}\coloneqq\left\{\boldsymbol{\mathbf{\Delta}},\boldsymbol{\mathbf{\psi}}\right\}$; $\boldsymbol{\mathbf{h}}_{{n}}$ is the ${n}$th frame of $\boldsymbol{\mathbf{H}}(\boldsymbol{\mathbf{X}};{\boldsymbol{\mathbf{\Delta}}})$, a down-sampled representation of $\boldsymbol{\mathbf{X}}$ computed by the document encoder; $\boldsymbol{\mathbf{e}}(\boldsymbol{\mathbf{q}};{\boldsymbol{\mathbf{\psi}}})$ is the vector representation of the query computed by the query encoder, and $\sigma(\cdot)$ is the logistic sigmoid function. Thus, $z_{{n}}\left(\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}};\boldsymbol{\mathbf{\theta}}\right)$ is interpreted as $P_{\boldsymbol{\mathbf{\theta}}}\left(y_{{n}}=1|\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}\right)$.

Given a training dataset containing of a set of spoken documents $\mathcal{X}$, the model is trained by stochastic gradient descent to minimize the negative log-likelihood of the indicators:

where the training queries are taken from $\mathcal{Q}$, the set of all unigrams, bigrams, trigrams in the transcripts of $\mathcal{X}$, and the word-level timestamps required for training are obtained by forced-alignment with an HMM-GMM-based ASR system trained on the KWS training data. In practice, the model is trained with a modified cross-entropy objective which was introduced in [7] (and which we will recap in Section III-C) because it has been shown to outperform the vanilla binary cross-entropy implied by (3).

The approach we propose for incorporating unpaired text into E2E KWS, JOSTER—depicted in Figure 1(b)—involves modifying the document encoder of BeKWS to accept not just acoustic inputs but also textual ones. To do so, we introduce a pair of modality encoders which transform input from their respective modalities into a shared space; a speech-only encoder which takes spoken documents as input, and a text-only encoder which takes written sentences as input. The output of either encoder can then be fed into a shared document encoder, and combined as in Equation 2 with the output of the (shared) query encoder to obtain probabilities of occurrence of the query in either spoken or written sentences.

For a spoken document, $\boldsymbol{\mathbf{W}}^{{\mathrm{audio}}}=\left(\boldsymbol{\mathbf{w}}_{1},\dots,\boldsymbol{\mathbf{w}}_{N}\right)$, we compute its representation $\boldsymbol{\mathbf{X}}^{{\mathrm{audio}}}\left(\boldsymbol{\mathbf{W}}^{{\mathrm{audio}}};\boldsymbol{\mathbf{\Delta}}_{{\mathrm{audio}}}\right)$ by passing it through an optional speech-only encoder with parameters $\boldsymbol{\mathbf{\Delta}}_{{\mathrm{audio}}}$. Henceforth, to reduce clutter, we drop the functional form $\boldsymbol{\mathbf{X}}^{\mathrm{audio}}\left(\boldsymbol{\mathbf{W}}^{{\mathrm{audio}}};\boldsymbol{\mathbf{\Delta}}_{{\mathrm{audio}}}\right)$, and simply write $\boldsymbol{\mathbf{X}}^{\mathrm{audio}}$ with the understanding that the dependency is implied. Note that, with the change of the model, we have had to make a slight change in notation: in Section III-A, $\boldsymbol{\mathbf{X}}$ denoted both the sequence of acoustic features and the document encoder input (since these are identical for BeKWS); here, $\boldsymbol{\mathbf{W}}^{\mathrm{audio}}$ denotes the sequence of acoustic features, while $\boldsymbol{\mathbf{X}}^{\mathrm{audio}}$ refers to the document encoder input which computed on $\boldsymbol{\mathbf{W}}^{\mathrm{audio}}$. For most of our experiments, we do not use a speech-only encoder at all. Rather, we use the text-only encoder to project written documents to the space of acoustic features, i.e., by default, $\boldsymbol{\mathbf{\Delta}}_{{\mathrm{audio}}}=\varnothing$ and $\boldsymbol{\mathbf{X}}^{{\mathrm{audio}}}=\boldsymbol{\mathbf{W}}^{{\mathrm{audio}}}$.

To compute the representation for a written document, $\boldsymbol{\mathbf{W}}^{{\mathrm{text}}}=\left({w}_{1},\dots,{w}_{N}\right)$, we first mask it to obtain $\tilde{\boldsymbol{\mathbf{W}}}^{{\mathrm{text}}}=\left(\tilde{w}_{1},\dots,\tilde{w}_{N}\right)$:

where _ is a special mask symbol. Then we incorporate a rudimentary duration model transforming the input to $\hat{\boldsymbol{\mathbf{W}}}^{{\mathrm{text}}}=(\hat{w}_{1},\dots,\hat{w}_{N})$ where each $\hat{w}_{n}$ is obtained by simply repeating $\tilde{w}_{n}$ $\rho$ times. For instance, the phrase $\boldsymbol{\mathbf{W}}^{{\mathrm{text}}}=\texttt{thecat}$ might be converted to $\tilde{\boldsymbol{\mathbf{W}}}^{{\mathrm{text}}}=\texttt{t\_ec\_t}$, and then, if $\rho=2$, to $\hat{\boldsymbol{\mathbf{W}}}^{{\mathrm{text}}}=\texttt{tt\_\,\_eecc\_\,\_tt}$. This final representation is then input into the text encoder—a neural network with an embedding lookup input layer—with parameters $\boldsymbol{\mathbf{\Delta}}_{{\mathrm{text}}}$ to obtain $\boldsymbol{\mathbf{X}}^{{\mathrm{text}}}\left(\boldsymbol{\mathbf{W}}^{{\mathrm{text}}};\boldsymbol{\mathbf{\Delta}}_{{\mathrm{text}}}\right)$. We determine the values of $\pi$ and $\rho$ experimentally (see Section IV-C for an analysis of their impact).

Having obtained the modality-specific representations, we can use Equation 2 to get the occurrence probabilities $P_{\boldsymbol{\mathbf{\theta}}}\left(y_{{n}}|\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}^{\mathrm{audio}}\right)$ or $P_{\boldsymbol{\mathbf{\theta}}}\left(y_{{n}}|\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}^{\mathrm{text}}\right)$ for any query $\boldsymbol{\mathbf{q}}$ where the parameters $\left\{\boldsymbol{\mathbf{\Delta}},\boldsymbol{\mathbf{\psi}}\right\}$ are shared by both the speech-text retrieval and the text-text retrieval.

As stated in Section III-A, for spoken documents, $\boldsymbol{\mathbf{y}}(\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}^{\mathrm{audio}})$ is defined by whether the query is spoken at a time span of the document. For written documents, $\boldsymbol{\mathbf{y}}\left(\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}^{\mathrm{text}}\right)$ is defined by whether the query occurs exactly at a given location. Using the example from above, with document sentence $\boldsymbol{\mathbf{W}}^{{\mathrm{text}}}=\texttt{thecat}$ and $\hat{\boldsymbol{\mathbf{W}}}^{{\mathrm{text}}}=\texttt{tt\_\,\_eecc\_\,\_tt}$, and a query $\boldsymbol{\mathbf{q}}=\texttt{cat}$,

We train the model jointly on a paired speech-text dataset, $\mathcal{X}^{{\mathrm{audio}}}$, and an unpaired text-only one, $\mathcal{X}^{{\mathrm{text}}}$, using stochastic gradient descent. At each training step, $k$, we sample the dataset $\mu$ uniformly from $\{{{\mathrm{audio}}},{{\mathrm{text}}}\}$ and minimize:

where $\left\{\boldsymbol{\mathbf{q}}^{\mu}_{k,1}\dots\boldsymbol{\mathbf{q}}^{\mu}_{k,L}\right\}$ is a mini-batch of $L$ queries sampled randomly from the set of unigrams, bigrams and trigrams of the dataset $\mathcal{X}^{\mu}$ ²²2Note that, as in the baseline, we consider multiple occurrences of the same training query to be distinct elements of the set, so that the probability of sampling a particular training query is directly proportional to the number of times it occurs in the training data.; $\left\{\boldsymbol{\mathbf{W}}^{\mu}_{k,l,1},\dots,\boldsymbol{\mathbf{W}}^{\mu}_{k,l,M}\right\}$ is a set of documents sampled from the dataset such that $\boldsymbol{\mathbf{W}}^{\mu}_{k,l,1}$ contains $\boldsymbol{\mathbf{q}}^{\mu}_{k,l}$ while the other $M-1$ documents are sampled randomly; $\boldsymbol{\mathbf{X}}^{\mu}(\cdot)$ is the output of the corresponding modality-specific encoder; $\boldsymbol{\mathbf{z}}(\cdot)$ is the model output as described by Equation 2; $\boldsymbol{\mathbf{y}}(\cdot)$ is the ground truth as described by Equations 1 and 5; and $f(\cdot)$ is the modified binary cross-entropy function defined as:

where $\phi$ is a hyper-parameter controlling the tolerance of the objective to easily-classified frames and $\lambda$ controls the relative weighting of positive to negative frames.

After the model is trained, we no longer require the text-only document encoder $\left(\boldsymbol{\mathbf{\Delta}}^{{\mathrm{text}}}\right)$, i.e., at search time, JOSTER becomes effectively identical to BeKWS. We post-process the output of the query and spoken document encoders for KWS using the procedure illustrated in Figure 2. As in BeKWS, for a given document, the query is detected if there exists “islands” of consecutive frames whose sigmoid outputs, $P_{\boldsymbol{\mathbf{\theta}}}\left(\boldsymbol{\mathbf{y}}|\boldsymbol{\mathbf{q}},\boldsymbol{\mathbf{X}}^{audio}\right)$, exceed some threshold. We set this threshold to 0.5 in all our experiments, although we found search performance to be stable for thresholds between 0.4 and 0.7. The first and last frame of the sequence are taken as the timestamps of the query hit and the median probability of the sequence is taken as the confidence. Finally, we discard hits which are shorter than 40$ms\times$query length in letters.

In this section, we compare the performance of JOSTER to BeKWS across languages and feature kinds. Table II shows the TWV for each of the five test languages. For the baseline (BeKWS), we note that replacing BNF as used in [11] with XLS-R features yields significant improvements across languages—on average, +10.2 MTWV on the dev sets and +9.3 ATWV on the eval sets. Furthermore, by pretraining the document encoder multilingually for KWS, using speed perturbation and increasing $M$ from 4 to 8 (in Equation III-C), the baseline performance is increased by an additional +6.8 dev MTWV and +7.6 eval ATWV on average across languages, showing that BeKWS with XLS-R features can be improved with multilingual KWS pretraining despite the XLS-R features being already multilingual. This tracks a similar finding about multilingual BNF in [11].

We find that JOSTER invariably improves the TWV by considerable margins compared to BeKWS in each setting (BNF, XLS-R, XLS-R + multilingual pretraining). For BNF, the improvements across languages average +4.7 for dev set MTWV and +4.4 for eval set ATWV. When using XLS-R features, the respective improvements increase slightly to +5 and +5.3. When finetuning the multilingually pretrained model with XLS-R features, we get average improvements of +3.7 and +3.5 by using JOSTER instead of BeKWS. Note that we use the same multilingual model—which is trained without unpaired text—to initialize the document encoders for both BeKWS and JOSTER.

The DET plots in Figure 3 provide an even more comprehensive picture of the performance difference. In each test language, JOSTER outperforms BeKWS across virtually all operating points of the plots; i.e., at any given recall rate, JOSTER incurs fewer false alarms than BeKWS, further strengthening the significance of the superiority of JOSTER.

As described in Section III-B, when computing the representation of written documents, we first mask with probability $\pi=0.3$ and repeat each token $\rho=2$ times. In this section, we quantify the significance of these choices on retrieval performance.

Figure 4 shows the MTWV as $\pi$ is varied with $\rho$ fixed to 2. We find that even without masking (at $\pi=0$), JOSTER already outperforms BeKWS by +3.5 MTWV. This runs counter to our original intuition that without masking, retrieval from written sentences would be too trivial to aid learning. Nevertheless, setting $\pi$ to $0.15$ further improves MTWV by an average of +1.9. Increasing $\pi$ further starts to worsen performance. We note that although MTWV varies with the masking rate, only at extreme values ($\pi>0.9$) does it get worse than the baseline, indicating that the joint training is robust across a large range of $\pi$. We surmise that having the text input is crucial, and the masking acts as extra regularization in the vein of dropout.

Figure 5 shows the performance of JOSTER as $\rho$ is varied with $\pi$ fixed to 0.3. JOSTER outperforms the baseline across all the settings of $\rho$ we tried. Although the average MTWV at $\rho=2$ is better than the MTWV at $\rho=1$, by 1.4, the latter may still be preferred as the computational cost of the text document pipeline increases linearly with $\rho$. Finally, we consider a more involved duration model (denoted $\bar{\rho}$ in the figure), where we set $\rho$ for each letter to be its average duration—estimated by forced-alignment with graphemic HMMs trained for each language. We find that this added complexity yields no TWV improvements. In fact, it generally degrades performance compared to fixed duration with $1\leq\rho\leq 4$.

Overall, we note that although both parameters can change the performance of the system, the variance is low enough that JOSTER still outperforms BeKWS over large ranges of either parameter.

In this section, we measure the impact of the number of negative examples in each training step. Instead of fixing $M=4$ for both paired and unpaired batches, we vary them in turn:

• •

  $M(\mathrm{audio})$: We set $M=4$ for unpaired batches and vary it between 2, 4 and 8 for paired batches.

• •

  $M(\mathrm{text})$: We set $M=4$ for paired batches and vary it between 2, 4 and 8 for unpaired batches.

Figure 6 shows the impact as of these variations. In both cases, we find that increasing $M$ increases the MTWV, with $M(\mathrm{audio})$ having higher impact compared to $M(\mathrm{text})$. This however comes at the cost of increased compute and memory cost for each training step. Note that in all our experiments, negative training utterances are sampled randomly. We hypothesize that better sampling of negatives could result in better training efficiency or even better search accuracy, but we leave investigation of any such sampling strategies to future work.

So far, we have fed the speech features directly into the shared encoder, i.e., there are no trainable speech-only encoder parameters. In this section, we reduce the number of shared layers. As we reduce the number of shared layers, we increase the number of speech-specific layers (including transferring any dropout or down-sampling components) so that the architecture and number of parameters used for the spoken document pipeline ($|\boldsymbol{\mathbf{\Delta}}|+|\boldsymbol{\mathbf{\Delta}}_{{\mathrm{audio}}}|$) do not change. For instance, when we remove two LSTM layers from the shared encoder, we use a two layer LSTM network with the same configuration as the speech-only encoder. We keep the text-only encoder configuration constant throughout.

Figure 7 shows that the MTWV generally improves as more layers are shared. It is particularly noteworthy that when no layers are shared ($\boldsymbol{\mathbf{\Delta}}=\varnothing$) by the two modalities’ document encoders (with the query encoder shared as always), the MTWV is almost identical to the baseline. This indicates that the performance improvements result from using the unpaired text to improve the (acoustic) representations learned by the document encoder rather than simply having more text data for training the query encoder.

In this section, we measure the impact as we change the amount of unpaired text used for the auxiliary task and report the results in Figure 8. First, we compare using the FLP text as has been done so far to using the LLP text, i.e., using the transcripts of the paired data as the “unpaired” text. The LLP text performs significantly worse than using the FLP text and, in three of the five languages, worse even than BeKWS.

Next, to test how much of this degradation is due to data size and how much of it results from using the same text, we create three random subsets of the FLP text (with the LLP text excluded) each with the same number of sentences as the LLP text. We report the MTWV of the best (S-max) and worst (S-min) performing of these splits for each language. We observe that even the best split performs much worse than the full FLP indicating the size of the augmentation text matters. However, we also observe that even the worst random split outperforms the LLP text, indicating that textual diversity is also crucial. Finally, we report a topline (Dev+Eval) where we use the text from the transcriptions of the Dev and Eval sets as the unpaired text for training, and find that, unsurprisingly, it outperforms using even the larger FLP text. While it is unrealistic to assume that the transcription of the test set can be obtained beforehand, this shows that further improvements can be obtained if it can be somewhat anticipated.

In this section, we quantify how much improvement we get on various queries depending on whether or not they exist in the unpaired text.

Figure 9 shows the ATWV difference between JOSTER and BeKWS across languages for queries that are:

1. 1.

   OO: Out of vocabulary of both the KWS training data and the unpaired FLP text

2. 2.

   OI: Out of vocabulary with respect to the KWS training data but in the FLP text vocabulary

3. 3.

   II: In vocabulary with respect to both KWS training data and the FLP text.

Note that for multi-word queries, out-of-vocabulary means at least one of the query words is out-of-vocabulary.

The worst average improvements over BeKWS (+0.84 on average ATWV) are achieved for OO queries (which form a minute proportion of all queries as shown in the OOV-F column of Table I), with performance even degrading for two of the five languages. For OI and II queries, we get consistent significant improvements (+5.7 and +5.3 average ATWV respective improvements).

Overall, we infer that JOSTER improves the document encoder’s representation of words which are in the augmentation text regardless of whether or not they actually exist in spoken form in the paired data.

So far, we have trained and tested exclusively with Babel data. In this section, we experiment with Turkish data from various domains with various configurations of paired and unpaired data. The objective for doing so is twofold:

1. 1.

   To what extent is the matching domain necessary? In other words, can we improve the TWV in one domain by using text from a different domain?

2. 2.

   Is text-only domain adaptation possible? Given paired training data in one domain and a test set in another domain, how much can we improve the test set performance using unpaired text from the target domain?

To answer these questions, we conduct experiments on two Turkish language datasets. In addition to the Turkish Babel dataset used in previous experiments, we use Turkish Broadcast News (BNTR) [52] for KWS training and testing.

To match the training data size of the Babel LLP corpus, we use a 10-hour subset of BNTR from the VOA channel¹¹¹¹11https://catalog.ldc.upenn.edu/LDC2012S06 for training. This training set has a vocabulary size of 16464. We select two 10-hour subsets from the remaining BNTR data as dev and eval sets. Since the BNTR dataset has no official keyword lists, we randomly select 1500 queries composed of equal proportions of unigrams, bigrams and trigrams for each of the dev and eval sets with OOV rates of 11.7% and 6.5% respectively. We experiment with three text corpora for unpaired training: Babel FLP text, BNTR—unpaired text from the Broadcast News dataset totalling around 180 hours and text from Turkish Wikipedia. The first two allow us to measure the impact of using text from the test domains, while Wikipedia stands as a control corpus.

Table III shows the results of training with various configurations of paired and unpaired data on the different test sets. First, we note that the BNTR results are generally better than the Babel ones, which is to be expected as the latter contains conversational speech from a telephone channel, while the former contains news recordings of professional newscasters.

For each test set, we get improvements by using JOSTER regardless of the unpaired text used for joint training. However, we get the largest improvements when we use text from the test domain to augment training. For instance, in the cross-domain setting where we train with the paired Babel data and test on BNTR, JOSTER with the Babel FLP text improves the dev MTWV and eval ATWV by +8.3 and +8.0 respectively compared to BeKWS. Augmenting with Wikipedia text results in further +4.4 and +3.5 dev and eval improvements compared to using the Babel FLP text. Finally, using BNTR (target domain) unpaired text provides further improvements of +6.1 and +6.5 compared to Wikipedia. This final result cuts the gap to a topline of using BNTR data for training by 65% and 60% on the Dev and Eval sets respectively.

In the converse cross-domain setting (training with BNTR paired data and testing on Babel), we observe a similar trend where JOSTER using Wikipedia improves on the performance of BeKWS, with further performance gains obtained from using BNTR text, and the best performance resulting from using Babel text. We note, however, that the performance improvements are not as dramatic in this case—likely due to the difficulty of transferring the BNTR-trained model to the difficult acoustic conditions of the Babel data.

Finally, when training and testing within the same domain, we observe that JOSTER generally improves the TWV compared to BeKWS even with unpaired text from other domains. This holds even for BNTR which already has a high baseline performance.

Overall, these results add an extra dimension to the results so far, showing that the proposed method performs well, not just across languages—as shown in previous sections—but also across domains within the same language. Furthermore, they suggest that training JOSTER with unpaired text from a domain most improves search performance on test sets in that domain, providing a good alternative when domain-specific data is limited.

In this section, we compare our proposed method with TTS-based unpaired text integration, where we use an off-the-shelf TTS model to synthesize speech for the unpaired text and train with the resulting data. Here, we experiment with English language corpora due the difficulty of obtaining high-quality open-source TTS systems for other languages. Specifically, we use the 10-hour Libri-light corpus [53] as the paired KWS training data, and test on the standard Librispeech test splits [54] with around 1300 randomly generated queries for each test split. We use the Coqui xTTS system [55] for speech synthesis. We use the 100-hour Librispeech training set as the unpaired data for JOSTER and TTS. For JOSTER, we also consider training with Wikipedia text.

Table IV shows the results of these experiments. JOSTER, even with Wikipedia text, improves across all dev and test sets, and further, although the best performance is achieved when the in-domain Librispeech-100 text is used. Similarly, using TTS for data augmentation significantly improves KWS compared to BeKWS.

Compared to JOSTER, we note that TTS performs better on the “clean” test sets and performs worse on the more acoustically-challenging “other” sets. This highlights a difference between the two approaches. JOSTER, being text-based, is more channel-agnostic and is more influenced by linguistic similarities between the unpaired text and the target. TTS, on the other hand, is also influenced by channel match between the output of TTS (which is typically clean speech by design) and the test audio. Although, the impact of TTS augmentation on KWS could plausibly be improved by augmentation with artificially generated noise, reverberations or room impulse responses, an in-depth exploration of TTS-based augmentation is out of the scope of this paper. Moreover, these add extra complications which JOSTER does not have.