Romanization Encoding for Multilingual ASR

Our model structure is illustrated in Figure 1, where we take a ZH-EN CS as an example. In this work, we employ the Fast Conformer model [30] combined with Recurrent Neural Network Transducer (RNNT) technology [31] as our baseline. The Fast Conformer is an optimized version of the original Conformer model. It features a new downsampling schema that significantly reduces computational requirements by approximately 2.9 times and enhances inference speed, while maintaining or even improving performance across various Speech and NLP tasks. RNNT excels in capturing sequence knowledge and is popular in both monolingual [32] and CS ASR [25] tasks. However, despite RNNT’s strengths, it tends to struggle with script-heavy languages that have large vocabularies, as it can severely limit batch sizes and slow down training [33].

The baseline, highlighted by a dashed rectangle inputs ZH characters and EN words. In contrast, our method replaces characters with Romanized text (Pinyin) before processing through a concatenated tokenizer for text representations (detailed in Section 3.2). Meanwhile the audio representation is learned by FastConformer encoders. We introduce another decoder that transcribes the romanized text to characters (described in Section 3.3).

Following the concatenated tokenizer approach introduced by [34], we leverage pre-trained monolingual tokenizers to construct a combined tokenizer. This setup ensures separate label spaces for each language, enabling good generalization capabilities. For instance, English tokens are assigned indices ranging from [0,1023], while Mandarin tokens are allocated indices starting from 1024 up to 1024 + vocab size. The aggregated approach, while achieving similar performance levels to the non-aggregated method, offers the added advantage of facilitating LID. It does so by providing pseudo language information during training, thereby enabling the model to learn LID representations internally.

We prepare 1,024 English BPE sub-word units using the LibriSpeech (LS) dataset [35] and roughly 5,000 Chinese characters from the AISHELL-2 (AS2) dataset [36]. These Chinese characters were then romanized into Latin characters referred to here as ‘Roman’ encoding for all languages using the PyPinyin³ toolkit. As seen in Table 1, this romanization process reduced the Chinese vocabulary size from 5,178 to 1,239, enabling us to create a balanced concatenated tokenizer for Mandarin and English. For Korean and Japanese, similar processing methods were employed using the kroman⁴ and pykakasi⁵ toolkit, respectively, to achieve comparable reductions in vocabulary size and to facilitate a unified approach to tokenizer construction across multiple languages.

UTF8mj

To translate the romanized units Roman back to original characters, we introduce an additional module in our system called the Roman to Character (R2C) decoder. This Transformer-based model is designed to learn the multi-to-multi mappings between romanized text and original characters. For English, a language that already uses the Latin alphabet, the inputs and outputs remain unchanged as shown in (d) of in Table 1.

Importantly, despite the E2E training approach, the R2C decoder functions independently of the RNNT encoder outputs, focusing solely on learning the sequence-to-sequence mapping. Only its loss is merged with the RNNT loss, allowing for the possibility of separate training with text data or integration with more advanced pre-trained translation models, such as LLMs. This flexibility also enables the module’s extension to other languages with complex scripts, like Korean and Japanese. During training, accurately labeled Roman sequences serve as inputs for the module. To streamline the process, the decoding stage exclusively uses the greedy search hypothesis from the RNNT decoder as input, simplifying the overall pipeline.

Code-switching data SEAME is a publicly available dataset designed for Mandarin-English speech recognition, containing Mandarin, English, and natural intra-sentential code-switching data from interviews and conversations. The dataset specifics, including the duration of each data type, are outlined in Table 2. “ZH” refers to the monolingual Mandarin segments, and “EN” to the monolingual English parts. The dataset includes approximately 60 hours of natural CS data, a quantity considered limited for training robust models. Notably, most of SEAME speakers are from Singapore and Malaysia, presenting accents different from Mainland China. For the purposes of model selection, 10% of the training set samples are randomly chosen to form a validation set.

Monolingual data AISHELL-2 is an extensive 1000-hour open-source Mandarin speech corpus, the speakers of which mainly are from Mainland China. LibriSpeech comprises 960 hours of English speech from native speakers. These two monolingual datasets are used in our experiments to enhance the performance of code-switching ASR systems.

Evaluation data For evaluation, we stick to the data division of SEAME established by [37], which includes a test set for Mandarin speech named test_man and another tailored to Southeast Asian accented English, labeled test_sge. The specific durations of these test sets are also detailed in Table 2. Additionally, to assess performance on monolingual data, test sets from AISHELL-2 (as2_test) and LibriSpeech (ls_clean) are utilized in our analysis.

To evaluate monolingual test sets, Character Error Rate (CER) is applied to Mandarin, while Word Error Rate (WER) is used for English. For CS test sets, we employ Mixed Error Rate (MER), which incorporates word-level measurements for English and character-level assessments for Mandarin.

In all of our experiments, we use the Adam [38] optimizer combined with a Cosine Annealing learning rate scheduler, including a warm-up phase of 10,000 steps. The learning rate is set to peak at 1.5e-3 and then decrease to a minimum of 1e-6. In addition, we incorporate SpecAug [39] during training process to enhance model robustness and performance. Model averaging is employed, and during evaluation, greedy search is utilized without the assistance of any external LM or re-scoring techniques. The detailed training recipe will be open-sourced in NeMo.

Performance of training solely on SEAME dataset is detailed in Table 3, showing that romanization encoding yields improved results for both the test_man and test_sge sets. In Table 4, we integrate monolingual datasets utilizing their rich acoustic and linguistic content to boost multilingual and CS ASR. The proposed Roman-based method outperforms the Char-based model in both of the test sets of SEAME, with 10.77% and 9.43% MER reductions respectively. Further analysis by dividing the CS test sets into Mandarin and English segments underscores the advantages of romanization encoding for both languages.

Moreover, to thoroughly assess the model’s proficiency in handling both CS and monolingual scenarios, results for monolingual test sets are presented, indicating an improvement in performance on the monolingual Mandarin test set, albeit with a slight decline on the monolingual English test set. To balance monolingual and code-switching (CS) data within a fixed 2085-hour training data, we upscale CS data to 285 hours and reduce both AISHELL-2 (AS2) and LibriSpeech (LS) data to 900 hours each. This adjustment result in significant MER reductions of 13.72% and 15.03% in CS test sets, demonstrating improvements over baseline models. Performance variations in monolingual sets were observed, largely due to the differing accents and speaking styles of speakers such as speakers from Singapore and Malaysia versus those from Mainland China [29]. Nevertheless, the model effectively retains its capability to process monolingual information, often delivering equal or superior performance.

The proposed romanization encoding approach is designed to be easily adaptable to various languages. This section demonstrates the effectiveness of our method with Korean and Japanese, which face challenges such as limited publicly available corpora, insufficient for training large ASR models, and lack of CS data. Through training a multilingual model on constraint datasets (less than 50 hours), we demonstrate our approach’s capability and efficiency in data-limited situations and its potential extension to other languages.

Mandarin-Korean ⁶⁶footnotetext: https://github.com/goodatlas/zeroth A Mandarin-Korean bilingual ASR model was trained using the entire 50-hour Zeroth Korean⁶ dataset and 50 hours data randomly selected from the AISHELL-1 dataset [40]. Evaluations on monolingual Mandarin (test_as1) and Korean (test_zeroth) test sets utilized CERs for performance measurement. Results in Table 5 indicate that the Roman-based bilingual ASR model maintains performance on the Korean test set while achieving better results on the AISHELL-1 test set compared to a character-based model.

Mandarin-Japanese We also experiment on Mandarin-Japanese Bilingual ASR, drawing training data randomly from 50 hours of the AISHELL-1 dataset and 50 hours from the Japanese ReazonSpeech [41] dataset. By using romanization encoding, the concatenated vocabulary size is reduced from 8,507 to 2,298. As we can see in Table  6, when evaluated on AISHELL1 (test_as1) and ReazonSpeech (test_reazon) test sets, Roman-based model can perform better than Character encoding one in a large margin, which further indicates effeteness and scalability of our proposed Romanization encoding for other script-heavy languages.

Evaluations of R2C module Our proposed method includes additional R2C module to transcribe Roman to characters. The total training parameters are at par with the baseline system since the size of RNNT outputs is decreased. Although the end-to-end inference speed for the proposed system can not be faster than the character encoding models but the training batch sizes can be set larger, which is essential for the Multilingual ASR model training. For instance, the reduction in the RNNT concatenated vocabulary size in the Mandarin-Korean Bilingual ASR model from 6,380 to 2,441, primarily due to Mandarin (as Korean mapping is nearly one-to-one), allowed for at least a 2X larger training batch size and more than 20% quicker RNNT inference compared to models using character encoding. We believe that this work could benefit not only the RNNT-based model but also the popular auto-regressive speech large foundational models.