Low-Resourced Speech Recognition for Iu Mien Language via Weakly-Supervised Phoneme-based Multilingual Pre-training

The Yao ethnic group is one of the ethnic minorities in China, with a population of over 3 million people, mainly distributed in Guangxi, Guangdong, Hunan, Guizhou, Yunnan and other regions. It is the most widely distributed ethnic minority in southern China, and also exists in Southeast Asian countries such as Vietnam, Thailand, and Cambodia. Yao is the language of the Yao ethnic group, but it lacks a completely unified standardization and consists of many dialects. Among these, Iu Mien is a major dialect and serves as the basis for the current official standard Yao language. Iu Mien belongs to the Yao branch of the Iu Mien-Yao sub-family within the Sino-Tibetan language family. Its written form is a phonetic syllabic system using basic Latin letters.

With the development of deep learning based artificial intelligence, the capabilities of automatic speech recognition (ASR) technology have also developed rapidly. However, current speech recognition technology often requires hundreds to thousands of hours of training data to obtain good performances. Due to the long-term scattered residence of the Yao population and the lack of an authoritative central dialect, the speech and language resources of the Iu Mien language are very scarce. In our practice, it takes non-trivial efforts to collect and transcribe even less than 10 hours of Iu Mien language. The development of Iu Mien language speech recognition systems is very challenging, while it is very important to reduce digital divides and culture inheritance.

The paradigm of pre-training (PT) followed by fine-tuning (FT), called the PTFT paradigm, has emerged in recent years as an effective way to solve the problem of limited training data for low-resource languages for ASR. In pre-training, training data for a number of languages are merged to train a multilingual model. The pre-trained model can then serve as a backbone, which can be further fine-tuned for crosslingual speech recognition. This PTFT paradigm helps the backbone to learn common knowledge for speech recognition from multiple, different languages during pre-training. So when fine-tuning the backbone model on the target language, decent ASR performance could be obtained with only a small amount of target language training data.

Currently, there are three main multilingual pre-training methods: self-supervised pre-training, subword-based supervised pre-training, and phoneme-based supervised pre-training. Self-supervised pre-training uses unlabeled audio data to train models to learn common representations of multilingual audio. Subword-based pre-training combines multilingual annotated audio data by creating a shared token set for multiple languages. Phoneme-based supervised pre-training typically uses International Phonetic Alphabet (IPA) as a common pronunciation annotation system, enabling effective multilingual supervised pre-training. The IPA is designed to provide a unified system of symbols to represent the basic sounds of different languages. Each IPA symbol corresponds to a specific phoneme, ensuring a one-to-one relationship between the symbol and the sound. With the IPA, it is possible to achieve consistent phonetic transcription across all languages [1]. Comparing the three pre-training methods, the phoneme-based approach not only allows for efficient model training but also maximizes the sharing of pronunciation features across different languages. However, obtaining high-quality IPA pronunciation annotations for many minor languages is challenging.

Our work is based on a recent study on weakly supervised phoneme pretraining, Whistle [2]. In [2], the effects of different pre-training methods on multilingual and cross-language speech recognition are analyzed and compared. A weakly phonetic supervision method (Whistle) is proposed. It is shown in [2] that phoneme-based supervised pre-training models are more data-efficient, revealing that they perform better in cross-language speech recognition with limited training data. Additionally, related pre-training models are publicly released, which we utilize to construct a Iu Mien language speech recognition model in this work.

To the best of our knowledge, there has been no studies on using the pre-training and fine-tuning paradigm to train Iu Mien language speech recognition models. The aim of this paper is to explore the potential of the Whistle modeling on Iu Mien Language, a Sino-Tibetan low-resource language. Specifically, this paper uses the Whistle model as a pre-trained backbone model and fine-tuning it with a small amount of Iu Mien language training data to construct a Iu Mien language speech recognition model. We compare the fine-tuning results on Iu Mien with those obtained using backbone models from other pre-training methods. Our main contributions are as follows.

• •

  This paper introduces the basic characteristics of the Iu Mien language, including its writing and pronunciation features, and explores how to train a Iu Mien language speech recognition model using a small amount of data.

• •

  Experiments demonstrate that the Whistle method is more data-efficient compared to other methods, resulting in better Iu Mien language speech recognition models with an limited amount of Iu Mien speech data.

Grapheme-based pre-training methods use graphemes as modeling units to construct a shared vocabulary for multilingual data. There are currently three basic grapheme modeling methods: characters [3], subwords [4], and words [5]. To retain some semantic information while avoiding the out-of-vocabulary (OOV) issue, the most commonly used method is to use subwords for modeling. For example, Whisper [6] uses BPE-based (Byte-Pair Encoding base) text tokenizer and weakly graphemic supervision on more than 680,000 hours of clean web data, which can recognize more than 97 languages. However, graphemes are related to the writing system of a language, and it may be difficult to train grapheme based models to learn common speech recognition capabilities between different languages. Self-supervised training is another major multilingual pre-training method. For example, models such as XLS-R [7] use a large amount of unlabeled multilingual audio data for training, and their training methods are similar to wav2vec 2.0 [8] or BERT [9]. Based on the pre-trained model, a linear layer is added to map the output to the target token list of speech recognition. The CTC approach [10] can then be used to fine-tune the model.

Supervised pre-training based on phoneme annotations is different from supervised pre-training methods based on graphemics, in that it can reduce the influence of the writing systems of different languages during model training. On the other hand, compared with self-supervised training, clear phoneme training goals can also help the model quickly learn general speech recognition knowledge. However, obtaining accurate pronunciation annotations for each language often requires a lot of expert knowledge. Recently, [2] proposed a new phoneme-based supervised pre-training method called Whistle. Whistle uses IPA phoneme annotations with a small number of errors generated by the G2P model as the target (weakly supervision) during model pre-training to train a multilingual speech recognition model. Whistle used data from ten languages (English, French, Italian, Spanish, Russian, Dutch, Turkish, Kyrgyz, Swedish, Tatar) from commonvoice11 [11] as training data. The Conformer model was used as the encoder, and the CTC algorithm was used for supervised pre-training. Finally, three pre-trained models of different sizes were trained: Whistle-small (90M), Whistle-medium (220M), and Whistle-large (543M). [2] conducted cross-language fine-tuning experiments on Polish and Indonesian, respectively, to demonstrate the effectiveness of this method. However, during the pre-training process of [2], no Sino-Tibetan language was incorporated as the pre-training languages, and the IPA annotations of the pre-training data did not contain tone information.

The writing system currently used by the Yao people is a newly created writing system, which was unified by China and the United States in 1984. It spells the Iu Mien dialect. Therefore, it can also be called the Iu Mien Unified Script (IMUC). The script is a phonetic syllabic script based on the Latin alphabet, and each syllable can be composed of 30 initials, 128 finals, and 8 tones. As shown in Table 1, IMUC is a phonetic script that uses subwords to represent different sounds. For each IMUC word, its structure is similar to the Chinese pinyin to represent words, consisting of one or more syllables, but unlike Chinese pinyin, Iu Mien only uses the 26 letters of the basic Latin alphabet, and does not use additional symbols to distinguish tones or vowels. IMUC generally uses Latin letters at the end of the word to represent tones. However, there is no corresponding letter for the mid-level tone. If a word ends without a tone-indicating letter, it indicates that the word has a mid-level tone. Other tones are represented by five letters: ’h’, ’v’, ’z’, ’x’, and ’c’.

This paper use a weakly-supervised phoneme-based multilingual pre-training model and then fine-tuning to improve the recognition performance of the Iu Mien language speech recognition model. As shown in Figure 1, we use an end-to-end speech recognition model based on the CTC approach. The acoustic encoder of the Iu Mien language speech recognition model is initialized using the pre-trained Whistle-small model. The fine-tuning training of the Iu Mien language speech recognition model is performed separately based on subword modeling and phoneme modeling.

We add a linear layer on top of the acoustic encoder, which is used to map the feature vectors of the encoder output to the target prediction space. For methods of fine-tuning based on subwords, we use the random initialization method to initialize the linear layer parameters. For the phoneme-based approach to fine-tuning, in order to share the parameters from the pre-training as much as possible, we used the method in Whistle to use the parameters of the linear layer from the pre-training backbone for initialization. For the k-th row vector in the linear layer weight matrix at the top of the encoder during pre-training, it is considered as the embedding vector corresponding to the k-th phoneme in the predicted phoneme list. For a target language phoneme that has appeared in the predicted phoneme lexicon at pre-training, we use the embedding vector of that phoneme obtained at pre-training to initialize the corresponding position of the linear layer weight matrix at fine-tuning.

This paper primarily investigates and compares the effectiveness of different pre-training methods in fine-tuning Iu Mien language speech recognition models using a small amount of Iu Mien language data. By using the weakly-supervised phoneme-based multilingual pre-training model, Whistle, for fine-tuning, the speech recognition performance for the Iu Mien language can be effectively improved, achieving better results compared to other pre-training methods. Thanks to the common pronunciation among different languages, phoneme based pre-training with non-Chinese-Tibetan languages can also successfully improve the performance on the Chinese-Tibetan Iu Mien language. Hopefully, this could reduce the data requirement for low-resourced speech recognition for a broader range of languages.

There are some interesting future work. The Whistle model used in this paper was pre-trained using only ten non-tonal languages, without considering how to integrate tonal information from different languages into the multilingual pre-training. The Iu Mien language, however, has eight tones. There have been some effort towards addressing this problem [16, 17]. Potentially, multilingual pre-training with tone modeling incorporated would benefit low-resourced speech recognition for tonal languages.