Pronunciation Assessment with Multi-modal Large Language Models

Data2vec2 [27] was employed as the audio encoder because of its versatility, efficiency, and outstanding performance in speech tasks. It comprises several convolutional blocks and multi-layer transformers. The convolutions in each block have 512 channels with strides of (5, 2, 2, 2, 2, 2, 2) and kernel sizes of (10, 3, 3, 3, 3, 2, 2). It transforms 16 kHz raw audio into a sequence of embeddings with a stride of 20 ms and a receptive field of 25 ms.

For each sample, given the raw audio $\mathbf{x_{s}}$, the speech features $\mathbf{h_{s}}$ was firstly extracted from the last hidden layer through the data2vec2 based speech encoder, which can be written as:

The m-adapter [28] was firstly introduced for speech translation tasks. We employed it as a modality adapter in our model to connect the speech and text modalities. It consists of two pooling convolutions and a multi-head self-attention (MSA) layer. The pooling operations are used to modify the speech sequence length, while the MSA captures global information.

This work follows the paper [29] by replacing the three independent pooling modules for Q, K, and V with a shared pooling module to streamline the architecture and improve the efficiency of the model. In general, given the speech features $\mathbf{h_{s}}$ extracted from the speech encoder, a convolutional pooling operation is first used to adapt the speech length. These adapted features are then fed into the MSA layers to obtain the modality adapter features $\mathbf{{h^{\prime}}_{s}}$, expressed as follow:

The intuition of this module is that the number of features in the time domain of the speech signal is much larger than the length of the text. A scaling method needs to be applied to the speech features to balance the amount of information between the two modalities.

Decoder-only LLM serves as a foundational component in the proposed method. We expect it to understand the learner’s speech, analyse the difference between speech and the canonical text, and then make decisions regarding accuracy and fluency scores accordingly.

For the text input of the LLM, we define the task-specific prefix as “$<Pronunciation~{}Assessment>$”. Additionally, in follow-up scenarios, The prompt text, as an important reference cue, can be considered as the text input of the LLM. Therefore, the final text input of LLM is defined as “$<Pronunciation~{}Assessment>the~{}prompt~{}text~{}is~{}...$”, and then the text input will be transformed into embedding features denoted as $\mathbf{{h}_{t}}$. Finally, we concatenate the embedding features with adapter-modified speech features $\mathbf{{h^{\prime}}_{s}}$ and then fed the concatenated features into the LLM to obtain the final prediction text $\mathbf{y}$ through a token decoding operation as equation(3), where the [;] denotes the concatenation of two vectors.

The proposed method adopts a two-stage training strategy. Previous studies have shown that the performance of traditional align-based methods can be affected by the ASR system, as accuracy scoring heavily relies on speech content recognition. Therefore, in the first stage, we train the multi-modal model on the ASR task to make the connection between the two modalities more robust, which may benefit the pronunciation assessment task. We took the ASR task-specific prefix “$<transcript>$” as the text input and then utilized limited ASR data to complete the ASR task training.

In the second stage, we fine-tuned the model trained in the first stage using a limited amount of scoring data based on the “$<pronunciation~{}assessment>$” assessment task-specific prefix and prompt text. For the ground truth, we structured it as “$accuracy:9~{}fluency:7$”, aiming for LLM to output scores in such a format to achieve multi-task scoring for both accuracy and fluency.

Two public datasets were used in this study. Librispeech [32], a commonly used ASR corpus, was utilized to train the multi-modal ASR in the first stage. It consists of about 1,000 hours of 16kHz read English speech derived from read audiobooks, with the dev-clean and dev-other subsets used as the validation set, and the test-clean and test-other subsets used as the test sets. To evaluate fluency and accuracy scoring in the second stage, we conducted experiments on Speechocean762 [33], an open-source speech assessment corpus consisting of 5,000 English utterances collected from 250 Chinese speakers. The whole scoring dataset is divided into 2,500 sentences for training and 2, 500 for testing. The score distribution of the test set is shown in Figure 2. The x-axis represents the human-labeled scores, and the y-axis represents the number of occurrences for each corresponding score.

The weight of the data2vec2-based speech encoder is initialized from data2vec2-large ¹¹1https://dl.fbaipublicfiles.com/fairseq/data2vec2/large_vox_960h.pt and the Qwen-based LLM [17] is initialized using pre-trained weights derived from Qwen-7B ²²2https://huggingface.co/Qwen/Qwen-7B. For the M-adapter, we chose one MSA layer with 16 heads. The kernel size and stride of the convolution were both set to 2, aiming to halve the length of speech features extracted from the speech encoder.

We used 8 A800s for the experiments. Each stage was trained for a maximum of two epochs on the corresponding task data. The ASR and pronunciation scoring training took about 9 and 0.2 hours, respectively. During both stages, the parameters of LLM were frozen, and the parameters of the speech encoder and modality adapter were updated.

ASR task was often used to evaluate the performance of multi-modal systems[23, 24, 26, 25]. In this subsection, we selected several representative multi-modal systems with strong ASR performance as our baselines, including SALMONN [23], WavLLM [24], Qwen-Audio [26], and SALM [25].

The comparison results between the proposed system and these baseline systems are shown in Table 1. For the result of the proposed method, the predict and ground-truth texts were normalized using Whisper’s text normalization method ³³3https://github.com/openai/whisper/tree/main/whisper/normalizers before computing WER. The results suggest that the proposed method outperforms all the baselines in terms of WER in the Librispeech test sets. This observation encourages us greatly for our upcoming pronunciation assessment tasks.

In addition, we briefly compared several different speech encoders based on the proposed method, with results shown in Table 3. The findings reveal that the data2vec2 model as the optimal choice for speech encoding performs significantly better compared to the performance of Hubert ⁴⁴4https://huggingface.co/facebook/hubert-large-ls960-ft and Whisper ⁵⁵5https://huggingface.co/openai/whisper-large-v2.

In this subsection, we present the performance of the proposed multi-modal model in pronunciation scoring, which is an align-free end-to-end approach. For a fair comparison, we selected two types of baseline systems: one is align-free end-to-end systems based on SSL features and the other one is align-based systems that rely on traditional DNN-HMM based ASR model trained with Librispeech dataset. We did not consider the align-based system using SSL features [6] as our baseline, since such an approach would make the entire align-based pipeline more complicated and less feasible for practical implementation.

We first conducted performance comparisons between the proposed multi-modal pronunciation scoring system and the align-based baseline systems presented in Table 2. The leading results validate the effectiveness of our proposed method. Moreover, compared to the complex pipelines of align-based systems, our proposed method offers a simpler way to evaluate learners’ pronunciation.

Then, we evaluate the performance comparisons with the align-free methods. The proposed method demonstrates competitive results with baseline systems in fluency scoring. Additionally, it fills the gap in accuracy scoring for align-free approaches on the Speechocean762 dataset.

In this subsection, we conducted experiments to determine the relative importance of ASR training in the first stage and prompt text in the proposed system for pronunciation scoring. The results are presented in Table 4.

Firstly, we can clearly see that these two factors slightly improve fluency scoring performance in system (a). The two factors focus more on the content of the speech. As verified in [7, 8], prosodic information is more important than speech content information for fluency scoring. Moreover, compared to SSL system [12], SSL+IDX [11]showed a slight performance difference due to the additional information provided by phoneme IDs, which represent speech content. This further confirms that the content does not have a significant impact on fluency scoring.

Compared to speech fluency, accuracy is more significantly affected by these factors. The results of systems (b) and (c) show that the impact of introducing the prompt text is greater than that of ASR training. Additionally, system (d) found that not using any strategies resulted in poor accuracy scoring performance.