Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation

Large language models (LLMs) have been proven to be highly effective in many natural language processing (NLP) tasks. By integrating external tools, LLMs can enhance their functionality and handle a wider range of tasks using the additional knowledge and capabilities [21, 22, 23, 24, 25, 26, 27, 28, 29]. For example, AnyTool [16] is an LLM-based agent that uses various APIs to answer user queries across different domains, such as providing specific information about a book, movie, or product, or offering personalized recommendations through connected recommendation APIs. Similarly, ViperGPT [14] employs LLMs for complex visual tasks by generating Python programs that coordinate vision-and-language models to process visual queries.

However, despite the advancements in NLP and computer vision, the exploration of using LLMs in the speech domain to integrate various speech modules and foundation models, particularly via program generation for speech/audio tasks, is relatively limited.

Moreover, while LLM-based agents have been developing rapidly, the construction of toolsets for these agents remains relatively underexplored. Existing approaches [17, 18, 19] typically create tools at the instance level, where a new tool is created tailored for a single or a few instances, overlooking the high-level similarity of the collected instances. This may introduce redundancy in the created toolset that may be difficult to de-duplicate. In addition, they typically require golden labels for the instances, which make it harder to collect instances.

In contrast, our proposed method takes a holistic approach, utilizing LLMs to identify essential sub-tasks from all collected instructions simultaneously. This approach significantly reduces the redundancy of the constructed toolsets compared to instance-level creation methods. Additionally, since our method only requires instructions that can be easily synthesized, the associated audio files and golden labels are not necessary, making it more data-efficient and simplifying the toolset construction process compared to existing methods.

Equipping LLMs with speech-processing toolkits is underexplored compared with NLP and computer vision domains. There is only a limited number of studies in this area. Among them, AudioGPT [15] is notable for using an LLM as a core controller to manage various pre-trained audio and speech models. Upon receiving a user query, AudioGPT analyzes it, classifies it into task families, and assigns an appropriate speech model for the task. The model’s output is then sent back to the user as the system’s response. However, AudioGPT has some limitations: 1) Limited generalizability: It assigns only one model per query, with no collaboration between speech models, which limits its ability to handle complex tasks beyond predefined task families. Additionally, the families are not sufficiently broad and diverse, further restricting the generalizability. For instance, AudioGPT can not deal with Dynamic-SUPERB tasks, which are too complex to be addressed by a single model from its task families and model collection. 2) Lack of flexibility: The response is generated and provided to the user in a black-box manner, disallowing the users to manipulate the model’s behavior and limiting the flexibility.

In contrast, Speech-Copilot effectively addresses these issues through a well-crafted toolset construction and program generation approach, where a toolset with fundamental speech modules is first constructed and the agent can then use these modules as basic blocks and dynamically combine them to solve various tasks based on the user query. This ensures the generalizability and versatility. By solving tasks through programming, Speech-Copilot allows users to modify the program according to their preferences, providing a higher degree of flexibility and enabling behavior manipulation to some extent. Furthermore, analyzing the agent’s reasoning steps in the programs enhances interpretability, making it easier for users to understand how solutions are derived.

The toolset construction phase involves two steps: task decomposition and task modularization.

During the program generation phase, an LLM-based program-generating agent $\pi$ is constructed. Given a textual query $q$ and audio input $a$ from the user that is unseen during the toolset construction phase, the objective of the agent $\pi$ is to generate a program $z$ $=$ $\pi(q|\{f_{i}\}_{i=1}^{M})$ based on the available modules $\{f_{i}\}_{i=1}^{M}$ from Sec. 3.1.2 that solves the target task $T$ specified by the query.

This process involves the following steps: (1) Identifying the task $T$ from the query $q$, (2) Determining the combination $h$ and a set of sub-tasks $\mathcal{T}_{q}$ $\subseteq$ $\{\mathcal{T}_{i}\}_{i=1}^{M}$ that satisfy $T$ $=$ $h(\mathcal{T}_{q})$, (3) Selecting the relevant modules $f_{q}$ $=$ $\{f_{i}|\mathcal{T}_{i}\in\mathcal{T}_{q},i\in\{1,2,\ldots,M\}\}$, and (4) Generating a program $z$ that utilizes the modules $f_{q}$ and integrates their results with the combination $h$. In practice, we develop the agent $\pi$ by guiding the LLM through these steps, providing the module documentation $\{f_{i}\}_{i=1}^{M}$, and specifying additional constraints in the prompt. These constraints include requiring the agent to provide reasoning and explanations for how it addresses each step and to generate the program $z$ in a specific format, enhancing the interpretability and ease of parsing the programs generated by the agent. Finally, the result $r$ of the query $q$ can be obtained by executing the program $z$ on the audio input $a$ with a Python interpreter.

We evaluate Speech-Copilot on Dynamic-SUPERB²²2https://dynamic-superb.github.io/ [20] because of its wide coverage of diverse, complex, and challenging speech and audio tasks. Dynamic-SUPERB is designed to assess universal speech models that perform diverse and complex tasks with both strong instruction-following abilities and complicated speech/audio-related understanding. It includes 55 tasks, involving complicated speech/audio-related reasoning and covering six aspects of the speech and audio modalities: audio, content, degradation, semantic, paralinguistic, and speaker. The aspects are explained as follows:

1. 1.

   Audio (AUD): This aspect assesses the model’s ability to interpret audio signals. Tasks in this aspect include detecting the sound of a specific object in an audio clip and classifying the sound into some categories.

2. 2.

   Content (CNT): This aspect evaluates the model’s understanding of speech content. Tasks involve speech command identification, language recognition, and so on.

3. 3.

   Degradation (DEG): The goal here is to measure the model’s ability to detect noise and reverberation in speech signals. Tasks include predicting the signal-to-noise ratio (SNR) at the utterance level and determining if speech signals are affected by reverberation.

4. 4.

   Paralinguistic (PRL): The tasks of this aspect aim to gauge the model’s understanding and the ability to make inferences based on paralinguistic information in speech. The tasks include but are not limited to accent identification, emotion recognition, etc.

5. 5.

   Semantic (SEM): The goal is to assess the model’s semantic understanding. This aspect involves intent classification, sarcasm detection, etc.

6. 6.

   Speaker (SPK): The aim is to evaluate the model’s capacity to extract speaker-related information. This involves tasks like speaker verification and multi-speaker detection.

Currently, Dynamic-SUPERB tasks are formulated as multiple-choice questions with one and only one golden label per question.

Dynamic-SUPERB tasks are evaluated in accuracy, where a hit occurs when the response of the evaluated model aligns with the golden label. Due to the generative nature of current models, e.g. LLMs or large audio-language models (LALMs) in Sec. 4.3, instead of a single option, the models tend to generate long-form responses that do not follow a certain format, making the conventional exact-match (EM) evaluation unsuitable. To this end, the evaluation policy of Dynamic-SUPERB uses an LLM, whose ability as the automatic evaluator has been studied [30, 31], to determine the alignment between the models’ predictions and the labels.

In this work, GPT-4o³³3The model version employed in this study is gpt-4o-2024-05-13. is used as the evaluator. During the grading process, the original task instructions, the response from the assessed models, and the golden labels are presented in the prompts to the evaluator. We also include some rules in the prompts that the evaluator should strictly follow, and the evaluator is then employed for judgment. The rules include:

1. 1.

   As each question in Dynamic-SUPERB has one and only one golden answer, it should be judged incorrect if the models choose no or multiple options, meaning that they should clearly select one and only one option for each question.

2. 2.

   The evaluator should provide reasons for their judgment.

3. 3.

   The evaluation should be summarized with a single ”Yes/No” in a specific format, where ”Yes” indicates that the prediction aligns with the label, and ”No” indicates it does not.

Besides automatic evaluation with GPT-4o, human verification of the evaluation results is conducted to ensure grading correctness and consistency. Finally, we follow the standard approach of Dynamic-SUPERB and report the average performances of the aspects.

We compare the performances of Speech-Copilot with those of several baseline models to verify the effectiveness of Speech-Copilot, including the toolset construction and program generation. The baselines include several recent publicly available large audio language models (LALMs), e.g. Qwen-Audio-Chat [32], SALMONN [33], LTU-AS [34] and WavLLM [35], and cascaded systems that employ LLM to solve the tasks based on the information from automatic speech recognition (ASR), automatic audio captioning (AAC), and other available speech models. Large audio-language models [36, 32, 33, 34, 35, 37, 38] extend the capabilities of standard large language models by incorporating audio and speech recognition features. This integration enables LALMs to process and respond to tasks involving sound and speech.

On the other hand, in the cascaded systems, we adopt GPT-3.5⁴⁴4The model version employed in this study is gpt-3.5-turbo-0125. [39] as the LLM and Whisper-large-v3 [40] and Qwen-Audio-Chat [32] for ASR and AAC, respectively. The cascaded systems are denoted as “ASR+LLM” and “ASR+AAC+LLM” for those using ASR results only and using both ASR and AAC simultaneously. We also compare Speech-Copilot with the cascaded system that provides all the information from our constructed modules listed in Sec. 5.1, except for the speaker verification module if there’s only one audio input, to the LLM, denoted as “All Attributes + LLM”. This baseline simulates the ultimate cascaded system where the LLM utilizes all the available information to make predictions.

Greedy decoding is applied to all the models in our experiments. Regarding the candidates of LLMs, in the stages of toolset construction, program generation, and evaluation, GPT-4o is adopted because of its strong language capabilities. On the other hand, when executing the generated programs, GPT-3.5 is employed if querying LLM is required for certain modules. This choice balances cost and model performance, i.e. we can use a more powerful model where it’s most needed while opting for a less expensive option elsewhere.

As for Whisper-large-v3, though prompting for Whisper is common in prior works [41, 42, 43], it is not employed in our experiments due to the unclear effect of prompting methods of Whisper [44]. For other models involved in this work, we used the default settings, with the generation strategy consistently set to greedy decoding.

We first demonstrate the results of our toolset construction phase. We compare our task decomposition method, where we take all the instructions into consideration at once, with instance-level toolset creation, in which the tools are created based on a single instruction. More than 50 task instructions generated by GPT-3.5 are used for the construction, and the results with GPT-4 [45] are shown in Table 2.

As expected, our decomposition method, which considers all instructions collectively, significantly reduces the size of the sub-task set compared to instance-level tool creation. This reduction facilitates subsequent modularization and implementation. In addition, the size can be further reduced if the LLM is required to reflect on whether there are similar sub-tasks that can be unified or combined into single speech-processing tasks. This encourages the LLM to unify tasks of similar nature. For instance, the LLM can combine tasks related to the number of speakers with speaker diarization sub-tasks, thereby reducing the number of required sub-tasks.

As for the task modularization, we again employ GPT-4o to transform the collected sub-tasks into modules with detailed documentation with requirements outlined in Sec. 3.1.2. The resulting modules are listed in Table 3. We select the models for those modules based on the principles clarified in Sec. 3.1.2. For some modules where suitable and publicly available models are unavailable, we employ Qwen-Audio-Chat as the foundation model and realize these modules by prompting Qwen-Audio-Chat with a fixed set of prompts. The selected models for the modules are listed in Table 3.

Regarding the implementation details, most of the modules can be realized with the selected models directly. We briefly explain the implementation for some modules that require special handling. Stress position identification, a module for identifying stress syllables in spoken words, lacks a specialized model designed for this module. We hypothesize that this task can be approximated by combining speech recognition and LLMs due to their strong linguistic knowledge [53]. Thus, we implement this module using Whisper-large-v3 and GPT-3.5. Sound classification is a module for classifying a wide variety of sounds, e.g. environmental sounds, animal sounds, etc. Due to the high diversity of sound categories, it is difficult to find a single model to handle all kinds of classification tasks, so we choose to use Qwen-Audio-Chat as the backbone. To maintain generalizability, instead of using specific prompts for certain kinds of sound classification, we require Qwen-Audio-Chat to generate detailed audio captions, from which GPT-3.5 is adopted to extract the desired information based on the specific task objective.

We then compare the performances of Speech-Copilot with the selected baselines on Dynamic-SUPERB, as shown in Table 1. Overall, Speech-Copilot achieves the highest average score across the 55 tasks and outperforms other baselines in 5 out of 6 aspects, indicating the efficacy of the constructed toolset and the problem-solving capability of the LLM-based program-generating agent.

We observe that the LALM baselines, i.e. Qwen-Audio-Chat, SALMONN, LTU-AS, and WavLLM, typically encounter significant issues that impact their performances. For instance, Qwen-Audio-Chat suffers from severe hallucinations about the spoken content in the provided audio when required to answer the questions directly. However, it can almost correctly identify the spoken content if asked to provide an audio caption of the audio input instead. This observation aligns with prior works [54] on the hallucination of LALMs, indicating the vulnerability of recent LALMs. We also notice that these models struggle with clearly selecting one and only one option for the questions. For example, SALMONN tends to list all of them in the response, which is unacceptable. Moreover, these LALMs underperform compared to cascaded systems. Even the simplest “ASR+LLM” systems outperform end-to-end LALMs, indicating that current LALMs are not yet capable of handling complex speech/audio understanding and reasoning tasks, such as those in Dynamic-SUPERB. In contrast, Speech-Copilot demonstrates strong robustness due to its modular design, achieving significantly better performance on the evaluation benchmark.

Comparing the performances between the cascaded systems, it seems that the overall performance will be better if more information from different speech models is provided to the LLM. However, they still underperform relative to Speech-Copilot. Notably, there is a significant performance gap between the “All Attributes + LLM” baseline, which is the most effective cascaded baseline in terms of the average score, and Speech-Copilot, despite using the same modules. The key difference lies in the utilization of information from these modules. The former incorporates all available information indiscriminately, regardless of the query’s purpose, while the latter selectively uses and combines relevant modules based on an analysis of the input queries. This highlights the importance of selecting related and useful information to avoid being misled by redundancy. Additionally, the computation cost of the former system is higher since it requires running all the modules for all data, showcasing the benefit of efficient information selection considering computation budgets. Such a selection process is achieved by query analysis and programming in Speech-Copilot, demonstrating the advantage of a program-generating agent.