MMM: Multilingual Mutual Reinforcement Effect Mix Datasets & Test with Open-domain Information Extraction Large Language Models

Information extraction (IE) Sarawagi et al. (2008) is a significant area of research within natural language processing (NLP). This field has evolved to encompass a variety of subtasks, including sentence classification , text classification , Named Entity Recognition (NER) (Qu et al., 2023; Nadeau and Sekine, 2007; Lample et al., 2016), sentiment analysis (Tan et al., 2023; Medhat et al., 2014; Rodríguez-Ibánez et al., 2023), relationship extraction (Wadhwa et al., 2023; Mintz et al., 2009; Etzioni et al., 2008), and event extraction (Gao et al., 2023; Xiang and Wang, 2019). Traditionally, these IE subtasks have been segregated into distinct categories for processing. In conventional multi-task IE (Sun et al., 2023; Zhao et al., 2020), datasets from various tasks are typically merged and subsequently fine-tuned using a unified model. This process culminates in the extraction of information from multiple subtasks, each directed by task-specific output heads. While this method effectively leverages the internal knowledge of the model across different IE tasks, it does not address the potential interconnections among the tasks themselves. This omission highlights a gap in understanding how these tasks might benefit from exploring their mutual relationships.

The Mutual Reinforcement Effect (MRE) Gan et al. (2023b) represents an emerging research direction in multitasking Information Extraction (IE). It primarily explores the interconnections among individual subtasks, thereby enhancing the performance of each task. MRE categorizes IE subtasks into two principal groups: text-level tasks and word-level tasks. For instance, conventional tasks such as sentence classification, text classification, and text sentiment analysis fall under text-level tasks. Conversely, tasks like NER, which involve classifying words, are categorized as word-level tasks.

Figure 1 illustrates a specific example of MRE applied to IE subtasks. The left side of the figure depicts labels for sentence classification, while the right side displays words alongside their corresponding labels extracted from the sentence. Thus, the left side corresponds to text-level tasks, and the right side to word-level tasks. Notably, each sentence features both a text-level label and word-level label entities pair. This differs from typical IE multitasking, which extracts information from various sub-tasks across different texts. Instead, MRE simultaneously performs text-level classification and word-level label-entities pairing on the same text.

We can elucidate the MRE with specific examples. Consider the sentence from Figure 1: ’Giant pandas are mammals, endemic to China.’ This sentence is categorized under ’nature,’ and it includes two annotated label-entities pairs: ’Animal Name: pandas’ and ’Nation: China.’ This exemplifies that when a sentence is classified as relating to nature, it likely contains terms associated with natural elements, such as species names and geographical locations. Similarly, when a sentence is classified as nature, there is a high probability that the sentence as a whole describes nature. The interplay between these two levels of text level classification and word-level entity recognition illustrates the MRE, as each level mutually reinforces understanding and classification accuracy. Expanding upon this, MRE is applicable to various subtasks in IE. For instance, in sentiment analysis, the presence of numerous words with positive sentiment typically indicates that the text overall conveys a positive sentiment. Conversely, if a text is overall negative, it likely contains words with negative sentiment. Integrating tasks at both the text and word levels not only aids in the model’s comprehension of the context but also improves performance across both tasks. This methodology probes whether LLMs can interpret texts similarly to human understanding, by assessing the semantics of individual words and then synthesizing these insights to classify the text as a whole (Gan et al., 2023c).

Figure 2 illustrates the composition of the translated Multilingual Mutual Reinforcement Effect Mix (MMM) Datasets. Notably, the SCPOS subdataset within the original Japanese dataset is substantially larger than others, which is why the figure does not depict the proportional sizes of the subdatasets; instead, it merely labels each subdataset within the MMM dataset. This dataset encompasses three languages and includes seven distinct subdatasets for each language, each corresponding to a specific information extraction task.

Among these, SCNM denotes the Sentence Classification and Named Entity Recognition Mix Dataset. SCPOS, which stands for Sentiment Classification and Part-of-Speech Dataset, involves comprehensive sentiment polarity classification at both the text and word levels. This subdataset is further segmented into four categories: sentiment-related words, adjectives and nouns, solely adjectives, and solely nouns. This categorization facilitates an examination of how words with different lexical properties influence the overall sentiment classification of a text.

Additionally, the TCREE dataset is dedicated to Text Classification and Relation & Event Extraction, leveraging mixed data types for enhanced analytical depth. Lastly, TCONER represents the Open-Domain NER and Text Classification, utilizing an existing open-domain dataset for annotations. This structure allows for a nuanced exploration of text classification and entity recognition across various domains.

This work begins by introducing a novel dataset translation framework. Initially, six MRE mix datasets were translated, and subsequently, the TCONER dataset was expanded. Furthermore, to address the suboptimal performance of existing large language models LLMs on IE tasks, we refined the training methods. We propose a new, streamlined, and efficient input-output scheme that standardizes the handling of all tasks, enhancing the model’s training process. The LLM was then trained using the MMM dataset. The outcomes indicate that the optimized LLM, referred to as OIELLM, surpasses the original training method in performance across various datasets. Moreover, expanding the use of more MRE mix datasets is shown to further enhance the knowledge utilization within the LLM, leading to improved performance across all tasks.

This paper presents several contributions are detailed as follows:

1. 1.

   Development of a Novel Framework for Dataset Translation: We propose a new framework that leverages existing LLMs to translate datasets for underrepresented languages. This framework facilitates the construction of Multilingual MRE Mix Datasets, significantly reducing the reliance on extensive human labor and addressing the issue of dataset scarcity in linguistic research.

2. 2.

   Expansion of MRE Mix Datasets: We have successfully expanded the MRE Mix datasets to include a newly constructed open-domain NER dataset. This addition serves as a critical component in enhancing the comprehensiveness and utility of the MRE Mix datasets, effectively placing the last piece in the puzzle for open-domain applications.

3. 3.

   Optimization of Input and Output Schemes for IE LLMs: Our research optimizes the traditional input and output mechanisms of information extraction LLMs. By employing the specially constructed MMM dataset, we not only reject the standard models trained on generic data but also significantly enhance the performance of LLMs within the domain of information extraction.

Datasets. To begin, the MRE mix dataset primarily originates from the SCNM Gan et al. (2023b) dataset in Japanese, followed by the SCPOS (Gan et al., 2023d) and TCREE Gan et al. (2023a) datasets. However, the exclusive use of the Japanese language across these datasets poses significant challenges for researchers attempting to further explore the MRE. Moreover, there has been a growing interest in employing LLMs for dataset construction (Tan et al., 2024; Wadhwa et al., 2023; Li et al., 2023; Laskar et al., 2023). Pioneering studies Huang et al. (2023) have demonstrated the efficacy of LLMs in data annotation, where LLM-annotated datasets have outperformed manually annotated counterparts. For instance, LLMs have been utilized to generate datasets for mathematical problems Lin et al. (2024) and to develop dataset labeling frameworks, such as FreeAL (Xiao et al., 2023a), where the data is initially labeled by LLMs and subsequently refined by smaller models before undergoing a final, more accurate labeling by LLMs again.

These methodologies leverage instructional learning and in-context learning to guide LLMs to respond to specific queries and from these responses, extract annotated labels, thereby creating a fully labeled dataset. Distinct from previous efforts, the MMM dataset represents the inaugural initiative to translate datasets from lesser-used languages into more widely spoken languages, such as English and Chinese. Furthermore, the newly developed TCONER dataset addresses a critical gap by providing the first open-domain Named Entity Recognition (NER) dataset within the existing framework of the MRE mix dataset.

LLM in Information Extraction. Since the introduction of Pretrained Language Models (PLMs), sequential-to-sequential (seq2seq) based IE models have gained prominence. These developments range from the initial UIE Lu et al. (2022) to later models such as USM Lou et al. (2023) and Mirror (Zhu et al., 2023). All these models are generative in nature, enabling them to handle multiple word-level IE tasks—such as NER, Relation Extraction, and Event Extraction simultaneously. The primary advantage of these generative IE models is their generalizability; they eliminate the need for task-specific fine-tuning across different tasks. Instead, a single model can address all IE subtasks by standardizing the format of inputs and outputs for various tasks. The model is trained across different IE subtasks using these unified formats, aiming to equip a single model with the capability to manage multiple tasks effectively.

With the advent of LLMs, new approaches to IE have emerged, which can be broadly divided into two categories. The first involves direct interaction with LLMs using prompts in a zero-shot or few-shot manner, where the model outputs the desired entities either through multi-round dialog-style prompts or through single-command-based prompts that extract entities in one go (Wang et al., 2023; Wei et al., 2023). The second approach involves fine-tuning LLMs using specialized datasets (Zhou et al., 2023; Xiao et al., 2023b).

Our research distinguishes itself by focusing more intensively on the MRE. We go beyond merely aggregating existing IE sub-datasets for model training. Instead, we develop specialized MRE-enhanced datasets, through which we not only demonstrate but also apply the efficacy of MRE in enhancing information extraction capabilities.

In this chapter we will explain how to translate MRE mix datasets in small languages into other languages. And how to construct TCONER datasets. And how you can minimize the use of manual labor with guaranteed quality.

In this chapter, we outline methodologies to enhance the performance of existing models and techniques for processing MRE mix datasets, aiming to surpass previous benchmarks. Before delving into the specifics of the Open-domain Information Extraction Large Language Model (OIELLM), it is imperative to justify the necessity for a distinct model tailored to MMM datasets.

Firstly, MRE mix datasets differ significantly from traditional IE tasks as they require simultaneous output of text-level labels and word-level label-entity pairs. Consequently, standard sequence labeling models are inadequate for handling these demands directly. Furthermore, existing generative IE models and methodologies have solely focused on producing word-level label-entities, neglecting text-level labels altogether.

The primary objective of MRE mix datasets is to investigate the interplay between text-level and word-level annotations. By leveraging this synergistic relationship, we aim to concurrently enhance the performance of both tasks. This model improves textual understanding by learning both tasks in tandem. Additionally, the MRE framework can contribute to model interpretability, drawing inspiration from cognitive processes that mimic human reasoning.

In summary, this study proposes the development of a novel model specifically designed for processing the MMM dataset. Furthermore, we aim to experimentally investigate whether the MRE positively influences various IE subtasks when using LLMs. Traditionally, IE tasks have been approached through the use of QA dialogues for extraction. However, this research adopts a distinct methodology. This departure is motivated by earlier foundational studies in generic generative IE with PLMs, where dialogue models were not utilized. Instead, these studies implemented a generic framework. Accordingly, we too will employ a modified input and output scheme tailored for the MMM dataset, diverging from the conventional dialogue-based approaches.

Figure 5 illustrates the input and output formats of our enhanced OIELLM. The fundamental unit of analysis in both input and output is words, reflecting our understanding of the tokenization principle utilized by LLMs, which typically focuses on words or phrases. By omitting the dialog prompt, we do not compromise the LLM’s comprehension of the task. This adjustment not only reduces the input-output length but also simplifies the LLM’s processing, thereby enhancing operational speed.

Each text processed is prefixed with task-specific instruction words, which define the task type and guide the model’s subsequent output generation. In our format, all task instruction words in the input are introduced by a special symbol "/", which serves to delineate the task words from the main text. This separation is crucial for distinguishing between text-level labels and word-level label-entity pairs in the output.

The combined text and task instruction words are then fed into the OIELLM, with the output comprising both text-level labels and word-level label-entity pairs. Our labeling convention adheres to the format used in the previous MRE mix datasets, utilizing ":" and ";" to ensure consistency and clarity.

In summary, by standardizing the input and output structures and clearly defining task instruction words, our modified OIELLM effectively processes all sub-datasets within the MMM framework.

In this chapter, we detail specific experimental procedures, including dataset sizes for the MMM dataset and methodologies for training the OIELLM model, along with the evaluation techniques used.

Table 1 presents the experimental results of three OIELLM models trained on 21 MMM sub-datasets. Notably, the model designated with an asterisk, OIELLM-8B, was trained using the LLaMA3-8B-Instruct framework, whereas the remaining models were based on the LLaMA3-8B-Base framework. These results demonstrate the enhanced performance of OIELLM in handling Japanese data after incorporating multilingual capabilities. Impressively, OIELLM’s performance surpassed that of GIELLM-13B-jp on half of the datasets, despite GIELLM-13B-jp being a model specifically tailored for Japanese. This observation supports the hypothesis that integrating multilingualism and multitasking can more effectively leverage the knowledge embedded in the pre-training of multilingual LLMs.

However, OIELLM’s performance on the TCONER task was suboptimal, which we attribute to insufficient training data. Given that open-domain tasks require extensive and diverse datasets compared to domain-specific tasks, the limited data may have hindered the model’s performance. This area will be a focus of our future research, aiming to understand and improve the data dependencies of OIELLM in open-domain contexts.

In this study, we introduce a framework that utilizes LLMs to translate datasets, thereby removing language barriers for research in less-represented languages. To address the deficiency of open-domain IE tasks in the MRE mix dataset, we constructed the TCONER dataset. Additionally, we trained the OIELLM model using the newly created MMM dataset.

Future work will focus on employing the MMM dataset to further explore the Mutual Reinforcement Effect. We will also continue to enhance the performance of the OIELLM model in open-domain information extraction tasks.

Due to resource constraints, we were unable to employ the higher-performing GPT-4-Turbo OpenAI (2023) model as the base for our dataset translation framework. Consequently, this model was also not utilized during the testing phase on the dataset. In future work, we aim to leverage a more advanced model, such as the GPT-4-Turbo, to evaluate the MMM dataset, provided that the necessary resources become available.