Empowering Few-Shot Relation Extraction with
The Integration of Traditional RE Methods and
Large Language Models

In this part, we aim to implement the data augmentation via LLMs, anticipated to enrich the training data for relation extraction. Specifically, drawing inspiration from [20], we construct prompts to tell the LLM the essential components of one RE training sample, i.e., context text, subject entity, object entity, subject entity type, object entity type and the relation. Then the LLM is guided to create more pseudo RE samples. Upon receiving the outputs from the LLM, we establish rules, such as regular expressions, to transform the augmented RE data into the desired format.

With the augmented datasets, we obtain more diverse data to train a traditional RE model. Here we adopt the Typed Entity Marker (punct) method proposed by [23] to denote the entity and context text, and further train a relation extraction model. Specifically, we utilize the symbols “@” and “#” to denote the start/end of the subject and object entities, and further adopt the symbols “$\ast$” and “$\wedge$” to indicate the subject and object entity types, respectively. The processed text is then fed into the pre-trained language model to obtain the representations of the subject and object entities ($h_{sub}$, $h_{obj}$) via the special token “@” and “#”. Finally, we pass ($h_{sub}$, $h_{obj}$) into a classification layer to derive the results.

In Section 4.1, we train a traditional relation extraction model, which allows us to implement a k-nearest neighbors (KNN) search method to retrieve more valuable samples from the training set. Specifically, we utilize the obtained entity representation $H=[h_{sub},h_{obj}]$ to represent each sample, and further obtain the representation and label pair $(H_{i},r_{i})$ on the training set, which we denote as a datastore $D$.

When inferring a new sample $j$, we utilize its entity representation $H_{j}$ to query $D$ according to the euclidean distance to obtain the $k$ nearest neighbors: $\mathcal{N}=\{(H_{i},r_{i})\}_{i=1}^{k}$, which we adopt as demonstrations for LLM inference.

After obtaining the effective demonstrations, we design prompts to provide the essential information to the LLM, thus generating the LLM results. Specifically, inspired by the various attempts about ICL [20], we first describe the target of the relation extraction task through a instruction. Then, the retrieved $k$ nearest neighbors $\mathcal{N}=\{(H_{i},r_{i})\}_{i=1}^{k}$ of current sample are followed, which provide the most relevant information to the LLM. Finally, we ask the LLM to predict the relation of current sample.

For extensive experiments, we conduct our experiments on three widely-used relation extraction datasets: TACRED [22], TACREV [1] and Re-TACRED [16]. More statistics about the datasets can be found in Table 1. Regarding the evaluation metrics, we adopt the micro-F1 scores of RE as the primary metric to evaluate models, considering that F1 scores can assess the overall performance of precision and recall [3, 9, 10, 21].

In this paper, we utilize the zephyr-7b-alpha [18] model on Huggingface as the LLM to conduct experiments.

In the Traditional RE Model part (Section 4.1), we adopt roberta-large [11] as the base architecture. The batch size is set to 4, and the model is optimized by AdamW [12] with a learning rate of $3e-5$. We train the model on the training set for 50 epochs and choose the best epoch based on the micro-F1 performance on the development set.

In the LLM Data Augmentation part (Section 4.1), we double the K-shot training set through LLMs. That is to say, for an 8-shot training set, we construct 8 pieces of pseudo data per relation, thereby creating the final augmented training set. In the KNN Demonstration part (Section 4.2), we set the number of retrieved nearest neighbors as $k=8$. In the Integrated Prediction module (Section 4.3), we set the number of retrieved samples for each relation as $m=4$.

We compare our methods with the state-of-the-art few-shot relation extraction methods. According to the modeling architecture, they can be grouped into three categories, including Traditional RE Methods (① $\sim$ ④), LLM-based Methods (⑤ $\sim$ ⑦) and Hybrid Methods (⑧).

• •

  ① TYP Marker [23] proposes to incorporate entity representations with typed markers, which presents remarkable performance on the RE task.

• •

  ② PTR [7] designs prompt tuning with rules for relation extraction tasks and applies logic rules to construct prompts with several sub-prompts.

• •

  ③ KnowPrompt [3] innovatively injects the latent knowledge contained in relation labels into prompt construction with the learnable virtual type words and answer words.

• •

  ④ GenPT [6] proposes a novel generative prompt tuning method to reformulate relation classification as an infilling problem, which exploits rich semantics of entity and relation types.

• •

  ⑤ GPT-3.5 [15], ⑥ LLama-2 [17], ⑦ Zephyr [18] are the advanced LLMs. We leverage the API for GPT-3.5, while adopt the 7B version for LLama-2 (llama-2-7b-chat-hf) and Zephyr (zephyr-7b-alpha). We utlize the prompt from [20] to conduct In-Context Learning.

• •

  ⑧ Unleash [20] proposes schema-constrained data generation methods²²2For fair comparison, we apply this data generation method to double the training set, which is the same as our settings introduced in the Implementation Details part. through LLMs, which boost previous RE methods (i.e., KnowPrompt) to obtain more competitive results.

It is worth noting that, for these LLM-based baselines (⑤ $\sim$ ⑦), due to the limitations of maximum tokens and the fact that these datasets have at least 40 relations, we utilize the one-shot demonstration per relation following the strategy proposed by [20]. In contrast, our DSARE method, as introduced in the Implementation Details part, requires a maximum of 16 demonstrations³³3In the KNN Demonstration part (Section 4.2), the number of retrieved nearest neighbors is $k=8$. And in the Integrated Prediction module (Section 4.3), we need a maximum of $2m=8$ additional demonstrations., which is much fewer than the number of the one-shot demonstration per relation setting ($>=40$), thus avoiding unfair comparison.