Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge

Semantic plausibility is a fundamental element for understanding event semantics due to the multi-faceted human intuition in the assessment (Resnik, 1996) and the infrequency, non-typicality and non-preference of plausible and implausible events (Padó et al., 2007). It describes what is likely, but not necessarily attested in a given world (Gordon and Van Durme, 2013). In contrast to selectional preference (Erk and Padó, 2010), which is characterized by the typicality of events, semantic plausibility is sensitive to certain properties that are not explicitly covered by selectional preference (Bagherinezhad et al., 2016).

Modeling semantic plausibility is the task of distinguishing events that are likely to happen and those whose occurrences are implausible. Previous efforts focus on injecting world knowledge about entity properties (Forbes and Choi 2017; Wang et al. 2018) and capturing abstractness in simple events (Eichel and Im Walde, 2023), which requires domain expertise for the annotation. Furthermore, entity properties such as size and weight may not be sufficient for the model to learn the relationship between the subject and object involved in an event. As we will show, knowledge about “what kind of action is occurring in the event” and “what kind of entities are participating in it”, which are given by the type labels and their detailed definitions, can be an effective alternative to the entity properties.

Ultra Fine-grained Entity Typing (UFET) is a multi-label classification problem of predicting fine-grained semantic types for entity mentions in text (Choi et al., 2018). The most significant challenge attributes to the massive label space of entity types (typically over 10k classes). Existing approaches can be categorized into two lines: modeling label dependencies and type hierarchies (Onoe et al., 2021; Zuo et al., 2022), as well as data augmentation with distant supervision (Dai et al., 2021; Zhang et al., 2022; Li et al., 2022a). Feng et al. (2023) propose CASENT, a sequence-to-sequence system that predicts ultra-fine entity types using probability calibration. The model is trained to predict a ground-truth entity type given an input entity mention in an auto-regressive manner. During inference, a calibration module computes a calibrated confidence score for each predicted candidate type label of the entity mention. The system then selects the final predictions from the candidate labels using a threshold on the confidence scores. CASENT achieves new state-of-the-art performance on the UFET dataset (Choi et al., 2018).

An event is defined as an occurrence of an action that causes the change of a state (Li et al., 2022b). Events are represented in various ways in different studies. In early years, an event is defined either as a proposition of subject and predicate in studies on temporal news comprehension (Filatova and Hovy, 2001), or as a (predicate, dependency)-pair in studies on script learning (Chambers and Jurafsky, 2008). Later, Balasubramanian et al. (2013) represent events by (subject, relation, object)-triples for event schema induction. Recent studies in information extraction define an event as a more complex structure form consisting of event trigger, event type, event arguments and argument roles (Li et al., 2022b). The trigger is the core unit of an event, typically the verb serving as the predicate. The event type describes the representative feature of the event and is usually the type of the trigger. Event arguments are participants involving in the event and other details about the event, such as time and place.

Event detection is the task of identifying event triggers in a given event mention and classifying them into event types. Although multiple benchmarks are proposed for event detection (Grishman et al., 2005; Wang et al., 2020), they are limited to the ranges of topics and suffer from small data sizes. Li et al. (2023) release GLEN, a new event detection dataset with a larger data size and a wider coverage of type labels.

To address the issue of unbalanced label distribution towards plausible events in one of our datasets, we employ a data augmentation strategy to increase the number of implausible events. This involves enriching the binary-class variant of the dataset with unduplicated implausible events randomly sampled from the multi-class variant of the dataset which also provides additional binary plausibility labels.

We perform UFET to obtain fine-grained entity types and their definitions using CASENT (Feng et al., 2023). For each event triple $(s,v,o)$, we produce two sentences, one with the text span for the subject s indicated using the special tokens <M> and </M>, the other with the text span for the object o indicated using the same special tokens. Then, CASENT predicts a set of fine-grained entity types for s in the first sentence and for o in the second sentence. After that, we extract the definitions (“description”) for the predicted type labels from a knowledge base (KB) built on WikiData²²2https://wikidata.org.

For example, given the event (trader, ensures, strategy), we produce two sentences “<M> Trader </M> ensures strategy.” and “Trader ensures <M> strategy </M>.” as the inputs to CASENT. The system outputs 5 entity types for the subject trader and 3 entity types for the object strategy together with their definitions as shown in Table 1.

For each natural language sentence built from an event triple $(s,v,o)$, we identify the event trigger (which is usually $v$) and predict its type label $t_{v}$ using the event detection model CEDAR accompanied with GLEN (Li et al., 2023). Furthermore, we extract the definition of the event type label from the KB accompanied with the dataset which is also built on WikiData. An event can have multiple event triggers. Each event trigger, however, can only have one event type.

Table 2 shows the predictions of CEDAR for the event (option, robs, accusation) as an example. The model identifies two event triggers robs and accusation, assigned respectively with the type robberty with its definition taking or attempting to take something of value by force or threat of force or by putting the victim in fear, and the type accusation with its definition act of accusing or charging another with a crime. Based on the structure of simple events, we assume that the event trigger is always the verb $v$. Thus, we consider only the event type for the verb in the model’s predictions.

To better integrate the knowledge elements into the input, we design the following 9 templates to construct natural language prompts containing the event sentence, entity types and the event type, along with their definitions.

1. 1.

   TEMPLATE_SENT = [EVT]{sent}[/EVT]

2. 2.

   TEMPLATE_SUBJ_BASIC = The subject “{subj}” has type [STYPE]{stype}[/STYPE],
   which means [DEF]{stype_desc}[/DEF].

3. 3.

   TEMPLATE_SUBJ_EXTEND = It can also have type [STYPE]{stype}[/STYPE], which means [DEF]stype_desc[/DEF].

4. 4.

   TEMPLATE_SUBJ_UNK = The subject “{subj}” has an unknown type.

5. 5.

   TEMPLATE_VERB = The verb “{verb}” has type [ETYPE]etype[/ETYPE], which means [DEF]etype_desc[/DEF].

6. 6.

   TEMPLATE_VERB_UNK = The verb “{verb}” has an unknown type.

7. 7.

   TEMPLATE_OBJ_BASIC = The object “{obj}” has type [OTYPE]otype[/OTYPE], which means [DEF]otype_desc[/DEF].

8. 8.

   TEMPLATE_OBJ_EXTEND = It can also have type [OTYPE]otype[/OTYPE], which means [DEF]otype_desc[/DEF].

9. 9.

   TEMPLATE_OBJ_UNK = The object “{obj}” has an unknown type.

Our experiments are conducted on PEP-3K (Wang et al., 2018) and PAP (Eichel and Im Walde, 2023). We use the values in the column label as ground truth labels. The train- and dev sets of the both datasets are merged for the fine-tuning, each resulting in 4911 and 614 examples. The model is evaluated separately on the test sets of the two datasets, each containing 307 and 308 examples.

In our initial exploration, we investigate the statistic and linguistic characteristics of two datasets to glean insights into their composition and semantic nuances. Our analysis encompasses an examination of the distribution of examples across training, development, and test splits for each dataset. Additionally, we employ word clouds as a visual tool to highlight the most frequently occurring words tied to both plausible and implausible events, offering a vivid portrayal of the datasets’ lexical landscapes. The outcomes of this investigation are presented in Appendix Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge.

We utilize the python library Gensim³³3https://github.com/piskvorky/gensim library to calculate semantic similarities between the sets of most frequently appearing words in the context of plausible and implausible events. This comparative analysis yields intriguing findings: the datasets exhibit contrasting semantic profiles, with the PAP dataset showcasing a pronounced dissimilarity between plausible and implausible terms, while the PEP-3K dataset reveals a striking similarity among its terms. This distinction not only sheds light on the inherent linguistic patterns within the datasets but also correlates with our subsequent experimental observations, where classifiers tend to exhibit enhanced performance on the PEP-3K dataset as opposed to the PAP dataset. Figure 3 depicts words similarities between top plausible and implausible words in the datasets.

We fine-tune RoBERTa-large⁴⁴4https://huggingface.co/roberta-large (Liu et al., 2019) on the merged dataset enhanced by entity type and event type knowledge (Evt+Ent). As ablation study, we also fine-tune the model on the data with only event type knowledge injected (Evt), as well as with only entity type knowledge injected (Ent). Figure 2 illustrates the architecture of our system.

For Evt+Ent and Ent, we fine-tune the model for 10 epochs with batch size 16, using AdamW optimizer (Loshchilov and Hutter, 2017). The training procedure has an initial learning rate of 1e-5, a weight decay of 0.01 and warm-up steps 10. The Evt group has almost the same set of hyperparameter values, except the warm-up steps being set to 100.

The baseline BS-ft is fine-tuned for 10 epochs with batch size 8 using the AdamW optimizer. The learning rate is 1e-5, with a weight decay of 0.01 and warm-up steps 10.

While previous works prefer to use accuracy (Acc) for the evaluation, we also report precision (P), recall (R), F1 and Area Under the Curve (AUC).