Can Pre-trained Language Models Understand Chinese Humor?

Humor is an advanced language art prevalently used in human languages. However, it is very challenging to let machines possess a sense of humor as humans, since it requires a deep understanding of semantics as well as cultural background. Nowadays, as the development of human-machine interaction systems and applications, how to let machines have a sense of humor has become an increasingly important topic in Natural Language Processing (NLP). Its success or failure may potentially forecast whether a Babel of human-machine interaction could finally be built.

Due to its importance, great efforts has been made on humor-relevant tasks in the NLP community, which mainly focuses on Humor Recognition and Humor Generation. Early work mainly relies on shallow linguistic features and templates to recognize or generate humors. For instance, Cattle and Ma (2018) and Yang et al. (2015) recognize humor with words associations and the latent semantic structures, while Aggarwal and Mamidi (2017) and He et al. (2019) generate poetic three liner jokes and puns through analyzing the structure and retrieve-and-edit approach, respectively. However, these methods rely on unaffordable human cost to design features or templates for different datasets, which can only recognize or generate a very limited range of humorous expressions.

As the popularity of pre-trained language models (PLMs), some recent work, such as Yu et al. (2018) and Rodriguez et al. (2021), make preliminary attempts to finetune PLMs for humor recognition and generation. Thanks to the powerful understanding and generation capabilities that PLMs have learned from massive amounts of data, they significantly reduce human cost and enable the recognition (or generation) on more types of humorous expressions. However, these simple endeavors do not substantially answer an important question: whether PLMs are capable of humor understanding?

This is a deep question worth exploring, which should be answered firstly when we utilize PLMs for various humor understanding and generation tasks. To answer this question, we would like to investigate the humor understanding ability of PLMs in the following several aspects: i) Whether PLMs can understand humor before or after fine-tuning? ii) Whether existing external knowledge can help improve PLMs’ humor understanding ability? iii) Whether PLMs can detect interpretable clue words that fit human intuitive understanding of humor? To this end, we need a well-designed evaluation framework and corresponding comprehensive dataset, both of which cannot be directly obtained from the existing humor-relevant tasks  (Cattle and Ma, 2018; Yang et al., 2015; Aggarwal and Mamidi, 2017; He et al., 2019).

In this paper, we first propose a three-step evaluation framework, each step of which is responsible for answering one of the above questions. Next, within this framework, we employ four representative humor-relevant tasks to conduct the systematically evaluation on PLMs, including humor recognition, humor type classification, humor level classification and punchline detection. Meanwhile, we construct a comprehensive Chinese humor dataset, which fully meets all data requirements of the four tasks and three steps evaluation framework. We choose to construct the Chinese humor dataset since Chinese humor is as worthy of study as English humor and more challenging. However, the existing Chinese humor datasets^1,2 are far less abundant than the English humor datasets, thus we want to fill in the gap for Chinese humor research.

Our empirical study based on this Chinese humor dataset suggests that: 1) By fine-tuning on the constructed humor dataset, the humor understanding ability of PLMs has been greatly improved. 2) Some external knowledge, such as Chinese pinyin information, has a positive effect on improving the PLMs’ performance on humor-related tasks. 3) Moreover, a portion of the detected clue words are considered being in line with human perception of humor, but there is much room for improvement in PLMs’ ability to understand humor. To summarize, our contributions in this paper are threefold:

• •

  We are the first work to systematically evaluate the ability of PLMs in understanding Chinese humor. For this purpose, a comprehensive framework with three evaluation steps and four evaluation tasks is designed.

• •

  We construct a more comprehensive Chinese humor dataset compared with the prior research, which fully meets all the data requirements of the proposed evaluation framework.

• •

  Our empirical study justifies the positive effect of fine-tuning and external knowledge for PLMs on humor-relevant tasks, which has important guiding value for future optimization of PLMs in humor understanding and generation.

In this paper, we focus on evaluating the ability of PLMs in understanding humor. Only when PLMs are capable of understanding humor can they generate more reasonable humorous texts, so we leave the investigation of the humor generation ability of PLMs for future work. As depicted in Fig. 1, we propose a comprehensive evaluation framework with three evaluation steps based on four evaluation tasks to investigate the capability of PLMs in understanding humor. In the following, we introduce them in detail.

To fully meet all data requirements of the evaluation framework, we construct a large-scale Chinese humor dataset, which consists of four sub-datasets as depicted in Table 1.

Humor Recognition Sub-dataset. The humor recognition sub-dataset contains humorous texts mainly from released data^1,222https://github.com/liuhuanyong/ChineseHumorSentiment and humorless text crawled from various platforms. On another hand, to construct negative examples, we mainly crawl short monologues or dialogues, such as hint fictions, fables, celebrity stories, or monologue-based diaries as the humorless texts. Such texts have similar language styles with the humorous ones. The lengths of the texts are ranged from 20 to 200 characters. Each humorless text is tagged by three recruited human volunteers to evaluate whether this piece of text is actually humorless. Texts with controversial labels given by the three volunteers will be discarded.

Humor Type Classification Sub-dataset. The humor type classification sub-dataset contains three types of humorous texts:

• •

  Harmonic humor: it means a word-pair has a similar pronunciation but different meanings in a piece of humorous text, such as {CJK*}UTF8gkai“季付 (meaning: pay quarterly, Chinese pinyin: jifu)” and “继父 (meaning: stepfather, Chinese pinyin: jifu)”,  “月付 (meaning: pay monthly, Chinese pinyin: yuefu)” and “岳父 (meaning: father-in-law, Chinese pinyin: yuefu)” in the first row of Table 3 ;

• •

  Ambiguous humor: it means at least two definitions of a word are simultaneously used in a piece of humorous text. For example, {CJK*}UTF8gkai “十分” can mean “very” and “ten scores” at the same time. in the second row of Table 3 ;

• •

  Incongruous humor: it means there is a semantic incongruity in a humorous text, which doesn’t follow humans’ expectation. For example, in the third row of Table 3, {CJK*}UTF8gkai “你看这个姑娘就很有素质一直很冷静嘛” which means “the girl is calm without dispute”, and the response is {CJK*}UTF8gkai “我住30楼，跑下来累了歇会儿再骂你” which means “the girl keeps calm because she is too tired to quarrel”. Such response has semantic incongruity which doesn’t follow humans’ normal expectation.

We first collect the data from the release data^1,2 which already have labels. In order to guarantee the accuracy of the type labels, we proofread and then discard the wrong ones with the help of the recruited three human raters.

Humor Level Classification Sub-dataset. The humor level classification sub-dataset classifies humorous texts into three levels of humor: weak, medium, and strong. We collect the data from the same places as the humor type classification sub-dataset, and also perform similar manual proofreading operations to guarantee the accuracy of the labels.

Punchline Detection Sub-dataset. Based on the humorous text in humor recognition sub-dataset, we construct humorous and humorless context-ending pairs.

The humorous ending is extracted by dividing each humorous text into two parts: previous context and punchline. According to the theories of humor (Perks, 2012; Berger, 1987; Forabosco, 1992; Spence, 2006; Wen et al., 2021), the reason why humor introduces laughter is that a piece of text presents an unexpected sentence which is incongruous with the previous context. Thus, we extract the unexpected sentence in a humorous text as the punchline with the help of human annotation. Specifically, we first enroll another three volunteers, and each of them is required to vote punchline sentences for all humorous texts. We then choose the sentence which has the highest vote as the final punchline sentence for each humorous text and discard the following content in this piece of text. If more than one sentences have equal high votes, we discard this piece of humorous text.

The normal ending is generated by a text generator. We input the previous context into a large language model such as CPM (Zhang et al., 2021b, a), and generate a normal ending which has similar length with the punchline. We also carry out human evaluation and machine evaluation to guarantee the quality of the generated normal endings. The steps of quality evaluation are as follows: i) We first enroll another three volunteers and randomly select 3,000 normal endings. Each of the volunteers needs to give a rating for the overall 3,000 normal endings based on a scoresheet shown in Table 2. We calculate Inter-rater agreement of Krippendorff’s Alpha (IRA) to ensure the confidence of human ratings. For the controversial ratings which have low agreements ($<$0.7) or a normal ending is rated below 0.85, we re-generate a new normal ending. ii) Next, we use simCSE (Gao et al., 2021) to calculate similarity scores to guarantee the generated normal endings have similar semantics with the corresponding punchlines and the previous contexts. The similarity score between the normal ending and the punchline ending is $s_{1}$, and the similarity score between context-punchline pair and context-normal-ending pair is $s_{2}$. The final similarity score $s$ is the average of $s_{1}$ and $s_{2}$. For some normal endings whose $s$ are below 0.85, we also re-generate new ones.

Fig 3 illustrates our framework for evaluating PLMs’ ability in understanding humor, which consists of three parts: evaluate original/fine-tuned PLMs, evaluate knowledge-enhanced PLMs, and interpret humor understanding in PLMs.

Following the proposed evaluation framework, we carry out experiments to investigate whether the pre-trained language models (PLMs) have the ability of humor understanding (see Sec. 5.1), whether external knowledge can improve their humor understanding ability (see Sec. 5.2), and the interpretability of the detected clue words which lead to PLMs’ correct prediction on humor understanding tasks (see Sec. 5.3).

Experiment Setup. Our experiments are carried on GeForce RTX 3090 GPU (on our machine) and TPU (on Google Colab) with Pytorch in Python. The sequence length is set to 200. We initialize the learning rate to 2e-5 and batch size from 4 to 32 according to the memory of the machine, and use early stopping with 20 epochs.

Baselines, Datasets and Metrics. We adopt some representative PLMs, including base and large versions of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020), T5 (Raffel et al., 2019), CPT (Shao et al., 2021) for humor recognition, humor type classification, and humor level classification tasks. For the punchline detection task, we adopt base and large versions of simCSE-BERT (Gao et al., 2021) and simCSE-RoBERTa (Gao et al., 2021), which are proved to perform better in text matching. We divide each dataset into a training set and a dev set at a ratio of 7 to 3, and make down-sampling for all training and dev set to balance sample numbers in different classes. We use accuracy with percentage as the metric.

Similar to happiness, sadness, and anger, humor is one of the essential emotions of humans. Therefore, we carry out further evaluations to investigate whether PLMs which have been fine-tuned on the humor dataset can achieve better performance on the downstream tasks. We choose four Chinese sentiment classification datasets ⁸⁸8https://github.com/SophonPlus/ChineseNlpCorpus/raw/
master/datasets/: 1) ChnSentiCorp-htl-all (Chn), which is a hotel review dataset with more than 5,000 positive reviews and more than 2,000 negative reviews; 2) Waimai-10k (Wai), which contains 4,000 positive and 8,000 negative user reviews; 3) Online-shopping-10-cats (Shop), which has 30000 positive and 30000 negative user comments with online shopping; 4) weibo-senti-100k (Wei), which has about 50000 positive and 50000 negative comments.

We fine-tune BERT, which has been fine-tuned on our constructed Chinese humor dataset, including all the four sub-datasets. The results are shown in Table 6. The baseline results are based on BERT, which are derived from their published research (Gan et al., 2021; Zhou et al., 2021; Zhang et al., 2020b, a).We observe that the performance increases a little after fine-tuning on the humor dataset (see the row “ft”). The results suggest that the sense of humor has common characteristics with other emotions, which thus humor-fine-tuned PLMs have a positive effect on other sentiment analysis tasks. We also find that there is no further improvement after injecting external knowledge in any above-mentioned ways (see the row “K-ft”). It suggests that Chinese pinyin is a unique and important characteristic for humor understanding, which is not very useful for other relevant downstream tasks.

PLMs demonstrate terrific capabilities on various downstream tasks (Chen et al., 2024b, 2023a, 2023b, 2022, a, 2023d, 2023c). We list two main related work as follows:

Humor Datasets and Corpora. Some research dedicate themselves to construct a large-scale humor datasets and corpora. For example, Engelthaler and Hills (2018) design a humor dataset which provides researchers with a list of humor ratings with 4,997 English words. Chiruzzo et al. (2020) present the development of a corpus of 30,000 Spanish tweets that were crowd-annotated with humor value and funniness score. Hossain et al. (2019) introduce a new dataset called Humicroedit that design simple replacement edits to make English news headlines funny. Hasan et al. (2019) introduce a multimodal English humor dataset to detect humorous expressions in TED talks. Different from the above research, we construct a Chinese humor dataset, which includes four sub-datasets, each of which can be used for one representative humor-relevant task.

Humor Recognition. Other research on humor focus on humor recognition which is to decide whether a given sentence expresses a certain degree of humor. For example, Mihalcea and Strapparava (2005) report text classification techniques are a viable approach to recognize humorous one-liners. Barbieri and Saggion (2014) design several linguistic features to automatically detect irony and humor in twitter. Cattle and Ma (2018) adopt the minimum, maximum, and average Word2Vec similarity between ordered word pairs to recognize humor and extract humor anchor. Yang et al. (2015) investigate the latent semantic structures behind humor in four aspects. Morales and Zhai (2017) propose a generative language model and design some key component to identify humor in reviews. Chen and Lee (2017) use semantic structural features and semantic distance features to predict audience’s laughter in TED Talk Data based on convolutional Neural Network. However, most of them design linguistic features to recognize humor. They ignore the powerful learning abilities of pre-trained language models (PLMs). Different from them, we design a comprehensive evaluation framework to make a research on PLMs’ humor understanding ability.

Humor understanding for PLMs is a challenging research in Natural Language Processing. In this work, we systematically investigate the humor understanding ability of PLMs with a designed comprehensive framework with three evaluation steps and four evaluation tasks. We also construct a comprehensive Chinese humor dataset, and our empirical study on it yields some valuable observations : 1) While the original PLMs can hardly understand humor, they could gain a certain degree of humor understanding ability through fine-tuning. 2) Linguistic knowledge such as Chinese Pinyin has a positive effect for PLMs in humor-relevant tasks. 3) The existing knowledge bases and commonsense bases can not provide much required knowledge for humor understanding. As a future work, we will find ways to collect more cultural knowledge for the optimization of PLMs in humor understanding.

This work was supported by Science and Technology Commission of Shanghai Municipality Grant (No. 22511105902), National Natural Science Foundation of China (No.62072323), Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103), Shanghai Science and Technology Innovation Action Plan (No. 22511104700), and National Natural Science Foundation of China (No. 62102095). Yanghua Xiao is also a member of Research Group of Computational and AI Communication at Institute for Global Communications and Integrated Media, Fudan University.