Presence or Absence:
Are Unknown Word Usages in Dictionaries?

focuses on the automatic identification of shifts in word meanings over time. For instance, the word ‘chill’ used to mean ‘cold’ for individuals growing up in the 60s, but for those in the 90s, it means ‘relaxed’. Many works proposed to detect meaning shifts by using static or contextualized embeddings (Eger and Mehler, 2016; Hamilton et al., 2016a, b; Martinc et al., 2020; Gonen et al., 2020; Kaiser et al., 2021; Montariol et al., 2021a; Teodorescu et al., 2022; Homskiy and Arefyev, 2022). Most work in LSCD has been done on an unsupervised task formulation (Schlechtweg et al., 2020) which neither involved a dictionary, nor providing interpretation or qualification of detected sense changes. While early work on static embeddings (Kim et al., 2014; Hamilton et al., 2016c) could qualify changes to a certain extent through nearest neighbors, it usually did not provide sense clusters in a dictionary-like manner. More recent work straightforwardly enables the induction of sense clusters through clustering of contextualized embeddings (Giulianelli et al., 2020; Kudisov and Arefyev, 2022; Montariol et al., 2021b; Arefyev and Bykov, 2021). More recently, Ma et al. (2024) presented a graph-based clustering approach to detect gained word senses with low frequency, and offered interpretability by visualizing cross-language semantic changes. The works by Giulianelli et al. (2023); Fedorova et al. (2024a) offer new ways of interpretability such as automatically generating sense definitions for usages from clusters. For an overview of recent model architectures incl. clustering approaches, see Zamora-Reina et al. (2022).

The above-described approaches all have in common that they do not involve a dictionary in their task formulation. However, a variety of dictionaries is available for different languages and time periods (e.g. Dal, 1955; Paul, 2002; OED, 2009) providing valuable information characterizing a language stage on the lexical level. Thus, a possible alternative task formulation for LSCD is to start from an existing dictionary and compare corpus usages against the dictionary entries in order to find usages not covered by the dictionary (Erk, 2006; Lautenschlager et al., 2024).

The shared task provides datasets for the two subtasks for Finnish, Russian and German languages. These datasets contain dictionary entries such as headwords (target words), the definitions of their meanings, word usages, the positions of the headwords within word usages, and time period (indicating whether word usages belong to an earlier or later time period).

For Finnish, the dataset is curated from Vanhan kirjasuomen sanakirja (Dictionary of Old Literary Finnish)¹⁰¹⁰10https://kaino.kotus.fi/vks/ and is split into train, dev and test sets. It includes word usages from earlier and later time periods (before 1700 and after 1700). For Russian, the dataset from an earlier time period is sourced from Explanatory Dictionary of the Living Great Russian Language (Dal, 1955) while the dataset from a later period is from CODWOE (Mickus et al., 2022). Again, the dataset is divided into train, dev and test sets. For German, the dataset is collected from DWUG DE Sense (Schlechtweg, 2023). The German dataset is only available in the test phase, meaning that no train and dev sets are provided. This setup is to put submitted systems to test in handling an unseen language. We provide data statistics for the AXOLOTL24 shared task in Table 3, where the data from earlier and later time periods are treated as two separate corpora.

The baseline system is unsupervised, although it still requires a number of hyperparameters. These hyperparameters include a threshold for the minimum similarity between a word usage and a dictionary definition based on their embeddings, as well as parameters required by Affinity Propagation, such as the choice of distance metrics to compute distances between clusters and the number of clustering iterations. The baseline system sets the similarity threshold to 0.3 and keeps the default parameters of Affinity Propagation unchanged for all languages. For our submitted system, two predefined hyperparameters are needed: the similarity threshold as for the baseline system, and the number of nearest neighbors required for generating a semantic graph and computing distances between clusters. After tuning on the development sets, we set the similarity threshold to 0.5 and the number of nearest neighbors to 5 for all languages. On a side note, the baseline system uses the sentence-level encoder LEALLA (Mao and Nakagawa, 2023) to produce word usage embeddings while our system uses the word-level encoder m-BERT (Devlin et al., 2019) to produce both word and word usage embeddings.

The baseline system is supervised and finetunes the model parameters of XGLM Lin et al. (2022) in the task of generating definitions for word usages. Doing so requires several hyperparameters, including learning rate and weight decay for the Adam optimizer Kingma and Ba (2014), and the number of epochs for training. The baseline system uses the default parameters of the Adam optimizer and sets the number of epochs to 1. Our submitted system, on the contrary, is fully unsupervised. For each target word, we take a set of word usages identified by our clustering approach in Subtask 1 and prompt LLMs to generate a collective definition for the usages of the target word. A predefined prompt is needed and we provide it in Figure 6. For LLMs, we experiment with GPT-3.5-turbo and GPT-4-turbo, LLaMA-2-7B and LLaMA-3-8B.

Note that our prompt is created from scratch and refined on a small selection of random instances in the development sets, meaning that our prompt is not optimal for the entire sets in any language. Our refinement process starts with an English prompt to instruct LLMs to generate Finnish, Russian and German definitions; however, LLMs often generate English definitions for non-English word usages; we address this by translating the English prompt into Finnish, Russian and German via Google Translate. Other factors for refinement include (a) the length of a definition, (b) determining when to stop generation in order to ensure that generated definitions are comparable in length to the ground-truth counterparts, and (c) the number of word usages for LLMs to generate a collective definition.

For Subtask 1, the Adjusted Rand Index (ARI) (Hubert and Arabie, 1985) and macro-F1 score are the two evaluation metrics for reporting and comparing system performances. ARI calculates how much a pair of word usages from the predictions belong to the same sense ID (or different sense IDs) as they should, while the macro-F1 score computes the precision and recall of word usages for each sense ID and then averages these scores across all sense IDs. Note that F1 only considers old senses in the “new” time period, meaning that mappings of word usages to novel senses are not evaluated. ARI considers both novel and old senses in the “new” time period.

For Subtask 2, generated definitions for those usages with novel senses are compared to their ground-truth counterparts by computing similarities between definition pairs. The AXOLOTL-24 shared task uses both lexical-based and embedding-based metrics to compute definition pair similarities. The metrics considered are BLEU (Papineni et al., 2002) and BERTScore (Zhang* et al., 2020). Other metrics appropriate for doing so include MoverScore (Zhao et al., 2019), BlonDe (Jiang et al., 2022) and DiscoScore (Zhao et al., 2023). The latter two metrics have shown to be well-suited for computing long-text pair similarities, particularly useful when dealing with lengthy definitions.

We made two submissions for Subtask 1, with minor difference between them. The only difference is that the second submission includes additional predictions for the unseen German language. Table 4 compares the results of our system and other teams. We see that our system, based on the unsupervised graph-based clustering approach, outperforms the unsupervised baseline system by a large margin in all the languages. We observe a big performance drop for the German language compared to other two languages. One of the reasons for this is due to historical data issues. Unlike the Russian and Finnish corpus usages—which have been carefully preprocessed by AXOLOTL-24 organizers, German usages are not cleaned up and contain spelling variations (e.g., nöthig instead of nötig), OCR errors, escaping double quotes and others. These issues would incur out-of-vocabulary tokens, potentially resulting in poor performance.

Lastly, our system performs poorly in terms of ARI on both Russian and German test sets, despite having better scores in macro-F1. The performance gap between F1 and ARI attributes to the scope mismatch between the two metrics: new sense IDs are excluded when computing F1, whereas both old and novel sense IDs are considered when computing ARI. This means unlike ARI, F1 would not penalize wrong prediction of novel sense IDs. As a result, although our system performs poorly for novel sense predictions in Russian (see the ARI_new result in Table 6), the F1 result (F1$=$0.570) is still quite high.

Note that the results from our system and other teams are not directly comparable as the system details of other teams are missing. For instance, it remains unclear whether their systems are unsupervised or not. Overall, we see the deep-change system achieves the best performance in all the three languages (including the unseen German language where train and dev sets are unavailable); however, their achievement is made through a total of 33 submissions and the leaderboard only reports their best performance; this indicates overfitting.

We refined our prompts for instructing GPT-3.5-turbo. This results in three submissions we made for Subtask 2, where the prompts in our final submission yield the best performance on the randomly selected instances from the Finnish and Russian dev sets. Note that the final prompts are the Finnish, Russian and German translations from the English version (see Figure 6).

Despite not using train sets, our unsupervised system, based on GPT-3.5-turbo, considerably outperforms the supervised baseline system in all setups (see Table 5). This might be because the train sets are not large enough for fine-tuning XGLM Lin et al. (2022). When compared with other teams, our system ranks first for Finnish and German, and ranks second for Russian. Again, it is unclear whether other teams take advantage of the train sets, and thus the direct comparison with other systems is not meaningful.

Figure 7 compares the results of several LLMs. Overall, we observe that our unsupervised system based on LLMs greatly outperforms the supervised baseline system in terms of BERTScore. However, our system performs worse than the baseline in BLEU. This is because our generated definitions are not lexically but semantically similar to their ground-truth counterparts. The reason for this is the following: BLEU cannot recognize text pair similarity when there is no lexical overlap between them (Reiter, 2018). This is particularly problematic when dealing with morphologically rich languages like Russian and Finnish. In such languages, high-quality generated definitions might differ greatly from ground-truth definitions in morphological forms; in this case, BLEU would wrongly assign low scores to high-quality definitions due to the absence of lexical overlap. This is demonstrated by our results, where BLEU scores (0.02-0.03) mean very few lexical overlaps between the generated and ground-truth definitions while BERTScore (0.65-0.67) suggest that definition pairs are indeed semantically similar.

Additionally, we observe the supervised baseline system performs best in terms of BLEU, particularly for Russian. This means the generated definitions are lexically similar to the ground-truth. This might be attributed to the memorization of training sets. We see that many ground-truth definitions contain words from corpus usages. During training, the baseline system might have learned to prioritize the use of words from corpus usages when generating definitions. Lastly, although GPT-4-turbo has shown to greatly outperform GPT-3.5-turbo in many NLP tasks, we demonstrate that the superiority of GPT-4-turbo is not considerable in Subtask 2, especially for Russian, so is the case for LLaMA-2-7B and LLaMA-3-8B.

The datasets provided in the shared task are quite small and contain very few word usages for each headword on average. This is indeed expected as the datasets are sourced from hand-crafted dictionaries where lexicographers only collect a small number of word usages for each dictionary sense entry due to the costly mapping process. Here we argue that it would be better to use such datasets only for evaluation purposes, rather than for dividing them into train sets. Furthermore, we call for an additional database containing a large amount of word usages for each headword to support the development of unsupervised systems, as we see their potential demonstrated by our unsupervised system, which greatly outperformed the supervised baseline system in Subtask 2.

Our system relies on m-BERT (Devlin et al., 2019), a text encoder invented five years ago, to produce embeddings for both word usages and words in Subtask 1. In recent years, many text encoders (Ni et al., 2022; Neelakantan et al., 2022) have been introduced and shown to perform much better than m-BERT in various NLP tasks. Other encoders such as XL-LEXEME (Cassotti et al., 2023) specialized in capturing lexical semantic changes also meet our needs.

The works by Balloccu et al. (2024); Ravaut et al. (2024) show that the results of LLMs can be misleading due to the data contamination issue, i.e., that test sets are included in the training data of LLMs. This issue might be present in the AXOLOTL-24 test sets for the two reasons: (a) the source base of the test sets is publicly accessible and (b) LLMs do not document their training data at all. Thus, it is unclear whether the headwords, word usages, and definitions in the test sets have been exposed to LLMs. Future work should design a measure to calculate data contamination rates of LLMs on the AXOLOTL-24 datasets.