We translate the full development set and 10% of the training set of English HaDes dataset (Liu et al., 2022) in German, with 1,000 and 876 instances, respectively.¹¹1Since the original test set labels were not published, we rely on training and development sets throughout our experiments. We also ensure the subsample of the training set retains the original label ratio of the training data. Each instance includes a text, marked word spans, position of marked word spans, and label to indicate whether the marked word spans causes hallucination. Examples compared to the original English HaDes dataset are shown in Table 1.

Following the well-established practice (Hung et al., 2022; Senel et al., 2024), we carried out a two-phase translation process: (1) we started with an automatic translation – followed by (2) the manual post-editing of the translations. We first automatically translate the development and training set portions for both text and marked word spans relying on DeepL Translator. We then incorporate native speaker with University degree and fluent in English, to post-edit the automatic translations to ensure the correctness of the translation – especially the directly preceding and succeeding context, and the correct determination of the marked word spans. Common errors identified in machine-translated texts include incorrect translations, missing words, grammatical mistakes, or contextual inaccuracies. Examples comparing the original English HaDes with the automatically translated and manually post-edited texts are shown in Table 2. Besides, as the position of marked word spans changes in the German text²²2German and English, both Germanic languages, differ in ways that impact dataset design. In German, compound words are written as single words, whereas in English, they are separated by spaces, affecting marked word spans. Additionally, German commonly uses particle verbs, where marked word spans are split by other parts of the sentence. In such cases, only the conjugable main part of the verb is marked, while the particle is ignored., the position of marked word spans is adjusted accordingly.

Additionally, to conduct Translate-Train experiment for cross-lingual transfer, full training set (8,754 instances) are automatically translated using DeepL Translator, without post-editing. However, only 6,344 instances (72.5%) remain, since the discarded ones contain incorrect marked word spans.

The parallel nature and substantial size of anHalten facilitate benchmarking of cross-lingual transfer methods for hallucination detection tasks. We investigate three common methods for downstream cross-lingual transfer (XLT) (Ebing and Glavaš, 2023; Senel et al., 2024): (1) Zero-Shot Transfer, where we assume the absence of labeled task instances in the target language. The model is trained exclusively in English and is expected to perform the task directly in German without prior exposure to German labeled data. This method relies on the model’s capability to generalize knowledge from English to German. (2) Few-Shot Transfer, where a limited number of labeled instances in the target language exist with the majority of training data in the source language. The model is trained on abundant English data and a small amount of German data jointly,³³3Compared to sequential fine-tuning (Lauscher et al., 2020; Hung et al., 2022), joint fine-tuning (Schmidt et al., 2022) on instances in both source and target language can achieve better performances with higher stability. helping it adapt to the specific nuances of the German language with limited annotated data. (3) Translate-Train, where training instances in source language are automatically translated (i.e., noisy) to target language leveraging the state-of-the-art machine translation model. While this approach relies on the quality of translation, it benefits from creating a substantial amount of training data in German, closely approximating a fully supervised learning scenario.

To facilitate modular and efficient XLT, adapter-based approach is proposed to learn specialized task and language adapters for high portability and parameter-efficient transfer to various tasks and languages (Pfeiffer et al., 2020b). For downstream XLT, a task adapter is stacked on the pre-trained source language adapter, where the parameters are only updated for the task adapter. During evaluation, the source language adapter is replaced by the pre-trained target language adapter. In our setup, the task adapter is trained by (1) the English-only data for Zero-Shot Transfer; (2) a joint training of English and a small portion of German data for Few-Shot Transfer; or (3) the machine-translated English-to-German data for Translate-Train. The adapter-based approach ensures that the model can efficiently adapt to new tasks with minimal parameter updates, maintaining the balance between performance and computational efficiency.

We evaluate multilingual pre-trained language models (PLMs) in XLT methods (§ 2.2) for token-level reference-free hallucination detection tasks. To simulate real-world applications, we evaluate on two sub-tasks: offline and online (Liu et al., 2022). In the offline setting, the model accesses both preceding and succeeding contexts of the marked word spans, suitable for detecting hallucinations in pre-generated texts. In the online setting, the model considers only the preceding context, enabling proactive prevention of hallucinations during on-the-fly text generation.

We follow Liu et al. (2022) and evaluate the XLT capabilities utilizing multilingual PLMs on hallucination detection tasks. The evaluation metrics include accuracy, precision, recall, F1, Area Under Curve (AUC), G-Mean (Espíndola and Ebecken, 2005), and Brier Score (BS) (Brier, 1950). These metrics provide a comprehensive evaluation of model performance, balancing correctness, and the ability to handle imbalanced classes.

Experiments are conducted on multilingual PLMs, namely multilingual BERT (mBERT) (Devlin et al., 2019) and XLMR (Conneau et al., 2020),⁴⁴4The weights of PLMs are loaded from HuggingFace: multilingual-bert-base-cased and xlm-roberta-base. using language adapters⁵⁵5The pre-trained adapters are selected from AdapterHub (Pfeiffer et al., 2020a) for English (en-wiki@ukp) and German (de-wiki@ukp). proposed by Pfeiffer et al. (2020b) to facilitate modular and efficient XLT.

To evaluate downstream XLT, the experiments are conducted with 5 runs in both offline and online settings, with a fixed context window of 200 tokens. In the online setting, the context includes the 200 tokens preceding the marked word spans. In the offline setting, it includes 100 tokens before and after the marked word spans.⁶⁶6Liu et al. (2022) observed that model performance for English HaDes dataset stabilizes around 80 tokens, with minimal performance differences between offline and online settings regarding context length. Thus, using 200 tokens would not limit performance, and increasing the context is unlikely to improve results. During training, the instances are randomly split into a 70/30 train and validation split, while the original label ratio of training data is retained for the split. We train for 10 epochs in batches of 8 instances, with learning rate $5\cdot 10^{-3}$, and a dropout ratio 0.2 is set to avoid overfitting.