LoFTI: Localization and Factuality Transfer to Indian Locales

For dataset creation, we compile pairs of entities $(e_{\text{ref}},e_{\text{tar}})$, where $e_{\text{ref}}$ is an entity from a reference location outside India and $e_{\text{tar}}$ is an entity from India that serves as a suitable substitute for $e_{\text{ref}}$. These pairs are curated by human annotators and cover diverse categories and hyperlocal regions.

Given the reference entity $e_{\text{ref}}$, a fact-based reference text $T_{\text{ref}}$ is obtained from the entity’s description on the internet. We use the Google API Client or Wikipedia for this purpose. If no entity description is found from these sources, human annotators are tasked with providing the reference text.

Given a reference text $T_{\text{ref}}$ and a target entity $e_{\text{tar}}$ (paired with $e_{\text{ref}}$) from a target location $L_{\text{tar}}$, text localization aims to generate a target text $T_{\text{tar}}$ localized to $L_{\text{tar}}$ that retains the stylistic and semantic features of $T_{\text{ref}}$. This process involves localizing the entities and facts present in $T_{\text{ref}}$ while ensuring factual correctness. For text localization, we employ the Mixtral-8x7b-instruct-v0.1.Q4_K_M model. Given the target location $L_{\text{tar}}$, target entity $e_{\text{tar}}$, and the reference text $T_{\text{ref}}$, we prompt the Mixtral model to generate the localized target text $T_{\text{tar}}$. The prompt used for text localization is given in Figure A2.

In addition to the reference and target text pairs, LoFTI also contains questions that capture common aspects shared by $T_{\text{ref}}$ and $T_{\text{tar}}$. Given a pair of text $(T_{\text{ref}},T_{\text{tar}})$, we generate these questions by identifying shared properties or descriptions of the entities mentioned in the text pairs. We use few-shot prompting on Mixtral-8x7b-instruct-v0.1.Q4_K_M model for common question generation and the prompt used is given in Figure A1.

To ensure the correctness of the LoFTI dataset, all the generations were carefully checked by human annotators at each stage. These annotators represent diverse demographics and have knowledge about samples from different geographic and hyperlocal regions. Each sample undergoes verification by three annotators. Guidelines used by the human annotators at each stage are detailed in Appendix A.1.

To evaluate entity correctness of a generated target text, the human annotator checks if the entity present in the target text is correctly localized to a target location given the reference entity in the reference text. Note that there can be multiple correct localized entities for a given target location. If the entity localization is correct, then a score of 1 is assigned, else it is 0. Thus, for each generated target text, $T^{i}$, a binary score $E^{i}$ is assigned. Across $N$ generated target text sequences, the entity correctness metric is computed as $\textbf{EC}=\frac{1}{N}\sum_{i=1}^{N}E^{i}$.

For each target text with $\text{EC}=1$, the common questions present in LoFTI are further used to evaluate the localization capability of the model. Human evaluators check if the target text correctly answers the common questions given the target location. Each question is evaluated separately and they assign a binary score of 1 if it is answered correctly, else it is 0.

For a generated target text $T^{i}$, let the number of predefined common questions be $m_{i}$ and the binary scores for these questions be $\{C_{j}^{i}\}_{j=1}^{m_{i}}$. Then, the common question correctness metric across $N$ texts is calculated as $\textbf{CQ}=\frac{1}{\sum_{i=1}^{N}m_{i}}\sum_{i=1}^{N}\sum_{j=1}^{m_{i}}C_{j}^{i}$. This metric aggregates the scores across all questions for all target texts, providing an overall measure of the model’s effectiveness in generating contextually accurate and relevant responses.

For each target text $T^{i}$ with $\text{EC}=1$, the human annotator checks if every detail in the text is factually correct and provides a binary score $F^{i}=1$ if every fact is correct, else $F^{i}=0$. The factual correctness metric across $N$ texts is calculated as $\textbf{FC}=\frac{1}{N}\sum_{i=1}^{N}F^{i}$.

We evaluate the performance of two state-of-the-art LLMs on LoFTI: Mixtral (Jiang et al., 2024) and GPT-4 (OpenAI, 2023). The Mixtral-8x7B LLM (Jiang et al., 2024) is a pre-trained generative sparse mixture-of-experts model. It has a decoder-only architecture with its feedforward block selecting from a set of 8 distinct groups of parameters. For our analysis, we utilize the quantized Mixtral model Mixtral-8x7b-instruct-v0.1.Q4_K_M²²2The quantized Mixtral-8x7b models Q6 and Q8 gave similar performance to Q4. with zero-shot prompting. Interestingly, we observed that few-shot prompting did not improve performance compared to the zero-shot setting and adding more localization examples appeared to confuse the model. We also evaluate the performance of the state-of-the-art GPT-4 model on LoFTI. We use the same prompt for both Mixtral and GPT-4 (detailed in Appendix A.7).

Table 2 compares the performance of state-of-the-art models GPT-4 and Mixtral on LoFTI using all three metrics that collectively help evaluate localization and factual transfer. We observe that GPT-4 significantly outperforms Mixtral in EC, FC, and CQ by 17%, 14%, and 27%, respectively. As hyperlocal scores increase, both models show a decline in accuracy, indicating the difficulty in recalling knowledge about these locales. Benefiting from training on diverse and larger datasets and tasks, GPT-4 surpasses Mixtral by achieving superior localization even at higher hyperlocal scores, resulting in more accurate outputs.

Table 3 shows examples highlighting the limitations of both Mixtral and GPT-4 at different hyperlocality values. For the example from hyperlocal score=1 (India), we observe that both models localize well but Mixtral tends to make errors in the factual details (e.g., the height of the waterfall). For hyperlocal score=2 (Maharashtra), Mixtral tends to hallucinate and creates an imaginary entity (“Padmashri Rahul Aware") while GPT-4 localizes correctly. For hyperlocal score=3 (Khandra), both models fails to localize the reference entity “Eric Otto Valdemar Lemming" correctly. Mixtral returns an entity from a different category and location (“Surendra Kumar Singh" is a politician from Madhya Pradesh), while GPT-4 returns an entity from the correct category but a different location (“Milkha Singh" is a track-and-field athlete from Chandigarh).

In Table 4, we compare the performance of Mixtral, Mixtral + RARR, Mixtral Revised, and GPT-4 on a subset of 250 randomly chosen samples from LoFTI using human evaluators.³³3We restrict this evaluation to a 250-sample subset due to annotation costs. However, we point to the Mixtral and GPT-4 scores in Tables 2 and 4 which are very similar, thus affirming that the 250-sample subset of LoFTI is representative of the full set. Attribution using factual evidence helps Mixtral + RARR in improving Mixtral generations, especially in the CQ and FC metrics, where the scores improve by $9\%$ and $13\%$, respectively. However, the length of text obtained by RARR attribution is usually more than the original length of the text, and it fails to preserve the style.

Mixtral Revised utilizes factual evidence similar to RARR but regenerates the text instead of editing it. Including factual evidence in the prompt enhances the Mixtral outputs and results in improvements in both FC and CQ. The approach focuses mainly on revising the factual correctness of the text while largely retaining the entity present in it. However, we still see an enhancement in EC as factual evidence provides a richer context for the effective localization of the entity. While both Mixtral Revised and Mixtral + RARR use evidence, the former re-generates the text and the latter edits the text by retaining the entity. Re-generation helps in obtaining a factually correct entity. GPT-4 surpasses all the Mixtral models due to its extensive training and diverse world knowledge. With increasing hyperlocal scores, even with GPT-4, performance degrades. Nonetheless, the revision step in Mixtral Revised significantly improves the scores across all metrics, particularly for regions with a hyperlocal score of 3.

In Table 4, we also analyze the capability of GPT-4 as an evaluator for the task of localized text transfer. Compared to humans, GPT-4 shows a $0.10-0.15$ increase across all the metrics and models, due to the presence of false positives. Table 5 illustrates this limitation using an example. Mixtral hallucinates and returns the entity "Mystic Moods". GPT-4 incorrectly claims it is a factually correct localization and assigns a score of 1 for all the metrics. The comparison clearly shows that GPT-4 is not a reliable evaluator for absolute numbers. However, we observe similar overall trends in both human and GPT-4 evaluations. This shows that GPT-4 could be used as an LLM evaluator for localized text transfer to study the trends across models. Table 6 shows a detailed example for all the models discussed.

LoFTI can also be used as a benchmark to evaluate localized question answering. Given a target location and a question, the model has to generate text that answers the question while being correctly localized to the given target location. To aid this task, we also provide the reference location and the reference text as an example to guide localization and the style of generation.

Table 7 shows the performance of promoting Mixtral on this benchmark task. Mixtral obtains accuracies of 64%, 63%, and 59% on the EC, CQ, and FC metrics, respectively. Consistent with our previous observations, the model encounters challenges in efficient localization as hyperlocal scores increase. Some examples of Mixtral generations are shown in Table 8. In Table 7, we also discuss the performance of GPT-4 as an evaluator for this benchmark task. GPT-4 nearly matches human evaluation when targeting India as a whole (hyperlocal score = 1), but highly overestimates scores for regions with hyperlocal scores of 2 and 3. The overall trends of human evaluation are maintained by GPT-4. We also show this comparison on the full LoFTI dataset for the Mixtral model in Table  A7.