Conversational Query Reformulation with the Guidance of Retrieved Documents

Jeonghyun Park Hwanhee Lee^$\dagger$
Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea
{tom0365, hwanheelee}@cau.ac.kr

Abstract

Conversational search seeks to retrieve relevant passages for the given questions in Conversational QA (ConvQA). Questions in ConvQA face challenges such as omissions and coreferences, making it difficult to obtain desired search results. Conversational Query Reformulation (CQR) transforms these current queries into de-contextualized forms to resolve these issues. However, existing CQR methods focus on rewriting human-friendly queries, which may not always yield optimal search results for the retriever. To overcome this challenge, we introduce GuideCQR, a framework that utilizes guided documents to refine queries, ensuring that they are optimal for retrievers. Specifically, we augment keywords, generate expected answers from the re-ranked documents, and unify them with the filtering process. Experimental results show that queries enhanced by guided documents outperform previous CQR methods. Especially, GuideCQR surpasses the performance of Large Language Model (LLM) prompt-powered approaches and demonstrates the importance of the guided documents in formulating retriever-friendly queries across diverse setups.

Jeonghyun Park and Hwanhee Lee^$\dagger$ Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea {tom0365, hwanheelee}@cau.ac.kr

^†^†footnotetext: ^$\dagger$Corresponding author.

Refer to caption — Figure 1: Overall framework of GuideCQR: We visualize content of top-1 ranked document. For easier understanding, we show keywords augmented from the top-1 document and 3 answer pairs from the top-3 document.

1 Introduction

In the context of conversational question answering (Zaib et al., 2022) task (ConvQA), the objective of conversational search is to retrieve passages that contain the necessary information to answer the current query within a multi-turn conversation. To understand the intent of the query, previous research focuses on transforming queries into stand-alone forms, making them independent and robust. Mo et al. (2023) This process, known as CQR allows for clearer expression and better understanding the query.

Generally, CQR methods fall into two categories: rewriting and expansion techniques. In the current era where LLMs dominate, leveraging LLMs for CQR methods is particularly prevalent. Jagerman et al. (2023) involves instructing a LLM to generate relevant passages related to the query. This method makes the query more reasonable and human-understandable by expanding the sentence. (Mao et al., 2023) also involves using LLMs to rewrite the query through different prompting methods and aggregation techniques.

However, these CQR methods focus on reconstructing queries to be the most understandable and reasonable only for humans. Although these approaches aim to create human-friendly and easily comprehensible queries, they may not always yield the optimal results that retrievers desire. To optimize CQR methods for their primary objective of generating the most effective search queries, it’s necessary to prioritize retriever-friendly queries over human-friendly ones.

In this paper, we propose GuideCQR, an effective framework designed to generate retrieval-friendly conversational queries by leveraging the guidance of retrieved documents. We find that the guided documents can provide a set of documents that enhance the query, optimizing query for better search performance. By using the signals with the related document set, we argue that we can make retriever-friendly query that enhances search outcomes. Inspired by this point, we first obtain the guided documents through a retrieval process using the baseline query reformulated by LLM. To extract more accurate signals for making retriever-friendly query, we also conduct a simple re-ranking process to make a better guided documents, refining the order of documents based on the similarity between query and document. And then, we reformulate the baseline query by augmenting keyword and generating expected answer. Finally, we filter both of them based on similarity with query and history utterances to remove redundant signals.

Our experimental results demonstrate that GuideCQR shows the competitive performance on widely used datasets compared to several contemporary methods. We observe that GuideCQR consistently achieves remarkable scores across various metrics. Specifically, GuideCQR outperforms all baselines on the CAsT-19 dataset for average score, even better than LLM-prompt-leveraged methods. It demonstrates the robustness of the GuideCQR framework in making queries more retriever-friendly. This significantly enhances search results in the field of CQR systems. Additionally, the ablation study confirms the importance of the guided documents and the steps involved in creating retriever-friendly queries, including keyword augmentation and expected answer generation.

2 GuideCQR

We present GuideCQR, a novel framework to reformulate conversational queries by leveraging the guidance of retrieved documents. Figure 1 illustrates an overview of our proposed query reformulation framework. The objective of GuideCQR is to make a retriever-friendly query by utilizing the guidance of retrieved documents. To achieve this, we structure GuideCQR into three stages: We begin by retrieving a guided documents based on the simple LLM-reformulated query. This step involves obtaining a set of documents. And we re-rank the retrieved documents to prioritize documents. We consider only the most relevant ones in further stages. (Sec 2.1) And then we augment keywords and generate expected answers from the documents, creating components that contribute to making the query more retriever-friendly. (Sec 2.2) Finally, we apply a filtering process that evaluates both keywords and answers based on their similarity scores relative to the baseline query and dialogue history. Then, we unify and concatenate them back to the baseline query. (Sec 2.3)

2.1 Guided documents Retrieval

2.1.1 Initial documents Retrieval

In the process of developing a retriever-friendly query, initially retrieved documents can play a crucial role as guiding resources, by providing foundational insights for the CQR process. For example, these signals can include critical keywords or contextual signs that are necessary to the search, such as "cure," "rate," and "curable" from the document shown in Figure 1. Inspired by these points, GuideCQR starts from getting guided documents to get meaningful signals to the retriever. We obtain the guided documents by retrieving the documents using the baseline query set generated by LLM. We denote baseline query as $Q_{baseline}$ ,

Q_{baseline}=Rewrite_{LLM}(RawQuery),

(1)

where $Rewrite\_LLM$ represents an operation that resolves omissions or coreferences using LLM and $RawQuery$ denotes the raw question in the dataset. Using the $Q_{baseline}$ , we obtain the guided documents composed of 2,000 documents for each sample ID.

2.1.2 Re-ranking documents

We further enhance the quality of guided documents by incorporating a re-ranking process to get the final guided documents. By adjusting the order of previously retrieved documents, we aim to capture better signals that enhance the search result. We employ Sentence-Transformer Reimers and Gurevych (2019) to re-rank 2,000 documents for each sample ID, carefully selecting the final guided documents with the top 10 documents by measuring the similarity between the query and each document. To mitigate potential biases arising from using a single model, we perform an additional re-ranking process with a different Sentence-Transformer.

2.2 Generating Retriever-friendly Query

Guided documents plays an important role in making retriever-friendly queries since this documents is retrieved by retriever so that we can extract retriever-friendly components from the documents. Based on the final guided documents, we create retriever-friendly queries with respect to two aspects: Augmenting Keyword, Expected Answer generation.

2.2.1 Augmenting Keyword

Keywords can be crucial signals for creating retriever-friendly queries, as these keywords represent the most significant and relevant terms within the document. So, we utilize the guided documents to augment keywords by two aspects. The first aspect focuses on determining the number of documents for augmenting keywords, which depends on the extent of guidance we seek. Second aspect utilizes the length of the keywords, specifically how long the keywords we choose to use. Based on this, we augment keywords K:

K=[k_{1},k_{2},k_{3},k_{4}...,k_{n}],

(2)

where k denotes unit keyword and n is the number of documents. For example, keywords such as "early", "throat", "cancer", "high", and "cure" can be crucial for the document shown in Figure 1.

2.2.2 Expected Answer Generation

Additionally, expected answers related to the query can also be crucial signals because they offer direct insights into the user’s desired outcomes, thereby enhancing the relevance and accuracy of the search outcome. Therefore, we also utilize the guided documents in generating expected answers as additional information to the query:

A=[a_{1},a_{2},a_{3},...,a_{n}],

(3)

where A represents answer list and $a_{n}$ denotes a unit answer extracted from a single document. In this process, we require both the query and the context. We use the query as the $Q_{baseline}$ , and the context as the top-k re-ranked guided documents. Conditioned on this documents, we then generate the single expected answer for each document such as "throat cancer has a high cure rate" for document shown in Figure 1 and merge all answers in a single sentence. Because the spans of each answer are all important, we regard the entire set of answers as a single unified answer rather than considering each one individually.

2.3 Filtering And Unify

We find that redundant elements can arise from the augmented keywords and the generated answer such as "early" shown in Figure 1. We explain this is because GuideCQR might augment the keywords or answers generated from irrelevant documents. Therefore, we conduct an additional filtering stage to effectively filter keywords and answers in the reformulated query. We define the FilterScore based on cosine similarity as follows:

\textit{QueryScore}=10(1-\textit{CosSim}(\textit{query},\textit{item})),

(4)

\textit{HistoryScore}=\text{Max}\left(10(1-\textit{CosSim}(\textit{history}[i]% ,\textit{item}))\right),

(5)

\textit{FilterScore}=\frac{\textit{QueryScore}+\textit{HistoryScore}}{2},

(6)

where query represents the embedding of the current turn’s query, and item refers to the embedding of either a keyword or an answer sentence, history is history query list of current utterance and CosSim denotes cosine similarity. We define the HistoryScore as the maximum cosine similarity value of the dialogue history query and words. Finally, we define the FilterScore as the average of the QueryScore and the HistoryScore, ranging from 1 to 10. History queries are important for understanding the current query because, in a dialog, it is more effective to comprehend the question based on the previous conversation. So we also consider history query by $HistoryScore$ , allowing us to traverse the entire dialogue from past to present. We remove keywords and answers that fall below a specified $FilterScore$ . Finally, we unify the remaining keywords and answers and integrate them into the $Q_{baseline}$ to construct the final reformulated query as follows:

Q_{final}=Concat([Q_{baseline},K_{filtered},A_{filtered}]),

(7)

where $K_{filtered}$ and $A_{filtered}$ are remaining keywords, answers.

	CAsT-19		CAsT-20
Methods	MRR	NDCG@3	MRR	NDCG@3	Avg
Transformer++	0.696	0.441	0.296	0.185	0.404
CQE-Sparse	0.671	0.423	0.423	0.271	0.447
QuReTeC	0.689	0.430	0.430	0.287	0.459
T5QR	0.701	0.417	0.423	0.299	0.460
ConvGQR	0.708	0.434	0.465	0.331	0.484
CONVINV	0.742	0.449	0.476	0.334	0.500
$Q_{baseline}$	0.722	0.452	0.534	0.372	0.520
Humans	0.740	0.461	0.591	0.422	0.553
LLM4CS	0.776	0.515	0.615	0.455	0.590
GuideCQR (ours)	0.820	0.544	0.581	0.426	0.592
- w/o keywords	0.770	0.498	0.538	0.403	0.552
- w/o answers	0.806	0.526	0.549	0.398	0.569
- w/o both	0.726	0.458	0.545	0.387	0.529

Table 1: Performance comparison on CAsT datasets: MRR, NDCG@3, and the final average of the four scores. The best results are in bold, while the second best are underlined.

3 Experiments

To demonstrate the effectiveness of GuideCQR, we present a comparative analysis over existing methods, including LLM-prompt powered and human-rewritten queries. We focus on key metrics that are critical for evaluating conversational search.

3.1 Datasets and Metrics

We use the TREC CAsT-19 (Dalton et al., 2020) and TREC CAsT-20 (Dalton et al., 2021) datasets, widely regarded as public conversational search datasets. Human experts from the TREC Conversational Assistance Track (CAsT) curate these datasets. CAsT-19 and CAsT-20 consist of 50 and 25 conversations, respectively. Each conversation comprises turns filled with complex, context-driven queries that often refer to prior interactions within the same session. Both datasets share the same document collection and provide passage-level relevance judgments, as well as human rewrites for each turn. Following methods from prior research (Mo et al., 2023) (Mao et al., 2023) (Jang et al., 2023), we use two evaluation metrics to assess the performance: Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain at three documents (NDCG@3). Also, we use pytrec_eval (Van Gysel and de Rijke, 2018) tool to calculate each score.

3.2 Baselines

We compare GuideCQR with the following query reformulation (QR) baselines: (1) Transformer++ (Vakulenko et al., 2021a): A GPT-2 based QR model fine-tuned on CANARD dataset. (2) CQE-Sparse (Lin et al., 2021b): A weakly-supervised method to select important tokens only from the context via contextualized query embeddings. (3) QuReTeC (Voskarides et al., 2020): A weakly-supervised method to train a sequence tagger to decide whether each term contained in a historical context should be added to the current query. (4) T5QR (Lin et al., 2020): A conversational query rewriter based on T5, trained using human-generated rewrites. (5) ConvGQR (Mo et al., 2023): A query reformulation framework that combines query rewriting with generative query expansion. (6) CONVINV (Cheng et al., 2024): framework that transforms conversational session embeddings into interpretable text using Vec2Text. (7) LLM4CS (Mao et al., 2023): query rewriting based on LLM and various prompting methods. We build GuideCQR upon this prompted QR method using (REW + Maxprob) and evaluate using Self-Consistency.

3.3 Implementation Details

As a baseline, we generate the $Q_{baseline}$ using OpenAI 3.5-turbo-16k combined with the Maxprob approach in LLM4CS. We simply generate this query by sampling the highest generation probability with LLMs, resolving only omissions and coreferences. Following previous studies (Mao et al., 2023) (Jang et al., 2023), we use ANCE (Xiong et al., 2021) pre-trained on the MSMARCO (Campos et al., 2016) as our retriever. To re-rank the guided documents, we select the mxbai-embed-large-v1 (Lee et al., 2024) and ember-v1 (llmrails, 2022) models as the first and second re-ranking Sentence-Transformer models, respectively. We use KeyBERT (Grootendorst, 2020) for augmenting keywords. And then we use roberta-base-squad2 (Chan et al., ) for generating the expected answer. Finally, we use OpenAI text-embedding-3-small for generating embeddings for keywords and answers in the filtering stage in GuideCQR. We use different embeddings at each stage to mitigate potential biases caused by a single model.

3.4 Performance Comparison

Main Results

As shown in Table 1, GuideCQR significantly enhances performance metrics compared to the $Q_{baseline}$ across CAsT datasets. GuideCQR outperforms all of the compared methods in terms of average score. Specifically, on the CAsT-19 dataset, GuideCQR outperforms both LLM and prompting-based reformulated queries, demonstrating its robustness. For CAsT-20, still its performance is almost second-best. These results highlight the solidity of retriever-friendly queries. The score for CAsT-20 is lower than CAsT-19 because the topics of CAsT-19 are more complex. So CAsT-20 is more challenging than CAsT-19. Therefore, the signals extracted from guided documents may not work well.

Ablation Study

We evaluate the impact of removing individual components involved in creating retriever-friendly queries. As demonstrated in Table 1, removing any part of GuideCQR leads to a decrease in performance, indicating that each component plays an important role in GuideCQR.

3.5 Analysis

Performance among Amount of Initial Guided documents

As in Table 2, we adjust the number of guided documents in the guided documents retrieval stage to validate the proper amount of the documents. The results indicate that increasing the amount of guidance consistently improves performance. However, retrieving too many documents for the guided documents requires a longer inference time. Therefore, we decide to choose 2,000 documents for our main experiment.

# of initial guided documents	MRR	NDCG@3	Avg
10	0.757	0.505	0.631
100	0.782	0.514	0.648
1,000	0.795	0.521	0.658
2,000	0.810	0.522	0.666

Table 2: Evaluation of the CAsT-19 dataset with varying amount of guided documents.

Performance among Top-k doc for Keyword and Answer

As in Table 3, we explore the effects of varying the number of documents in augmenting keywords and generating the expected answer stage. For example, if we extract each keyword and answer from the top 2 documents, we generate a keyword and answer pair for each document, resulting in a total of 2 pairs. Consistent with previous studies, we find that using more documents generally provides additional signals that can enhance performance. However, an excessive number of documents may lead to a loss in the overall score.

	Keyword			Answer
	MRR	NDCG@3	Avg	MRR	NDCG@3	Avg
Baseline	0.722	0.452	0.587	0.722	0.452	0.587
top-2	0.783	0.501	0.642	0.767	0.483	0.625
top-3	0.804	0.521	0.662	0.775	0.490	0.632
top-4	0.810	0.522	0.666	0.769	0.500	0.634
top-5	0.802	0.523	0.662	0.775	0.496	0.635

Table 3: Evaluation of the CAsT-19 dataset with varying number of documents for augmenting keywords.

Performance among Keyword Span Length

We also investigate the impact of varying keyword spans in augmenting the keyword stage and present the result in Table 4. We observe that increasing the keyword span generally enhances performance to a certain extent. However, we find that too many signals can degrade performance due to inclusion of the redundant information to the query.

	CAsT-19		CAsT-20
	MRR	NDCG@3	MRR	NDCG@3
Baseline	0.722	0.452	0.534	0.372
5 span	0.769	0.499	0.577	0.415
10 span	0.788	0.520	0.537	0.398
15 span	0.810	0.522	0.541	0.400
20 span	0.801	0.520	0.546	0.398

Table 4: Evaluation with varying span of keywords. We augment all keywords from top-4 documents.

Query Relevance Analysis

To show the effectiveness of the keyword augmentation step, we investigate the precision score as follows:

Precision=\frac{N_{4}}{N_{\text{total}}},

(8)

where $N_{4}$ represents unique matched keywords found in the guided documents that have qrel (query relevance) score four and $N_{total}$ represents the total number of unique augmented keywords. Note that the document that has a qrel score of 4 is regarded as the most relevant document with user queries in CAsT datasets. Therefore, keywords matched in these documents can prove that augmenting keywords plays an important role in making retriever-friendly queries. The results in Table 5 demonstrate that a longer keyword span results in a higher precision score. This underscores the critical role of keyword augmentation in GuideCQR.

Span Length	Precision
5	0.6684
10	0.7018
15	0.7024
20	0.7045

Table 5: Precision scores for different keyword spans. We augment all keywords from top-4 documents of CAsT-19.

Performance among FilterScore

We test different $FilterScore$ settings. Before filtering, we use top-4 span 15 keywords and top-6 answers for CAsT-19, and top-5 span 5 keywords and top-4 answers for CAsT-20. And GuideCQR uses these keywords and answers as our final query set. As in Figure 2, excessive filtering degrades performance, highlighting the importance of selecting an appropriate filter score. We achieve our final results with a $FilterScore$ of 1.19 for CAsT-19 and 0.525 for CAsT-20.

4 Conclusion

We present the GuideCQR, a novel query reformulation framework for conversational search. GuideCQR effectively reformulates conversational queries to be more retriever-friendly by leveraging a guided documents. GuideCQR not only enhances the alignment of queries with user intent but also improves the precision and relevance of search results. Experimental results across various datasets and metrics confirm the capability of GuideCQR over human and LLM-powered methods.

Acknowledgement

This research was supported by Institute for Information & Communications Technology Planning & Evaluation (IITP) through the Korea government (MSIT) under Grant No. 2021-0-01341 (Artificial Intelligence Graduate School Program (Chung-Ang University)).

References

Campos et al. (2016) Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading comprehension dataset. ArXiv, abs/1611.09268.
(2) Branden Chan, Timo Möller, Malte Pietsch, and Tanay Soni. roberta-base for qa.
Cheng et al. (2024) Yiruo Cheng, Kelong Mao, and Zhicheng Dou. 2024. Interpreting conversational dense retrieval by rewriting-enhanced inversion of session embedding.
Dalton et al. (2021) Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2021. Cast 2020: The conversational assistance track overview.
Dalton et al. (2020) Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. Cast-19: A dataset for conversational information seeking. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1985–1988.
Grootendorst (2020) Maarten Grootendorst. 2020. Keybert: Minimal keyword extraction with bert.
Jagerman et al. (2023) Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653.
Jang et al. (2023) Yunah Jang, Kang-il Lee, Hyunkyung Bae, Seungpil Won, Hwanhee Lee, and Kyomin Jung. 2023. Itercqr: Iterative conversational query reformulation without human supervision. arXiv preprint arXiv:2311.09820.
Lee et al. (2024) Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. Open source strikes bread - new fluffy embeddings model.
Lin et al. (2021b) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. Contextualized query embeddings for conversational search. In In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1004–1015.
Lin et al. (2020) Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020. Conversational question reformulation via sequence-to-sequence architectures and pretrained language models.
llmrails (2022) llmrails. 2022. ember-v1.
Mao et al. (2023) Kelong Mao, Zhicheng Dou, Fengran Mo, Jiewen Hou, Haonan Chen, and Hongjin Qian. 2023. Large language models know your contextual search intent: A prompting framework for conversational search. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1211–1225.
Mo et al. (2023) Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. 2023. Convgqr: Generative query reformulation for conversational search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4998–5012.
Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
Vakulenko et al. (2021a) Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021a. Question rewriting for conversational question answering. In In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 355–363.
Van Gysel and de Rijke (2018) Christophe Van Gysel and Maarten de Rijke. 2018. Pytrec_eval: An extremely fast python interface to trec_eval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 873–876.
Voskarides et al. (2020) Nikos Voskarides, Dan Li, Pengjie Ren, Evangelos Kanoulas, and Maarten de Rijke. 2020. Query resolution for conversational search with limited supervision. In In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 921–930.
Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Zaib et al. (2022) Munazza Zaib, Wei Emma Zhang, Quan Z Sheng, Adnan Mahmood, and Yang Zhang. 2022. Conversational question answering: A survey. Knowledge and Information Systems, 64(12):3151–3195.

Appendix A Dataset Statistics

Table 6 presents statistics for the CAsT datasets, including the number of conversations, queries, and documents.

	CAsT-19	CAsT-20
# Conv	50	25
# Questions	479	216
# Documents	38M	38M

Table 6: Dataset Statistics for CAst-19 and CAst-20.

Appendix B Query Analysis

We present example queries reformulated our method and humans in Table 7.

RawQuery	What are its other competitors?
Human-rewritten	What are Netflix’s other competitors than Blockbuster?
$Q_{baseline}$	What are Netflix’s other competitors?
$Q_{final}$	What are Netflix’s other competitors? netflix competitors subscription streaming amazon instant hulu plus amazon netflix competes viaplay hbo nordic cmore asia netflix competes hooq sky demand competitors apple itunes amazon cinemanow hulu lovefilm google netflix fandangonow ultraflix voddler competitors Amazon Instant Video Service and the Hulu Plus service Presto, Stan and Quickflix FandangoNow, UltraFlix, and Voddler : Competitors brick and mortar video rental stores

Table 7: Query Analysis for CAsT-19 - Conv_id: 49, turn_id: 5