tcb@breakable
LLM-enhanced Reranking in Recommender Systems
Abstract
Reranking is a critical component in recommender systems, playing an essential role in refining the output of recommendation algorithms. Traditional reranking models have focused predominantly on accuracy, but modern applications demand consideration of additional criteria such as diversity and fairness. Existing reranking approaches often fail to harmonize these diverse criteria effectively at the model level. Moreover, these models frequently encounter challenges with scalability and personalization due to their complexity and the varying significance of different reranking criteria in diverse scenarios. In response, we introduce a comprehensive reranking framework enhanced by LLM, designed to seamlessly integrate various reranking criteria while maintaining scalability and facilitating personalized recommendations. This framework employs a fully connected graph structure, allowing the LLM to simultaneously consider multiple aspects such as accuracy, diversity, and fairness through a coherent Chain-of-Thought (CoT) process. A customizable input mechanism is also integrated, enabling the tuning of the language model’s focus to meet specific reranking needs. We validate our approach using three popular public datasets, where our framework demonstrates superior performance over existing state-of-the-art reranking models in balancing multiple criteria.
Einführung
Reranking is a fundamental technique in the field of recommender systems (Liu et al. 2022; Zhang et al. 2019; Qin et al. 2022). Its importance lies in its ability to refine and enhance the results generated by ranking models, ultimately providing users with the most relevant and personalized recommendations. In the typical recommendation process, interaction features (e.g., user attributes and item properties) are initially utilized by a ranking model to generate a candidate list for the user. Subsequently, a reranking model is applied to further scrutinize the association between candidate items, delving into the nuances and intricacies of user preferences, and ultimately producing the final recommendation list. Currently, existing reranking models predominantly focus on improving the accuracy aspect of recommendations (Ai et al. 2018; Pei et al. 2019; Lin et al. 2022). Although accuracy is crucial, it is equally important to consider extensive aspects of the recommended list in practical use, such as diversity (Carbonell and Goldstein 1998; Chen, Zhang, and Zhou 2018; Yan et al. 2021) and fairness (Wu et al. 2021; Zehlike et al. 2017). Diversity ensures that users are exposed to a varied range of items, while fairness guarantees equal representation and exposure of different categories or sellers. Though several research (Carbonell and Goldstein 1998; Chen, Zhang, and Zhou 2018; Wu et al. 2021) try to combine one of them with the accuracy aspect for modeling, how to better consider and balance more aspects simultaneously remains a problem.
![Refer to caption](x1.png)
Specifically, existing reranking models do suffer from several limitations. Firstly, it is challenging to comprehensively consider and balance the complex combination of multiple aspects in modeling, primarily due to the substantial semantic gap between these aspects (Fan et al. 2023; Wang et al. 2022a). This is because each aspect scrutinizes the recommendation list via unique attribute dimensions, highlighting intricate semantic relationships and distinctions. This complexity underscores the substantial gap that exists between different aspects. Furthermore, scalability issues present another major hurdle, inhibiting the application of a singular model across diverse recommendation settings that may prioritize different aspects or functional rules. This challenge is particularly pronounced when introducing novel aspects or custom reranking rules, such as backward rules or stop conditions, not initially anticipated during the model’s development. Moreover, the inability to personalize the amalgamation of various aspects further limits the personalization of existing models, as noted in prior research (Pitoura, Stefanidis, and Koutrika 2022; Zhang and Hurley 2008). Once deployed, the output tendency of a specific model on different aspects is fixed and cannot be intelligently adjusted according to evolving business or user preferences.
To overcome these barriers, an optimal solution would involve creating a versatile reranking framework capable of simultaneously accounting for diverse aspect combinations and semantic nuances. Such a framework would flexibly accommodate the unique demands of different contexts and user needs, offering a more dynamic and tailored reranking solution.
Currently, with the rapid development of Large Language Models (LLMs) (Min et al. 2023; Floridi and Chiriatti 2020), extensive research has been undertaken to evaluate the capabilities of LLMs in various contexts (Chang et al. 2023; Wang et al. 2023). Previous studies (Kocoń et al. 2023; Sun et al. 2023; Ma et al. 2023) highlight that zero-shot LLMs may not achieve the same level of proficiency as specialized models in tasks such as information extraction and recommendation. This limitation is often attributed to their restricted token count and difficulties in processing extensive contexts that include thousands of items (Liu et al. 2023). Despite these challenges, these studies have shown that LLMs can perform comparably or even surpass supervised benchmarks in reranking tasks. This efficacy is credited to their robust semantic understanding capabilities within concise contexts involving a limited number of items. Thus, the application of LLMs in the reranking phase of recommendations could serve as a key strategy to merge different aspects by enhancing semantic understanding.
Nonetheless, several significant challenges arise when modeling a reranking framework using LLMs. The first challenge involves ensuring the scalability of the framework in a well-organized and flexible manner, allowing it to accommodate current aspect requirements while also remaining adaptable to potential future aspects. The second challenge revolves around formulating a mechanism capable of automatically combining diverse aspect requirements in accordance with specific recommendation settings or user preferences, ultimately achieving genuine personalization.
To address these challenges, we propose LLM4Rerank, an innovative reranking framework that harnesses the power of zero-shot LLMs for more precise reranking. Specifically, LLM4Rerank represents various aspect requirements in reranking as distinct nodes, allowing the framework to automatically incorporate these nodes in a Chain-of-Thought (CoT) manner (Wei et al. 2022; Yao et al. 2023; Besta et al. 2023). The advantage of this approach is twofold: it ensures scalability, allowing for the seamless inclusion of new nodes to address emerging aspect requirements. To demonstrate this, in addition to the accuracy aspect, diversity, and fairness aspects are also added for LLM4Rerank modeling in this paper. Additionally, the LLM used in LLM4Rerank can automatically determine the next node to consider, guided by the current reranking history and an additional sentence input referred to as the “Goal”, which is provided by the user or deployer, representing the overall focus and objective of the ongoing reranking process. This dynamic process enables LLM4Rerank to achieve enhanced personalization in the reranking process.
In summary, in this paper, our contributions could be summarized as follows:
-
•
To the best of our knowledge, this work is the first endeavor to automatically integrate multiple aspects and could thus measure different aspects in a unified semantic space comprehensively through a multi-hop reranking procedure employing LLMs.
-
•
We propose LLM4Rerank, a novel framework that can handle the complex combination of various aspect requirements, such as accuracy, diversity, and fairness, within the reranking process. LLM4Rerank offers the potential for superior performance, scalability, and personalization in reranking.
-
•
Experiments conducted on three widely used industrial datasets demonstrate that LLM4Rerank outperforms existing baselines in all aspects considered. This validates its efficacy and superiority in enhancing performance, scalability and personalization within the reranking process of recommender systems.
![Refer to caption](x2.png)
.
Framework
This section outlines the problem formulation for the reranking task in recommendations, followed by a comprehensive overview of LLM4Rerank and its principal components.
Problem Formulation
The reranking task plays a pivotal role in recommender systems. As depicted in Figure 1, consider as the set of users and as the set of items available for recommendation. For clarity, this paper represents each user and item by a feature vector ( and , respectively). Initially, a ranking model generates a candidate item list with items for each user. To improve recommendation performance, a reranking process (Pei et al. 2019; Liu et al. 2022) is applied to analyze the relationships among items within the initial list . This analysis aims to generate a refined list with items, denoted as , from the initial list. The goal of reranking models is to enhance user-item relevance by optimizing a defined objective function:
(1) |
where is the scoring function that evaluates the relevance of an item for a user , considering aspects such as accuracy, diversity, and fairness. Through this optimization, the reranking model selects the optimal recommendation list from the candidate lists , tailored to individual user preferences, thereby significantly enhancing the quality of recommendations.
In this study, to facilitate equitable comparisons across various reranking baselines, the Generalized Matrix Factorization (GMF) model (He et al. 2017) is employed as the uniform global ranking model following previous study (Lin et al. 2022). This approach ensures that all reranking models operate on identical candidate item lists.
LLM4Rerank Overview
In this section, we introduce the proposed reranking framework, LLM4Rerank, illustrated in Figure 2. The framework receives three types of inputs: user information (user info), which includes features such as gender and age; the candidate item list ; and a sentence termed “Goal”, which outlines the prioritized aspects for reranking. LLM4Rerank is structured as a fully connected function graph, excluding the “Stop” node, comprising various nodes as shown in Figure 2 (a). Each node represents a potential reranking step that generates a reranking list by LLM, taking into account specific aspect-related or functional requirements as depicted in Figure 2 (b). Each edge within the function graph signifies a potential pathway for node-to-node transition, ensuring connectivity among all nodes, with the exception of the “Stop” node. To exemplify LLM4Rerank’s scalability, we integrate not just an “Accuracy” node but also “Diversity” and “Fairness” aspect nodes into the framework, alongside two functional nodes: “Backward” and “Stop” for practical functionalities. The node architecture is meticulously crafted to permit the LLM to sequentially evaluate diverse nodes, thereby optimizing the reranking outcome to fulfill multiple aspect requirements comprehensively. Moreover, to prevent memory loss and enhance the LLM’s assessment of aspect combinations, a historical reranking pool is utilized (Figure 2 (c)). This pool records the outcomes from each node in sequence, serving as an auxiliary reference for subsequent reranking at each node. Ultimately, when the “Stop” node is reached, the reranking process is completed. The output at this stage is precisely the latest reranking results from the historical reranking pool, represented as .
Nodes Construction
To facilitate the Large Language Model (LLM)’s systematic analysis of complex aspect requirements in reranking, this structure aims to establish distinct nodes for specific requirements. Such an arrangement enables the LLM to process these requirements in a Chain-of-Thought approach (Wei et al. 2022; Yao et al. 2023; Besta et al. 2023). Nevertheless, this approach presents two primary challenges: First, customizing the node structure to maintain scalability when incorporating additional requirements; and second, empowering the LLM to automatically select its subsequent reranking step. To address these challenges, we introduce a generic node structure. It comprises a reranking step, paired with an ancillary indicator that signifies the direction of the forthcoming step identified by the next node’s name. This configuration permits the LLM to automatically navigate through the LLM4Rerank framework, making decisions based on the presently available information.
This section outlines our strategy for addressing the challenge of node structure customization with the aim of augmenting the scalability of the LLM4Rerank framework. Specifically, we introduce a generic node structure that serves as the foundation for all nodes as shown in Figure 2 (b). This generic node represents a single step of reranking under LLM considerations. The inputs to a generic node encompass semantic representations of user information, candidate items, the “Goal” sentence that defines the personalized focus for the entire reranking process, and the whole historical reranking pool if available. Outputs from this node are twofold: the immediate reranking results for the current node, represented by a list of item IDs, are integrated into the historical reranking pool alongside the current node’s name, serving as a reference for subsequent steps. Additionally, an indicator (i.e., the next node’s name in this paper) specifying the subsequent node for reranking is produced, thereby achieving an automatic step-by-step procedure. Within each node, LLM4Rerank initially crafts a prompt tailored to the specific reranking criteria and inputs at hand with a predefined template. Subsequently, LLM4Rerank would interact with LLM to obtain the two outputs based on the generated prompt.
Aspect Nodes
To facilitate the LLM in executing reranking tasks tailored to distinct aspect requirements, we employ a prompt-based template approach within the proposed generic node structure. This method allows for the instantiation of specific nodes dedicated to evaluating different aspects within the reranking process. Consequently, each node is designed to systematically address one of these key aspects, ensuring that the reranking outcomes reflect a balanced consideration. In this study, to demonstrate the scalability of LLM4Rerank, we implement three aspect nodes dedicated to reranking: “Accuracy”, “Diversity”, and “Fairness”.
-
•
Accuracy Node: This node is designed to fulfill the performance criteria of the final recommendation list during the reranking phase. As such, the prompt templates are crafted to underscore the correlation between users and items. Figure 3 presents a straightforward template instance employed within the node. Furthermore, given the paramount importance of recommendation accuracy - a fundamental aspect indispensable in recommender systems - the accuracy node has been established as the initial point within the LLM4Rerank framework. Consequently, every reranking procedure commences with the accuracy node, ensuring a foundational focus on precision from the outset.
Figure 3: Example prompt template of the accuracy node. Figure 4: Example prompt template of the diversity node. -
•
Diversity Node: This node is specifically designed to address the diversity criteria for the final recommendation list during the reranking phase. In this study, we assess the diversity of the reranking outcomes by evaluating the extent to which a particular attribute of the items is varied within the final list. We employ the -NDCG metric (Clarke et al. 2008) for this purpose. Consequently, an illustrative example of the template used in the diversity node is depicted in Figure 4.
-
•
Fairness Node: This node is designated to meet the fairness objectives within the final recommendation list at the reranking phase. In our study, fairness of the recommendation outcomes is operationalized as the average score disparity across two sample groups, segregated by a distinct characteristic, and evaluated using the Mean Absolute Deviation (MAD) metric (Zhu, Hu, and Caverlee 2018). Given that the LLM inherently generates reranking lists rather than numerical scores, we allocate scores ranging linearly from 1 to 0 to the items in the final recommendation list. These scores are subsequently utilized to compute the MAD for fairness assessment. For an in-depth methodological exposition, readers are directed to Section Implementation Details. Figure 5 provides a straightforward template illustration for the fairness node.
Figure 5: Example prompt template of the fairness node.
Functional Nodes
Recent research has demonstrated the efficacy of reflection in optimizing the output of LLMs (Shinn, Labash, and Gopinath 2023; Ji et al. 2023). To augment the logical capabilities of LLM4Rerank in the reranking process and introduce specialized functionalities, we have developed two functional nodes specifically aimed at facilitating reflection and termination within the reranking sequence.
![Refer to caption](x6.png)
-
•
Backward Node: This node empowers the LLM to selectively ignore a reranking outcome deemed suboptimal during the evaluation of previous reranking efforts. Within this framework, LLM4Rerank deletes the latest reranking result from the historical reranking pool and advances to the subsequent node as dictated by the LLM’s output directives. An illustrative template example of this node’s operation is provided in Figure 6.
-
•
Stop Node: This node governs the termination of the LLM4Rerank output sequence. When the LLM4Rerank designates this node as the incoming step, it signifies the conclusion of the complete reranking process. Subsequently, this node extracts the most recent reranking outcome from the historical reranking pool, presenting it as the definitive reranking result. Note that since this node only functionally signals the end of reranking and does not require access to LLM, the prompt template is not required for this node.
Automatic Reranking Process
To leverage the LLM for reranking based on a diverse set of aspect requirements, we have designed distinct nodes, each addressing specific aspect criteria. Nonetheless, delineating a predefined path from one node to another for every reranking task is both inefficient and challenging to achieve. Thus, to accommodate unique user preferences and significantly improve personalization, an automatic reranking process has been developed, which mainly consists of the following three sub-processes:
-
•
Setting “Goal”: To accommodate personalized requirements and facilitate LLM4Rerank’s scalability across varied contexts, a manually entered sentence, referred to as the “Goal,” is incorporated as one of the preliminary inputs for each reranking process. As illustrated in Figure 2, the “Goal” indicates the main focus of a specific reranking process. By interpreting the semantic connections between the “Goal” and the respective nodes, LLM is enabled to automatically select the most appropriate nodes for any given reranking task.
-
•
Automatic Transition Across Nodes: For each node, upon receiving replies from LLM, LLM4Rerank would obtain the current reranking results, along with an indicator (i.e., the next node name) for the subsequent node, thereby ensuring a fluid and automatic transition across various nodes.
-
•
Conditions to Stop Reranking: To mitigate the risk of prolonged inactivity and to address errors stemming from possible unrecognized semantic inaccuracies in the LLM’s responses, two termination criteria have been established within the framework. The first criterion is triggered when the LLM autonomously identifies the “Stop” node as the subsequent step. The second criterion activates upon the LLM’s navigation through a predetermined number of nodes, set by a hyper-parameter, excluding the “Backward” node from this count. Fulfillment of either condition marks the completion of the reranking process. Subsequently, this node retrieves and presents the most recent reranking outcome from the historical reranking pool as the definitive result.
By applying these sub-processes, the whole automatic reranking process of LLM4Rerank is established as illustrated in Algorithm 1.
Input: User information , Candidate item list , the reranking focus , Maximum node count
Output: Final reranking result
Note: represents the execution of functions in node with input .
Experiments
In this section, we conduct experiments on three widely recognized industrial datasets to explore the following research questions:
-
•
RQ1: How does LLM4Rerank compare to established reranking baselines across accuracy, diversity, and fairness aspects?
-
•
RQ2: Can LLM4Rerank automatically identify and prioritize a specific blend of aspect requirements for reranking tailored to individual preferences?
-
•
RQ3: Does LLM4Rerank’s automatic reranking framework offer clear benefits over a predetermined reranking pathway?
Experimental Setup
Dataset
We conduct our experiments using three widely recognized public datasets: ML-1M 111https://grouplens.org/datasets/movielens/1m/ (Harper and Konstan 2015), KuaiRand (KuaiRand-Pure) 222https://kuairand.com/ (Gao et al. 2022), and Douban-Movie 333https://www.kaggle.com/datasets/fengzhujoey/douban-datasetratingreviewside-information (Zhu et al. 2020, 2019). For each dataset, we employ the leave-one-out method (He et al. 2017; Bayer et al. 2017; Gan et al. 2021), a widely adopted approach in the literature, for dividing the data into training, validation, and testing sets. According to previous studies (Lin et al. 2022), we select the Generalized Matrix Factorization (GMF) model as the global ranking model to generate candidate lists for each user, with each set comprising 20 items. To ensure equitable comparisons across deep learning and LLM-based models, we omit features lacking explicit semantic information and exclude users with fewer than five interactions. For deep learning-based models, we utilize a standard embedding technique (Wang et al. 2021; Guo et al. 2021) to transform various features into vector inputs. Conversely, for LLM-based models, the semantic information of the feature (e.g., the feature’s name) is utilized as the input. Table 1 presents the statistics of both datasets subsequent to preprocessing.
Dataset | Interactions | Users | Artikel |
---|---|---|---|
ML-1M | 1,000,209 | 6,040 | 3,883 |
KuaiRand | 102,433 | 10,494 | 7,583 |
Douban-Movie | 759,652 | 2,606 | 34,893 |
Baseline
Model | ML-1M | KuaiRand | Douban-Movie | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
HR | NDCG | -NDCG | MAD | HR | NDCG | -NDCG | MAD | HR | NDCG | -NDCG | MAD | |
GMF | 0.4156 | 0.1853 | 0.1005 | 0.0613 | 0.4417 | 0.2314 | 0.1627 | 0.1588 | 0.5723 | 0.3150 | 0.2516 | 0.4006 |
DLCM | 0.5781 | 0.2354 | 0.1378 | 0.0549 | 0.6893 | 0.3080 | 0.1767 | 0.1026 | 0.6827 | 0.4102 | 0.3581 | 0.2619 |
PRM | 0.6986 | 0.3246 | 0.1653 | 0.0436 | 0.8083 | 0.3904 | 0.1869 | 0.1032 | 0.6979 | 0.4167 | 0.3477 | 0.2509 |
MMR | 0.4675 | 0.2588 | 0.2104 | 0.0265 | 0.4928 | 0.2606 | 0.1877 | 0.1569 | 0.6538 | 0.3873 | 0.3744 | 0.2539 |
FastDPP | 0.4719 | 0.2561 | 0.1942 | 0.0263 | 0.5728 | 0.2913 | 0.1882 | 0.0660 | 0.6635 | 0.4038 | 0.3818 | 0.2820 |
FairRec | 0.4805 | 0.2007 | 0.1243 | 0.0199 | 0.6083 | 0.2761 | 0.1540 | 0.0318 | 0.6771 | 0.4021 | 0.3119 | 0.1752 |
RankGPT | 0.5584 | 0.2587 | 0.1799 | 0.0564 | 0.6583 | 0.2910 | 0.1557 | 0.1256 | 0.6635 | 0.3967 | 0.3448 | 0.2472 |
GoT | 0.5730 | 0.2714 | 0.1942 | 0.0486 | 0.7184 | 0.3198 | 0.1788 | 0.1211 | 0.6827 | 0.4135 | 0.3592 | 0.2195 |
LLM4Rerank-A | 0.7031* | 0.3320* | 0.2294 | 0.0434 | 0.8252* | 0.4229* | 0.2032 | 0.1969 | 0.7041* | 0.4301* | 0.3806 | 0.2446 |
LLM4Rerank-D | 0.6875 | 0.3292 | 0.2407* | 0.0571 | 0.8058 | 0.4143 | 0.2223* | 0.0969 | 0.6701 | 0.4019 | 0.3837* | 0.2757 |
LLM4Rerank-F | 0.5584 | 0.2328 | 0.1411 | 0.0193* | 0.7282 | 0.3276 | 0.1825 | 0.0271* | 0.6598 | 0.3917 | 0.2970 | 0.1696* |
LLM4Rerank-ADF | 0.6364 | 0.3058 | 0.2051 | 0.0250 | 0.8000 | 0.4117 | 0.2163 | 0.0530 | 0.6877 | 0.4105 | 0.3664 | 0.1975 |
Model | ML-1M | Node Used | Path Used | ||||||
---|---|---|---|---|---|---|---|---|---|
-NDCG | MAD | Acc | Div | Fair | Fav Path | Fav Prop | Ave Length | Max Stop Prop | |
DF | 0.2110 | 0.0255 | 21% | 47% | 32% | A-D-F | 12% | 3.13 | 21% |
D-F | 0.2337 | 0.0416 | 20% | 59% | 21% | A-D-D-F | 19% | 3.31 | 9% |
F-D | 0.1456 | 0.0242 | 21% | 27% | 52% | A-F-D-F | 18% | 3.39 | 11% |
In this section, we assess LLM4Rerank’s capability to address diverse aspect requirements by comparing it with the following baseline methodologies:
-
•
GMF (He et al. 2017) extends matrix factorization into a non-linear framework, serving as the primary global ranking method in this study. The GMF results represent recommendations prior to the application of any reranking process.
-
•
DLCM (Ai et al. 2018) enhances reranking efficacy by employing a recurrent neural network alongside an attention-based loss function to comprehend local ranking dynamics, aiming primarily at improving accuracy within recommendation outcomes.
-
•
PRM (Pei et al. 2019) leverages a transformer architecture with self-attention mechanisms to refine the entire recommendation list by acknowledging the inter-item influences, thereby concentrating on augmenting accuracy.
-
•
MMR (Carbonell and Goldstein 1998) aims to balance query relevance with the reduction of redundancy in reranked documents, employing a maximal marginal relevance score to bolster the diversity aspect in recommendation outcomes.
-
•
FastDPP (Chen, Zhang, and Zhou 2018) expedites Determinantal Point Processes (DPP) for Maximum A Posteriori (MAP) inference, facilitating the efficient production of diverse recommendation sets. This model focuses on the diversity aspect of the recommendation result.
-
•
FairRec (Wu et al. 2021) introduces a fairness-aware recommendation framework that employs decomposed adversarial learning and orthogonality regularization. It aims to alleviate bias concerning sensitive user attributes, thereby fostering fairness in recommendations without compromising overall performance.
-
•
RankGPT (Sun et al. 2023) investigates the application of LLMs in ranking tasks within information retrieval, employing a novel instructional permutation generation method alongside a sliding window strategy. This model is distinguished by its focus on accuracy. Note that, as a zero-shot LLM baseline, the permutation distillation method in the original paper is not implemented.
-
•
GoT (Besta et al. 2023) proposes a graph-of-thought approach to enhance LLMs’ prompting efficacy by structuring generated content as a graph. This facilitates synergistic outcomes, thought distillation, and feedback loop integration, aligning LLM reasoning more closely with human cognitive processes. Unlike LLM4Rerank, GoT adheres to predetermined node-to-node inference paths without historical data consideration. By applying a fixed path, “Accuracy-Diversity-Fairness-Stop.” GoT serves as a zero-shot LLM baseline focusing on the combination of accuracy, diversity, and fairness aspects in this paper.
Implementation Details
In the evaluation of the accuracy aspect, we adopt widely recognized metrics: Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG). For assessing the diversity aspect, we apply the commonly used metric -NDCG. To evaluate fairness, we use the Mean Absolute Difference (MAD) (Zhu, Hu, and Caverlee 2018). The MAD calculation is formalized as:
(2) |
where and represent the predicted ratings for two distinct groups, and denotes the total number of ratings for group . Within the ML-1M dataset, we utilize the “genre” feature for diversity analysis and the “year” feature for fairness assessment, categorizing films into two groups based on their release year: pre-1996 and post-1996 (Zhu, Hu, and Caverlee 2018; Kamishima and Akaho 2017). For the KuaiRand dataset, “upload_type” serves as the criterion for diversity, while “video_duration”-divided into less than 60,000 ms and greater than 60,000 ms-serves for fairness evaluation. To fine-tune deep learning-based models for optimal performance, we set the learning rate at 0.001 and engage in a grid search to determine the best hyper-parameters. For zero-shot LLM baselines and LLM4Rerank, Llama-2-13B (Touvron et al. 2023) is selected as the default LLM backbone.
Overall Performance (RQ1)
This section presents a comprehensive performance comparison of LLM4Rerank against various baselines, as detailed in Table 2. The comparative analysis reveals that:
-
•
DLCM and PRM achieve acceptable performance in terms of accuracy, as indicated by HR and NDCG metrics. PRM, leveraging a transformer architecture for user-item relevance evaluation, surpasses DLCM in accuracy.
-
•
MMR and FastDPP demonstrate effectiveness in enhancing diversity, as quantified by the -NDCG metric. These models excel in diversifying user reranking lists by emphasizing item similarity and list-wide diversity.
-
•
FairRec exhibits strong performance in promoting fairness, measured using the MAD metric. Through the integration of decomposed adversarial learning and orthogonality regularization techniques, FairRec ensures more equitable recommendations across different user groups.
-
•
RankGPT shows commendable performance, underscoring the capability of zero-shot LLMs in reranking tasks. Conversely, GoT, employing a Chain-of-Thought approach, yields superior outcomes by facilitating a sequential analysis of multiple aspects.
-
•
LLM4Rerank, through personalized “Goal” setting and an automatic reranking process, significantly surpasses baselines, validating its comprehensive efficacy. LLM4Rerank adeptly merges various aspect requirements for reranking, illustrating its versatility. While LLM4Rerank-ADF may not lead in any single aspect, its overall balanced performance across all dimensions confirms the advantage of integrating LLMs with an automatic reranking framework. This approach effectively harmonizes different aspect demands via semantic comprehension, delivering optimized results across accuracy, diversity, and fairness.
Aspect Combination Analysis (RQ2)
In this section, we delve into experiments designed to assess whether LLM4Rerank can automatically tailor its reranking strategy to incorporate a specific blend of aspect requirements, guided by different user-defined “Goal”. Our investigation on the ML-1M dataset encompasses tests with LLM4Rerank under varied “Goals” reflecting distinct prioritizations on the diversity and fairness aspects:
-
•
DF: Assign equal importance to diversity and fairness aspects.
-
•
D-F: Prioritize diversity, with subsequent emphasis on fairness.
-
•
F-D: Prioritize fairness, with subsequent emphasis on diversity.
In this experiment, the maximum node count is set to 5. The outcomes, presented in Table 3, indicate that LLM4Rerank proficiently adjusts its reranking paths based on different “Goals”, facilitating a dynamic, weighted integration of aspect requirements. This capability significantly bolsters the personalization of the reranking process. It is noteworthy that the “Accuracy” node is consistently involved across all reranking outcomes, underscoring that every reranking sequence commences with the accuracy node. This initial step ensures the foundational accuracy in user-item matching is maintained. Moreover, it is noted that in the LLM’s favorite path for different “Goals”, the prioritized aspects dominate, indicating that LLM4Rerank framework can drive the LLM to think and capture the importance relation of aspects in “Goals”, and influence the reasoning focus of the LLM. Additionally, it can be noted that there are few inference paths that end because the inference node reaches its maximum value. This shows that in the current setting with 3 different aspect nodes, 3-4 thinking steps are enough for the LLM to give the result naturally.
![Refer to caption](x7.png)
Ablation Study (RQ3)
In this section, we undertake ablation studies on the ML-1M dataset to elucidate the impact of LLM4Rerank’s various components on overall performance. The experiments aim to dissect the model’s architecture by systematically removing certain features, thereby highlighting their individual contributions. We focus on the “Accuracy” aspect as a case study and align our investigation with a specific “Goal”: Pay attention to the accuracy aspects. The following variants of LLM4Rerank are considered for comparison:
Model | HR | NDCG | -NDCG | MAD |
---|---|---|---|---|
LLM4Rerank-A | 0.7031 | 0.3320 | 0.2294 | 0.0434 |
LLM4Rerank-H | 0.6410 | 0.3142 | 0.2275 | 0.0496 |
LLM4Rerank-AR | 0.6413 | 0.3191 | 0.2200 | 0.0464 |
LLM4Rerank-N | 0.6533 | 0.3079 | 0.2141 | 0.0515 |
-
•
LLM4Rerank-A: As detailed in Table 2, including all sub-structures and focusing on the accuracy aspect.
-
•
w/o historical reranking pool (-H): Exclude the historical reranking pool, removing the capability to reference previous reranking outcomes.
-
•
w/o automatic reranking (-AR): Adopting a static reranking path of ’Accuracy-Accuracy-Stop’.
-
•
w/o other aspect nodes (-N): Omit all nodes except for “Accuracy” and “Stop” nodes.
The findings from Table 4, allow us to draw several conclusions:
-
•
The absence of the historical reranking pool (LLM4Rerank-H) leads to a marked decline in performance, underscoring the importance of a holistic view in sequential decision-making. This feature enables LLM4Rerank to recall and evaluate previous choices, enhancing the model’s strategic depth.
-
•
The removal of the automatic reranking process (LLM4Rerank-AR) results in a significant performance drop, validating the utility of adaptive pathways in addressing diverse aspect requirements. The automatic reranking mechanism allows LLM4Rerank to dynamically determine subsequent steps based on the entirety of current information, thus optimizing the reranking sequence.
-
•
Eliminating other aspects and functional nodes (LLM4Rerank-N) also precipitates a notable decrease in performance. This highlights the value of a comprehensive review mechanism, as facilitated by the “Backward” node, in mimicking human-like decision-making processes. Meanwhile, in comparison with -AR, the LLM can still decide how many times it can access this node before ending the reranking process. The performance improvement verifies that LLM4Rerank can still benefit from dynamic node visit times, even if LLM4Rerank only has a single aspect node.
These results illuminate the critical roles played by LLM4Rerank’s substructures in augmenting reranking performance, particularly in tailoring the process to specific aspect focuses. The study underscores the model’s sophisticated architecture, designed to flexibly integrate and balance various reranking criteria.
Case Study
In this section, we present a concrete case study to further illustrate how the LLM4Rerank framework works and whether it can really balance different aspects of reranking as shown in Figure 7. In this figure, we report the two most common paths for LLM4Rerank under the different “Goals”: The first one (A-D-F) considers accuracy, diversity, and fairness simultaneously; The second one (A-A-B-D) focuses more on the accuracy aspect, followed by the diversity aspect. The evaluation is based on the average result on the specific path. As shown from the first path, according to the guidance of the “Goal”, LLM4Rerank passes through the “Accuracy”, “Diversity”, and “Fairness” nodes, respectively, and then ends reranking. After the step of diversity reranking, not only the “-NDCG” metric becomes higher, but also the “HR” and “NDCG” metrics. This may be because in the experiment, LLM can synthesize the current reranking considering not only the current aspect but also the historical reranking results. In addition, the positive relation between “-NDCG” and the “NDCG” metrics may also influence both aspect results when considering them together. From the second path, it can be seen that the addition of functional nodes such as “Backward” helps LLM to think more systematically. When it senses that there is almost no change in the diversity aspect after continuous access to the “Accuracy” node, it considers returning to the previous step and setting the next step as the diversity node.
Hyper-parameter Analysis
![Refer to caption](x8.png)
![Refer to caption](x9.png)
![Refer to caption](x10.png)
![Refer to caption](x11.png)
Recent studies have illuminated the challenges LLMs face in comprehensively processing long contexts laden with dense information (Liu et al. 2023). As the number of candidate items in a ranking sequence increases, so does the volume of semantic information, potentially overwhelming LLMs. This could explain the diminished efficacy observed when zero-shot LLMs are directly implemented in recommender systems that catalog millions of items. In light of this, in this section, we explore the impact of the hyper-parameter “candidate item number”, initially fixed at 20, on the reranking performance within the ML-1M dataset, as demonstrated in Figure 8 with LLM4Rerank-ADF.
The findings indicate a degradation in LLM4Rerank’s performance across various aspects as the “candidate item number” escalates. This outcome not only underscores the current limitations of LLMs in parsing long contexts but also reinforces their aptitude for tasks entailing more concise contextual information, such as reranking, over direct application in extensive recommendation or ranking frameworks. By constraining the number of items and focusing on rich semantic content within a single request, LLM4Rerank effectively narrows the semantic chasm across different aspect requirements, thus delivering more coherent reranking outcomes that enhance the overall recommendation quality.
Related Work
This section offers a review of the current methods of reranking strategies in recommendations.
Reranking in Recommendations
Reranking emerges as a critical post-processing strategy within recommender systems by introducing supplementary criteria to optimize the initial sequence of candidates. For example, the Deep Listwise Context Model (DLCM) (Ai et al. 2018) employs a recurrent neural network to sequentially process candidate items, thereby accruing contextual insights. Similarly, the Personalized Re-ranking Model (PRM) (Pei et al. 2019) utilizes the Transformer architecture (Vaswani et al. 2017) for encoding, enabling the modeling of extensive inter-item relationships. PRM enhances its efficacy by integrating personalized encoding mechanisms through the adoption of pre-trained embeddings.
Furthermore, the reranking phase is instrumental in addressing varied aspectual requirements, such as diversity (Yan et al. 2021; Gan et al. 2021) and fairness (Yadav, Du, and Joachims 2021; Raj and Ekstrand 2022). These aspects are increasingly recognized for their pivotal role in enriching user experience and ensuring that recommendations are congruent with overarching business objectives. For instance, the Maximal Marginal Relevance (MMR) (Carbonell and Goldstein 1998) model formulates a reranking schema emphasizing diversity by balancing document-query similarity against inter-document similarity. Conversely, FastDPP (Chen, Zhang, and Zhou 2018) innovates with an efficient greedy Maximum A Posteriori (MAP) inference for Determinantal Point Processes (DPP), facilitating the generation of both relevant and diverse recommendation sets. Additionally, FairRec (Wu et al. 2021) pioneers a fairness-focused recommendation framework employing decomposed adversarial learning and orthogonality regularization to mitigate biases and promote equity in the reranking process.
Moreover, recent advancements in LLMs indicate that LLM excels in reranking tasks with shorter contexts (Ma et al. 2023; Sun et al. 2023; Xiong et al. 2023; Lin et al. 2023) than in tasks with longer contexts. This discovery makes it possible to use LLM for better and more personalized reranking. For instance, DQ-LoRe (Xiong et al. 2023) uses dual queries for exemplar selection in prompting, RankGPT (Sun et al. 2023) employs a sliding window for ranking with long context, and the Graph of Thoughts (Besta et al. 2023) framework improves reranking outcomes with a fixed graph (Wei et al. 2022; Wang et al. 2022b) which allows for a complicated step-to-step data processing approach.
However, existing research typically focuses on one aspect, “Accuracy”, and occasionally includes one more aspect, such as “Diversity”. They overlook the need to integrate multiple aspects of different applications. In contrast, LLM4Rerank employs a function graph for automatic reranking, offering scalability and personalization in combining various aspects to suit different situations.
Fazit
In this paper, we introduce an LLM-based automatic reranking framework designed to enhance recommender systems through auto-reranking. Central to our approach is the development of a generic node structure, which serves to represent various aspect requirements and functions as distinct nodes within the system. This structure facilitates the construction of a function graph that orchestrates the automatic reranking process, complemented by a historical reranking pool that enables retrospective analysis of reranking decisions. Additionally, a “Goal” sentence is utilized to direct the integration of different nodes, ensuring that the framework can dynamically amalgamate multiple aspect requirements. This design allows LLM4Rerank to deliver superior performance, scalability, and personalization in the reranking process. Experimental validation on three widely recognized industrial datasets underscores the efficacy of the proposed LLM4Rerank framework.
References
- Ai et al. (2018) Ai, Q.; Bi, K.; Guo, J.; and Croft, W. B. 2018. Learning a deep listwise context model for ranking refinement. In The 41st international ACM SIGIR conference on research & development in information retrieval, 135–144.
- Bayer et al. (2017) Bayer, I.; He, X.; Kanagal, B.; and Rendle, S. 2017. A generic coordinate descent framework for learning from implicit feedback. In Proceedings of the 26th international conference on world wide web, 1341–1350.
- Besta et al. (2023) Besta, M.; Blach, N.; Kubicek, A.; Gerstenberger, R.; Gianinazzi, L.; Gajda, J.; Lehmann, T.; Podstawski, M.; Niewiadomski, H.; Nyczyk, P.; et al. 2023. Graph of thoughts: Solving elaborate problems with large language models. arXiv preprint arXiv:2308.09687.
- Carbonell and Goldstein (1998) Carbonell, J.; and Goldstein, J. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, 335–336.
- Chang et al. (2023) Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Zhu, K.; Chen, H.; Yang, L.; Yi, X.; Wang, C.; Wang, Y.; et al. 2023. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109.
- Chen, Zhang, and Zhou (2018) Chen, L.; Zhang, G.; and Zhou, E. 2018. Fast greedy map inference for determinantal point process to improve recommendation diversity. Advances in Neural Information Processing Systems, 31.
- Clarke et al. (2008) Clarke, C. L.; Kolla, M.; Cormack, G. V.; Vechtomova, O.; Ashkan, A.; Büttcher, S.; and MacKinnon, I. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, 659–666.
- Fan et al. (2023) Fan, W.; Zhao, X.; Wang, L.; Chen, X.; Gao, J.; Liu, Q.; and Wang, S. 2023. Trustworthy Recommender Systems: Foundations and Frontiers. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 5796–5797.
- Floridi and Chiriatti (2020) Floridi, L.; and Chiriatti, M. 2020. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30: 681–694.
- Gan et al. (2021) Gan, L.; Nurbakova, D.; Laporte, L.; and Calabretto, S. 2021. EMDKG: Improving Accuracy-Diversity Trade-Off in Recommendation with EM-based Model and Knowledge Graph Embedding. In IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 17–24.
- Gao et al. (2022) Gao, C.; Li, S.; Zhang, Y.; Chen, J.; Li, B.; Lei, W.; Jiang, P.; and He, X. 2022. KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management, CIKM ’22, 3953–3957.
- Guo et al. (2021) Guo, H.; Chen, B.; Tang, R.; Zhang, W.; Li, Z.; and He, X. 2021. An embedding learning framework for numerical features in ctr prediction. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2910–2918.
- Harper and Konstan (2015) Harper, F. M.; and Konstan, J. A. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4): 1–19.
- He et al. (2017) He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; and Chua, T.-S. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, 173–182.
- Ji et al. (2023) Ji, Z.; Yu, T.; Xu, Y.; Lee, N.; Ishii, E.; and Fung, P. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, 1827–1843.
- Kamishima and Akaho (2017) Kamishima, T.; and Akaho, S. 2017. Considerations on recommendation independence for a find-good-items task.
- Kocoń et al. (2023) Kocoń, J.; Cichecki, I.; Kaszyca, O.; Kochanek, M.; Szydło, D.; Baran, J.; Bielaniewicz, J.; Gruza, M.; Janz, A.; Kanclerz, K.; et al. 2023. ChatGPT: Jack of all trades, master of none. Information Fusion, 101861.
- Lin et al. (2023) Lin, J.; Dai, X.; Xi, Y.; Liu, W.; Chen, B.; Li, X.; Zhu, C.; Guo, H.; Yu, Y.; Tang, R.; et al. 2023. How Can Recommender Systems Benefit from Large Language Models: A Survey. arXiv preprint arXiv:2306.05817.
- Lin et al. (2022) Lin, Z.; Zang, S.; Wang, R.; Sun, Z.; Senthilnath, J.; Xu, C.; and Kwoh, C. K. 2022. Attention over self-attention: Intention-aware re-ranking with dynamic transformer encoders for recommendation. IEEE Transactions on Knowledge and Data Engineering.
- Liu et al. (2023) Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilacqua, M.; Petroni, F.; and Liang, P. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172.
- Liu et al. (2022) Liu, W.; Xi, Y.; Qin, J.; Sun, F.; Chen, B.; Zhang, W.; Zhang, R.; and Tang, R. 2022. Neural re-ranking in multi-stage recommender systems: A review. arXiv preprint arXiv:2202.06602.
- Ma et al. (2023) Ma, Y.; Cao, Y.; Hong, Y.; and Sun, A. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08559.
- Min et al. (2023) Min, B.; Ross, H.; Sulem, E.; Veyseh, A. P. B.; Nguyen, T. H.; Sainz, O.; Agirre, E.; Heintz, I.; and Roth, D. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2): 1–40.
- Pei et al. (2019) Pei, C.; Zhang, Y.; Zhang, Y.; Sun, F.; Lin, X.; Sun, H.; Wu, J.; Jiang, P.; Ge, J.; Ou, W.; et al. 2019. Personalized re-ranking for recommendation. In Proceedings of the 13th ACM conference on recommender systems, 3–11.
- Pitoura, Stefanidis, and Koutrika (2022) Pitoura, E.; Stefanidis, K.; and Koutrika, G. 2022. Fairness in rankings and recommendations: an overview. The VLDB Journal, 1–28.
- Qin et al. (2022) Qin, J.; Zhu, J.; Chen, B.; Liu, Z.; Liu, W.; Tang, R.; Zhang, R.; Yu, Y.; and Zhang, W. 2022. RankFlow: Joint Optimization of Multi-Stage Cascade Ranking Systems as Flows. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 814–824.
- Raj and Ekstrand (2022) Raj, A.; and Ekstrand, M. D. 2022. Measuring fairness in ranked results: An analytical and empirical comparison. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 726–736.
- Shinn, Labash, and Gopinath (2023) Shinn, N.; Labash, B.; and Gopinath, A. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366.
- Sun et al. (2023) Sun, W.; Yan, L.; Ma, X.; Ren, P.; Yin, D.; and Ren, Z. 2023. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent. arXiv preprint arXiv:2304.09542.
- Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
- Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. Advances in neural information processing systems, 30.
- Wang et al. (2021) Wang, R.; Shivanna, R.; Cheng, D.; Jain, S.; Lin, D.; Hong, L.; and Chi, E. 2021. Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, 1785–1797.
- Wang et al. (2022a) Wang, S.; Zhang, X.; Wang, Y.; and Ricci, F. 2022a. Trustworthy recommender systems. ACM Transactions on Intelligent Systems and Technology.
- Wang et al. (2022b) Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2022b. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations.
- Wang et al. (2023) Wang, Y.; Zhong, W.; Li, L.; Mi, F.; Zeng, X.; Huang, W.; Shang, L.; Jiang, X.; and Liu, Q. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966.
- Wei et al. (2022) Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824–24837.
- Wu et al. (2021) Wu, C.; Wu, F.; Wang, X.; Huang, Y.; and Xie, X. 2021. Fairness-aware news recommendation with decomposed adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 4462–4469.
- Xiong et al. (2023) Xiong, J.; Li, Z.; Zheng, C.; Guo, Z.; Yin, Y.; Xie, E.; Yang, Z.; Cao, Q.; Wang, H.; Han, X.; et al. 2023. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. arXiv preprint arXiv:2310.02954.
- Yadav, Du, and Joachims (2021) Yadav, H.; Du, Z.; and Joachims, T. 2021. Policy-gradient training of fair and unbiased ranking functions. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1044–1053.
- Yan et al. (2021) Yan, L.; Qin, Z.; Pasumarthi, R. K.; Wang, X.; and Bendersky, M. 2021. Diversification-aware learning to rank using distributed representation. In Proceedings of the Web Conference 2021, 127–136.
- Yao et al. (2023) Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
- Zehlike et al. (2017) Zehlike, M.; Bonchi, F.; Castillo, C.; Hajian, S.; Megahed, M.; and Baeza-Yates, R. 2017. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, 1569–1578.
- Zhang and Hurley (2008) Zhang, M.; and Hurley, N. 2008. Avoiding monotony: improving the diversity of recommendation lists. In Proceedings of the 2008 ACM conference on Recommender systems, 123–130.
- Zhang et al. (2019) Zhang, S.; Yao, L.; Sun, A.; and Tay, Y. 2019. Deep learning based recommender system: A survey and new perspectives. ACM computing surveys (CSUR), 52(1): 1–38.
- Zhu et al. (2019) Zhu, F.; Chen, C.; Wang, Y.; Liu, G.; and Zheng, X. 2019. DTCDR: A framework for dual-target cross-domain recommendation. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 1533–1542.
- Zhu et al. (2020) Zhu, F.; Wang, Y.; Chen, C.; Liu, G.; and Zheng, X. 2020. A Graphical and Attentional Framework for Dual-Target Cross-Domain Recommendation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, 3001–3008.
- Zhu, Hu, and Caverlee (2018) Zhu, Z.; Hu, X.; and Caverlee, J. 2018. Fairness-aware tensor-based recommendation. In Proceedings of the 27th ACM international conference on information and knowledge management, 1153–1162.