Recall-Augmented Ranking: Enhancing Click-Through Rate Prediction Accuracy with Cross-Stage Data

The Cross-Stage User/Item Selection Module select the most similar users and relevant items. The selection process can be abstracted into two steps and we take the selection of recall items as an example. First, similarity is measured between the target item and each recall item by similarity function $f(\cdot)$. Then top-k relevant recall items can be selected based on the similarity score, which can be formalized in Equation 1,  2. We conclude the key notations and the descriptions in Table 1.

An intuitive idea is using the embedding and search k-nearest neighbor by inner product. However, the huge number of multiplications makes real-world deployment impractical. Considering the selection complexity, we use the SimHash function in our experiment.

SimHash, leveraging locality-sensitive properties, ensures similar outputs for similar inputs through random projection and signed axes, simplifying embeddings to binary fingerprints. This process is detailed in Equation 3,  4, where $\boldsymbol{e}^{i}_{k}$ stands for the embedding of the $k^{th}$ recall item and $m$ is the $m^{th}$ hash function in the hash function set. It reduces storage and speeds up selection by using hamming distance for efficient comparison.

Co-Interaction Module provides a fine-grained set-to-set modeling. It improves upon the simplistic equal weighting of all selected recall items, which overlooks hierarchical information. We introduce a matching matrix to assess user-item interest compatibility. The matching score is represented as high-level latent vectors’ inner product, as is shown in Equation 5. Then we compute the matching matrix in Equation 6, where $Sigmoid(\cdot)$ is used to map the matching scores to (0,1).

To provide the model with a clearer indication of which recall items are more important, the signal $y_{ui}^{ep}$ is utilized to supervise the training of the matching matrix. As the exposed signal is very sparse and we define $y_{ui}^{ep}$ in Equation 8.

Finally, the matching matrix is averaged by row and by column to obtain the item and user weighting vector. We obtain user common interest $v^{c}_{ui}$ and user diverse interest $v^{d}_{ui}$ by multiplying weighting vectors with corresponding embeddings. Then user enriched representation $v^{enr}_{u}$ is obtained by concatenating $v^{c}_{ui}$ and $v^{d}_{ui}$.

The loss function of RAR can be represented by Equation 12, where $\mathcal{L}_{clk}$ aims to predict CTR accurately and $\mathcal{L}_{ep}$ aims to provide a clearer indication to the model of which recall items are most important. $\alpha\in[0,1]$ is a tunable parameter for balancing the two losses. Both two losses are cross-entropy loss and supervise the training process in a point-wise manner. All modules of RAR are trained jointly by minimizing the joint loss function on the training dataset.

We analyze the efficiency of the RAR in this section. Co-Interaction Module first gets the matching matrix and weighting scores by multiplying the embedding vectors of selected look-alike users and recall items, followed by a weighted sum using the weighting vectors. Therefore, three matrix multiplications are needed, and the time complexity is $O(B\cdot k_{l}\cdot k_{r}\cdot d)$. As for the selection module, the time complexity depends on the selection method utilized. Since conducting $XOR$ and counting the number of bits in 1 in SimHash can be accomplished in $O(1)$, the overall time complexity of RAR is $O(B\cdot(l+r)\cdot d+B\cdot k_{l}\cdot k_{r}\cdot d)$, where $d=Max(d_{1},d_{2})$. Since $k_{l}<<l$, $k_{r}<<r$, it can be approximated as $O(B\cdot(l+r)\cdot d)$.

Datasets. We conduct experiments on three public datasets. KKBox is a challenge dataset for music recommendation. Movielens contains users’ tagging records on movies. CandiCTR-Pub is a publicly available industrial dataset that is both practical and large-scale. Apart from CandiCTR-Pub, which already includes recall item sets, we manually construct recall item sets and look-alike user sets for KKBox and MovieLens by employing a pretrained matching model(e.g., DSSM) for inner product calculations between user-item and user-user pairs. Our implementation builds upon FuxiCTR and follow the public benchmark (Zhu et al., 2022) and previous works (Wang et al., 2022; Zheng et al., 2022; Lin et al., 2023).

Base models. We consider both high-order feature interaction and ensemble models. We choose IPNN, WDL, DeepFM, DCN, xDeepFM, AutoInt+, DeepIM, DCN-V2 as our base models, which has been evaluated in the BARS benchmark; see references in (Zhu et al., 2022, 2021).

Baselines. FRNet (Wang et al., 2022) learns context-aware feature representations by capturing cross-feature relationships, becoming the new SOTA. CIM (Zheng et al., 2022) encodes all candidate items into a context vector by transformer to characterize users’ implicit awareness.

Metrics. We apply the most popular metrics AUC and gAUC (weighted sum AUC, grouped by users) to evaluate the performance.

We evaluate RAR on existing models, including many SOTA methods, which is shown in Table 2. RAR notably surpasses other methods, with xDeepFM+RAR improving AUC by up to 4.7% across datasets, demonstrating the efficacy of using cross-stage data for richer user representations.

We investigate the effectiveness of different components of RAR in Table 3. Three typical base models are selected to ensure generalizability and fairness.

• •

  Removing the channel of look-alike users: RAR-user replaces the channel of look-alike users with the target user only. Table 3 shows a 2.6% gAUC and 2.5% AUC increase over raw CTR models, highlighting the benefit of incorporating recall items into ranking models for diversified user representations. However, RAR-user’s comparison to RAR indicates further potential by leveraging user common interest introduced by look-alike users.

• •

  Removing the User and Item Selection Module: RAR-select, removing the Cross-Stage Selection Module, truncates the look-alike user set and recall item set to match RAR’s scale. Table 3 reveals a 0.4% gAUC and 2.3% AUC boost over base CTR models, which indicates the importance of a careful selection for further accuracy improvement.

• •

  Removing the Co-Interaction Module: RAR-aux-wght, omitting the Co-Interaction Module and related losses, uses simple sumpooling for user interest representation and shows weaker performance. RAR-wght, dropping weighting vectors but keeping auxiliary loss, significantly outperforms RAR-aux-wght, highlighting the auxiliary loss’s role in guiding hierachical information of recall items. RAR’s superiority over RAR-wght underscores the value of both matching loss and weighting vectors.

We present a wall-time comparison of RAR and CIM, the latter of which has proven effective in a real-world search advertising system. Findings detailed in Table 4 reveal that CIM_short, utilizing a truncated context input of 50 recall items, exhibits the least time consumption, whereas CIM_long, processing the full set of 305 recall items, incurs the most time. RAR’s selection modules effectively filter noise from extensive recall pools and look-alike user sets without reducing information. Additionally, as Section 4.2 demonstrates, the performance gain of RAR is substantial.