UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches

Quality Metric is designed to evaluate the predictive accuracy of user summaries in forecasting future activities. The evaluation involves splitting each user activity timeline into past and future activities (see Figure 1). Summaries are generated from the past activities, and the quality of these summaries is measured by how well they predict the future activities. The quality of a user summary, $Q_{s}$, is computed by aggregating performance across multiple future activity prediction tasks.

Here, $q_{s,t}$ denotes the binary prediction outcome for summary $s$ on task $t$ from the set of tasks $T_{s}$; $m$ is the threshold for the number of correct predictions required; $I[.]$ is the indicator function, where a result of 1 indicates a ”Good” summary, and 0 indicates a ”Bad” summary.

To evaluate a summarization approach on a generated summary set $S$, the Quality Metric (QM) is calculated as the percentage of summaries classified as ”Good” based on the qualities of these summaries.

The Instruction Following Metric evaluates how well the user summaries adhere to specific constraints, such as a word limit, as introduced in other works (Skopek et al. 2023). For a summarization approach, the Instruction Following Metric (IFM) is defined as the proportion of summaries within the set $S$ that meet the word limit constraint $X$.

While this metric was originally proposed in earlier studies, it remains a useful measure for assessing how effectively a summarization approach adheres to the given prompt instructions.

The Information Density Metric assesses the conciseness and informativeness of user summaries, combining elements of both the Quality Metric and the Instruction Following Metric. This metric evaluates the balance between the length of a summary and its effectiveness in predicting future activities. For each summary $s$ in the set $S$, and each associated task set $T_{s}$, the Information Density Metric (IDM) is calculated by dividing the average task prediction accuracy by the length of the summary.

This metric provides a quantitative approach to evaluate the trade-off between informativeness and brevity in user summaries, ensuring that summaries are both concise and meaningful.

This section outlines the dataset preparation and the prediction tasks used for evaluation.

User timelines from the three datasets were filtered based on activity count $L_{\Phi}$ to ensure a balance between data sufficiency and computational efficiency:

• •

  $L_{\Phi}\geq N_{low}$: Timelines with fewer than 50 activities were excluded to ensure sufficient context for generating robust summaries.

• •

  $L_{\Phi}\leq N_{up}$: Timelines with more than 200 activities were truncated to the most recent 200 to focus on recent behavior patterns and manage computational load.

These thresholds ($N_{low}=50$ and $N_{up}=200$) were chosen to balance informative summaries with processing efficiency. Table 1 shows the number of examples used for evaluation after filtering.

Four prediction tasks were defined to evaluate the quality of the summaries generated from these datasets. Each task was structured as a multiple-choice question with one correct answer and four randomly selected incorrect options. The order of all choices was randomized during execution to ensure robustness.

• •

  $t_{1}$: Predict the user’s next activity (e.g., next watched movie name, next purchased product name) based solely on the summary, assessing the summary’s ability to encapsulate immediate user behavior.

• •

  $t_{2}$: Predict the user’s next activity considering both the summary and the $N_{r}$ most recent activities, evaluating the summary’s effectiveness in conjunction with recent user data.

• •

  $t_{3}$: Predict the category of the user’s next activity (e.g., next watched movie genre, next purchased product category) based on the summary alone, testing the summary’s ability to generalize user preferences.

• •

  $t_{4}$: Predict the category of the user’s next activity using both the summary and the $N_{r}$ most recent activities, examining how well the summary integrates with recent behavior to provide accurate insights.

These tasks were designed to comprehensively assess different aspects of summary quality. The default value for $N_{r}=20$ and the threshold $m=3$ as defined in Equation 1.

To validate the effectiveness of the proposed Quality Metric (refer to Section 3.1), we conducted a human rating exercise.

We selected 200 examples from each of the three summary-extended datasets (MovieLens, Yelp, and Amazon Reviews), resulting in a total of 600 examples. These examples were based on user activity histories from English-speaking residents in the United States, and the summaries were generated using the single-step summarization approach (see Figure 3(1)) with the Gemini Advanced model (Team et al. 2023).

Six human raters (three male and three female), all English-speaking USA residents familiar with common user activities participated in the evaluation. Each dataset was evaluated by two raters (one male and one female) to enhance robustness and minimize bias, with each rater assessing 100 examples.

Given the subjective nature of summary evaluation, particularly for criteria like ”Fact Hallucinations Verification,” a unified rating guideline was implemented to standardize the process across all raters. This guideline was designed to ensure consistency and reliability in the ratings, classifying summaries as either ”Good” or ”Bad” based on the following criteria:

• •

  Query Hallucinations Verification: Assesses whether the summary accurately reflects the specified summary query. For instance, if the query is to ”summarize a user’s movie-watching preferences,” but the summary only lists movie-watching activities without indicating any preferences, the summary would fail this check.

• •

  Fact Hallucinations Verification: Evaluates whether the details in the summary align with the context of the input activities. For example, if a user’s recent activity history shows a focus on purchasing toys, but the summary inaccurately states that the user mainly purchased office supplies, it would be marked as inaccurate.

• •

  Top Category Recall: Checks whether at least one of the top three categories, such as movie genres, is mentioned in the summary. Very popular categories like ”Restaurant” in the Yelp dataset are excluded from the top three to ensure meaningful assessment.

This structured approach, with clearly defined evaluation criteria and qualified raters, ensures that the human annotation process is consistent and capable of producing reliable comparisons between prediction results and human ratings.

After obtaining prediction results and human ratings for the same user summaries, we categorized each summary into one of four possible outcomes based on the alignment between the prediction and the human rating. We then calculated the Metric-Annotator Agreement (MAA) using Equation 7, which represents the percentage of cases where the evaluation metrics and human annotators agreed on the quality of the summary. Higher scores indicate better alignment.

In this equation, the four possible outcomes are as follows:

• •

  $TP$ (True Positive): Both the prediction and the human rating classify the summary as ”Good.”

• •

  $FP$ (False Positive): The prediction classifies the summary as ”Good,” but the human rating classifies it as ”Bad.”

• •

  $FN$ (False Negative): The prediction classifies the summary as ”Bad,” but the human rating classifies it as ”Good.”

• •

  $TN$ (True Negative): Both the prediction and the human rating classify the summary as ”Bad.”

As shown in Table 2, the proposed Quality Metric demonstrated strong alignment with human ratings, with above 70% agreement across all datasets.

To further evaluate the performance of the proposed benchmark metrics, we applied them to three popular models: Gemini 1.5 Pro (Reid et al. 2024), GPT-4o (Achiam et al. 2023), and Claude 3 Haiku (Anthropic 2023). Detailed comparisons can be found in Appendix B.

In addition to UserSumBench benchmark metrics, we also considered other reference-free evaluation metrics that do not require ground-truth summaries for user summary evaluation. These metrics compare the generated summary against user activities without relying on a reference summary.

• •

  ROUGE-2 (Lin 2004): Measures the overlap of bigrams between the generated summary and user activities.

• •

  ROUGE-L (Lin 2004): Measures the Longest Common Subsequence (LCS) between the generated summary and user activities.

• •

  BLEU (Papineni et al. 2002): Evaluates the precision of n-grams in the generated summary compared to user activities.

• •

  BertScore (Zhang et al. 2019): Uses BERT embeddings to compute precision, recall, and F1 score based on the similarity of words in the generated summary and user activities.

  • -

    BertScore-precision: Measures the precision of embedding overlap.

  • -

    BertScore-recall: Measures the recall of embedding overlap.

  • -

    BertScore-F1: Combines precision and recall to provide an F1 score.

• •

  BLEURT (Sellam, Das, and Parikh 2020): Uses a pre-trained model to evaluate the quality of the generated text based on human judgments compared to user activities.

• •

  AutoEval (Chiang and Lee 2023): Adopts an LLM to automatically evaluates summaries without requiring reference summaries, focusing on coherence and relevance.

To further validate our approach, we calculated the Pearson Correlation Coefficient between human ratings and the reference-free metrics, including the proposed summary quality measurement $Q_{s}$ (see Equation 1). In this comparison, BLEURT utilized a pre-trained BLEURT-base-128 model (Sellam, Das, and Parikh 2020) for evaluation, while AutoEval was based on the Gemini 1.5 Pro model. The $Q_{s}$ also employed the Gemini 1.5 Pro model for future activity predictions. As shown in Table 3, our metric demonstrated a more reliable and higher correlation with human ratings than other metrics, as they are specifically tailored to the characteristics of user timeline activities. Despite not having a perfect correlation, the $Q_{s}$ remains highly useful for tasks like weakly supervised learning (Zhou 2018), where ground-truth summaries are limited.