Product Description and QA Assisted Self-Supervised Opinion Summarization

In the e-commerce domain, reviews play a vital role in making informed decisions. However, due to the recent proliferation of online reviews, going through all the product reviews before making a decision is challenging. Opinion summarization provides a solution by summarizing the opinions presented in the reviews (Hu and Liu, 2006; Wang and Ling, 2016; Angelidis and Lapata, 2018).

However, text summarization (Nallapati et al., 2016; See et al., 2017; Liu and Lapata, 2019) usually contains reference summaries which are very difficult to obtain at a large scale for opinion summarization. As a result, recent studies (Bražinskas et al., 2020; Elsahar et al., 2021) enable self-supervision by curating synthetic pairs out of review corpus by sampling one of the reviews as a pseudo summary and considering the remaining reviews as the input.

Self-supervised Opinion Summarization.  Recent approaches use self-supervision by considering one of the reviews as a pseudo-summary. Bražinskas et al. (2020) randomly selected $N$ reviews per entity to construct $N$ pseudo-summary, reviews pairs. Amplayo and Lapata (2020) sampled a review randomly and generated noisy versions of it as input reviews. Amplayo et al. (2020) used aspect and sentiment distributions to sample pseudo-summaries. Elsahar et al. (2021) selected reviews similar to a randomly sampled pseudo-summary as input reviews, based on TF-IDF cosine similarity. Wang and Wan (2021) aimed at reducing opinion redundancy and constructed highly relevant reviews pseudo-summary pairs by learning aspect and sentiment embeddings to generate relevant pairs. Im et al. (2021) used synthetic dataset creation strategy similar to Bražinskas et al. (2020) and extended it to multimodal version. Ke et al. (2022) captured the consistency of aspects and sentiment between reviews and pseudo-summary using constrained sampling. Siledar et al. (2023a) use lexical and semantic similarities for creating synthetic datasets. Our work is most similar to Elsahar et al. (2021) in using cosine similarity to select input reviews and pseudo-summary pairs. However, we use review embeddings to compute similarity instead of TF-IDF scores. Additionally, our pseudo-summary selection considers additional sources such as product description and question-answers as well. Our synthetic dataset creation strategy ensures that the pseudo-summary selection is highly relevant to all our input sources. Recent opinion summarization systems (Bhaskar et al., 2023; Hosking et al., 2023) include a large number of reviews. However, we limit our work to a fixed number of reviews to enable a fair comparison with previous approaches.

Preliminaries. For a specific product or an entity, $R=\{r_{1},...,r_{N}\}$ is the set of $N$ reviews, $D$ represents the product description, and $Q=\{q_{1},...,q_{M}\}$ represents a set of $M$ question-answer pairs such that $q_{i}$ represents the $i^{th}$ concatenated question and its corresponding answer.

Before discussing the details of our framework, we formalize the synthetic dataset creation process used to train these models. In the absence of supervised datasets, most recent approaches (Bražinskas et al., 2020; Im et al., 2021) resort to self-supervision wherein {input reviews, pseudo-summary} pairs are constructed.

Nach Bražinskas et al. (2020), we can assume that a review $r\in R$ can serve as a summary for a set of reviews $T\subseteq R-\{r\}$. This lets us create training points $(T,r)$ i.e. {input reviews, pseudo-summary}, similar to what the model will experience during inference. $T$ is fixed to size $k$, enabling comparison with existing works.

However, in the presence of additional sources such as product description $D$ and question-answer pairs $Q$, we slightly modify this definition. Instead of synthetic pairs, we construct synthetic quadruplets of the form: {input reviews, product description, question-answers, pseudo-summary}.

Algorithm 1 details the process of generating synthetic quadruplets. We generate multiple such quadruplets out of reviews $R$, product description $D$, and question-answer pairs $Q$ for a specific product. The overall idea for synthetic dataset creation is to choose relevant quadruplets for training. Here we define relevance as the quadruplet that best aids our model in learning the task of opinion summarization using multiple sources.

The intuition is to first select a pseudo-summary $r$ that is the closest to both $D$ and $Q$. We measure closeness in terms of cosine similarity $sim$ between their embeddings (SBERT (Reimers and Gurevych, 2019)). This selection ensures that the pseudo-summary $r$ contains information relevant to both $D$ and $Q$ so that the model learns to pick information from these two sources as well during training. Next, using the pseudo-summary $r$ selected, we look for its closest $k$ set of reviews that can act as its input reviews set $T$, which ensures that the model learns the task of summarization.

More formally, we first compute a matrix $M\in\mathbb{R}^{N\times N}$ by computing cosine similarity between embeddings of each review pair $(r_{a},r_{b})$ where $r_{a},r_{b}\in R$. We make all the diagonals of $M$ as zero to remove self-comparisons using $diag$ function. Next, we compute $ds\in\mathbb{R}^{N\times 1}$ by computing cosine similarity between the embeddings of each review $r_{a}$ and $D$. We also compute $qs\in\mathbb{R}^{N\times 1}$ by computing cosine similarity between the embeddings of each review $r_{a}$ and all $q\in Q$ and taking a $mean$ of it respectively. Finally, we compute $ss\in\mathbb{R}^{N\times 1}$ as $\lambda_{1}\cdot ds+\lambda_{2}\cdot qs$ where $\lambda_{1},\lambda_{2}$ are parameters set to $0.5$ for our experiments. We select $R_{p}\subseteq R$ reviews for forming $p$ synthetic quadruplets by taking the top-p scores from $ss$. For each review $r\in R_{p}$, we get the top-k reviews $T$ from $R-\{r\}$ using scores corresponding to the review $r$ from $M$. This lets us form synthetic quadruplet instances such as $\{T,D,Q,r\}$ for model training.

Figure 1 represents our multi-encoder framework, where each source passes through its separate encoder to generate separate attentions: $\mathbf{a_{R}}=\text{S-Attn}(\text{Enc}(T)),\;\mathbf{a_{D}}=\text{S-Attn}(\text{Enc}(D)),$ and $\;\mathbf{a_{Q}}=\text{S-Attn}(\text{Enc}(Q))$. The fused attention $\mathbf{a_{f}}$ is then computed as:

where $\odot$ represents element-wise multiplication, $\alpha$ and $\beta$ act as gates regulating the flow of information from product description and question-answers, computed as: $\alpha=\phi([\mathbf{a_{R}};\mathbf{a_{D}}]\mathbf{W_{\alpha}})$ and $\beta=\phi([\mathbf{a_{R}};\mathbf{a_{Q}}]\mathbf{W_{\beta}})$ where $\mathbf{W_{\alpha},W_{\beta}}$ are learned parameters and $\phi(\mathbf{x})=\text{RELU}(\text{tanh}(\mathbf{x}))$ is the activation, following Im et al. (2021).

Automatic Evaluation.  We use the ROUGE-{1,2,L} F1 score Lin (2004) (R1, R2 & RL) to assess the generated summary quality. Tables 3, 11 & 12 present the results on Amazon and its variants, Oposum+ and its variants, and Flipkart and its variants respectively. In general, we observe that our MEDOS model performs better than baselines and outperforms MultimodalSum on all nine test sets. Better results on GPT-RDQ versions are expected as our model and these test sets use all sources for generating summaries. However, we observe that even on the original and GPT-R test sets our models perform much better. The reason for this we believe is that under the presence of multiple sources, our models are better at figuring out what information is essential and needs to be presented in the summary. Our approach to creating synthetic datasets plays a vital role in this. By showing the model the most relevant summary that takes into consideration all the sources, our models are able to learn better the task of opinion summarization as evidenced by the results. Next, almost for all cases, we observe that MEDOS performs better than the combination of simple concatenation approach and single encoder models (BART-concat & T$5$-concat). The MEDOS model due to its multi-encoder framework is able to selectively choose relevant information from the product description and question-answers. Additionally, we observe that single encoder models encounter context limitations in most cases thereby being unable to leverage the additional sources fully.

We proposed a novel approach to create synthetic datasets by harnessing information from reviews and additional sources such as product description and question-answers. This method enables supervised training of models without the necessity of expensive annotated training datasets. Our proposed framework MEDOS uses separate encoders for selectively fusing information from these sources to generate an opinion summary. For evaluation, due to the absence of any test sets that contained such additional sources and annotated summaries, we extended the already available e-commerce test sets with additional sources and leveraged ChatGPT to annotate summaries. This resulted in six additional test sets with $\mathbf{662}$ opinion summaries in total. Results show that our synthetic dataset approach and MEDOS framework outperforms the SOTA model on average by $\mathbf{14.5\%}$ and the simple input concatenation baseline by $\mathbf{6.5\%}$ across all nine test sets. Through qualitative and comparative analysis we demonstrated that our model-generated summaries are more informative and emphasize the importance of including additional sources for comprehensive summaries.

One future work is to expand these frameworks to encompass more reviews and all available sources, creating thorough product summaries.

Our work, although uses a multi-encoder framework, is still currently limited by the size of the input. In e-commerce, reviews generally tend to be in the tens of thousands which could not be supported directly by the current model architectures. There has been research on increasing the context limits of the latest large language models, however, the performance of such models needs to be tested in the context of handling larger inputs for the task of opinion summarization. It becomes even more challenging to integrate additional sources found on product pages on e-commerce websites to provide an overall well-rounded product summary. Finally, we did not consider large language models (LLMs) in our work as our goal was to push for improvements in smaller models for multi-source opinion summarization utilizing only the available product corpus without the need for expensive large-scale annotated datasets and compute-intensive large-scale models. Our models do not use any LLM signals or LLM-generated data for training and rely only on the product corpus for learning the task of multi-source opinion summarization.

We perform our experiments on existing opinion summarization datasets as well as extend the test sets by generating summaries using ChatGPT. Some of the examples in these datasets might not be appropriate for everyone. Our models may also propagate these unintended biases due to the nature of the datasets. We urge the research community to use our models and these test sets with caution and we are fully committed to removing any discrepancies in the existing datasets in the future.