Rethinking Transformer-based Multi-document Summarization: An Empirical Investigation

We modify the source documents instead of the summarization models to the format of: $\mathcal{D}=\{\mathbf{d}^{1},\mathbf{s},\mathbf{d}^{2},\mathbf{s},...,\mathbf{s},\mathbf{d}^{N}\}$, where $N$ is the number of documents in a document set $\mathcal{D}$, the superscript $\mathbf{d}^{n}$ represents the n-th document in the set, and $\mathbf{s}$ denotes the special tokens. We investigate different Transformer models on two MDS datasets and eleven evaluation metrics to explore the impact of the document boundary separators qualitatively and quantitatively. We analyze and compare the prediction uncertainty from different datasets and different formats of source documents by inspecting entropy values during summary generation. We aim to understand how decisions by adding document boundary separators are reflected in the model’s uncertainty. In the generation process, each predictive position $\mathbf{X}_{i}$ has an outcome probabilistic distribution $\mathbf{x}_{i1},...,\mathbf{x}_{im}$, $m$ is the number of a corpus pool. We use entropy as an uncertainty measurement which can be calculated as follows:

Because the size of the corpus pool is large and the prediction distribution is usually long-tailed Xu et al. (2020), we sort the prediction distribution $\mathbf{X}_{i}$ in descending order and get a minimal set of tokens where the sum prediction values are larger than 0.95, and then normalize the distribution. We calculate the entropy value based on the new distribution $P^{\prime}(\mathbf{x}_{ij})$. The utilization of entropy as a measure allows us to gauge the distribution of probabilities across different tokens within the predictive positions of the summaries. Higher entropy values indicate a wider spread of probabilities, suggesting that the model is less certain about the most appropriate token to choose. Conversely, lower entropy values suggest that the model is more confident in its token predictions. The quantification of uncertainty through entropy measurements and its qualitative analysis enables us to assess how the introduction of document boundary separators influences the performance of the summaries generated by Transformer-based models. This holistic approach helps us unravel the nuanced impact of document boundary separators on the MDS process and gain valuable insights into the behavior of these models in handling multiple document inputs.

Transformer structures have become an essential component of many state-of-the-art natural language processing models. However, the design of the Transformer architecture can vary dramatically, and different structures may impact the performance of the model on different tasks. In this study, we aim to evaluate the effectiveness of different Transformer structures for MDS tasks. Specifically, we focus on two types of structures: flat Transformer and hierarchical Transformer.

The flat Transformer consists of a single layer of self-attention and feed-forward neural network layers that process the input tokens sequentially. In contrast, the hierarchical Transformer has a more complex structure, where the input tokens are first grouped into sentences or documents, and then processed by local and global Transformer layers. To explore the hierarchical Transformer structure, we investigate two different granularities of high-level Transformer: sentence-level and document-level. Building on the work of Liu Liu and Lapata (2019), we make modifications to the local Transformer layers to encode individual documents. The global Transformer layers are then able to exchange information at the sentence or document level.

Our analysis is motivated by the need to better understand how different Transformer structures can impact the performance of MDS models. By comparing the performance of the flat Transformer and hierarchical Transformer structures, we aim to identify which structure is more effective for multiple document summarization data.

In summarization tasks, the encoder plays a crucial role in extracting representations from the input text, while the decoder is responsible for generating the output summary, which requires producing coherent and meaningful language. Given the intricate nature of summary generation, the decoder’s role demands fine-grained control and precision, making it potentially more sensitive than the encoder. To explore the sensitivity of the encoder-decoder in Transformer-based summarization models, we add Gaussian noise at the parameter space of the encoder or decoder. We devise this experiment based on the intuition that a module (whether it’s the encoder or decoder) exhibits varying sensitivity to noise, thereby signifying the differing degrees of importance each module holds for overall performance. Formally, we have:

In this study, we aim to investigate the impact of different training strategies on Transformer models for abstractive summarization. While we have previously examined the components of Transformer models, the specific influence of training strategies remains unexplored. Our objective is to identify the most effective training strategies by leveraging the inherent characteristics of MDS datasets, without the need for external data sources. To create pseudo data utilizing the characteristic MDS, we adopt a straightforward approach. We treat one document from a given document set as a pseudo-summary while considering the remaining documents as input documents. This process is iterated, systematically selecting each document in the set as a pseudo-summary, until all input documents have served as pseudo-summaries. Consequently, we generate multiple sets of pseudo-document-summary pairs, which we refer to as pseudo-MDS dataset. The original MDS dataset is denoted as the original dataset in the subsequent analysis.

To evaluate the effectiveness of different training strategies, we design three distinct approaches. Firstly, we train the MDS model exclusively on the pseudo dataset. Secondly, we mix the pseudo dataset with the original dataset, creating a comprehensive mega dataset, on which the MDS model is trained. Lastly, we employ a two-step process, initially training the model on the pseudo dataset and subsequently fine-tuning it on the original dataset.

For abstractive MDS, a persistent challenge arises from the inclination of models to produce repetitive sentences or words during the summarization process. This tendency creates a loop that is difficult to break, hampering the generation of accurate summaries. To analyze what may cause repetitive problems, we delve into an analysis of prediction uncertainty, examining uncertainty scores throughout the generation process and localizing uncertainty to certain positions in a repetition behavior.

To quantify uncertainty, we employ Equation 1, which calculates the uncertainty score for each time slot during the summarization generation. By applying this equation, we obtain a measure of uncertainty that corresponds to the level of doubt or ambiguity associated with the generated output. The analysis focuses on observing how the uncertainty score evolves in response to the occurrence of repetition phenomena.

We evaluate the performance of three Transformer models: Vanilla Transformer (VT) Vaswani et al. (2017), Vanilla Transformer with copy mechanism (VTC), and modified Hierarchical Transformer (HT) Liu and Lapata (2019). These models are assessed on two widely used MDS datasets: Multi-XScience Lu et al. (2020) and Multi-News Fabbri et al. (2019). To comprehensively analyze their performance, we employ eleven evaluation metrics: ROUGE Lin (2004) including ROUGE-1 (R-1), ROUGE-2(R-2), ROUGE-L (R-L), ROUGE-SU (R-SU), ROUGE-WE (R-WE) Ng and Abrecht (2015), BLEU Papineni et al. (2002), S³ Peyrard et al. (2017) including pyramid (pyr) and responsiveness (resp) scores, BertScore (BS) Zhang et al. (2020), Relevance (Rel) Peyrard (2019), Redundancy(Red) Peyrard (2019).

We investigate the VT, VTC, and HT models on both datasets and report the eleven evaluation metrics to explore the impact of the document boundary separators. From Table 1, interestingly, we find that adding separators reduces models’ performance in half of the cases (3 out of 6). For example, model VT with separators performs relatively worse on Multi-News (the results of 8 evaluation metrics are worse among 11 evaluation metrics); model VTC performs relatively worse on both Multi-XScience (the results of 9 evaluation metrics are worse among 11 evaluation metrics) and Multi-News (the results of 8 evaluation metrics are worse among 11 evaluation metrics) when with separators. These results indicate input documents with separators are not very helpful for flat Transformer models. However, we can perceive that the HT model achieves better performance on both datasets with document boundary separators.

Another interesting finding is the most commonly used ROUGE, in a few cases, shows the opposite result from other evaluation metrics. For instance, on the Multi-XScience dataset, the VT (with document boundary separators) shows better ROUGE results than VT (without document boundary separators) but contradicts the results on “R-WE", “BLEU”, “S³”, ”BertScore", “Redundancy” and “Relevance". It indicates that the ROUGE-centric evaluation system needs to be updated and the measurement of summarization can not rely solely on ROUGE.

We also discover the relations between document boundary separators and token uncertainty scores. Figure 1 shows the uncertainty scores of generated tokens of VTC models on both datasets. Surprisingly, the figure reflects that separators are associated with high uncertainty score actions which means the separators increase the predictive uncertainty of models. Possible because the separators have no semantic relations with the source documents and separators may be regarded as noise to increase the predictive uncertainty. The median uncertainty score of the Multi-News is larger than the Multi-XScience aligning with the size of datasets.

We investigate (1) the effectiveness of different Transformer architectures: flat Transformer (VT,VTC) and hierarchical Transformer (HT); (2) the influences of different granularities within hierarchical Transformer structure. The results are also found in Table 1. In most evaluation metrics, the HT model can not achieve as good results as two flat Transformer models on both datasets. The two potential reasons are: (1) the pipeline of the HT model is longer than the flat Transformer models which makes the HT model hard to train. (2) the Multi-XScience and Multi-New datasets are not long document summarization datasets. The average document length of Multi-XScience and Multi-New are 778.08 and 2103.49. From the experimental results, we can conclude that the HT model is more suitable for lengthy documents, implying that flat Transformer models are a good choice for tasks with shorter documents.

As mentioned in Section 2.2, to evaluate the influences of different granularities within the hierarchical Transformer structure, we modify the local Transformer layers to encode individual documents. Figure 2 shows the performances of document-level and sentence-level HT models. All the metrics are showing better performances with the document-level HT compared to the sentence-level HT as the green line exceeds the boundary of the orange line in every dimension (redundancy is the lower the better). The apparent trend implies that a higher level of granularity is more favorable for the hierarchical Transformer structure.

To investigate the hypothesis in section 2.3, we select the VTC model as the foundation for evaluating the effectiveness of the encoder-decoder structure on the Multi-XScience and Multi-News datasets. By examining Table 2, we observe large differences in performance when introducing noise to the encoder and decoder in highly noisy scenarios (with $\alpha=1\text{e-}1$ and $\alpha=1\text{e-}2$). Specifically, in noisy conditions, we find that adding noise to the decoder has a more substantial impact on performance compared to adding noise to the encoder. However, as the noise levels decreased, the performance gaps between the two approaches narrowed. This observation supports our initial hypothesis that the decoder is more sensitive than the encoder. The potential reasons are: (1) errors or inaccuracies in the decoder can have a cascading effect on subsequent tokens generated during decoding. This error propagation phenomenon can make the decoder more sensitive to small perturbations, as any mistakes or noise introduced during decoding can amplify and affect the overall quality of the generated summary; (2) Transformer-based models often employ an attention mechanism that allows the decoder to focus on different parts of the encoded input during the decoding process. The decoder’s sensitivity is crucial in effectively attending to relevant information, and even slight perturbations in the encoded input can impact the attention weights and subsequently influence the decoding process. Consequently, it underscores the crucial role played by the decoder in summarization tasks. These findings shed light on the high importance of the decoder’s contribution to the overall summarization process.

The experimental results presented in Table 3 provide an overview of the performance of the VTC model trained using different pretraining strategies on the Multi-XScience and Multi-News datasets. In the table, the VTC is trained on the original document set and golden summary pairs. The “finetune" strategy refers to the training of the model on the pseudo dataset (introduced in Section 2.4) first and then fine-tuning on the original dataset. The “self-supervised" strategy denotes training the VTC model exclusively on the pseudo dataset. The “mix" strategy illustrates training the model using a combination of the pseudo dataset and the original dataset. By comparing the results obtained from these different training strategies, we aim to identify the most effective approach for each dataset.

For the Multi-XScience, the results show that the VTC (pretrain-finetune) strategy outperforms the VTC trained on the original dataset across most metrics, indicating the effectiveness of the pretrain-finetune strategy in improving summarization quality. In contrast, the VTC (self-supervised) exhibits lower performance compared to the VTC (pretrain-finetune), suggesting that just self-supervised training is less effective for this dataset.

Similarly, for the Multi-News dataset, the results imply the VTC model achieves good performance across all metrics, with higher scores on the VTC (pretrain-finetune) strategy, showcasing improved summarization quality. Conversely, the VTC (self-supervised) and VTC (mix) strategy yields lower performance compared to the other strategies.

The comparison of these different training strategies reveals that the pretrain-finetune approach consistently leads to better summarization performance compared to the baseline VTC model and other training strategy, highlighting its effectiveness in improving summarization quality.

To find the potential reason why the finetune strategy works well, we visualize the feature distributions of three training strategies: VTC, VTC (self-supervised) VTC (finetune) using Principal Component Analysis (PCA) as illustrated in Figure 3. For the Multi-News, the features come from the encoder of the VTC (self-supervised) and the VTC (finetuning) exhibits overlapping, while maintaining distance from the plain VTC. In contrast, for the Multi-XScience, the VTC (finetune) is more similar to the plain VTC but still noticeably distinct from the VTC (self-supervised). This observation is consistent with the performance results presented in Table 3. In the case of the Multi-XScience, finetuning the model after self-supervised training significantly improves the model’s performance compared to the VTC. However, when the model is only pretrained using self-supervised learning, it performs worse than the VTC. This discrepancy can be attributed to the fact that the features of the finetuned model closely align with the VTC model’s distribution since both models possess better representations for the final prediction. Conversely, for the Multi-News, the finetuned model exhibits only marginal improvements over the VTC. This observation also explains the overlap between features from the finetuned model and the self-supervised model, as finetuning adjusts the feature distribution towards the ‘genuine’ distribution, albeit to a limited extent.

We examine the correlation between repetition and uncertainty in the process of generating summaries. To assess uncertainty, we compute a score for each token generated. Two summaries are presented: one featuring repetition and the other as a standard summary without repetition. The outcomes are depicted in Figure 4. The X-axis represents token indexes, while the Y-axis illustrates uncertainty scores for each token. In summary #1, where no repetitions occur, the uncertainties of tokens remain within a “normal" range. This suggests that the model successfully avoids repetitive patterns, resulting in lower uncertainty scores throughout the summary generation process. Conversely, in summaries #2, we observe a distinct pattern. As the repetition of tokens or phrases begins, the uncertainty scores escalate rapidly.

By comparing uncertainty scores across different time slots, we gain insights into the relationship between repetition and uncertainty in abstractive summarization. When a repetition phenomenon occurs, we observe notable changes in the uncertainty score, indicating a correlation between the two factors. Specifically, as the model generates repetitive sentences or words, the uncertainty score tends to increase. This increase in uncertainty suggests that the model becomes less confident and more uncertain about the appropriateness or relevance of the repeated elements within the summary. By understanding this relationship, we can devise strategies to mitigate repetition and subsequently enhance the quality of generated summaries. By reducing uncertainty through the minimization of repetition, we pave the way for more accurate and reliable abstractive summarization.