M2DS: Multilingual Dataset for Multi-document Summarisation

Despite being essential for various applications, MDS datasets are relatively scarce compared to SDS datasets. However, the following key datasets have significantly influenced summarisation research. DUC and TAC datasets had set early benchmarks in the news domain [28, 2]. The Multi-News [10] dataset offers substantial size and traceability in the news domain. WikiSum [24] leverages Wikipedia and search engine results for abstractive summarisation challenges. Multi-XScience [27] blends arXiv ³³3https://arxiv.org papers and Microsoft Academic Graph [37] (MAG) for scientific writing challenges. BigSurvey [25] and MS2̂ [6] contribute to scientific writing, focusing on comprehensive summaries and consolidating conflicting evidence, respectively.

Domain-specific datasets like Rotten Tomatoes [19] and WikiHow [17] diversify summarisation research into movie reviews and knowledge base articles. In customer reviews, Opinosis [11] and OPOSUM [3] are significant, with Opinosis providing professional-written golden summaries for model training and evaluation, and OPOSUM including domain and polarity information across six product categories.

However, a notable gap exists in these datasets—they primarily cater to English, highlighting the pressing need for multilingual datasets. Embracing linguistic diversity can drive global advancements in summarisation research, marking a fertile ground for innovation and exploration. The development of summarisation datasets in multiple languages stands as a promising avenue for future research and inclusivity in the field.

Transformer architecture-based models, particularly those pre-trained on large datasets, have gained attention for their ability to capture inter-document relationships and generate informative summaries. Examples include BERTSUM [26], using a hierarchical encoder, BART [20], designed as a denoising auto-encoder, PEGASUS [42], leveraging self-supervised learning, and T5 [33], a text-to-text transformer.

In the MDS domain, PRIMERA [41], based on the LongFormer Encoder-Decoder (LED) [4] architecture, stands out, surpassing previous models with a synthetic summary generation strategy during pre-training. DAMEN [30], tailored for the medical domain, combines BERT models with discriminative methods. CGSUM [5] introduces a citation-guided summarization approach for scientific papers.

Despite these advancements, challenges persist in accurately reflecting conflicting information, especially in multi-document scenarios [7]. In multilingual MDS, progress is limited, often relying on linear programming models, and summaries are often in English rather than the original languages, limiting language coverage [29].

The Workshop on Multilingual Summarisation (MultiLing)⁴⁴4https://aclanthology.org/venues/multiling/ within the ACL anthology has been a crucial focal point in multilingual summarisation research and the 2013 workshop specifically focused on Multilingual MDS [12]. During this event, a Multilingual MDS corpus was constructed, featuring languages like Arabic, English, Greek, Chinese, Romanian, Czech, Hebrew, and Spanish. The corpus creation involved selecting English texts and employing a sentence-by-sentence translation approach for the featured languages [21, 9].

In terms of model concepts, Marina et al.(2013) [29] introduced a novel text representation model extending the classic Vector Space Model [34] to Hyperplane and Half-spaces. They reformulated the extractive summarisation problem as an optimisation task using linear programming, addressing the challenge of representing a large number of extracts without explicit computation. The optimal solution was found by minimising a distance function in polynomial time. While an evaluation was not conducted, the authors suggested potential assessments using Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scores [23]. This nuanced approach to multilingual MDS, focusing on innovative text representation models and optimisation strategies, has laid a foundation for further exploration and evaluation in subsequent research endeavours [29].

In recent years, there has been a notable increase in research exploring the benefits of summarising across diverse languages, particularly in bilingual settings [39]. Our focus is directed towards datasets relevant to multilingual summarising, covering SDS, MDS, and Cross-Lingual Summarization (CLS).

While multilingual SDS has seen significant progress, research in multilingual MDS is limited. CLS, involving generating summaries in one language for documents in another language, has gained momentum with the development of multilingual SDS. Many multilingual SDS efforts have transitioned to include CLS components in their datasets [21, 9, 39, 14].

Key multilingual SDS datasets include MLSUM [36], featuring 1.5 million news articles across multiple languages; XL-Sum [14], a diverse dataset containing 1.35 million articles in 44 languages; WikiLingua [18], one of the largest parallel multilingual summarization datasets; MLGSum  [40], drawing from various news providers; and M3LS, comprising over 1.11 million multilingual multi-modal instances across 20 languages. Despite their contributions for SDS, the exploration of multilingual MDS is limited due to the absence of a dedicated high-quality dataset.

In the rapidly changing digital landscape, the significant increase in online news articles has led to a growing demand for concise and informative content. To address this need, the Multilingual Multi-document Summarisation Dataset (M2DS) has been introduced. Emphasising language inclusivity, the dataset focuses on linguistic diversity and uses BBC News as the primary source due to its global coverage and articles available in multiple languages.

We utilised the M3LS dataset to extract links of parsed articles in each language. This dataset served as a valuable foundation for creating our dataset by providing corresponding Twitter page links for each BBC news article. To ensure the reliability of the M3LS dataset [39], the authors conducted a manual assessment of article and summary quality, evaluating factors such as informativeness, length, and the ability to capture essential information. This assessment involved a meticulous review of 100 articles in four languages from their dataset. For each article, the authors carefully read the text and assigned a score between 1-5 to the golden summary, with 5 representing the best possible summary that captures most of the crucial information from the given article and vice versa. Notably, more than 70 articles across the evaluated languages received a score of over 4 out of 5 in their analysis [39].

Assuming uniformity in the quality of articles published by BBC across various domains, the authors extrapolated that this high-quality standard holds true for every language in their dataset [39]. This verification process ensured the overall quality and reliability of the BBC articles and their summaries derived from the M3LS dataset for our study.

Once we completed the first stage, we had a SDS dataset similar to the M3LS dataset, consisting of (1) BBC articles extracted from the links included in the M3LS dataset, (2) the corresponding summaries, and (3) links to related articles which are listed in the original articles. The transformation from an SDS dataset to a MDS dataset involved extracting article-summary pairs from related links, which is illustrated by Figure 1. To ensure the quality of these summaries and the relatedness of articles in each cluster, we manually verified a sample of 10 clusters, containing 2-10 articles per cluster for English, Tamil, and Sinhala languages.

Our dataset spans from 2010 to 2021, incorporating articles sourced from the M3LS dataset dated between 2010 and 2021. To expand temporal coverage, we collected links from the front page of the BBC News site, focusing on articles from December 2021 to December 2023, and ensured non-duplication. To handle duplicates, we meticulously removed repeated links, guaranteeing the uniqueness of each document cluster. Consequently, the dataset contains a diverse range of articles, free from duplication. The dataset is structured in the Hugging Face DatasetDict format, offering ease of access⁵⁵5The dataset can be found at https://huggingface.co/datasets/KushanH/m2ds and https://osf.io/7gjtm/files/osfstorage. The code and pre-trained models are available at https://github.com/KushanMH/m2ds..

The M2DS dataset encompasses articles in Sinhala, English, Japanese, Korean, and Tamil languages. Our aspiration is for the M2DS dataset to serve as a catalyst, sparking research interest in languages that have received less exploration. Each language-specific cluster within the dataset comprises two to ten documents.

The M2DS dataset comprises 180,000 documents organized into 51,500 clusters, with an average of 3.5 documents per cluster. English-language news articles contribute the highest number of documents at 67,000, whereas Sinhala has the lowest count at 23,500. The average documents per cluster vary across languages, with Japanese news having the lowest at 2.6 and Sinhala having the highest at 4.2. (See Table 1).

As the M2DS dataset is the first of its kind, we decided to compare it with existing MDS datasets. We conducted a comprehensive comparison with existing MDS datasets across various domains, considering factors such as the total number of documents, number of clusters, and the number of documents per cluster. This approach provides valuable insights into the positioning of our dataset within the landscape of English-centric MDS datasets.

To offer a holistic view, we present both aggregated numbers and language-specific statistics. This multi-faceted analysis allows for a nuanced understanding of the dataset’s characteristics in comparison to established datasets as shown in Table 1. Comparatively, when assessing M2DS against other MDS datasets, our dataset stands out with a significant overall number of documents. However, on a language-wise comparison, it exhibits a relatively lower count per language, emphasising the importance of considering linguistic variations in dataset analysis.

It is important to note that certain statistical metrics, such as average sentence length, average token count, and average word count per article and per cluster, were not included in the comparison. The rationale behind this omission is the inherent linguistic differences across languages. For instance, languages like Japanese and Korean may convey the same meaning with a lesser number of words, or in some cases, they might encapsulate an entire sentence with a single character. Consequently, direct comparisons based on these metrics could be misleading due to the diverse linguistic structures and expressions employed by different languages.

In the context of Multilingual MDS, there is a noticeable gap in the literature concerning the absence of transformer-based models. Recognising the robustness of such models, the approach in this study involved evaluating pre-trained models to establish baselines. Model selection was guided by an extensive literature review, considering factors like model performance, ROUGE scores, publication year, and venue.

The evaluation focused on three summarising models: PRIMERA, PEGASUS, and LED. PRIMERA demonstrated superior performance in previous studies, while PEGASUS showed superior sentiment understanding, particularly on the Rotten Tomatoes dataset. LED, a widely used pre-trained model, served as a baseline in the existing literature (See Table 2).

Among the models under consideration, PRIMERA emerged as a promising choice due to its distinctive approach to MDS [16]. Seeking to minimize dependency on dataset-specific modeling, it consolidates multiple documents into a single extended sequence, employing the LED architecture known for its computational efficiency. PRIMERA incorporates a sparse “local+global” attention mechanism in the encoder and introduces special document separator tokens (<doc-sep>) to indicate document boundaries. Inspired by models like PEGASUS, PRIMERA adopts a unique masking strategy based on the Entity Pyramid framework, to address the limitations in selecting representative information for summarisation [41, 20, 4].

For our baseline models, we explore simpler extractive approaches and statistical methods alongside pretrained models. In the extractive category, we employ LEAD-3 and RANDOM [39]. LEAD-3 extracts the first three sentences from the source text as the final summary, while RANDOM recursively selects words randomly from the source text until the threshold summary length is reached. These approaches serve as unbiased reference points for understanding and comparing more complex models. In the statistical approach, we experiment with CENTROID, inspired by [32]. CENTROID ranks sentences based on centrality scores derived from the words within each sentence, utilising TF-IDF scores to measure word similarity. We extract top sentences from each ranking until the threshold summary length is achieved.

Moving to pre-trained models, we select PRIMERA, PEGASUS, and LED, training them on each language’s respective training set. For tokenization, we use a space-based tokenizer for Sinhala and Tamil, the original tokenizer for PRIMERA and LED in other languages, and a space-based tokenizer for PEGASUS in all languages except English. For English, we report results both with and without fine-tuning.

Additionally, we present baseline scores for Llama 2, which is an open Large Language Model (LLM)  [38]. Llama 2, an updated version of Llama 1 and a formidable 7 billion-parameter causal decoder-only model, is introduced by Meta AI ⁶⁶6https://ai.meta.com. We limit ourselves to using open LLMs to ensure reproducibility within the research community.