OpenDebateEvidence: A Massive-Scale Argument Mining and Summarization Dataset

The High School and College Policy Debate, also known as the "cross-examination debate" (CX), involves two teams of students arguing for and against a specific policy proposal based on an annually changing broad resolution. Each debate round lasts about 90 minutes, comprising eight speeches (four by each team and two by each speaker) and cross-examination periods. The structure includes constructive speeches followed by refutations, with cross-examination periods allowing debaters to clarify arguments or challenge assumptions. New arguments are restricted to constructive speeches.

During a debate, teams present evidence from various sources to support their arguments. This evidence is usually in the form of written "cards" ¹¹1Policy Debaters used to literally cut their evidence out of magazines and glue it onto physical cards; while this has fallen out of fashion, the name stuck., such as research publications, academic articles, news reports, or government documents. Figure 1 shows an example of a "card." The quality and quantity of evidence used in a debate round often determine the winner. Policy Debate is unique among competitive debate styles in that the quality of the speech act is secondary²²2Leading to a peculiar phenomenon known as ”speed reading” or ”spreading” to be normalized: https://en.wikipedia.org/wiki/Spreading_(debate) compared to the quality, quantity, and factuality of the evidence.

Lincoln-Douglas Debate (LD), a one-on-one format with a bimonthly topic, originated from the historic debates between Abraham Lincoln and Stephen Douglas. Popular in high school and college competitions, LD debates share structural similarities with Policy Debate but feature shorter speech times and cross-examination periods. LD debates emphasize ethical and moral reasoning, focusing more on philosophical arguments rather than policy implications. However, they still prioritize the quality and quantity of evidence presented.

Public Forum Debate is a two-on-two format debating a monthly topic designed to be accessible to a broader audience. Compared to Policy and LD debates, Public Forum rounds have shorter speaking times and place less emphasis on evidence. Public Forum Debate constitutes a smaller portion of the evidence in OpenDebateEvidence and was not included in DebateSum.

Significant prior work in argument mining has focused on competitive formal debate. IBM’s Project Debater has been a leading effort, publishing extensively on argument detection (Ein-Dor et al., 2019), argument quality (Gleize et al., 2019), key point analysis/summarization (Bar-Haim et al., 2020; Magnusson & Friedman, 2021), and autonomous debating systems (Slonim et al., 2021). However, their work does not focus on the real-world competitive debate evidence found in our dataset.

Other notable contributions include VivesDebate, a multilingual audio dataset of debate tournaments (Ruiz-Dolz & Iranzo-Sánchez, 2024); ArgAnalysis35K, focusing on single argument analysis pairs in evidence-free parliamentary debate (Joshi et al., 2023); IAM (Integrated Argument Mining), a highly annotated dataset for integrated argument mining tasks with only $1{,}000$ articles (Cheng et al., 2022); and DebateSum, a dataset with $240{,}566$ examples focusing on pre-season debate evidence (Roush & Balaji, 2020). Additionally, several legal summarization datasets have been developed, including ArgLegalSumm (Elaraby & Litman, 2022), Multi-LexSum (Shen et al., 2022), and datasets targeting Indian and British case law (Shukla et al., 2022), which together total fewer than $10{,}000$ examples.

Other resources include logos.app Community (2024c), debate.cards (Community, 2024b), and contention.ai (Community, 2024a), which index various debate evidence and generate new evidence from web searches. Datasets targeting biased or query-focused summarization include QBSUM, a Chinese dataset with $49{,}000$ samples (Zhao et al., 2021); QMSum, which studies meeting summarization with $1{,}808$ samples (Zhong et al., 2021); and LMGQS, a dataset with over 1 million documents converted to query-focused summarization (Xu et al., 2023). In contrast, our dataset is fully human-created and human-annotated by active debate competitors.

Compared to these datasets, OpenDebateEvidence offers a significantly larger scale and scope, with over 3.5 million documents enriched with detailed metadata.

OpenDebateEvidence is sourced from the OpenCaseList project (Hardy, 2024), an online platform where high school and college debate teams disclose and open-source their evidence. The dataset contains over 3.5 million documents, covering all NSDA debate topics from 2012 to 2023.³³3A list of these topics can be found here. Each document corresponds to a single piece of evidence used in a debate, categorized by debate format (Policy, LD, Public Forum), and includes comprehensive metadata such as author, date, title, source, citation details, and the debate round in which it was used⁴⁴4An example of some of these downloads can be found here.

The dataset also includes standardized tags to describe the type of argument made by the document, such as topicality, disadvantages, advantages, and counter plans, along with details of the structure and location in the debate file from which the document was extracted. To protect privacy, identifying information has been anonymized.

Debate evidence is stored in the .docx file format, requiring a specialized parsing process to extract relevant information. The parsing pipeline begins by unzipping the .docx file to access the internal XML files. Ensuring accurate preprocessing is paramount for maintaining dataset quality. This process involves detailed steps to preserve the integrity and consistency of the data, including tokenization, simplification, and structuring of text blocks, followed by extracting and organizing individual debate cards into a structured format that captures both metadata and content.

The XML files are parsed to extract formatting details such as underlining, bold, and highlighting. Next, the document undergoes tokenization, creating a structured representation with text blocks representing paragraphs or coherent units of text along with their formatting information. A simplification step removes unnecessary formatting and merges adjacent tokens with similar styling.

To extract individual debate cards, the parsing procedure identifies card boundaries based on formatting and structure, extracting components such as the tag, citation, and body text. This information is organized into a structured format that captures the metadata and content of each debate card. Finally, the parsed dataset is converted back into a cleaned Hugging Face dataset, providing a human-readable version of the dataset. This structured dataset serves as the foundation for further natural language processing tasks.

After parsing the dataset and extracting individual cards, identifying and removing duplicates is essential to ensure data quality. Deduplication involves comparing the textual content of each card to identify those sharing significant portions of their text. This process enhances dataset usability by eliminating redundancy, ensuring each unique argument is represented only once.

The deduplication algorithm splits each card’s text into sentences. These sentences are then preprocessed by removing non-letter characters and converting them to lowercase. Short sentences below a certain length are filtered out to focus on meaningful content.

The algorithm retrieves and compares card IDs with a significant number of shared sentences. If the number of matching sentences exceeds a predefined threshold and their positions within the cards are within a certain range, the cards are considered duplicates. Duplicate clusters are formed by identifying all cards connected through shared sentences. A representative card is then selected from each cluster based on factors such as sentence count and content quality, and duplicates are removed iteratively.

The OpenDebateEvidence dataset offers a comprehensive collection of over 3.5 million documents categorized by debate format (Policy, Lincoln-Douglas, and Public Forum). Each document is enriched with extensive metadata, including author, date, title, source, citation details, and debate round information. Standardized tags describe the type of argument, such as topicality, disadvantages, advantages, and counterplans.

Policy Debate evidence constitutes approximately two-thirds of the dataset, Lincoln-Douglas Debate evidence comprises about one-third, and Public Forum Debate evidence makes up a smaller percentage. Spanning topics from 2012 to 2023, the dataset represents over 1,400 schools and includes contributions from more than 3,200 authors.

Key statistics of the dataset are provided in Table 1, and more detailed statistics and information can be found in Appendix F.

Each evidence document is organized with a “hat,” “pocket,” and “tag” to represent its role within a debate case.

The “pocket” indicates the top-level speech section the evidence supports, such as “1NC” for the first negative constructive speech. The “hat” denotes the broad argument category, like “Oil Disadvantage,” which aligns with a structured argument against an affirmative case. The “tag” provides a concise, biased summary of the specific argument made by the evidence. Debaters often create these tags first and then find the evidence that fits the tag.

This metadata encodes the rhetorical structure and purpose of the evidence in a practical and real-world context. The “hat” and “pocket” provide the argument’s context, while the “tag” offers a concise summary of the core claim.

For argument mining, this metadata offers valuable semantic annotations for training models on argument components and relations. “Hats” and “pockets” help models learn the overarching structure, while “tags” summarize key points.

For summarization, the hierarchical metadata enables multi-level summaries: “pockets” for high-level overviews, “hats” for key categories, and “tags” for concise core claims. The biased nature of “tags” illustrates how debaters rhetorically summarize their claims and arguments. OpenDebateEvidence is particularly rich as it includes both hierarchical biased abstractive and token-level extractive summaries.

We employed three recent fine-tuning techniques for adapting our models to OpenDebateEvidence: Low-Rank Adaptation (LoRA) (Hu et al., 2021), Representation Fine-Tuning (ReFT) (Wu et al., 2024), and Orthogonalization (Arditi et al., 2023). These methods are chosen for their parameter efficiency and ability to prevent catastrophic forgetting. The details of these techniques are provided in Appendix D.

We perform our experiments on three datasets: OpenDebateEvidence, DebateSum, which is also a dataset of Policy Debate Evidence, and the billsum dataset from Kornilova & Eidelman (2019), a dataset of US legislation and summaries, to illustrate our fine-tuned models capabilities at performing argumentative summarization in many contexts.

We conducted two types of experiments: traditional NLP evaluation metrics and using GPT-4o as a judge model. All experiments were conducted on a 4xA100 machine from Microsoft Azure with parallelism, attention optimization, and 16-bit quantization enabled. All decoding/sampling settings were kept default. The seed value of "42" was used wherever possible.

The results of our experiments are shown in Tables 2, 3 and 4. Both the Mistral-7B and LLaMA3-8B models achieved promising results, with the LLaMA3-8B models generally outperforming the Mistral-7B models across all ROUGE metrics. Notably, fine-tuning on a larger subset of the dataset ($1{,}000{,}000$ examples) significantly improved performance, as evidenced by the LLaMA3-8B LoRA (1M Ex) model. This model achieved the highest scores across traditional NLP metrics and the LLM as Judge evaluations, demonstrating that extensive fine-tuning on domain-specific data is crucial for optimizing model performance in argument mining tasks.

Our results demonstrate the significant impact of using the OpenDebateEvidence dataset on model performance. On all three datasets, our fine-tuned models showed substantial improvements over the baseline models. The LLaMA3-8B LoRA model, fine-tuned on $1{,}000{,}000$ examples from OpenDebateEvidence, achieved the highest scores across all traditional NLP metrics and the LLM as Judge evaluations. This highlights the importance of extensive fine-tuning on domain-specific data.

The extensive collection of real-world debate arguments and rich metadata in the dataset provides a unique opportunity to study argument quality. By analyzing factors such as duplicate count, position in the debate round, and debate outcomes, researchers can develop models to automatically assess persuasiveness, relevance, and overall quality. This could lead to tools that offer real-time feedback and suggestions for debaters, improving debate communities’ inclusivity by reducing reliance on expensive human judges.

The hierarchical structure of arguments in the dataset enables research into multi-level argument summarization. Models can generate summaries at various granularities, from concise one-sentence summaries to detailed overviews. This aligns with the emerging interest in query-focused and hierarchical summarization in the NLP community.

With its diverse collection of arguments and counterarguments, OpenDebateEvidence is valuable for developing argument generation models. By studying successful debaters’ patterns and strategies, researchers can create systems that generate persuasive and relevant arguments on given topics. Additionally, the dataset’s balanced coverage of affirmative and negative arguments enables the development of rebuttal generation models that counter opposing arguments.

While primarily focused on competitive debate formats, the argumentation skills and techniques in the dataset are applicable across domains such as legal reasoning, policy-making, and online discussions. Researchers can develop general argument mining models for diverse argumentative texts, advancing areas like legal document analysis, opinion mining, and fact-checking. Integrating OpenDebateEvidence with fact-checking and misinformation detection datasets could yield robust models for identifying and countering misleading claims in public discourse.

The dataset captures both arguments’ logical structure and debaters’ rhetorical strategies and emotional appeals. By studying the interplay between rational argumentation and affective language, researchers can develop sophisticated models for understanding sentiment’s role in persuasion and decision-making. This has applications in political science, marketing, and human-computer interaction.

OpenDebateEvidence holds significant potential for debate coaching and education. Analyzing successful arguments and strategies can help coaches identify best practices and develop more effective training programs. The dataset can also serve as a resource for creating educational materials such as argument templates, case studies, and interactive learning tools. This will support aspiring debaters in skill development.