-
Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights
Authors:
Elphin Tom Joe,
Sai Dileep Koneru,
Christine J Kirchhoff
Abstract:
In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide…
▽ More
In this research short, we examine the potential of using GPT-4o, a state-of-the-art large language model (LLM) to undertake evidence synthesis and systematic assessment tasks. Traditional workflows for such tasks involve large groups of domain experts who manually review and synthesize vast amounts of literature. The exponential growth of scientific literature and recent advances in LLMs provide an opportunity to complementing these traditional workflows with new age tools. We assess the efficacy of GPT-4o to do these tasks on a sample from the dataset created by the Global Adaptation Mapping Initiative (GAMI) where we check the accuracy of climate change adaptation related feature extraction from the scientific literature across three levels of expertise. Our results indicate that while GPT-4o can achieve high accuracy in low-expertise tasks like geographic location identification, their performance in intermediate and high-expertise tasks, such as stakeholder identification and assessment of depth of the adaptation response, is less reliable. The findings motivate the need for designing assessment workflows that utilize the strengths of models like GPT-4o while also providing refinements to improve their performance on these tasks.
△ Less
Submitted 2 July, 2024;
originally announced July 2024.
-
Reproducibility, Replicability, and Transparency in Research: What 430 Professors Think in Universities across the USA and India
Authors:
Tatiana Chakravorti,
Sai Dileep Koneru,
Sarah Rajtmajer
Abstract:
In the past decade, open science and science of science communities have initiated innovative efforts to address concerns about the reproducibility and replicability of published scientific research. In some respects, these efforts have been successful, yet there are still many pockets of researchers with little to no familiarity with these concerns, subsequent responses, or best practices for eng…
▽ More
In the past decade, open science and science of science communities have initiated innovative efforts to address concerns about the reproducibility and replicability of published scientific research. In some respects, these efforts have been successful, yet there are still many pockets of researchers with little to no familiarity with these concerns, subsequent responses, or best practices for engaging in reproducible, replicable, and reliable scholarship. In this work, we survey 430 professors from Universities across the USA and India to understand perspectives on scientific processes and identify key points for intervention. Our findings reveal both national and disciplinary gaps in attention to reproducibility and replicability, aggravated by incentive misalignment and resource constraints. We suggest that solutions addressing scientific integrity should be culturally-centered, where definitions of culture should include both regional and domain-specific elements.
△ Less
Submitted 13 February, 2024;
originally announced February 2024.
-
The evolution of scientific literature as metastable knowledge states
Authors:
Sai Dileep Koneru,
David Rench McCauley,
Michael C. Smith,
David Guarrera,
Jenn Robinson,
Sarah Rajtmajer
Abstract:
The problem of identifying common concepts in the sciences and deciding when new ideas have emerged is an open one. Metascience researchers have sought to formalize principles underlying stages in the life-cycle of scientific research, determine how knowledge is transferred between scientists and stakeholders, and understand how new ideas are generated and take hold. Here, we model the state of sc…
▽ More
The problem of identifying common concepts in the sciences and deciding when new ideas have emerged is an open one. Metascience researchers have sought to formalize principles underlying stages in the life-cycle of scientific research, determine how knowledge is transferred between scientists and stakeholders, and understand how new ideas are generated and take hold. Here, we model the state of scientific knowledge immediately preceding new directions of research as a metastable state and the creation of new concepts as combinatorial innovation. We find that, through the combined use of natural language clustering and citation graph analysis, we can predict the evolution of ideas over time and thus connect a single scientific article to past and future concepts in a way that goes beyond traditional citation and reference connections.
△ Less
Submitted 11 September, 2022; v1 submitted 25 February, 2022;
originally announced February 2022.
