Assessing the Effectiveness of GPT-4o in Climate Change Evidence Synthesis and Systematic Assessments: Preliminary Insights

Climate change is one of the most pressing challenges that several regions across the world have to face in the coming decades Lee et al. (2023). Adapting to climate change is essential for ensuring long-term sustainability Styczynski et al. (2014). For decision-makers to effectively respond to this challenge, they must carefully plan their strategies based on well-documented and assessed climate adaptation evidence. This involves reviewing a vast array of scientific documents and case studies that detail adaptation efforts in different regions. The Intergovernmental Panel on Climate Change (IPCC) has formalized the assessment of this evidence through the publication of time-sensitive reports. These reports play a crucial role in informing international treaties and country-specific legislative actions.

While traditionally such assessments relied on domain experts working voluntarily in teams tasked with annotating the documents for specific aspects of climate change, this is changing of late with the incorporation of machine learning in the evidence gathering and synthesis process Berrang-Ford et al. (2021); Sietsma et al. (2024). This is because the exponential growth of scientific literature over time has made the process of managing and synthesizing the evidence increasingly challenging Bornmann et al. (2021). Automating the annotation of large volumes of scientific data can save valuable researcher time and reduce the assessment cycle, allowing decision-makers to receive quicker and more up-to-date information.

Recent advancements in neural network methods have shown to be very useful in processing documents and extracting useful information from documents in open domain data such as Wikipedia articles Martinez-Rodriguez et al. (2020). However, scientific documents present unique challenges due to their complex domain-specific terminologies and concepts. Addressing these challenges requires training models with high-quality, human-labeled data at a sufficient scale. Recent advances in Large Language Models (LLMs) have shown promise in overcoming these challenges. Their diverse training across various topics has made them effective at extracting information from scientific documents and supporting researchers in enhancing time efficiency. Although they perform well in extracting information from scientific documents in certain domains, they still struggle to fully understand the evidence presented in these articles Koneru et al. (2023).

Given the importance of accurate information extraction for decision-making, it is crucial to have a reliable model that ensures factual accuracy during the extraction phase. In this preliminary work, we explore the potential of using GPT-4o, a state-of-the-art LLM by OpenAI, without explicit domain specific training, as extractors of useful information from documents related to climate adaptation. Our contributions are as follows

• •

  We empirically evaluate the utility of GPT-4o for annotating climate-related texts for systematic assessments. This evaluation involves comparing the performance of GPT-4o to human annotators.

• •

  We evaluate the annotation capability across different levels of information complexity. For this, we test the extraction of features at varying levels of expertise: low, medium, and high. For low-level information, we identify direct feature that can be extracted without any domain expertise. Medium-level feature requires the model to use a taxonomy for extraction, while high-level feature requires an understanding of prior complex domain-specific information to make a decision.

In recent years, LLMs have gained significant attention across the world due to their ability to complete tasks on which no explicit training was provided. The variety of applications being explored by this technology has spawned a new interest in deploying them across different domains such as medical science, business, education etc. Several studies have investigated the use of LLMs as text annotators, primarily in open domain settings Ding et al. (2022); He et al. (2023). However, there is limited research on evaluating their performance in applications requiring domain expertise, specifically information extraction from scientific articles Dagdelen et al. (2024). Studies have experimented with incorporating LLMs into data annotation pipelines, particularly for annotating texts that require domain expertise. These efforts have shown potential to reduce the time and overall cost of annotation Goel et al. (2023). For instance, LLMs have been used to extract information about nanorod structure procedures from scientific texts Walker et al. (2023) and to extract information from clinical trials Ghosh et al. (2024).

In the context of Climate Change research, recent work such as the creation of Expert Confidence in Climate Statements (CLIMATEX) dataset Lacombe et al. (2023) to weigh assessment tasks in a few-shot learning setting has shown limited accuracy. However, prior to the task of assessment is the collection and streamlining of evidence from the peer reviewed literature where for example, LLMs can be useful for Named Entity Recognition (NER) Mallick et al. (2024) from the vast corpus to help organize the evidence more efficiently for synthesis. Other similar trajectories in the use of LLMs in climate change research involve fact-checking of climate change claims Leippold et al. (2024) or the utilisation of trained domain-specific climate models such as to synthesize interdisciplinary research on climate change Thulke et al. (2024) or the use of LLM agents to extract information from a database to improve climate change related information analysis Kraus et al. (2023).

To evaluate the information extraction capabilities of GPT-4o in the context of climate change data from scientific publications, we compared the annotation agreement between human-created labels and the information extracted by GPT-4o. Table 2 presents a summary of the evaluation metrics, and our findings are detailed below:

In this study, we tried to cover a diverse set of information extraction tasks in the context of climate change adaptation research to understand the feasibility of using GPT-4o, but it is in not an exhaustive list. Additionally, our findings are specific to the climate change adaptation literature and in food sector, limiting their generalizability to other domains or sub-fields of climate change research. Further studies are necessary to assess the applicability of GPT-4o across a broader range of climate change topics. We did not explore complex prompting techniques, such as Chain of Verification Dhuliawala et al. (2023), which could potentially enhance the accuracy and reliability of GPT-4o outputs. Incorporating such advanced techniques in future research might address some of the challenges we encountered, such as misclassification and the extraction of irrelevant information. Our evaluation was conducted exclusively on GPT-4o, and we did not test other LLM models, which may perform differently. Future research should include a comparison of multiple LLMs to determine if our findings are consistent across different models and architectures.

From our study we find that there are opportunities and challenges for the deployment of pre-trained LLMs in the climate evidence synthesis and assessments. We assessed the efficiency of GPT-4o’s role as an annotator and find that tasks requiring beyond low levels of expertise are challenging for GPT-4o. Future work should explore using methods to integrate knowledge for medium expertise level and learning from human feedback to improve the model performance on extraction on information that requires high levels of expertise. Further, models with such capacity when trained on task specific data, could play a complementary role in the task of adaptation tracking by governments and global agencies and eventually help in timely securing funding for necessary adaptation responses. However, it is important to emphasize that these models cannot completely replace the expert-driven process, rather a human-in-the loop system would be extremely beneficial for ensuring the integrity and effectiveness of this process.

We would like to thank the anonymous reviewers for their helpful feedback and suggestions, which have greatly improved this paper. We also want to express our appreciation to the creators of the GAMI dataset. Their hard work in gathering and organizing the dataset was essential for our work.