Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain

The advent of Large Language Models (LLMs) in both practical applications and academic research has marked a significant paradigm shift in the Natural Language Processing (NLP) community’s focus towards these models and their potential. Recent years have witnessed a surge in studies employing, assessing, evaluating, and leveraging LLMs for various objectives [5]. These include traditional NLP tasks like sentiment analysis [35] and text classification [24], among others. Despite their remarkable ability to generate contextually consistent outputs using natural language and enable a broad range of tasks —from straightforward proofreading to more complex challenges like generating code for algorithms— LLMs are not without their flaws, being prone to errors, misunderstandings, and hallucinations [19]. In particular, when it comes to task- or domain-specific performance, especially in classification tasks, LLMs demonstrate conflicting results as compared to other well-grounded machine learning techniques such as transformer-based models and Naive Bayes [34]. If some works highlight LLMs’ proficiency in achieving comparable results to the current state-of-the-art [24, 7], other studies show that their extensive capabilities may lead to suboptimal performance againstmore specialized models [6, 26]. These observations have driven the NLP community towards exploring also small language models (SLMs), noted for their compact architecture and fewer parameters compared to larger models [34].

When it comes to domain-specific applications, despite the extensive discourse surrounding LLMs, there remains a scarcity of exploration into their application for environmental and climate change (CC)-related texts, a domain of growing significance within the NLP field [25, 12]. The urgency of addressing CC and ecological crisis through NLP techniques has begun to emerge as a crucial point of research. This includes tasks such as sentiment analysis [23], and stance detection [31]. Moreover, the scope of interest should further broaden to encompass a wider array of environmental issues, from deforestation to plastic pollution, underscoring the importance of this research beyond CC to a more holistic environmental perspective [13, 11].

To address these issues, we aim to explore the efficacy of open and close Large and Small Language Models (L/SLMs) in three ecology-related text classification tasks (Eco-Relevance, Environmental Impact Analysis, and Stance Detection)[11]. In particular, we utilize three representative generative models for these categories: GPT-3.5-Turbo-0125 (closed LLM); Llama-2-13b-chat-hf (open mid-LLM), Gemma-2b-it (open SLM), and compare their performance with the baseline consisting of the results for the same tasks of three small, non-generative language models. Moreover, this assessment extends to examining the models’ self-evalution capabilities through the analysis of their prompt-elicited confidence scores to measure their calibration levels¹¹1The code is available here: https://github.com/stefanolocci/LLMClassification/. Self-evaluation, the ability of language models to assess the validity of their own outputs, has proven to be crucial for enhancing accuracy and content quality when properly calibrated [21]. In summary, we pose three research questions:

RQ1: How accurately do state-of-the-art generative L/SLMs, both open-source and closed-source, identify texts related to ecology, analyze environmental impacts, and determine the stance on environmental issues?

RQ2: Compared to a baseline set by BERT-based classifiers, do these models demonstrate comparable accuracy in classifying texts within the domain of environment and CC?

RQ3: How well-calibrated are the verbalized confidence scores of LLMs and SLM in these environmental classification tasks?
Our results indicate that while LLMs generally outperform the SLM, BERT-based models still generally surpass both LLMs and SLM. Notably, the performance of large generative models, especially GPT, is significant, particularly in terms of recall. The calibration assessment revealed mixed results: GPT produced well-calibrated outputs, LLama exhibited moderate calibration, and Gemma struggled to adhere to the prompt template within multilabel settings. These findings aim to contribute to our understanding of the capabilities of L/SLMs in environmental text classification, enriching the ongoing discussion on leveraging NLP for ecological and CC research.

Given the considerable amount of work and vertiginous growth of material on the topic of LLMs, we report a limited number of studies relevant to our research. LLMs in Classification Task Text Classification, a fundamental NLP task, has progressed from traditional methods such as Naïve Bayes [18], to advanced models like BERT [9, 17]. Recent studies on LLMs in text classification present mixed outcomes when compared to traditional supervised methods. For instance, [34] explores the performance of different classes of language models in text classification, often favoring smaller models. [2] investigates LLMs across languages and tasks without a clear best performer, while [6] discusses ChatGPT’s limitations in biomedical fields compared to domain-specific models. Nonetheless, advances in prompting techniques, such as Clues and Reasoning prompts [6], and strategic prompts [7], show potential in improving LLMs’ classification abilities. [24] highlights GPT-4’s comparable performance in propaganda detection to the current state-of-the-art.
LLMs Self Evaluation The significant hype around LLMs has encouraged numerous studies on their capabilities [5] and self-evaluation skills, including calibration—the alignment between a model’s confidence and its prediction accuracy. Calibration is crucial for a model’s reliability, especially in determining when to defer to an expert [29]. Studies [29, 16] indicate LLMs’ verbalized confidences are often better calibrated than conditional probabilities, yet [33, 21] highlight their tendency towards overconfidence. Our research evaluates the accuracy and calibration quality of LLMs’ verbalized confidence scores.
LLMs in Environmental Domain Regarding the intersection of LLMs and the environmental domain, similar to other NLP subfields, efforts in this direction have been focused mostly on climate change (CC) topics rather than the broader environmental and ecological fields (which still include CC). Among the notable works, [4] and [36] propose evaluation frameworks for analyzing LLM responses to CC topics; [14] built a prototype tool for localized climate-related data leveraging LLMs; [15] developed an AI-based tool for fact-checking of CC claims utilizing an array of LLMs; [28] introduced a family of domain-specific LLMs designed to synthesize interdisciplinary research on climate change. Unfortunately, these models require significant hardware resources to operate, making them difficult to access.

In this study, we addressed three Research Questions (RQs) regarding the performance of advanced generative language models—GPT-3.5.Turbo-0125, Llama-2-13b-chat-hf, and Gemma-2b-it—in ecological domain tasks: Eco-Relevance, Environmental Impact Analysis (EIA), and Stance Detection. Our findings reveal: RQ1 showed variable effectiveness across models, with GPT demonstrating particular strength in Eco-Relevance, outperforming others. RQ2 assessed if these models could exceed a BERT-based benchmark. GPT excelled in the EIA task, yet no model consistently surpassed all baseline metrics. Gemma significantly underperformed compared to the other models. RQ3 evaluated the calibration of verbalized confidence scores, noting GPT’s consistent reliability. Conversely, LLaMA and Gemma’s calibration varied, indicating a need for refinement. This study highlights the strengths and areas for improvement of generative models in environmental classification, contributing to the dialogue on NLP’s role in addressing ecological and climate issues, and underscores the potential of LLMs in domain-specific tasks.