Geode: A Zero-shot Geospatial Question-Answering Agent with Explicit Reasoning and Precise Spatio-Temporal Retrieval

Advancements in Large Language Models (LLMs) have ushered a new age of unprecedented improvement in text analysis, classification, and completion. Especially with the self-attention mechanisms [23], parsing and contextualization of text based prompts has led to the rise of automated chat-bots and Question-Answering (QA) agents, such as chatGPT which can answer the user questions to a high degree of precision. In addition to text based capabilities, LLMs are advancing to interact and learn from other forms of data such as images, video, and audio paving the way for Large Multi-modal Models (LMMs) such as Gemini [5].

Owing to the recent and rapid development of Large Language Models (LLMs), developing a Large Geospatial Model (LGM) has become an important and tractable goal. These models leverage spatial information, integrating various data types such as satellite imagery, GIS (Geographic Information Systems) data, and environmental readings. The end goal using this data is to understand and predict patterns, trends, and relationships within geographical contexts.

IBM and NASA recently released a geospatial foundation model called Prithvi [13] that can achieve zero-shot multi-temporal cloud gap imputation, flood mapping, and wildfire scar segmentation. Researchers have also explored contrastive spatial pretraining paradigms like CLIP [20], for geospatial visual representations with promising results. However, due to a variety of modalities with different semantics and evolution dynamics, developing unified representations of geospatial data has been a challenge computationally and logistically. Additionally, language-guided interaction with geospatial data and spatio-temporal reasoning is a relatively unexplored area.

Although, pre-trained LLMs are repositories of extensive geospatial knowledge, yet unlocking this wealth presents a nuanced technical challenge. While conventional methodologies rely on leveraging geographic coordinates, such as latitude and longitude, to pinpoint specific locations, the crux lies in the LLMs’ capacity to comprehensively grasp and interpret these numerical representations within real-world contexts. Furthermore, this challenge escalates when queries necessitate accessing real-time or spatio-temporal datasets, such as meteorological records, which may surpass the LLM’s capability to interact with and extract requisite information.

Therefore, it is essential to develop a system that can integrate geospatial data and effectively address user inquiries demanding intricate coordination between the LLM and multiple data sources across diverse geospatial modalities. Moreover, this system must also have the ability to retrieve spatio-temporal data accurately from necessary sources, enabling the sophisticated analysis and execution of complex geospatial tasks. In this context, we propose Geode–a methodology and LLM system designed to perform zero-shot Question-Answering (QA) with explicit reasoning capabilities for complex geospatial queries, coupled with precise spatio-temporal data retrieval from open-source geospatial data sources.

Geographical Information Systems (GIS) integrate and analyze diverse geospatial data to generate digital thematic maps through computational techniques [14]. GIS leverages fundamental principles of geography, cartography, and Geode, empowering end-users to formulate queries, analyze spatial information, visualize data in maps, and present the final results as detailed thematic digital maps (e.g., Clarke [4]; Maliene et al. [16]). The data handled by GIS systems encompasses a vast array of modalities, including vector and raster data formats, geodatabases, hyperspectral and multispectral data, and unstructured data. Notable examples of such systems include GeoSpark [27], geoMesa [11, 12], GeoTrellis [3], and RasterFrames [2]. These systems are typically employed to perform complex distributed processing on massive volumes of geospatial data across large clusters of high-performance computing systems, providing mission-critical insights for environmental protection, climate prediction, commerce, defense, and numerous other domains. However, these systems lack the ability to perform Natural Language Processing (NLP) tasks to address geospatial queries effectively.

Prior works in literature have focused on developing an LLM based interface to perform Natural Language Processing (NLP) tasks on Geospatial data. Mai et al. [15] showcased the practical applications of large language models (LLMs) in the geospatial domain which include tasks like recognizing fine-grained addresses, forecasting time-series data related to dementia records, and predicting urban functions. Zhang et al. [28] uses GeoGPT, an autonomous AI tool built upon GPT-3.5, designed to autonomously collect, process, and analyze geospatial data using only natural language instructions. However, both these works primarily utilized pre-trained LLMs without exploring the potential of fine-tuning these models to create a specialized foundation model tailored for geospatial applications.

In a separate study, Deng et al. (2023) developed K2, a language model specifically fine-tuned on a corpus of geoscience texts. This specialized model demonstrated remarkable performance on various natural language processing (NLP) tasks within the geoscience domain. However, the capabilities of K2 are still confined to common NLP tasks, such as question answering, text summarization, and text classification, limiting its applicability.

In contrast to the above works, a more recent approach taken in GeoLLM by Manvi et al. [17], employs an innovative method that can efficiently extract the vast trove of geospatial knowledge inherently embedded within LLMs by fine-tuning the LLMs on prompts carefully crafted with auxiliary map data obtained from open-source geospatial data repositories. Furthermore, by fine-tuning multiple LLMs, the authors of GeoLLM aim to quantify and rigorously assess the extent of geospatial knowledge encapsulated within these models. Additionally, they seek to evaluate the scalability assessment of this knowledge for a wide range of practical geospatial tasks, including population density prediction.

Despite the fine-tuning with auxiliary map data, GeoLLM encounters limitations in tasks necessitating the retrieval of spatio-temporal data across varied modalities, sourced from both real-time and offline sources. For instance, answering a question related to the air quality in a specific geographical location requires an ability to access and extract the real-time information, which potentially limits the ability of LLM to reason about complex queries on geospatial data. Moreover, the zero-shot reasoning and complex mathematical computations (such as finding standard deviation) of LLMs are poor despite providing extensive prompting.

In response, with Geode, we aim to bridge the shortcomings by providing the LLM the capability to interact and retrive real-time data from diverse data modalities to complete the required tasks. In addition, the LLM is capable of performing complex reasoning and provide a more accurate answers to the geospatial queries.

Geode is a system designed to answer geospatial user queries that may involve multiple modalities and require complex reasoning. We identified a set of key insights that informed the design of Geode as a system. We observe that the zero-shot reasoning and mathematical computation capabilities of LLMs are poor [19, 29, 25]. This can be improved to some extent with special prompting techniques like few-shot, chain-of-thought [24] and tree-of-thought prompting [26]. This is the reason why we opted to leverage explicit geospatial experts to compute answers to narrow subsets of the geospatial query space. This also leads to high degree of compositionality and extensibility to the system, as with implementation of on only a few additional experts can enable the system to answer a whole new set of queries in query space. Additionally, since geospatial inference should not be only limited to retrieval, we unify ML inference, retrieval and explicit reasoning within Geode.

Our evaluation approach includes several different queries which test the capability of Geode to perform QA which requires retrieval and augmentation of spatio-temporal data. While Geode is able to deliver zero-shot responses for the easier queries, the more complex ones may require more prompting and elaboration on particular prompt aspects. We will be using the publicly available GIS datasets [10] to train and evaluate our proposed model.