Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study

To construct a comprehensive and challenging spatial task dataset, we employed a structured and efficient data collection strategy. The creation process involved multiple stages, starting with a team of three experts, including the authors, determining the types of questions to include. To ensure the questions reflect real-world application needs and provide broad coverage as well as significant challenges, we based them on six key dimensions:

Conceptual topics:

These questions primarily assess the model’s understanding of fundamental geographic and spatial concepts, assessing its ability to accurately comprehend and apply the core theories and practices of Geographic Information Systems (GIS)[wang2022thoughts][bolstad2012gis].

Explanatory subjects:

These questions evaluate the model’s ability to explain complex technical information and data. Explanatory questions require the model not only to understand information but also to clearly convey its meaning to users[hochmair2023perspectives].

Intellectual topics:

These questions examine the model’s ability to extract and process specific information from text[hochmair2023perspectives]. They test the model’s information retrieval and knowledge application skills, which are crucial for assessing its practical utility.

Operational problems:

These questions assess the model’s ability to apply its knowledge in real programming tasks. Operational questions test the model’s practical skills, especially in software development and coding.

Inference problems:

This type of question evaluates the model’s navigation and logical reasoning abilities in complex spatial environments. These questions require the model to understand spatial layouts and perform complex logical judgments and reasoning.

Applied problems:

These questions primarily test the model’s ability to solve problems in real-world application scenarios. Applied questions emphasize the model’s practicality and performance in real-world contexts[juhasz2023chatgpt].

Next, we designed spatial tasks aligned with the six aforementioned dimensions. Our task design strategy leveraged previous research and online resources such as Wikipedia to ensure the breadth and depth of the test content, thoroughly evaluating the models’ ability to handle various types of information. For example, in conceptual topics, we determined the categories of spatial literacy and GIS concepts based on WANG Jiayao et al. ’s summary of GIS theoretical knowledge and entrance exams for college and graduate students[wang2022thoughts]. For explanatory subjects, we drew on the research of Hartwig H. Hochmair et al. on code and function interpretation[hochmairacorrectness] and Manasi Sharma et al. ’s study on the application of LLMs to numerical trajectory data[yamada2023evaluating], establishing tasks for function interpretation, code explanation, and numerical trajectory recognition. In intellectual topics, we based the geographic feature search task on Zilong Liu et al. ’s use of DBpedia abstracts as real-world corpora to assess LLMs’ understanding of geographic diversity[liu2024measuring], and the toponymic identification task on Yingjie Hu et al. ’s evaluation of LLMs’ ability to accurately extract geographic locations from disaster-related social media information[hu2023geo]. For operational problems, we determined the task types of code generation and code translation based on Hartwig H. Hochmair et al. ’s research on the practical application of code in spatial tasks[hochmairacorrectness]. In inference problems, we defined the spatial understanding task type based on Yutaro Yamada et al. ’s research on evaluating LLMs’ representation and reasoning of spatial structures through natural language navigation tasks[yamada2023evaluating], and the route planning task type based on Mohamed Aghzal et al. ’s evaluation of LLMs’ ability to infer correct paths on a grid[aghzal2023can]. For applied problems, we determined the mapping task type based on practical operations with Mapbox and the understanding of specific parameters in Mapbox map links. After determining the categories, team members designed specific questions and reference answers based on academic papers related to each task and the provided code or examples. Each expert was responsible for different task categories, ensuring diversity and comprehensive coverage of the dataset questions.

During the design and collection process, we selected and compiled 900 questions, including rare and complex ones, distributed across 12 different categories, covering various aspects from GIS concepts to programming skills. To ensure the accuracy and practicality of the questions, the entire dataset underwent multiple rounds of rigorous review and validation. Specifically, in categories like spatial literacy, GIS concepts, and mapping, we required that the designed questions have not only theoretical value but also practical application value. The final step involved verifying the matching of questions, answers, and task categories to ensure that each question was appropriately set for real-world applications. It is important to note that some categories, such as spatial literacy and GIS concepts, involve subjective questions that do not have standard answers.

Next, this section will detail the structure and functionality of each category, along with corresponding question examples.

In this spatial task dataset test, we included several top large language models from both domestic and international sources. For international models, we selected three from OpenAI: gpt-3.5-turbo, gpt-4-turbo-2024-04-09, and gpt-4o. Among them, gpt-3.5-turbo is regarded as a milestone in large language models due to its coherent and human-like responses, making it a benchmark for industry performance evaluations. gpt-4-turbo-2024-04-09 is widely considered one of the most powerful language models globally, ranking highly on multiple model test leaderboards such as MMLU and MATH. gpt-4o (”o” stands for ”omni”), OpenAI’s latest model, offers faster responses and has reached new performance heights. Additionally, we selected ANTHROPIC’s claude-3-sonnet-20240229 model, which excels in understanding and generating poetic texts and represents the best combination of performance and speed in the claude series. For domestic models, we chose moonshot AI’s moonshot-v1-8k and zhipuai’s glm-4. As one of the top domestic models, glm-4 performs exceptionally in semantic understanding, code generation, and general knowledge processing, surpassing gpt-4 in some aspects. moonshot-v1-8k is also a leading large language model domestically.

Our test is divided into two rounds, aiming to comprehensively evaluate the capabilities of various large language models in handling spatial tasks based on natural language prompts. The first round is a zero-shot test, which does not involve any prompt tuning to assess the initial performance of the models. This allows us to observe the models’ direct response capabilities to spatial tasks. We used the following system prompt for testing: You are a quiz assistant answering the following questions and outputting the results without any additional explanation.

Following the first round of testing, we will conduct prompt strategy tuning for spatial tasks where performance was suboptimal to evaluate and enhance the large language models’ performance in complex spatial tasks. These strategies include:

One-shot:

In the One-shot test, the model receives an example question and its answer to establish context understanding before immediately facing a new question. This method tests whether the model can quickly learn from a single example and apply it to a new scenario without additional data or extended training. This strategy is particularly useful for evaluating the model’s response speed and adaptability when encountering unknown, one-off questions, effectively revealing the model’s mechanism for absorbing and processing new information[li2023one].

Combined Techniques Prompt:

This strategy uses a combination of model-specific identity prompts, detailed test category descriptions, and specific example questions and answers to create a rich informational environment for the model. This helps the model build a more accurate context for understanding and solving problems and utilize multiple information sources to improve decision accuracy. Combined Techniques Prompt is an advanced testing method aimed at evaluating how models perform complex tasks in an information-rich, structured environment and exploring how different types of information interact to optimize model performance.

Chain of Thought:

The Chain of Thought (CoT) method requires the model to receive and process an example problem with a detailed solution process before answering questions. This strategy emphasizes the model’s ability to understand and generate logical answers by simulating human problem-solving thinking processes to enhance answer transparency and interpretability. CoT is particularly suitable for handling problems requiring advanced reasoning and logical thinking, demonstrating how the model performs in a continuous logical sequence and systematically processes and solves complex problems[wei2022chain][wei2022emergent].

Zero-shot-CoT:

Zero-shot-CoT testing involves providing the model with step-by-step thinking instructions (e.g., ”Let’s think step by step”) without prior specific training or example input, directly guiding the model to develop a thought process. This method’s core is to evaluate how the model autonomously constructs a logical chain of problem-solving without direct prior examples. It aims to test the model’s autonomous learning and innovation capabilities, particularly its adaptability and solution strategies when facing entirely new and complex problems[kojima2022large].

Through these methods, we aim to further explore and understand the potential and limitations of various models in handling specific spatial tasks.

As language models are increasingly applied to handle various complex tasks, ensuring these models provide unbiased and consistent answers has become a significant challenge. To address this need, this study developed an automated script specifically for standardizing the answer collection process, aiming to minimize human error and the potential influence of conversation history through technical means. This script directly calls the APIs of various models, interacting with them using text input only, and adheres to a single-turn dialogue mode. The single-turn dialogue mode was chosen to minimize the possible impact of conversation history on the answers, ensuring that each answer is generated independently of previous dialogue context. This operation mode is particularly suitable for evaluating the model’s ability to respond to immediate questions rather than its performance in multi-turn, continuous dialogues. Additionally, to ensure consistency of experimental conditions and to promote more predictable and consistent responses from the models, we uniformly set the temperature parameter to 1 for all models tested. This parameter setting aims to enhance the large language models’ cognitive ability in generating answers, thereby allowing for a more accurate assessment of their true performance levels.

Specifically, when testing the gpt-3.5-turbo model, we observed that the model might generate different answers to the same question in different dialogues, particularly for questions with a high degree of subjectivity. This variation is primarily due to the random sampling technique used during decoding. Although these differences are typically minor, to improve the simplicity and efficiency of data processing, we chose to collect only one answer per question. Below is a concrete example demonstrating the process of conducting a single-turn dialogue with the gpt-3.5-turbo model via an API call:

This code example clearly demonstrates how to construct an API request to ensure consistency and standardization of parameters for each request, thereby enhancing the reliability and validity of the study.

Accuracy statistics and qualitative analysis will be used to evaluate the chatbot’s responses. In statistical analysis, most answers can be clearly classified as correct or incorrect, such as multiple-choice questions or reasoning questions with clear answers. However, in open-ended tasks like explaining GIS concepts or describing trajectory data visualization, the chatbot’s responses may only approximate the correct answer. Such cases, typically found in explanatory or conceptual tasks, will be specially marked with a score of 1. Qualitative analysis will explore the challenges encountered during task execution, along with related examples and illustrations. To accurately assess the model’s ability to handle different types of questions, we use the following specific scoring method to quantify the accuracy of the responses:

• •

  Fully Correct Answer: When the model provides an answer that is completely correct and meets the expectations, we give it 2 points. This indicates that the model not only understood the core of the question but also generated an accurate and correct answer.

• •

  Partially Correct Answer: For answers that are partially correct or partially meet the requirements of the question, we give it 1 point. This situation typically occurs when the model captures part of the key points of the question but does not fully cover all the correct details or mishandles some information.

• •

  Completely Incorrect Answer: If the model’s answer is unrelated to the correct answer or completely wrong, we give it 0 points. This indicates a fundamental misunderstanding of the question by the model or that the generated response is entirely off the mark.

Human reviewers play a crucial role in the scoring process, especially when dealing with partially correct answers. Certain task categories, such as ”Spatial Literacy” and ”GIS Concepts,” often include multiple possible correct answers or require subjective judgment in the responses. In these cases, human judgment is essential for determining the final score. To ensure objectivity and consistency in scoring, we require all reviewers to engage in detailed discussions before scoring and decide the final score through a voting process. For specific task categories involving code execution, such as ”Code Generation” and ”Code Translation,” we have made appropriate adjustments to the scoring criteria:

• •

  Score of 2: The code runs perfectly, and the output format is correct, fully meeting the expected result.

• •

  Score of 1: The code is basically correct but has minor errors or deficiencies, such as syntax errors, logic errors, or format errors.

• •

  Score of 0: The code fails to run, or the result deviates significantly from the expected outcome, indicating a clear deficiency in the model’s understanding and generation of code.

Through this detailed and systematic scoring mechanism, we can more accurately measure and compare the ability and effectiveness of different models in handling various types of problems. This approach not only helps us identify each model’s strengths and weaknesses but also provides valuable data support for future model optimization and development.

To comprehensively evaluate and compare the ability of large language models in handling spatial tasks, we introduced Weighted Accuracy (WA) as a key evaluation metric. This metric assigns different weights to the scores of each response to accurately reflect the importance and difficulty of the tasks. The calculation formula for Weighted Accuracy is as follows:

The variable n(s2) represents the number of answers with a score of 2, n(s1) represents the number of answers with a score of 1, and n(s0) represents the number of answers with a score of 0.

By applying this formula, we can accurately quantify and compare the efficiency and accuracy of different models in handling spatial-related problems. This provides valuable data support and insights for the optimization and development of future models. This approach emphasizes the importance of a comprehensive assessment of model performance, ensuring that our evaluation is as thorough and impartial as possible.

After the first round of zero-shot testing, we evaluated the total scores and weighted accuracy (WA) of 12 subspace tasks, as shown in the table. In this evaluation, gpt-4o performed exceptionally well, achieving the highest number of scores of 2 and the highest overall WA. Following closely is gpt-4-turbo-2024-04-09, whose scores are almost identical to gpt-4o. Although Claude-3-sonnet-20240229 exceeded glm-4 in the number of questions scored 2, glm-4’s overall weighted accuracy remained slightly higher. The overall slightly lower performers are Moonshot-v1-8k and gpt-3.5-turbo. In the evaluation of 900 spatial tasks, gpt-4o provided completely correct answers for 627 questions and partially correct answers for 24 questions, with an overall weighted accuracy of 71%. gpt-4-turbo-2024-04-09 had an overall weighted accuracy of 69.7%, just 1.3% lower than gpt-4o. Comparatively, gpt-3.5-turbo performed the worst, with an overall weighted accuracy of only 43.8%. As one of the leading large models domestically, glm-4’s performance followed gpt-4o and gpt-4-turbo-2024-04-09, with an overall weighted accuracy of 62.4%. Claude-3-sonnet-20240229 and Moonshot-v1-8k had overall weighted accuracies of 62.1% and 53.2%, respectively. Analyzing the score distribution, we observed that the number of questions scored 1 was significantly lower than those scored 0 and 2, indicating that the models were able to provide clear answers in most cases. It is noteworthy that the performance gap between gpt-4o and gpt-3.5-turbo is substantial, which aligns with the conclusions of gpt-4o’s latest report.

Figure 4 shows the overall accuracy (WA) of various models across different spatial sub-tasks. The figure indicates that the performance order of the models remains consistent across most spatial sub-tasks. Additionally, Table 2 provides a detailed breakdown of the results for each spatial sub-task. gpt-4o and gpt-4-turbo-2024-04-09 performed the best in six different categories and achieved 100% accuracy in code explanation tasks. In contrast, gpt-3.5-turbo ranked last in performance across six categories.

The overall performance of the models on explanatory, knowledge-based, and conceptual tasks was quite good, especially in GIS conceptual tasks, where all models achieved an overall accuracy (WA) of over 80.0%. This performance demonstrates that the training corpora of large language models encompass extensive GIS knowledge and showcase the models’ strong capability in understanding and applying this knowledge. However, in numerical trajectory recognition tasks, gpt-3.5-turbo had an accuracy of only 31.0%, significantly lower than the other models. This discrepancy may be related to its training dataset, parameter settings, or training algorithm. Additionally, when it comes to operational tasks involving specific programming languages and code details, gpt-3.5-turbo and other weaker models may underperform due to insufficient specialized training.

The models generally performed poorly on reasoning and application tasks. Especially in route planning tasks, which require the models to not only understand the specific requirements but also perform complex logical reasoning, such as considering multiple possible routes and evaluating various environmental factors. Even the best-performing model, gpt-4o, had a weighted accuracy (WA) of only 12.4%, while the worst-performing model, Moonshot-v1-8k, had a WA of just 0.8%. Additionally, in spatial understanding tasks, gpt-3.5-turbo had a WA of only 1.8%, significantly lower than the other models. In mapping tasks, which require models to generate Mapbox links for specified areas or perform map visualization based on specific commands, the performance gap between the models was significant. gpt-4o performed the best with a WA of 61.9%, while Moonshot-v1-8k performed the worst with a WA of only 7.5%, 54.4% lower than gpt-4o.

Overall, although the major models performed well in explanatory, knowledge-based, and conceptual tasks, particularly demonstrating excellent accuracy in GIS conceptual tasks, their performance in reasoning and application tasks was generally poor. Path planning and mapping tasks especially highlighted the challenges models face in complex logical reasoning and handling advanced spatial information. These results indicate that while current large models can achieve high accuracy in certain areas, they still require further optimization and training for tasks that demand high-level reasoning and specialized knowledge.

To conduct a more detailed model evaluation, we further divided the spatial task dataset into three levels based on difficulty: Level I (easy), Level II (medium), and Level III (hard). Through in-depth discussions with an expert panel, we established specific criteria for classifying the difficulty of the questions: 1) Questions correctly answered and scored 2 by at least five models were classified as easy.2) Similarly, questions correctly answered and scored 2 by at least two but no more than four models were marked as medium difficulty.3) Questions correctly answered and scored 2 by at most one model (usually the best-performing model, such as gpt-4o) were defined as hard, as these questions are often the most challenging. In these three difficulty levels, the number of questions was 395 (easy), 275 (medium), and 230 (hard), totaling 900 questions. Next, we will evaluate the performance of each model based on these difficulty levels. The evaluation criteria are as follows:

• •

  If the model answers an easy question completely correctly (score of 2), it is considered to have correctly solved one easy question.

• •

  If the model answers an easy question partially correctly (score of 1), it is considered to have correctly solved 0.5 of an easy question.

• •

  The evaluation criteria for medium and hard questions are the same.

According to Figure 5, as expected, the models generally performed well on easy questions. gpt-4-turbo-2024-0409 performed the best with an accuracy rate of 0.99, while gpt-4o and Claude-3-sonnet-20240229 also had high accuracy rates of 0.98. On medium difficulty questions, gpt-4-turbo-2024-0409 also led with an accuracy rate of 0.73. gpt-3.5-turbo and Moonshot-v1-8k performed poorly, with an accuracy rate of only 0.35. In handling hard questions, all models performed generally poorly. gpt-3.5-turbo stood out slightly with an accuracy rate of 0.03, while gpt-4o and gpt-4-turbo-2024-0409 had similar performances, both at 0.13. This indicates that even the latest models faced challenges with the hardest questions. Overall, gpt-4-turbo-2024-0409 and gpt-4o performed well across all three difficulty levels, while gpt-3.5-turbo performed poorly.

It is noteworthy that using the models’ answers to classify the difficulty of questions is a simple and efficient method. Additionally, another possible approach is to seek the opinions of an expert panel to determine the difficulty level of each question. The main purpose of classifying the dataset by difficulty is to evaluate the performance of future models on this dataset, monitor their progress over time, and compare it with human performance. For example, if a model only shows high accuracy on easy questions, we cannot consider it a powerful model based on this alone. Conversely, if the model also performs well on medium and hard questions, we can consider it an outstanding and effective model for handling spatial tasks. Therefore, the ability to handle questions of varying difficulty is crucial when evaluating a model’s capabilities.

After conducting the first round of zero-shot testing, we found that the models performed poorly in path planning, spatial understanding, and mapping tasks. Additionally, based on the previous difficulty classification results, we observed that the number of hard questions in these three tasks was relatively high. Therefore, we will conduct specific prompt optimization tests for these three tasks. For the route planning and spatial understanding tasks, we used four prompt strategies for optimization testing: One-shot, Combined Techniques Prompt, Chain of Thought (CoT), and Zero-shot-CoT. When discussing the mapping task, considering that code-related issues are not suitable for prompt strategy optimization, we focused only on the first type of mapping task(see Table 4): where the model accesses online map services to construct and return a map link for a specific area. Since this type of mapping task is not suitable for testing with reasoning-based prompt strategies, we selected One-shot and Combined Techniques Prompt strategies for optimization testing. For the path planning task, examples of system prompts for each prompt strategy are shown below:

1. (a)

   One-Shot
   Question: You are in a 4 by 4 two-dimensional array. You have to avoid some obstacles: (0,3). From (1,1) to (3,0). Just use “up, down, left, right” to indicate your route.
   Answer: path: down, down, left

2. (b)

   Combined Techniques Prompt
   As a path planning expert with extensive experience in GIS, you will be responsible for designing and analyzing optimal paths that avoid obstacles.
   In the “path planning” category, the task is to navigate from a starting point to an end point while avoiding obstacles placed in the environment. The process involves computing an efficient route, usually in a gridded virtual environment, by using simple directional commands such as “up”, “down”, “left”, “right”.
   Question: You are in a 4 by 4 two-dimensional arrays. You have to avoid some obstacles: (0,3). From (1,1) to (3,0). Just use “up, down, left, right” to indicate your route. Your output is formatted as: path:
   Answer: path:down,down,left

3. (c)

   Chain of Thought (Cot)
   Question: You are in a 4 by 4 two-dimensional arrays. You have to avoid some obstacles: (0,3). From (1,1) to (3,0). Just use “up, down, left, right” to indicate your route. Your output is formatted as: path:
   Answer: From the starting point of (1,1), you need to reach the end of (3,0). In the process of moving, we need to avoid the obstacle located at (0,3). First move down one step to (2,1), then continue down one step to (3,1), and finally move one step to the left to reach (3,0) to complete the path. So the answer is: path: down, down, left.

4. (d)

   Zero-shot-CoT
   Let’s think step by step.

Route Planning Task:

By comparing the test results (as shown in Figure 6), we observed that optimizing input prompts significantly improved the models’ overall accuracy. For gpt-4o, in the initial unoptimized zero-shot test, the model’s weighted accuracy was only 12.4%. After the second round of prompt optimization, its weighted accuracy increased to 50.8%, 43.3%, 87.5%, and 68.3% under the One-shot, Combined Techniques Prompt, CoT, and Zero-shot-CoT strategies, respectively. We observed that the CoT and Zero-shot-CoT strategies had a particularly significant effect on improving the model’s weighted accuracy. For gpt-4-turbo-2024-04-09, similar improvement trends were observed in the corresponding tests. gpt-3.5-turbo achieved its highest weighted accuracy of 25.8% under the Zero-shot-CoT strategy but showed a performance decline in the One-shot test. Similarly, Moonshot-v1-8k’s weighted accuracy dropped to 0 in the One-shot test, and it did not exceed 10% under the CoT and Zero-shot-CoT strategies. Claude-3-sonnet-20240229 showed no significant improvement with the Combined Techniques Prompt and Zero-shot-CoT strategies, achieving its best performance of 9.1% with the CoT strategy. For glm-4, it achieved its best performance of 36.6% with the CoT strategy. Notably, its weighted accuracy decreased in both the One-shot and Combined Techniques Prompt tests, dropping to 0 in the Combined Techniques Prompt test.

Spatial Understanding Task:

After prompt optimization (see Figure 8), different prompt strategies significantly improved the WA of gpt-3.5-turbo, with the CoT strategy being the most effective, reaching a WA of 37.8%, which is a 36% increase from the zero-shot test’s 1.8%. gpt-4o and gpt-4-turbo-2024-04-09 showed similar growth trends, both achieving their highest weighted accuracy under the Zero-shot-CoT strategy. For Claude-3-sonnet-20240229, the prompt strategies did not significantly improve its WA, with the CoT strategy being the most effective at 51.2%, an 8.8% increase from the zero-shot test’s 42.4%. Additionally, it is worth noting that domestic large models performed poorly in this task after prompt optimization, with different prompt strategies even leading to a decline in performance. Moonshot-v1-8k only showed improvement under the Combined Techniques Prompt strategy, increasing from 46.1% in the zero-shot test to 50.6%. glm-4’s performance decreased across all prompt strategies, with the lowest WA of 32.9% under the Zero-shot-CoT strategy, a 15.6% drop from the zero-shot test’s 48.5%.

Mapping Task:

By comparing the test results (see Figure 9), we found that the gpt-4o, gpt-4-turbo-2024-04-09, and Claude-3-sonnet-20240229 models showed similar growth trends under the One-shot and Combined Techniques Prompt strategies. The highest weighted accuracies (WA) for these three models were 84.7%, 76.3%, and 66.1%, respectively. Notably, the Combined Techniques Prompt strategy significantly improved the gpt-3.5-turbo model, with its WA increasing from 22.0% in the zero-shot test to 62.7%. The One-shot strategy also brought substantial improvement to the Moonshot-v1-8k model, with its WA soaring from 10.1% in the zero-shot test to 76.3%, matching the performance of gpt-4-turbo-2024-04-09. However, the glm-4 model performed relatively poorly, with its performance even declining under the One-shot strategy. In the Combined Techniques Prompt strategy, its performance only slightly increased from 37.2% in the zero-shot test to 40.7%.

When tasked with ”creating a Mapbox map link that shows a standard view of downtown San Francisco” we found that both gpt-4o and gpt-4-turbo-2024-04-09 accurately captured the semantic essence of the task and successfully returned a Mapbox map link of downtown San Francisco. However, there was a difference in how each model determined the specific location of ”downtown San Francisco”: gpt-4-turbo-2024-04-09 chose the New Conservatory Theatre Center, while gpt-4o selected the San Francisco Museum of Modern Art as the map center. Although both models scored 2 points for this task, this difference highlights the subtle variations in how different models understand real-world scenarios.

In the first round of zero-shot testing, we observed an interesting phenomenon in the Numerical Trajectory Recognition task. When handling the question ”What is the directional description of the coordinate sequence [(5, 0), (5, 15), (15, 15), (15, 5), (5, 5)] in order from left to right?” gpt-3.5-turbo showed a different understanding compared to other models. Specifically, it considered the move from (5, 5) to (5, 0) as the last direction change, thus generating five directional descriptions for the question. To further investigate whether this phenomenon was common, we selected five different cases and conducted three rounds of testing on gpt-3.5-turbo. In these tests, we focused on the number of directional descriptions rather than their accuracy. The results showed that in 15 tests, only 5 successfully generated the correct number of descriptions (highlighted in red).

Our results indicate that the gpt-4o model demonstrates excellent overall performance, especially in complex reasoning tasks, where it stands out. The gpt-4-Turbo-2024-04-09 model follows closely. However, in specific subtasks, such as toponym recognition, the moonshot-v1-8k model even surpasses gpt-4o and gpt-4-Turbo-2024-04-09. For instance, when addressing the question, ”The Tokyo Tower was badly damaged in last night’s earthquake. Which parts of the sentence represent location descriptions?”, moonshot-v1-8k correctly identified ”Tokyo Tower” as the location description. In contrast, gpt-4o incorrectly identified both ”Tokyo Tower” and ”last night” as location descriptions. This illustrates that in tasks related to semantic recognition and understanding, moonshot-v1-8k is comparable to gpt-4o. Although glm-4 did not excel in any specific subtask, its overall performance ranked third. As powerful AI tools, large language models possess significant capabilities, but they also have their strengths and weaknesses. Therefore, selecting a language model for practical applications depends on specific needs and scenarios. For example, if addressing a complex reasoning problem such as path planning, we might prefer the gpt-4o model combined with COT prompt strategies to effectively solve the issue. This highlights the practical significance and application value of our research.

Traditional difficulty classification methods typically rely on the understanding of various experts and multiple rounds of voting to determine the difficulty levels of questions. This approach is inefficient and highly subjective, especially when dealing with datasets containing multiple task categories and a large number of questions. In this study, we adopted an innovative strategy based on the performance of models in the first round of zero-shot testing to classify the dataset difficulty. This method integrates the test results of six different large language models to categorize the difficulty levels of the dataset and further discuss the outcomes. We believe this strategy is not only more efficient but also significantly promotes subsequent research. Difficulty classification is extremely beneficial for follow-up research, as it allows researchers to more accurately assess the effectiveness of different models in handling simple, medium, and difficult problems. Through this approach, researchers can deeply analyze the models’ capabilities and limitations in complex situations. Difficulty classification guides further model development, especially when it is found that the current model performs poorly on high-difficulty problems. Researchers can then specifically enhance the model’s reasoning abilities or improve its performance on complex data. Additionally, difficulty classification enables researchers to design test and training sets more rationally, making them more challenging or balanced. This not only tests the models’ limits but also enhances their robustness.

By analyzing the performance of different models on tasks of three difficulty levels (easy, medium, hard), we found that the main differences between the models lie in their responses to medium and hard questions(see Table 3). To further explore these differences, we randomly selected medium and hard questions from different task categories and compared the performance of moonshot-v1-8k and gpt-4o(see Table 6). In path planning tasks, the difficulty increases with the size of the grid. In spatial understanding tasks, difficulty mainly depends on the complexity of the described space and the number of elements involved; more elements mean higher complexity and thus higher difficulty. We observed that gpt-4o consistently outperformed moonshot-v1-8k on these more complex problems. We believe this disparity may be due to several factors: first, gpt-4o’s training parameters may be more optimized; second, its training framework might be more advanced; and third, the quantity and quality of training data used by gpt-4o may be greater. These factors likely contribute to gpt-4o’s superior performance on complex problems.

In the ”Example Analysis” section, we found that different large language models vary in their semantic understanding and analysis of text. We compared gpt-4o with gpt-4-turbo-2024-04-09. For the mapping task, although the input question ”creating a Mapbox map link that shows a standard view of downtown San Francisco” is purely textual, it fundamentally requires a deep understanding of unstructured maps combined with structured text. Comparing the results from both models, we see that gpt-4o’s response highlights significant landmarks and commercial areas in San Francisco, such as Union Square, Golden Gate Theater, and the San Francisco Museum of Modern Art. These are typical downtown landmarks, indicating high pedestrian and commercial activity. In contrast, gpt-4-turbo-2024-04-09’s response, although covering a broader area, including residential and cultural zones west of downtown, focuses more on residential areas with fewer commercial landmarks. Thus, it does not emphasize downtown commercial and cultural activities as much as gpt-4o. Why do these differences occur? We believe the primary reason is that while gpt-4-turbo performs well in text data extraction due to its robust natural language understanding capabilities, it struggles with scenarios involving unstructured data like images. Conversely, gpt-4o seamlessly integrates data from different modalities, efficiently identifying and extracting needed information from both structured text and unstructured data like images.

It is widely recognized that large language models can significantly improve output quality by optimizing input prompts. However, our research reveals significant differences in the sensitivity of domestic and international models to prompts. In this experiment, we defined mapping, path planning, and spatial understanding as complex spatial tasks. Most test results for these three complex spatial tasks showed that international models, such as gpt-4o, exhibited ”versatility,” improving performance under various prompt strategies. This was particularly evident in reasoning tasks, where using chain-of-thought (CoT) and zero-shot-CoT significantly enhanced performance. In contrast, domestic models, particularly glm-4, showed different responses in spatial understanding tasks, with each prompt strategy seemingly leading to a performance decline. This difference may be due to several factors: 1) Differences in architecture design between glm-4 and models like gpt-4o may lead to varied performance when handling prompt strategies.2) glm-4 may be relatively weaker in basic spatial understanding and reasoning abilities compared to models like gpt-4o. Therefore, even with prompt strategy optimization, it may not significantly improve performance and could potentially degrade due to information overload or improper processing.3) glm-4 may have bottlenecks when handling complex tasks, and the introduction of prompt strategies could further increase task complexity, affecting performance. Therefore, to fully realize the potential of different large language models, it is crucial to carefully design prompts for specific problems.

To accurately assess and compare the performance of different large language models on spatial tasks, our study had two core requirements: 1) establishing an independent and diverse spatial task dataset, and 2) designing a comprehensive testing methodology, including the refinement of variable design and evaluation strategies. However, the spatial task categories used in this study are not comprehensive. Previous research has conducted more detailed analyses of other spatial task categories, such as point of interest (POI) recommendation, vector data analysis[zhang2024geogpt][chen2024mapgpt], and map analysis. Additionally, our study primarily focused on text-based spatial tasks, while real-world applications of spatial tasks often involve multimodal elements such as charts and maps. Since some models do not support images as input, our tests did not include image-related tasks. In the second round of prompt optimization testing, due to certain limitations in the spatial task outputs, we were unable to adopt more complex prompt strategies, such as Self-consistency: sampling multiple solutions and then performing majority voting[wang2023self], using complex chains instead of simple chains as context examples[fu2022complexity], and decomposing complex tasks into simpler tasks to solve them sequentially[khot2022decomposed]. Therefore, introducing multimodal test data and designing more comprehensive testing methodologies are key directions for future work.