Using Large Language Models in Public Transit Systems: San Antonio as a Case Study ^††thanks: Citation: Authors. Title. Pages…. DOI:000000/11111.

The advent of artificial intelligence (AI) and machine learning (ML) has ushered in a new era of technological advancements that are transforming various sectors, such as cyber security [1, 2, 3], healthcare [4, 5, 6], and public transportation [7, 8, 9]. Among these innovations, large language models (LLMs), such as OpenAI’s GPT series [10, 11], have demonstrated exceptional capabilities in natural language processing, understanding, and generation. These models can analyze vast amounts of data [12, 13], generate human-like text [14], and facilitate complex decision-making processes [15, 16], making them potentially invaluable tools for enhancing public transit systems.

Public transit systems are the backbone of urban mobility, providing essential services to millions of passengers daily [17, 18]. Efficient and reliable public transportation is crucial for reducing traffic congestion, minimizing environmental impact, and promoting equitable access to mobility [19]. However, transit agencies often face challenges such as fluctuating passenger demand, route optimization, real-time communication with passengers, and efficient resource allocation [20, 21]. Traditional methods of addressing these issues may fall short due to their limited scalability and adaptability.

San Antonio, one of the fastest-growing cities in the United States, presents a unique case study for examining the integration of LLMs in public transit. The city’s rapid population growth has increased the demand for efficient public transportation solutions [22, 23]. The deployment of LLMs offers a promising avenue for addressing current challenges and future demands in public transportation as well as many other domains.

This study aims to investigate the potential of LLMs to improve various aspects of San Antonio’s public transit system. Below are some of the potentials from employing LLMs in public transportation:

• •

  Optimize Route Planning and Scheduling: Evaluating how LLMs can analyze historical and real-time data to optimize routes and schedules, thereby reducing wait times and improving service reliability.

• •

  Enhance Passenger Communication: Exploring the use of LLMs for real-time interaction with passengers, providing personalized travel assistance, updates, and recommendations.

• •

  Improve Operational Efficiency: Assessing the impact of LLMs on resource allocation, including the deployment of buses and drivers, to enhance overall operational efficiency.

The integration of LLMs in public transit systems holds the potential to revolutionize urban mobility by making transportation more efficient, responsive, and user-friendly. This study not only contributes to the academic understanding of AI applications in transportation but also provides practical insights for transit authorities and policymakers. By focusing on San Antonio, a city representative of many growing urban areas, the findings can be generalized and applied to other cities facing similar challenges.

Furthermore, the research highlights the broader implications of AI in public services, emphasizing the importance of ethical considerations, data privacy, and the need for continuous evaluation and improvement. As cities worldwide grapple with the complexities of modern urbanization, the lessons learned from San Antonio’s experience with LLMs can serve as a valuable guide for future innovations in public transit systems.

In conclusion, this study endeavors to bridge the gap between cutting-edge AI technologies and practical applications in public transportation, demonstrating how LLMs can be harnessed to create smarter, more adaptive, and passenger-centric transit networks. The following sections delve deeper into the theoretical framework, detailed methodology, findings, and implications of this transformative approach to public transit management.

The integration of large language models (LLMs) like OpenAI’s GPT-4 into public transit systems is a burgeoning field that aims to enhance the efficiency, accessibility, and user experience of public transportation. LLMs can process and analyze vast amounts of data, generate human-like text, and understand complex queries, making them suitable for a range of applications in public transit. This literature review explores the current state of research on the deployment of LLMs in public transit systems, focusing on areas such as passenger information services, operational efficiency, and accessibility improvements.

One of the primary applications of LLMs in public transit is in improving passenger information services. Studies have demonstrated that LLMs can enhance the quality and accuracy of real-time information provided to passengers. For instance, researchers explored the use of GPT in generating real-time updates and personalized travel advice for passengers, [24], [25], [26], [27]. Their findings indicated that LLMs could effectively handle complex passenger queries and provide accurate, context-aware responses, thereby improving the overall passenger experience.

Furthermore, researchers highlighted the potential of LLMs in multilingual support for transit systems, [28], [29], [30]. Given the diverse linguistic backgrounds of urban populations, LLMs like GPT-4 can be trained to provide information in multiple languages, ensuring that non-native speakers have equal access to transit information. This capability not only improves user satisfaction but also promotes inclusivity and accessibility.

The paper, [31] presents an evaluation of large language models (LLMs), specifically ChatGPT, in interpreting and retrieving information from General Transit Feed Specification (GTFS) data. The study demonstrates that ChatGPT can effectively understand and respond to various queries about public transit schedules and services, showcasing its potential in enhancing transit information systems. However, the paper also highlights areas for improvement, such as the model’s occasional inaccuracies and the need for further fine-tuning to handle complex and domain-specific transit queries more reliably.

The paper, [32] explores the potential of using ChatGPT and similar large language models (LLMs) to revolutionize intelligent transportation systems. It argues that LLMs could significantly enhance various aspects of transportation, such as traffic management, passenger assistance, and operational efficiency, but also points out the challenges related to data privacy, model accuracy, and integration with existing systems.

Most LLMs today rely on learning-based methods. For example, the well-known ChatGPT [10] leverages the Transformer [33] architecture and generative pre-training (GPTs) [34, 35, 36, 11]. The output these models is inherently tied to the data they were trained on. Consequently, incorrect LLM responses can stem from multiple factors, such as limited information on a specific topic within the pre-training data or an LLM architecture (including its embedding method) incapable of correctly processing the user’s input. Therefore, differentiating between pre-trained models and architectures is crucial when evaluating learning-based LLMs.

This project aims to assess LLMs’ ability to understand GTFS (General Transit Feed Specification) and other public transportation information in two ways:

1. 1.

   Performance of Common Pre-trained Models: We will evaluate a pre-trained LLM model "as-is" by posing transportation-related questions and analyzing the accuracy of its responses. This assesses the model’s ability to leverage its existing knowledge of GTFS data and public transportation information. Errors in this experiment might indicate either limited information within the pre-training dataset on the topic or an LLM architecture unsuited for handling the specific topic or questions. We denote this as the "understanding" task in our experiments.

2. 2.

   Impact of LLM Architecture: To delve deeper into the cause of errors, we propose a second experiment, assuming the LLM models have not encountered relevant information during pre-training. Before posing a specific transportation-related question, we will provide the necessary GTFS data and public transportation information to the LLMs and instruct them to answer based on the provided information. We will then re-ask the questions that resulted in failures during the first experiment. We denote this as the "information retrieval" task in our experiments.

The findings from these tasks will offer valuable insights into the cause of errors. For instance, if the LLMs can answer the questions correctly in the second experiment but not the first, it suggests insufficient pre-training data on the specific topic within the models. Conversely, the results might indicate that even with adequate data, the LLM models struggle with the questions, potentially due to architectural limitations.

This work evaluates the ability of Large Language Models (LLMs) to understand public transportation information through two tasks: "understanding" and "information retrieval." The LLMs achieved accuracy ranging from $47.97\%$ to $98.44\%$ on the understanding task and $60.53\%$ to $90.48\%$ on information retrieval. The high performance on some understanding tasks suggests that pre-trained LLM models have acquired a significant amount of transportation-related information from their training datasets. However, the large gap between the best and worst performing tasks also indicates that the models might have been trained on an imbalanced dataset, with significantly less information on certain areas. While relevant information is given, modern LLM models can handle task about to unknown data, suggested by the high performance on the information retrieval task. However, their ability to do so seems to be significantly reduced when the task complexity increases.

This work demonstrated the use of large language models in public transit systems holds great promise for transforming how these systems operate and serve their users. From improving passenger information services and operational efficiency to enhancing accessibility, LLMs offer a wide range of applications that can significantly benefit public transit. However, the large performance gaps between the best and worst performing tasks needed to be address before using it in the real-world. In addition, addressing ethical concerns and ensuring the responsible use of these technologies will be essential as this field continues to evolve. With continued research and development, LLMs have the potential to play a pivotal role in the future of public transportation.

This work is supported by the National Science Foundation under Grant No. 2131193. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation