E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

Previously, three primary methods existed for incorporating automated UI navigation on mobile devices. These options included smart assistants developed by mobile phone manufacturers such as Siri, as well as macro recording tools Rodrigues (2015); Rodrigues and Guerreiro (2014); Li (2021) and Programming by Demonstration (PBD) systems Cypher and Halbert (1993); Lieberman (2001); Guibert et al. (2004); Maués and Barbosa (2013); Li et al. (2017a). They can all translate user intentions into low-level operations and automate execution. The smart assistant is restricted to calling only the built-in applications on mobile phones and a few select external applications that are in collaboration with mobile phone manufacturers. This limitation significantly constrains its range of application scenarios. The macro recording tool’s Rodrigues (2015); Rodrigues and Guerreiro (2014); Li (2021) capabilities are limited to playing back user-recorded operations. It lacks the ability to handle tasks with altered parameters or customized actions. The PBD system not only supports the automatic generation of execution scripts through user demonstrations but also provides corresponding interfaces for users to edit scripts Cypher and Halbert (1993); Lieberman (2001); Guibert et al. (2004); Maués and Barbosa (2013); Li et al. (2017a). However, despite its usefulness, the system still has a learning curve, and the scripts are not easily applicable across various applications with similar functions.

The increasing popularity of large-scale language models and multi-modal language models, including GPT Achiam et al. (2023), LLaMA Touvron et al. (2023), BaiChuan Baichuan (2023), LLaVA Liu et al. (2024, 2023), MiniGPT4-V Zhu et al. (2023); Chen et al. (2023), and others, has led to a growing interest among researchers in utilizing these models as intelligent agents for automating UI navigation Wang et al. (2023a); Kim et al. (2024); Wen et al. (2023b); Zhang et al. (2023b); Wen et al. (2023a); Lee et al. (2023); Yan et al. (2023); Zhan and Zhang (2023); Hong et al. (2023); Yang et al. (2023).

Among them, some works Wang et al. (2023a); Kim et al. (2024); Yan et al. (2023); Wen et al. (2023a) utilize trained LLMs/MLLMs like GPT. They achieve automatic navigation on mobile devices by prompt and incorporating the knowledge of UI navigation domain in pre-trained LLMs/MLLMs. Furthermore, Li et al.  Li et al. (2023b)introduced structured self-reflection into the UI navigation agent to improve its planning capabilities, while Zhang et al. Zhan and Zhang (2023) used Chain-of-Action prompts to improve the performance of multi-modal agents on UI navigation tasks. In addition, Zhang et al. Zhan and Zhang (2023) and Hong et al. Hong et al. (2023) based on pre-trained LLMs/MLLMs, fine-tuned instructions for the content in the UI navigation field to enhance the accuracy of the model’s navigation decisions.

There are currently some studies focusing on evaluation data collection and benchmark construction in the field of UI navigation Shi et al. (2017); Liu et al. (2018); Yao et al. (2022); Rawles et al. (2024); Bai et al. (2021); Deka et al. (2017). MiniWob Shi et al. (2017) and MiniWob++ Liu et al. (2018) are established benchmarks in the field of computer UI navigation research. It requires agents to perform specific tasks in the computer environment they build through instructions such as clicks and inputs. MiniWob++ goes a step further by providing programmatically defined rewards for each decision made during execution. In addition, WebShop  Yao et al. (2022) provides the UI navigation community with a simulated e-commerce environment. In this environment, agents need to navigate multiple types of web pages and issue different actions to find and purchase products based on instructions. These environments, datasets, and benchmarks focus primarily on navigation and decision-making on web pages Shi et al. (2017); Liu et al. (2018); Yao et al. (2022). For mobile phones, UIbert Bai et al. (2021) and RICO Deka et al. (2017) provide practical page understanding benchmarks that can effectively evaluate the target detection and recognition capabilities of models or agents on the page. However, they lack the intentions and operational actions actually performed on the page and cannot evaluate the agent’s Navigation and decision-making skills. The AitW dataset Rawles et al. (2024) fills this gap by providing an extensive collection of over 6 million images and corresponding actions performed on the Android operating system. However, its focus is mostly limited to first-party applications, such as settings, clock, and Google Maps, with minimal support for third-party applications with different design styles. This limits the comprehensiveness of the evaluation capabilities of this dataset and the generalizability of models trained on this dataset.

Generally, navigating a GUI can be seen as a series of decision-making tasks when interacting with a webpage or app.

Decision-making tasks on UI pages. For a web page denoted as $S_{t}$, it hosts a range of interactive controls. We identify the collection of all possible interactions within these elements as the action set $A_{t}$. Upon selection and execution of an action $a_{t}$ from $A_{t}$ by either a user or an agent, the web page transitions to its updated state, labeled as $S_{t+1}$. We use the notation $S_{t+1}=S_{t}\wedge a_{t}$ to represent that by executing action $a_{t}$ on state $S_{t}$, the system transitions to the new state $S_{t+1}$.

Navigation tasks on UI pages. Given an initial page $S_{0}$ and a final state $S^{*}$, the objective of the UI navigation task is to ensure that $S_{T}=S^{*}$, which is achieved by sequentially applying a series of decisions $A=\{a_{0},a_{1},\cdots,a_{T-1}\}$ to transition from $S_{0}$ to $S_{T}$ through the operations $S_{0}\wedge a_{0}\wedge a_{1}\wedge\cdots\wedge a_{T-1}$. For each $t\in\{0,\cdots,T-1\}$, the action $a_{t}$ is chosen from the set $A_{t}$, which contains all possible interactive actions available on page $S_{t}$. We usually call $S^{*}$ the intention or purpose of the UI navigation task.

Therefore, the aim for a tool or model designed for UI navigation is to develop a decision-making function $\hat{a}_{t}=f(S^{*},S_{0})$ that achieves $S^{*}=S_{T}=S_{0}\wedge\hat{a}_{0}\wedge\hat{a}_{1}\cdots\wedge\hat{a}_{T-1}$, all within a limited number of steps $T$. Previous research Rawles et al. (2024); Humphreys et al. (2022) has also incorporated historical operation trajectories and states into the input for decision-making functions in the study of UI navigation tasks. For these approaches, the decision-making function can be represented as $\hat{a}_{t}=f(S^{*},S_{0},\cdots,S_{t-1},a_{0},\cdots,a_{t-1})$. Additionally, for ease of annotation and comprehension, the target state $S^{*}$ is typically described using a single sentence $p$.

We design an annotation systems for annotators to interact with and record tasks to record the real human’s behaviour on tiny apps. The annotation system establishes a real-time connection with an Android emulator in the backend, while the annotators interact with the frontend which contains a mobile interface and task description. The entire data collection process is as follows: (1)The backend server synchronizes screenshots from the backend Android device to the frontend interface. (2)Annotators act on the interface according to specified tasks, such as clicking buttons, scrolling pages, entering content, and navigating back, etc. (3)The backend server records the actions and synchronizes the current screenshot, operation coordinates, and text to the cloud as a record. (4)The backend server sends the operation instructions of the tagging personnel to the Android virtual machine and actually performs the corresponding operations. (5)After the emulator performs the actions, it updates the screenshot to the frontend.

Our dataset is composed of 49,023 operational traces across diverse mini-programs within super apps. It encompasses both single-step and multi-step traces, spanning 27 sectors including catering, retail, healthcare, and government services, and extends to over 20,000 distinct tiny-apps and urls. For each operation trace, we will provide the corresponding operation purpose $p$, an indicator of whether the purpose was achieved, and a series of operation steps. At the same time, for each operation step, we will provide page screenshots, page layout analysis results and corresponding actions.

To gain a deeper insight into our dataset’s composition, we have visualized one of its trajectories in Figure 2. Additionally, Figure 3 illustrates the data associated with each step of the trajectory alongside the data production pipeline.

We employ three evaluation methods to assess the performance of various models on the E-ANT, specifically designed for LLMs/MLLMs. These methods include Zero-shot inference and Fine-tuning. In addition, we trained an XYLayoutLM model Gu et al. (2022) trained using the behavioral cloning method, which is a baseline for non-generative methods.

We randomly selected 1000 trajectories as our testset (about 5%), and for fine-tuning experiments, we selected 4000 trajectories (about 20%) that are not included in the testset as training data. We present our experimental results in the table 1. We can notice that for zero-shot inference, although most models can correctly infer the decision under a single step on some pictures, they perform poorly in terms of accuracy in the trajectory-level. At the same time, for the same zero-shot inference, LLaVA did not show a higher level of competitiveness than stronger LLMs such as GPT3.5 and Qwen72B, even though it added image modal input. This may be because under the premise of carrying layout parsing text input, increasing the basic ability of the text base itself is more important for decision-making than adding a modal input. In addition, the fine-tuned LLaVA can show higher accuracy than the un-fine-tuned version. Combining the understanding of data through GUI with decision-making data effectively enhances its accuracy.

Typically, human users navigate through GUI navigation tasks in a two-step approach: initially by comprehending the contents of the page (page understanding), and subsequently determining the necessary actions (action decision-making). This indicates that a profound understanding of the page significantly enhances the model’s ability to connect the input image with specific actions.

Generally speaking, if you want the model to obtain a good understanding of UI pages, you need to rely on additional training data, including page understanding data sets such as UI-Bert and RICO. However, a challenge arises due to the stylistic divergence between the UI designs in these datasets and those encountered in current UI navigation tasks. This discrepancy may lead to a disjointed integration of the two critical phases of understanding and action.

In fact, the image trajectory data utilized for GUI navigation tasks is valuable for understanding image pages. However, it lacks the necessary annotation information for each element within the image. Simultaneously, employing annotators to add further annotations to all images and their respective elements would entail a significant investment in terms of both manpower and financial resources. Another approach involves leveraging the existing MLLM for auxiliary annotation. However, we’ve observed that the current MLLM struggles to fully and accurately identify elements within GUI images. This challenge arises from the characteristic of the MLLM’s training data, which primarily consist of regular images and not specialized GUI content like app screenshots. To enhance the image trajectory data from GUI navigation tasks into high-quality image page understanding data without incurring additional costs, this paper introduces a bootstrap data augmentation method. This method leverages the existing layout parsing model and the MLLMs to expand data.

In this subsection, we describe our approach to training our multimodal UI navigation model using UI navigation data. Our methodology is structured around two critical processes: generating page understanding data and creating chained decision-making data. The goal of the first process is to enrich the model’s comprehension of webpage content, while the second process is designed to improve the model’s ability to link navigation objectives with ultimate decision-making actions.