A3VLM: Actionable Articulation-Aware Vision Language Model

Unlike the robot-centric action representation in RT-1 [4], RT-2 [3] and ManipLLM [21], A3VLM uses an object-centric representation that focuses on the articulations and affordances of the movable parts within an object. Compared to a single articulation detection pipeline such as FlowBot++ [38], A3VLM can predict the affordance of each part of the object and locate the appropriate part to manipulate based on the desired task. GapartNet [13] has a similar representation. However, GapartNet uses nine different types of articulation prototypes, whereas we unify articulation structures into two basic types: prismatic articulation and revolute articulation.

Actionable parts, affordances, and articulation structures in A3VLM are represented as a triad: (Bounding box $\mathcal{B}$, Axis $\mathcal{A}$, Semantic label $\mathcal{S}$). Bounding box $\mathcal{B}$ locates an actionable part of interest in the given image. Axis $\mathcal{A}$ represents the articulation structure of the part. Semantic label $\mathcal{S}$ refers to the articulation type (prismatic or revolute), the link name and the action type. Examples of this representation are shown in Fig. 2.

In practice, the 3D bounding box $\mathcal{B}$ is represented by its eight vertices $\{(x_{i},y_{i},z_{i})\}_{i=1,\ldots,8}$. Here, $(x_{i},y_{i})$ is the 2D projected position of vertex $i$ in the given image’s plane, and $z_{i}$ is normalized to the range (0,1) using the maximum and minimum depth values. Axis $\mathcal{A}$ is represented using two edge points $\{(\alpha_{i},\beta_{i},\gamma_{i})\}_{i=1,2}$ in the 3D space, and it is normalized using the same method used for the bounding box $\mathcal{B}$.

As training a VLM requires colossal resources in terms of data and computation, we do not train A3VLM from scratch. Instead, we fine-tune an established VLM. To fine-tune a VLM, we need to build an instruction-following dataset, where the input is an image and a text prompt, and the answer should be structured text. In this section, we will discuss how to construct this instruction-following dataset. As we mentioned in Sec. 3.1, we use a triad $(\mathcal{B},\mathcal{A},\mathcal{S})$ to describe the location, articulation structure, and action affordance of a movable part. In practice, we do not ask the VLM to generate everything in one inference step but separate the tasks into four different types of sub-tasks. This separation allows the VLM to focus on one concept during each inference.

Within each image, we provide an annotation $(\mathcal{B},\mathcal{A},\mathcal{S})$ for each visible and movable link. We categorize all links into prismatic and revolute types. For revolute links, axis $\mathcal{A}$ is the rotation axis provided in the URDF. For prismatic links, we use the prismatic direction provided in the URDF as the axis direction, ensuring that axis $\mathcal{A}$ passes through the 3D center of the link. After determining axis $\mathcal{A}$, we project the link points along $\mathcal{A}$ and compute a minimal 2D bounding box for the projected shape. We use the longer edge of this bounding box as the x-axis, the shorter edge as the y-axis, and axis $\mathcal{A}$ as the z-axis for bounding box $\mathcal{B}$. The center of bounding box $\mathcal{B}$ is the 3D center of the link. The width, height, and length of bounding box $\mathcal{B}$ are computed based on the distance between the furthest points of the link and the center. Semantic information $S$ stores the articulation type, name, and affordable action of the link. Fig. 3 shows multiple annotation examples from different categories in the PartNet-Mobility dataset.

Noticeably, the affordable actions of the links in the PartNet-Mobility dataset are not provided. We therefore select the actions from a robotic skill library as defined in [2]. To ensure that the selected skills are compatible with the corresponding links of the objects, we use GPT-4 for skill selection. The prompt used during this process can be found in Appendix A.

Following these definitions, we construct four different sub-tasks: (1) Detection, (2) REC-Link, (3) REG-Joint, and (4) REC-Action. Each sub-task consists of an image, a text question, and a text answer. Examples of each sub-task question can be found in Tab. 1. The detection task requires the VLM to locate all manipulable parts within an image by outputting a bounding box $\mathcal{B}$ for each part. Since this task generates more than one bounding box, it does not follow the traditional definition of REC tasks. The REC-Link task involves locating a part/link based on a description. The REG-Joint task asks the VLM to provide the joint axis $\mathcal{A}$ and the joint type $\mathcal{S}$ for a part specified by a bounding box $\mathcal{B}$. REC-Action requires the VLM to locate a movable part by providing a bounding box $\mathcal{B}$ and corresponding action type $\mathcal{S}$ according to an action task, such as “Open the storage.”

A limitation of the original PartNet-Mobility dataset is the absence of texture details. As depicted in Figure 3, most objects are rendered in plain gray, which does not reflect real-world conditions. To address the simulation-to-reality (Sim2Real) gap, we employed ControlNet [39] to generate more realistic images, using depth maps as the primary control signal due to their ability to convey both geometric and semantic information. For objects with minimal depth variance, we utilized semantic segmentation as the control signal. To enhance the diversity of the generated images using Stable Diffusion, we employed ChatGPT to generate a broader range of detailed descriptions, enriching the contextual input necessary for producing varied visual outputs. Specific prompts used with ChatGPT and examples of the augmented data are detailed in Appendix E.

As previously mentioned, A3VLM utilizes an object-centric representation. To translate this into a robot movement, we need to define specific action primitives. A3VLM is designed for use with various types of robots; therefore, it is not optimized for any particular type of manipulator, such as a parallel or a suction gripper. An independent generic grasp-pose proposer is required to generate a list of grasp-pose candidates. During manipulation, we utilize the triad $(\mathcal{B},\mathcal{A},\mathcal{S})$ generated by A3VLM along with the grasp pose candidates.

We define three types of action primitives: Rotate, Slide, and Scroll. For a given link, if its corresponding joint type is prismatic, we select the slide action; if it is revolute, we choose the rotate action, unless the target link is semantically labeled as a bottle cap or scroll button, in which case we opt for the scroll action. If the action selected is “scroll”, we ensure that the grasp pose overlaps with the rotation axis $\mathcal{A}$. Otherwise, we randomly select a grasp pose within the bounding box $\mathcal{B}$ to serve as the contact point $\mathcal{C}$. We then generate a trajectory using $\mathcal{C}$ and $\mathcal{A}$ for each action type, as illustrated in Fig 5. These trajectories constitute our generated actions.

We fine-tuned the A3VLM model using the SPHINX framework [12] on eight NVIDIA A100 (80 GB) GPUs. The fine-tuning was completed in three epochs, which took approximately 24 hours. The visual encoders were kept frozen throughout the fine-tuning phase to maintain the integrity of the pre-trained features. We utilized the SPHINX-1K model, sourced directly from the official repository, as our pre-trained base. Training was conducted with a batch size of 4 and a learning rate set to $2\times 10^{-5}$.

To evaluate the action capabilities of A3VLM, we modified upon the settings of ManipLLM [21]. This benchmark utilizes Sapien [34] as the simulator and PartNet-Mobility [28] objects as the target objects. Object’s joint values will be initialized to the middle value. We use a flying Franka Panda Robot’s gripper with suction ability as the manipulator.

Evaluation Metric. The goal of this benchmark is to evaluate the model’s ability to interact with the environment. In each task, we will load an articulated object from PartNet-Mobility dataset. Then, we will drive the manipulator to interact with the articulated object. The base of the object is fixed, so the robot needs to locate a movable part and understand the correct direction to drive the part. Regarding the definition of successful task, we follow the definition in ManipLLM, where we measure the movement of a part in the articulated objects as $d$. If, after manipulation, the part’s movement $d$ exceeds a threshold $\sigma$, we define it as a success. We use the same threshold as ManipLLM, where $\sigma=0.01$²²2ManipLLM uses two thresholds $\sigma=0.01$ and $\sigma=0.1$. But their main evaluation is performed under $\sigma=0.01$..

Baselines. The robot manipulation success rates are compared for 20 training categories (used during training) and 10 testing categories (excluded from training). We compare our proposed method with five different baselines: (1). Where2Act [27]: This method processes input from point-clouds to evaluate scores for each point. The highest-scoring point is chosen as the contact point. Additionally, it predicts 100 possible orientations for the end-effector, selecting the orientation with the top score to establish the contact pose. To ensure a fair comparison, we modified the originally used parallel gripper into a suction gripper. (2). UMPNet [36]: In alignment with UMPNet’s methodology, we carry out manipulations at the predicted contact point, positioning the end-effector’s orientation perpendicular to the surface of the object. (3). Flowbot3D [10]: This approach identifies the motion direction within a point cloud, referred to as “flow”. The point demonstrating the greatest flow magnitude is selected as the point of interaction, and the flow’s direction dictates the orientation of the end-effector. (4). Implicit3D [40]: This framework formulates a manipulation strategy for subsequent tasks by employing the Transporter network to identify keypoints on 3D articulated objects. These keypoints facilitate the determination of the end-effector’s pose. (5). ManipLLM [21]: ManipLLM is the current state-of-the-art method on this benchmark. It uses a vision language model to predict the contact point and forward direction of a suction gripper given an RGBD image and a language prompt.

Action Primitives Details. Different from the mentioned baselines, A3VLM models the action in an object-centric way. To be more specific, for each object, we first detect a list of action parts with corresponding bounding boxes $\mathcal{B}$, axes $\mathcal{A}$, joint types, and link names $\mathcal{S}$. We select a random action part from the list and use its bounding box $\mathcal{B}$ and axis $\mathcal{A}$ to generate two robot trajectories. For example, for the handle of a faucet, we will generate trajectories to rotate it clockwise and counter-clockwise. We will execute these trajectories in two attempts. The task is regarded as successful if it succeeds in either try.

Result. Table. 2 shows the performance of A3VLM and five other baselines. From the table, we can see that A3VLM outperforms all baselines by a large margin for most object categories. We think this improvement comes from two folds. One fold comes from the accurate grounding of actionable parts and articulation structure and the other fold is the introduce of action primitives. Action primitives enable A3VLM to perform different actions towards different articulated objects. This result demonstrates the effectiveness of A3VLM for manipulating articulated objects.

Real-World Inference Test. To test the stability and accuracy of A3VLM in the real world, we perform an inference test for A3VLM on many objects across different categories. It turns out A3VLM is able to correctly detect the movable parts of the objects and accurately recognize the articulation structure of those parts. As shown in Fig. 6, A3VLM consistently generates correct inferences on 20 different real-world objects. It is worth-noting that A3VLM is able to correctly perform inference on objects with reflecting or transparent surface (e.g. microwave oven, pot and coke bottle), which are very challenging for point-cloud-based methods due to their inaccurate depth.

Real-World Robot Manipulation. To test the effectiveness of the manipulation ability of A3VLM, we selected five different objects from the aforementioned twenty tested categories. We used a Kuka robot equipped with a RealSense D415 depth camera and a Robotiq three-finger gripper. We followed the action primitive mentioned in Sec. 3.5. Five trials with different initial positions are performed for each object. Manipulation is considered successful if the manipulated part has moved a sufficient distance. Success rate results can be found in Tab. 3, and experiment start and end states are shown in Fig. 7. As shown in Tab. 3 and Fig. 7, A3VLM successfully manipulated the articulated objects in the real world with simple action primitives. Further implementation details on the robot experiments can be found in Appendix B.