MeshSegmenter: Zero-Shot Mesh Semantic Segmentation via Texture Synthesis

Mesh segmentation aims to segment the semantic regions according to the corresponding text descriptions. The input is a untextured mesh $M_{\text{unTextured}}$, characterized by vertices $V\in\mathbb{R}^{n\times 3}$ and faces $F\in{\{1,\cdots,n\}}^{m\times 3}$, where $m$ is the number of faces. Users provide an object class text $T_{\text{o}}$ and a grounding text $T_{\text{g}}$. The objective of 3D zero-shot segmentation is to accurately segment the regions specified by the text within the given mesh.

Current 3D semantic segmentation models based on untextured meshes can only utilize shape information for segmentation, which is easily influenced by the lighting conditions and the viewpoints. We propose leveraging generated textures to incorporate texture information for mesh segmentation, rather than relying solely on shape information to address this issue. As shown in Fig. 2, we propose a novel 3D mesh semantic segmentation framework, consisting of three parts, including text-guided texture synthesis, 2D zero-shot semantic segmentation, and face confidence revoting strategy.

In text-guided texture synthesis, we generate textures for untextured mesh $M_{\text{unTextured}}$ based on object class text $T_{\text{o}}$. The texture information comes from the pre-trained generative model Stable Diffusion [38] which has great prior texture information. We render the textured meshes $M_{\text{Textured}}$ and the untextured meshes $M_{\text{unTextured}}$ to 2D images in different viewpoints and segment the 2D images according to the object class $T_{\text{o}}$ and the grounding description $T_{\text{g}}$ by the 2D zero-shot segmentation model GroundingDINO [24] and SAM [18]. Finally, we fuse the segmentation results from different views and obtained the segmented faces. Considering the inconsistency among the predictions from different views, especially from the extreme hard-case views, we further propose to fuse the 2D segments by the face confidence voting module and achieve consistency among different views.

Our method segments the textured mesh and text-guided texture synthesis first generates textures given the raw untextured meshes guided by the text descriptions. As shown in Fig. 2, the untextured meshes only provide the structure information. It is hard to localize the semantic regions without colors, for example, the car doors that are integrated with the car body structure. Therefore, we leverage the texture information generated from the pre-trained generative model such as Stable Diffusion [38] to help the mesh segmentation. The generative model is trained on large-scale of data therefore is good at generating textures.

It should be noted that the descriptions used in this module are different from the grounding descriptions. For example, our model employs “a photo of $\{T_{o}\}$ and $\{\}$ view” as the prompt for the diffusion model, while the grounding descriptions are like “hair”, “door” or “tail”. TEXTure[37] introduces different view prompts to facilitate better generation conditioned on viewpoints, which include “front”, “left”, “back”, “right”. Further details on the prompt settings for other orientations are available in the supplementary material.

Our work utilizes both $M_{\text{unTextured}}$ and $M_{\text{Textured}}$ meshes to provide geometric and texture information, respectively. The processing workflow is consistent for both types of meshes. For simplicity, we will explicitly enumerate only the segmentation steps for $M_{\text{unTextured}}$.

Get the rendered $K$ images, $x_{k}$ is the rendered image, and $\text{VisFace}_{k}$ is visible face set in the $k$-th viewpoint. Randomly generating viewpoints during rendering can have a detrimental effect on the efficiency of object semantic segmentation. This can result in the introduction of extreme perspectives, which can cause inaccuracies in semantic segmentation. To address this concern, we define a camera rendering trajectory that aims to strike a balance between the semantic segmentation results from rendered images and the coverage of all object viewpoints. In particular, we first render $M_{\text{unTextured}}$ from $K$ viewpoints $V\in\{v_{1},\dots,v_{l}\}$. We set radius $r$ as 2, and for the polar angle $\theta$ being $75^{\circ}$ and $115^{\circ}$ respectively, we have the azimuthal angle $\phi$ uniformly sampled $\frac{K}{2}$ times in a range of $0^{\circ}$ to $360^{\circ}$. The rendering process is formulated as:

GroundingDINO[24] is a novel approach to open-set object detection that combines the strengths of DINO architecture and grounded pre-training model. It takes an image $x_{k}$ and a 2D region-level description $T_{\text{g}}$ as input and accurately segments the region specified by the text into bounding boxes $b_{k}$ and corresponding 2D confidence scores $c_{k}$. The text-guided detection process is formulated as:

Segment Anything Model (SAM) [18] is a foundational approach to image segmentation that enables promotable segmentation. This means the model can be trained on a pre-training task and then used to respond appropriately to any prompt at inference time, making it highly adaptable to a range of downstream tasks. An inference approach is to use the bounding box as input and SAM[18] outputs the mask $\text{Mask}_{k}$ for the region of interest within the bounding box $b_{k}$. The process is formulated as:

Our model is specifically developed for performing zero-shot semantic segmentation of 3D objects, where the task involves delineating a region specified by text from an object. These regions are required to be partial regions of the object. Therefore, when detection produces bounding boxes that cover the entire object, we classify these as evident segmentation errors. These detection results are promptly deleted.

Based on the results of a powerful 2D detection and segmentation model, for each rendered image, we can obtain a set of bounding boxes $b_{k}$, 2D confidence scores $c_{k}$, and accurate masks $\text{Mask}_{k}$ for the regions described in text within that viewpoint. Additionally, using Nvidia Kaolin, we can obtain a face index map for that particular viewpoint. This allows us to identify visible faces $\text{VisFace}_{k}$ and determine whether they are within the mask region.

A simple approach would be to aggregate the faces of the mask region across different viewpoints to obtain the final segmentation result. However, this approach would introduce errors from incorrectly detected and segmented faces in a particular viewpoint, thus contaminating the overall segmentation result. Therefore, we propose a technique called Face Confidence Revoting.

In Face Confidence Revoting, we selectively consider the faces within the mask region with their corresponding confidence scores, while penalizing visible faces outside the mask region. This approach allows the correct segmentation results of neighboring faces to compensate for the errors in detection and segmentation within the current viewpoint. We denote the global confidence scores from $M_{\text{unTextured}}$ oder $M_{\text{Textured}}$ as $g_{m}$ For the visable faces in k-th view $v_{k}$, we set the local confidence $l_{k}$ of the masked faces to $c_{k}$ and the unmasked faces to $-c_{k}$. The global confidence of the m-th face is the sum of multi-view local confidence:

MeshSegmenter leverages the $M_{\text{unTextured}}$ and $M_{\text{Textured}}$ generated from the generative models to obtain the overall semantic segmentations. Consequently, our strategy involves setting the overall confidence scores $o_{m}$ as the average of the $g_{m}$ values of the untextured and textured meshes. When the value of $f_{m}$ exceeds $o_{\text{threshold}}$. It is considered to belong to the region determined as a text.

$R_{\text{text}}$ is a collection of faces for a text-specified region. We set the threshold as 0 for all cases to ensure a fair comparison. Finally, we averages the confidence of each face with their neighbor faces to obtain a smooth segmentation.

We leveraged a single Nvidia A40 GPU for each experiment, involving a single object, its corresponding description, and a grounding text. We use Nvidia Kaolin to render multi-view 2D images from $M_{\text{Textured}}$ and $M_{\text{unTextured}}$. We compare MeshSegmenter with 3DHighlighter[9], SATR[2], and ZSC[1]. Since ZSC[1] is not open-source, we use its official results for comparison. We utilize existing 2D detection model GroundingDINO[24] and segmentation model SAM[18], selecting the segmentation results corresponding to the bounding box with the highest confidence score from different perspectives as our baseline.

Qualitative Comparisons. MeshSegmenter demonstrates superior performance in zero-shot mesh semantic segmentation compared with 3DHighlighter[9] and SATR[2], as shown in Fig. 4. While 3DHighlighter[9] is limited to segmenting based on a single text query, MeshSegmenter can accurately segment a mesh in single and multiple queries. When dealing with multiple queries, MeshSegmenter does not rely on the competition of semantic segmentation for neighboring text queries and can independently and accurately segment each semantic region. We show the comparisons with SATR[2] in Fig. 5 and the comparisons with ZSC [1] in Fig. 6 (b). Texture systhesis can effectively enhance the performance of zero-shot semantic segmentation, particularly for regions with minimal geometric signals, as shown in Fig. 6 (a).