Fine Structure-Aware Sampling: A New Sampling Training Scheme
for Pixel-Aligned Implicit Models in Single-View Human Reconstruction

Methods that reconstruct a human body mesh from a single image can be broadly classified into two classes: Parametric methods and non-parametric methods.

Parametric methods (Kanazawa et al. 2018; Kolotouros et al. 2019) recover human body shapes by predicting parameters belonging to a chosen human parametric model (e.g. SMPL-X (Pavlakos et al. 2019)). However, these parametric methods can only produce cloth-less and hairless human body meshes. While methods like Multi-garment Net (Bhatnagar et al. 2019) and Bcnet (Jiang et al. 2020) try to predict clothes on top of a human parametric model, these methods are unable to produce accurate clothed meshes.

On the other hand, non-parametric methods do not use a human parametric model. As aforementioned, a subclass of non-parametric methods that has attracted significant attention from the research community is the pixel-aligned implicit models. PIFu (Saito et al. 2019) is the first of such models, and it is able to reconstruct highly accurate clothed human meshes from a single image.

After PIFu, other pixel-aligned implicit models have been proposed. These include PIFuHD (Saito et al. 2020), StereoPIFu (Hong et al. 2021), S-PIFu (Chan et al. 2022a), IntegratedPIFu (Chan et al. 2022b), and more.

The original PIFu (Saito et al. 2019) uses an encoder-decoder architecture. An illustration of the PIFu architecture is given in Fig. 3 (Not including the predicted normal maps and the magenta arrows, fonts, and boxes). The encoder is a 2D CNN (usually a stacked hourglass network (Newell, Yang, and Deng 2016) consisting of 4 stacks). From a RGB image, the encoder produces a set of feature maps (i.e. feature volume). During training and testing, 3D sample points are sampled within the 3D camera space (of the RGB image). The (x,y) coordinates of the sample points are then used to index the feature volume to obtain feature vectors, with each feature vector corresponding to a sample point. The z coordinate of the sample point, together with the feature vector obtained by the sample point, would then be passed to the decoder, which is a multi-layer perceptron (MLP). The MLP will predict a value from 0 to 1 where value$>0.5$ means the sample point is ‘inside’ a groundtruth human body mesh, value$<0.5$ means the sample point is ‘outside’, and value=0.5 means the sample point is exactly on a human body surface. During testing, the 3D camera space of a RGB image is evenly sampled to get a 3D grid of evenly spaced sample points. By predicting these sample points’ values, we get a 3D grid of values. By applying the Marching Cube algorithm (Lorensen and Cline 1987) on this grid of values, we obtain a reconstructed 3D human mesh.

During training, PIFu uses what is known as the spatial sampling scheme to generate sample points and assign labels to these sample points. The spatial sampling scheme mainly generates sample points by displacing mesh surface points of a groundtruth human mesh with normally-distributed noise. These sample points can be displaced in any direction and in any magnitude. An illustration of these sample points is shown in Fig. 1a. Sample points in this scheme are given binary labels (either 0 or 1).

In IntegratedPIFu (Chan et al. 2022b), the authors identified problems with the spatial sampling scheme and proposed the Depth-Oriented Sampling (DOS) training scheme. They showed that the binary labels (as opposed to continuous labels) and unconstrained displacement of mesh surface points in spatial sampling scheme caused thin but obvious body features like the ears and fingers to be missing in the reconstructed meshes. Their proposed DOS scheme introduced continuous labels and constrained displacement of surface points. Continuous labels provide more information and clearer supervision to a pixel-aligned implicit model during training. Constrained displacement of mesh surface points (i.e. surface points are only displaced in the camera-direction) narrows down the meaning of each label, and this makes learning simpler for a pixel-aligned implicit model. These two advantages allow a pixel-aligned implicit model to reconstruct human meshes that retain thin body features like ears and fingers, unlike spatial sampling. However, DOS has serious flaws that will be explained in the next section.

As explained in Section 2.2, DOS is a sampling training scheme that was recently proposed in IntegratedPIFu (Chan et al. 2022b) to succeed the spatial sampling scheme.

To teach pixel-aligned implicit models to reconstruct thin body features (e.g. ears, fingers), DOS constrains sample points’ labels to only the camera direction (see Fig. 1b), making it easier for models to interpret and learn from the labels. For example, if a sample point has a label of 0.7 in DOS, it means that a short distance away (in positive or negative camera-direction) from this sample point, we will find a mesh surface. Conversely, if labels are not constrained to camera-direction only, then a label of 0.7 does not really pinpoint where the mesh surface is (as any direction is likely).

But because the sample points’ labels are determined by the shortest distance between a sample point and a mesh surface in the camera direction, rather than the shortest distance in any direction, a DOS-trained pixel-aligned implicit model is trained to only identify surfaces that are front-facing and back-facing. Lateral surfaces become difficult for a DOS-trained model to identify. Thus, DOS suffered from problems such as high-frequency, wavy artefacts on the side-facing surfaces of its reconstructed human meshes. Moreover, badly reconstructed side-facing surfaces also mean thin body features (e.g. ears, fingers) tend to get incorrect shapes.

To overcome the drawbacks of DOS, we propose FSS. FSS solves the above issues with its 5 key features.

The sample points’ normals, which can be computed, are regrettably not used in existing sampling training schemes. The closest related work that tried is PHORHUM (Alldieck, Zanfir, and Sminchisescu 2022). PHORHUM slightly modifies the spatial sampling scheme such that normals of sample points that lie exactly on the mesh surface are used to regularize the training process. In contrast, our proposed idea will use the normals of all our sample points.

In reality, the normals of sample points are very useful for inferring the underlying structure of a human body mesh (see Fig. 5). Hence, as an extension to our FSS scheme, we exploit the normals of sample points to further enhance the training process of pixel-aligned implicit models.

We define the normal of a sample point as the normal of the mesh surface point that is nearest to that sample point in the camera direction (Refer to Fig. 5).

We use the normals of sample points to supervise and enhance the representations in the Feature Volume in Fig. 3. To do so, we use the sample points to index the Feature Volume (refer to Fig. 6) and obtain a set of feature vectors. The feature vector indexed by a sample point, along with the z coordinate of this sample point, will be fed into a small neural network that will attempt to predict the normal vector of this sample point. The mean squared error between the predicted and groundtruth normal vectors is computed, and the derived supervision signals are used to train the representations in the Feature Volume. The small neural network only acts as a dummy as it is only used during training.

A point to highlight is that there is a difference between the predicted normal maps that are used as inputs to a pixel-aligned implicit model and our proposed use of normals of sample points (NSP). While the former directly injects normal information into the model, the latter regularizes and improves the representations in the Feature Volume such that ridges, peaks, and troughs on a clothed body are registered and retained in these representations. The impact of NSP is supported by our ablation studies that will be shown later.

Works, such as (Hong et al. 2021; Peng et al. 2021; Chan et al. 2022b), observed that pixel-aligned implicit models tend to produce reconstructed meshes with implausible thickness. An intuitive solution to this problem is to teach the pixel-aligned implicit model to directly learn the thickness of different body parts (at different orientations). Thus, FSS proposes a mesh thickness loss (MTL) signal to do that.

It is, however, not trivial to introduce this loss. This is because predicted meshes (and their thicknesses) are not available during train time. During training, for each image, a pixel-aligned implicit model will process and predict the labels of (typically) only 8000 sample points. During testing, to predict a human mesh, the same model would need to process and predict for (typically) $256^{3}$=16,777,216 sample points. Thus, the time cost alone makes it infeasible to compute the mesh thickness of predicted meshes at train time. Moreover, since the marching cube algorithm used to produce the predicted mesh is not differentiable, it is unclear how we could backpropagate the error between predicted meshes’ thickness and groundtruth meshes’ thickness.

To solve the issue, we first introduce an architectural modification of the PIFu framework. In Fig. 3, we see that the Encoder is a Stacked Hourglass Network consisting of 4 stacks. In PIFu, unlike what is shown in Fig. 3, 4 Hourglass stacks (in orange) are used. Tri-PIFu stack (in green) is a new type of stack that we proposed to replace 3 of the 4 Hourglass stacks. Each of the 4 stacks, regardless of their types, will produce a separate Feature Volume (see Fig. 7).

As shown by the blue dotted box in Fig. 7, an Hourglass stack would produce a Feature Volume that is indexed via bilinear interpolation to retrieve feature vectors. Feature vectors are concatenated with z coordinates and fed into a MLP that predicts occupancy values that range from 0 to 1.

In contrast, as shown by the red dotted box in Fig. 7, a Tri-PIFu stack does not use the MLP, and we do not concatenate the feature vector with the z coordinate. Instead, we use the sample point’s (x,y,z) coordinates to index the Feature Volume via trilinear interpolation to directly retrieve an occupancy value that ranges from 0 to 1.

The Feature Volume produced by a Tri-PIFu stack is interpreted as a 3D space with shape of (D, H, W), where D, H, W represents its depth, height, and width respectively. Tri-PIFu’s aim is to model an implicit function of a human mesh surface inside this 3D space. To do so, we applied sigmoid activation function on the Tri-PIFu stack’s outputs, thereby ensuring every value or element in the Feature Volume falls within [0,1]. This ensures that any sample point that index the Feature Volume via trilinear interpolation will always obtain a value within [0,1]. Once trained, the stack will model an implicit function (of a predicted human mesh surface) inside its Feature Volume. Aside: The predicted surface is at the 0.5 (not 0) level-set of the implicit function.

With the implicit function in the Feature Volume, it is easy to get a metric of mesh thickness at any (x,y) position. We simply sum up the Feature Volume in the z (or D) dimension to obtain a 2D plane of shape (H, W). Each value or element in this 2D plane would be a consistent approximation of the mesh thickness at that (x,y) position. We refer to this 2D plane as the mesh thickness plane.

We can easily compute the groundtruth mesh thickness plane using the groundtruth mesh. The mean squared error between the groundtruth and predicted mesh thickness planes is then computed during training. This loss signal is our MTL signal. MTL signals are an extension to our FSS scheme as the signals provide additional supervision to the training process. MTL will be ablated later. More on MTL (e.g. why we still use 1 Hourglass stack) in Supp. Mat.

In our experiments, we use the THuman2.0 dataset (Yu et al. 2021) as the training set for both our models and other competing models. THuman2.0 dataset contains 526 high-quality scans (or meshes) of ethnic Chinese human subjects. We use a 80-20 train-test split of these meshes. For each training mesh, we first render a RGB image of the mesh’s front view using a weak-perspective camera. We then render 10 other images by evenly fanning out (i.e. changing the yaw) from the first RGB image, in both clockwise and counter-clockwise directions. This set-up is similar to the ones used in IntegratedPIFu (Chan et al. 2022b) and S-PIFu (Chan et al. 2022a), which are benchmarks that we aim to compete against in our experiments.

In addition, we made use of the BUFF dataset (Zhang et al. 2017) to evaluate all the models. None of the models are trained using the BUFF dataset. Like what the authors did in IntegratedPIFu, we conducted systematic sampling (based on sequence number) on the BUFF dataset, giving us 101 human meshes to be used for evaluating the models. By using systematic sampling, we avoided getting meshes that have both the same human subject and the same pose.

We trained two models. The first uses the architecture in Fig. 3. The second is same as the first but uses the HRI component proposed by IntegratedPIFu to incorporate high-res images. We compare our models against existing models on single-view clothed human reconstruction. The existing models include PIFu (Saito et al. 2019), PIFuHD (Saito et al. 2020), and IntegratedPIFu (Chan et al. 2022b). To be fair to all models, predicted normal maps are used in all models, including PIFu. Following (Saito et al. 2019, 2020; Chan et al. 2022b), we use Chamfer distance (CD), Point-to-Surface (P2S), and Normal reprojection error (Normal) as the metrics in our quantitative evaluation. In addition, to compare our methods against methods that use SMPL-X meshes as priors, we added S-PIFu (Chan et al. 2022a) in our quantitative evaluation. Due to space constraint, S-PIFu is compared qualitatively with ours in our Supp. Mat. We also compared against ICON (Xiu et al. 2022) in our Supp. Mat.