Anatomically Constrained Implicit Face Models

Prashanth Chandran
DisneyResearch|Studios
[email protected] Gaspard Zoss
DisneyResearch|Studios
[email protected]

Abstract

Coordinate based implicit neural representations have gained rapid popularity in recent years as they have been successfully used in image, geometry and scene modeling tasks. In this work, we present a novel use case for such implicit representations in the context of learning anatomically constrained face models. Actor specific anatomically constrained face models are the state of the art in both facial performance capture and performance retargeting. Despite their practical success, these anatomical models are slow to evaluate and often require extensive data capture to be built. We propose the anatomical implicit face model; an ensemble of implicit neural networks that jointly learn to model the facial anatomy and the skin surface with high-fidelity, and can readily be used as a drop in replacement to conventional blendshape models. Given an arbitrary set of skin surface meshes of an actor and only a neutral shape with estimated skull and jaw bones, our method can recover a dense anatomical substructure which constrains every point on the facial surface. We demonstrate the usefulness of our approach in several tasks ranging from shape fitting, shape editing, and performance retargeting.

1 Introduction

Deformable face models are an important tool in the arsenal of visual effects artists dealing with facial animation. As they are ubiquitously used both in high-end production workflows and lightweight consumer applications, building expressive face models for various applications continues to remain an active area of research [17]. Face models today can range from simple linear global shape models [4, 27, 29] to highly complex local models that incorporate the underlying facial anatomy through physical simulation [48, 15, 44] or through anatomical constraints [47].

In this work, we concern ourselves primarily with the high-quality facial animation workflow where actor specific linear blendshape models [27] continue to remain the most commonly used tool for creating facial animations [33, 47, 10]. We propose a new class of actor specific shape models named the Anatomical Implicit face Model (AIM) which provides several unique advantages over the existing actor specific face models, and can be used as a drop-in replacement for traditional blendshape models.

An actor specific blendshape model is a collection of 3D shapes of the given actor performing a number of facial expressions, usually created by face scanning [2] or by an artist. While the user-friendliness of such actor specific blendshape models contributes to their wide adoption, it is a well known limitation that such models often require hundreds of shapes to accurately model complex facial deformation [27]. To address these shortcomings, local blendshape models [42, 47, 10] were proposed. By splitting the face into regions, and allowing the individual regions to deform independently, local shape models are able to capture complex deformations with a limited number of shapes.

While local models address the lack of expressivity in global shape models, state-of-the-art methods in facial performance capture [47] and retargeting [10] often incorporate anatomical constraints on the facial surface to plausibly restrict the range of the skin deformations. The anatomical constraints employed by these models [47, 10] provide a few hidden advantages that end up contributing towards their practical success. For example, in the context of facial performance capture, Wu et al. [47] demonstrated that including anatomical constraints derived from the relationship between the facial skin and underlying bones (skull and mandible) helps to separate the rigid and non-rigid components of facial deformation, leading to better face performance capture. In the context of facial performance retargeting, Chandran et al. [10] made use of such an anatomically constrained local face model to restrict a retargeted shape to lie within the space of anatomically plausible shapes of the target actor.

Despite their practical success, anatomical constraints are often formulated in practice as regularization terms that have to be satisfied as part of complex optimization problems involving several objectives. As a result, fitting these anatomical face models to a target scan or an image for instance, is a computationally intensive procedure taking several minutes per frame on a CPU, or requires hand crafted GPU solvers [20]. Furthermore anatomy constraints are enforced only in sparse regions of the face, whereas in reality the facial skin surface is more densely constrained by the underlying anatomy, and simulating this dense interaction between the anatomy and facial skin through physical simulation can be even more computationally intensive [39, 48].

In this paper, we propose the Anatomical Implicit face Model; a framework that allows for a holistic representation of both the facial anatomy and the skin surface using simple implicit neural networks and facilitates the learning of a continuous anatomical structure that densely constrains the skin surface. Our model formulation, inspired by the anatomical local model (ALM) of Wu et al. [47], can further disentangle deformation arising from rigid bone motion (jaw motion) and non-rigid deformations created by muscle activations. Our model also addresses the computational bottleneck of the ALM model by explicitly deriving the skin surface from the anatomy, instead of formulating it as a constrained optimization problem. By ensuring that a point on the skin surface is always reconstructed through the underlying anatomy, our method provides several unique features in comparison to existing implicit face models, such as anatomy based face manipulation (see Section 5). Before describing the details of our anatomical formulation in Section 3, we discuss related work in Section 2.

2 Related Work

3D Morphable Models

Facial models used in animation make up for an extremely well studied body of work with the earliest works dating back to the late 1970s [18]. We therefore refer to the excellent survey of Egger et al. [17] for an in-depth review of the state-of-the-art methods, and provide only a concise summary in this section. Facial blendshapes [18, 27] have been conventionally used as a standard tool by artists to navigate the geometric space of human faces. The seminal 3D linear morphable model proposed by Blanz and Vetter [4] used principal component analysis to describe the variation in facial geometry and texture, which was later extended to multilinear models, jointly modeling identity and expression by Vlasic et al. [43] and later by Cao et al. [7]. Today a very commonly used morphable face model is the FLAME model [29] which incorporates identity, expression and corrective blendshapes in addition to modeling bone motion with linear blend skinning. Due to its flexible nature, the FLAME model is widely used by face reconstruction algorithms today [19]. Finally Chai et al. [8] recently created the HIFI3D++ morphable model which is built from a union of scans from several previously proposed models.

In the past few years, numerous face models leveraging the power of deep neural networks to model the nonlinear deformation of the human face have also been proposed. While the initial work in this area by Ranjan et al. [38] focused on the use of specialized graph convolutional networks to operate on shapes, several later approaches proposed further modifications to the network architecture to improve the accuracy in shape representation [14, 55, 5, 22]. To make these deep morphable models intuitive to use, Chandran et al. [9] subsequently proposed the Semantic Deep Face Model which treats a collection of neural networks like a multilinear model to achieve identity-expression disentanglement. Extensions of such a semantically controllable model to deal with topology changes [12] and temporal sequences of geometry [11] have also been proposed. Deep neural models that jointly model the facial geometry and appearance with semantic controls have also been proposed [28].

Implicit Face Models

Owning to the massive success of coordinate based neural networks in representing images [40, 30], 3D shapes [35] and arbitrary scenes [31], today’s research on parametric face models primarily focuses on implicit representations. Yenamandra et al. [49] proposed i3DMM as an initial exploration of using coordinate based networks for modeling full head geometries. This was followed by IMFace [51] which disentangled facial geometry into separate identity and expression embeddings with the help of individual deformation fields. More recently, Neural Parametric Head Models (NPHM) [21] proposed a method which improves the fidelity of neural implicit representations by jointly training an ensemble of local neural fields centered around anchor points. Implicit neural representations have also successfully been employed in learning an animatable avatar of a human face from only monocular video as demonstrated by IMAvatar [52] and Point Avatar [53]. Wang et al. [45] also proposed MoRF, which is a Neural Radiance Field [31] conditioned on an identity code allowing for photorealistic free viewpoint rendering of the full head in a fixed expression. Recently Buhler et al. [6] also explored how such multi-identity radiance fields can be fit to sparse images to recover a volumetric head model. Finally coordinate based neural networks have also been successfully employed in creating animatable human body models [16, 34, 3, 23].

Anatomically Constrained Face Models

The anatomical local model proposed in the context of monocular facial performance capture by Wu et al. [47], first introduced the coupling of the anatomical bone structure to the skin surface and modeled the effect of skin patches sliding over the bone through soft anatomical constraints. This formulation was later adapted by Chandran et al. [10] for facial performance retargeting. Qiu et al. proposed SCULPTOR [37], a multi-identity joint morphable model of facial anatomy and skin learned from a database of computed tomography (CT) scans. Recently Choi et al. proposed Animatomy [15], a muscle fiber based anatomical basis for animator friendly face modeling applications. Lastly we recognize several physically based face models [48, 44, 41, 39] which inherently have the ability to model anatomy constraints through simulation.

We draw inspiration from the three classes of facial morphable models discussed above and propose the Anatomical Implicit face Model: a blendshape based, implicit, anatomically constrained face model targeted towards high-quality actor specific face modeling. Our method can be seen as general extension of local blendshape models [10] to a continuously evaluable implicit function, and represents a set of actor blendshapes through a novel anatomical formulation. Unlike traditional patch-based models, our framework allows us to approximate complex shapes without requiring the user to specify patch layouts and other hyper-parameters. Our solution is based on simple coordinate based MLPs enabling efficient training and inference, and provides computational benefits over previous anatomically formulated face models [47]. Finally to the best of our knowledge, our method is the first to explore anatomical constraints inside an implicit facial blendshape model.

Refer to caption — Figure 1: Our approach consists of a model learning stage (Section 4.1) and a model fitting stage (Section 4.2). In the model learning stage, a set of an actor’s blendshapes are memorized by an ensemble of MLPs by our *Anatomical Implicit face Model* (AIM). In the second model fitting stage, the memorized model can be used as power shape prior to fit the actor model to target shapes.

3 Anatomical Model Formulation

The core idea of our approach is to formulate a learning scheme for an implicit neural representation that can reproduce an actor blendshape model while automatically learning the underlying facial anatomy and constraining the skin surface to this learned anatomy. Crucial to our learning scheme is our anatomically constrained face model that geometrically couples the underlying facial anatomy to the enclosing skin surface which we describe next.

We assume that we are given a set of $N$ 3D scans $({S_{0},S_{1},S_{2},..,S_{N-1}})$ of an actor represented as meshes. Without loss of generality, let $S_{0}$ be the shape with a neutral expression (or the rest pose). Each shape $S_{i}$ consists of $V$ vertices, and all shapes share the same vertex connectivity. For simplicity we exclude the index of the vertex in a shape in our notation and present our formulation as operating on surface points $\mathbf{s}\in\mathbf{R}^{3}$ . Let $\mathbf{s_{0}}\in\mathbf{R}^{3}$ and $\mathbf{s_{i}}\in\mathbf{R}^{3}$ be corresponding points on the skin surface for the neutral expression and expression $i$ respectively. In most previous methods for learning neural face models, a skin surface point $\mathbf{s}$ is learned as a displacement from a base face surface [9, 12, 21] or simply as points lying in an arbitrary 3D space [51, 52, 45]. Contrary to such approaches, we propose to learn the skin surface $\mathbf{s}$ using implicit neural representations that arrives at the facial skin surface through a formulation that combines anatomic constraints, linear blend skinning (LBS), and expression blendshapes into a single framework.

For our model formulation, we take inspiration from the anatomic constraints first proposed for non-neural face models [1, 47], particularly that of Wu et al. [47]. They establish a link between the skin surface and the anatomic bones by modeling the thickness $d_{i}\in\mathbf{R}$ of the soft tissue between a bone point $\mathbf{b_{i}}\in\mathbf{R}$ and the skin surface $\mathbf{s_{i}}$ . These constraints are defined in sparse regions of the face where a skin point can be trusted to have bone underneath. We draw inspiration from their simple formulation and make some important deviations that enable us to jointly learn both the surface of the underlying skin anatomy and the enclosing skin surface for every point on the skin through end-to-end learning. Specifically, we arrive at a point on the skin surface as follows

\mathbf{s_{0}}=\mathbf{b_{0}}+d_{0}\mathbf{n_{0}}

(1)

where $\mathbf{s_{0}}$ is the position of a surface point corresponding to $\mathbf{s}_{i}$ but on the neutral shape $S_{0}$ , $\mathbf{b_{0}}$ , $d_{0}$ , and $\mathbf{n_{0}}$ are the bone point, soft tissue thickness and the bone normal at $\mathbf{s_{0}}$ . While Eq. 1 allows us to reconstruct points on the neutral face geometry, to adequately represent skin surfaces under arbitrary facial expressions, we need to account for surface deformation arising from the rigid motion of underlying facial bones (skull and mandible), and the non-rigid skin motion arising from muscle activations, skin sliding, and self collisions. To accommodate these additional degrees of freedom in skin deformation, we incorporate standard linear blend skinning, and expression blendshapes similar to the FLAME model [29]. Therefore given an anatomically reconstructed point on the neutral skin surface $\mathbf{s_{0}}$ , we can now compute the position of the same point in an arbitrary expression $\mathbf{s_{i}}$ as

\mathbf{s_{i}}=\text{LBS}(\mathbf{s_{0}},T_{b},k)+\mathbf{e_{i}}

(2)

where LBS refers to the standard linear blend skinning operator that rigidly transforms the anatomically reconstructed neutral surface point $\mathbf{s_{0}}$ with a transformation $T_{b}$ and a skinning weight $k$ , $\mathbf{e_{i}}\in\mathbf{R}^{3}$ is the corrective displacement that is added on top of the skinned result to account for deformations that cannot be explained by skinning alone. A visual overview of our approach to anatomically build up the facial skin surface is shown in Fig. 2.

At this point we have established how to arrive at points on the skin surface $\mathbf{s_{i}}$ for a shape in an arbitrary facial expression $S_{i}$ by starting from the underlying anatomy $\mathbf{b_{i}}$ . It is important to note that the anatomic constraints as defined by Wu et al. [47] can only be computed on regions with an underlying bone, and thus, regions like the cheeks are not anatomically constrained in their approach. An essential feature of our approach that distinguishes it from all previous works is that we enforce anatomic constraints for every point on the skin surface; even in regions where there is no underlying biological bone structure. For this purpose we redefine the anatomy in our work as a rigidly deforming region underneath the skin surface that is not restricted to only the manifold of the skull and mandible bones. Since this structure does not exist in reality and is, therefore, not available for supervised learning, we formulate a learning framework where such rigidly deforming surface can be learnt only from the sparse set of anatomic constraints that can be computed between the skin and the underlying bones. As we will see in Section 5, learning this anatomic surface from data leads to several interesting applications in shape manipulation and performance retargeting that were previously challenging to obtain without expensive physical simulation [48] or extensive volumetric data capture [37].

4 Anatomical Implicit Face Model

At a high level, our method is comprised of two stages: first, a model learning stage (Section 4.1) and second, a model fitting stage (4.2). In the model learning stage, we bake a collection of expression blendshapes from an actor into an implicit neural network that uses the anatomical model formulation described in Section 3. Our model fitting stage uses this learned Anatomical Implicit face Model (AIM) and optimizes for coefficients that deform the model to match test time constraints like 3D shapes, 2D landmarks and so on. The overview of our approach is shown in Fig. 1.

4.1 Model Learning

To learn our anatomical implicit face model, we assume we are given a template shape $C$ , a registered set of $N$ shapes $({S_{0},S_{1},S_{2},..,S_{N-1}})$ of a single actor in the same topology of the canonical shape. Additionally we fit a template skull and jaw only to the neutral shape using the method of Zoss et al. [56]. The template shape $C$ can either be the neutral shape of the actor or a generic face shape, and the number of shapes provided can be arbitrary. We use a collection of 20 shapes in our work. A visual summary of our training data is shown in Fig. 3. Our objective in the learning stage is to use a coordinate based neural network to memorize the given shapes through the anatomical formulation in Section 3. Given the high representation power of periodic implicit neural networks [40], we use the SIREN coordinate network; an MLP with sinusoidal activation functions, as our base architecture. An ablation study on alternate network choices is provided in section 5.5.

Given a point $\mathbf{c}\in\mathbf{R}^{3}$ on the template shape $C$ , we use three independent MLPs denoted by $\mathbf{B}$ , $\mathbf{D}$ , and $\mathbf{N}$ to predict the anatomy point $\mathbf{\widetilde{b}}_{0}\in\mathbf{R}^{3}$ , the soft tissue thickness $\widetilde{d}_{0}\in\mathbf{R}$ , and the anatomy normal $\mathbf{\widetilde{n}}_{0}\in\mathbf{R}^{3}$ . These predicted anatomic properties are then used to reconstruct the position of a point on the neutral skin surface $\mathbf{\widetilde{s}}_{0}$ as

	$\displaystyle\mathbf{\widetilde{b}}_{0}=\mathbf{B}(\mathbf{c})$		(3)
	$\displaystyle\widetilde{d}_{0}=\mathbf{D}(\mathbf{c})$		(4)
	$\displaystyle\mathbf{\widetilde{n}}_{0}=\mathbf{N}(\mathbf{c})$		(5)
	$\displaystyle\mathbf{\widetilde{s}}_{0}=\mathbf{\widetilde{b}}_{0}+\widetilde{% d}_{0}\mathbf{\widetilde{n}}_{0}.$		(6)

As discussed in Section 3, to further account for the rigid and non-rigid deformations of the skin surface, the anatomically constructed neutral skin point $\mathbf{\widetilde{s}}_{0}$ has to be skinned and further displaced with residual expression deformations. We therefore employ two additional MLPs $\mathbf{K}$ and $\mathbf{E}$ that predict the skinning weight $\widetilde{k}\in\mathbf{R}$ and the corrective displacements basis $\mathbf{\mathcal{B}_{e}}\in\mathbf{R}^{(N-1)\times 3}$ respectively. Note here that, as an implementation detail, we predict the expression displacements for all $N-1$ blendshapes (excluding the neutral) at once from $\mathbf{E}$ . The corrective expression displacement $\mathbf{\widetilde{e_{i}}}\in\mathbf{R}^{3}$ for shape $i$ can be extracted from this output by indexing $\mathbf{\mathcal{B}_{e}}$ appropriately.

	$\displaystyle\widetilde{k}=\mathbf{K}(\mathbf{c})$		(7)
	$\displaystyle\mathbf{\mathcal{B}_{e}}=\mathbf{E}(\mathbf{c})$		(8)
	$\displaystyle\mathbf{\widetilde{e_{i}}}=\mathbf{\mathcal{B}_{e}}[i]$		(9)
	$\displaystyle\mathbf{\widetilde{s}}_{i}=\text{LBS}\left(\mathbf{\widetilde{s}}% _{0},\widetilde{T_{b}},\widetilde{k}\right)+\mathbf{\widetilde{e_{i}}}$		(10)

Here $\widetilde{T_{b}}\in\mathbf{R}^{9}$ is a 6-DOF jaw bone transformation optimized along with the training of the MLPs to account for rigid motion of the mandible. Here we parameterize the jaw bone rotation $\widetilde{T_{b}}$ following the continuous 6D representation [54].

4.1.1 Training Objectives

We next describe the training objectives to learn actor expression blendshapes along with the underlying anatomy structure for each skin surface point.
Skin Position Loss The skin position loss penalizes the difference between the estimated skin point $\mathbf{\widetilde{s}}_{i}$ and the ground truth skin point $\mathbf{s}_{i}$ .

\mathbf{L}_{\text{S}}=\lambda_{\text{S}}||\bf{\widetilde{s}}_{i}-\bf{s}_{i}||_% {2}^{2}

(11)

We set $\lambda_{\text{S}}=1.0$ for all our experiments.
Anatomy Regularizer Since we can roughly estimate the skull and jaw geometry on the neutral shape using the method of Zoss et al. [56], we compute sparse anatomic constraints [47] and loosely regularize the learned anatomic properties to stay close to these estimates only in regions where the constraints can be accurately computed (i.e. skin regions with an underlying bone).

\mathbf{L}_{\text{A}}\!=\!\lambda_{b}||\bf{\widetilde{b}}_{0}-\bf{b}_{0}||_{2}% ^{2}+\lambda_{d}||\widetilde{d}_{0}-d_{0}||_{2}^{2}+\lambda_{n}||\bf{% \widetilde{n}}_{0}-\bf{n}_{i}||_{2}^{2}

(12)

We set $\lambda_{\text{b}}=\lambda_{\text{d}}=\lambda_{\text{n}}=1.0$ for all our experiments, and observe that this constraint only regularizes 5-10% of all the vertices generated by the model on average (see Supplemental).
Thickness Regularizer We regularize the soft tissue thickness $\widetilde{d}$ predicted by the model in unconstrained regions to remain as small unless dictated otherwise by the skin position loss.

\mathbf{L}_{\text{D}}=\lambda_{D}^{Reg}||\widetilde{d}_{0}||_{2}^{2}

(13)

We set $\lambda_{D}^{Reg}=7.5\mathrm{e}{-4}$ for all our experiments.
Symmetry Regularizer To exploit the symmetry of the face, we regularize the predictions of the anatomy MLP $\mathbf{B}$ to be symmetric. We achieve this by requiring that reflecting the input points $\mathbf{c}$ along the plane of symmetry provides the same result as reflecting the predicted anatomy points $\mathbf{\widetilde{a}}$ .

\mathbf{L}_{\text{Sym}}=\lambda_{sym}||\bf{B}(\bf{R(c)})-R(\bf{B}(\bf{c}))||_{% 2}^{2}

(14)

where $R$ is an operator that reflects a point along the plane of symmetry. We set $\lambda_{sym}=1\mathrm{e}{-4}$ for all our experiments. Note that we do not regularize symmetry on the predicted thickness or anatomy normals thereby allowing the model to still be able to represent asymmetric faces.
Optional Skinning Weight Regularizer Finally inspired by [52], we use an optional loss that encourages the estimated skinning weights $\widetilde{k}$ in regions like the forehead that are guaranteed to not be affected by the rigid deformation of the jaw bone to be zero.

\mathbf{L}_{\text{K}}=\lambda_{k}||\mathbf{K}(\mathbf{c}^{*})||_{2}^{2}

(15)

here $\mathbf{c}^{*}$ refers to a small region on the canonical shape $C$ which includes the forehead. We set $\lambda_{K}=1\mathrm{e}{2}$ for all our experiments.

Our final model energy $\mathbf{L}_{\text{Model}}$ is a summation of the above losses and is minimized using gradient decent [26] to train our ensemble of coordinate MLPs end-to-end.

\mathbf{L}_{\text{Model}}=\mathbf{L}_{\text{S}}+\mathbf{L}_{\text{A}}+\mathbf{% L}_{\text{D}}+\mathbf{L}_{\text{Sym}}+\mathbf{L}_{\text{K}}

(16)

4.2 Model Fitting

While the aforementioned model can recover interesting anatomic properties of the face with only sparse supervision, it is not very useful unless it can be deformed to match user constraints and serve as a shape prior for an actor facial geometry.

After training our anatomical implicit face model on a collection of $N$ shapes, the coefficients that are required to deform it include a jaw bone transformation $\mathbf{T_{b}}^{*}\in\mathbf{R}^{9}$ , coefficients $\mathbf{w}^{*}\in\mathbf{R}^{N-1}$ that can be used to blend the corrective expression displacements $\mathcal{B}_{e}\in\mathbf{R}^{(N-1)\times 3}$ , and an optional global head transformation $\mathbf{T_{g}}^{*}\in\mathbf{R}^{9}$ . Following equation (10), we can therefore evaluate our anatomical implicit face model as

\mathbf{s}^{*}=\mathbf{T_{g}}^{*}\left(\text{LBS}\left(\mathbf{\widetilde{s}}_% {0},\mathbf{T_{b}}^{*},\widetilde{k}\right)+\sum_{N-1}\mathbf{w}^{*}\mathbf{B}% _{e}\right)

(17)

where $\mathbf{T_{g}}^{*}$ , $\mathbf{T_{b}}^{*}$ and $\mathbf{w}^{*}$ are the only unknowns, and the rest can be queried from a pre-trained AIM. We consider two scenarios for model fitting which include i) fitting our model to a sequence of 3D scans e.g. from a facial performance, and ii) fitting our model to 2D landmarks detected on a video [13, 46].

For both scenarios, inspired by the state-of-the-art findings of Kim et al. [50], we employ neural reparameterized optimization [25] and solve for the weights of a simple MLP that predicts the unknown parameters instead of directly optimizing for them. Specifically when given a sequence of $J$ frames with 3D/2D constraints, we optimize for $J$ frame codes $\mathbf{z_{j}}\in\mathbf{R}^{f}$ which, when fed as input to a simple 4-layer MLP $\mathbf{F}_{T}$ with GeLU [24] activations, predicts the head $\mathbf{T_{g}}^{j}$ and jaw $\mathbf{T_{b}}^{j}$ poses for each frame. Additionally as the coefficients $\mathbf{w}^{j}$ are local and spatially varying depending on the template query point $\mathbf{c}$ , we use a separate 4-layer MLP $\mathbf{F}_{W}$ which predicts the coefficients $\mathbf{w}^{j}$ by taking both the frame code $\mathbf{z_{j}}$ and the query point $\mathbf{c}$ as input.

	$\displaystyle[\mathbf{T_{g}}^{j},\mathbf{T_{b}}^{j}]=\mathbf{F}_{T}(\mathbf{z_% {j}})$		(18)
	$\displaystyle\mathbf{w}^{j}=\mathbf{F}_{W}(\mathbf{z_{j}},\mathbf{c})$		(19)

Unlike the method of Kim et al. [50] where the reparameterized optimization was used mainly for improved performance, this neural optimization is even necessary in our case to restrict the number of optimized variables as the number of spatially varying coefficients $\mathbf{w}^{*}$ used to evaluate our anatomical implicit face model can vary drastically depending on the number of constraint points (see Section 5).

4.2.1 Fitting Objectives

3D Position Constraint For fitting our trained model to 3D constraints coming from a facial performance of an actor, we minimize the euclidean distance between the estimated skin point $\mathbf{s}^{*}$ and the ground truth skin point $\mathbf{s}^{GT}$ .

\mathbf{L}_{\text{Pos}}^{3D}=\lambda_{3D}||\mathbf{s}^{*}-\mathbf{s}^{GT}||_{2% }^{2}

(20)

2D Position Constraint For fitting our model to 2D constraints such as facial landmarks estimated by a pre-trained landmark detector [13, 46], we project the estimated skin point $\mathbf{s}^{*}$ to screen space using known camera intrinsics $\psi$ and calculate the euclidean distance in 2D between the project point $\psi(\mathbf{s}^{*})$ and the corresponding landmark.

\mathbf{L}_{\text{Pos}}^{2D}=\lambda_{2D}||\psi(\mathbf{s}^{*})-\mathbf{p}||_{% 2}^{2}

(21)

$\mathbf{p}\in\mathbf{R}^{2}$ is a detected landmark corresponding to point $\mathbf{s}^{*}$ .
Coefficient Regularizer As the complexity of our implicit anatomical face model can be arbitrarily large, we regularize the estimated blending coefficients $\mathbf{w}^{*}$ to be small with a weak L2 regularizer.

\mathbf{L}_{W}=\lambda_{Reg}^{w}||\mathbf{w}^{*}||_{2}^{2}

(22)

We set $\lambda_{Reg}^{w}=0.75$ for all our experiments.
Temporal Regularizer Finally when optimizing for coefficients on sequential data, we regularize the optimized frame codes $\mathbf{z_{j}}$ to remain similar between adjacent frames.

\mathbf{L}_{T}=\lambda_{Reg}^{t}||\mathbf{z_{j}}-\mathbf{z_{j-1}}||_{2}^{2}

(23)

We set $\lambda_{Reg}^{t}=0.05$ for all our experiments.

Our final fitting energy $\mathbf{L}_{\text{Fitting}}$ is therefore

\mathbf{L}_{\text{Fitting}}=\mathbf{L}_{\text{Pos}}^{3D}+\mathbf{L}_{\text{Pos% }}^{2D}+\mathbf{L}_{\text{W}}+\mathbf{L}_{\text{T}}

(24)

Depending on the 3D or 2D fitting scenario, we set $\lambda_{2D}$ or $\lambda_{3D}$ to 0 respectively.

4.3 Implementation Details

In the model learning stage, we optimize our implicit coordinate networks for $1\mathrm{e}{4}$ iterations with a learning rate of $2\mathrm{e}{-3}$ . This takes approximately 10 minutes to converge on a single Nvidia RTX 3090 for an actor model with 40,000 vertices and 20 blendshapes.

In the model fitting stage, we use a learning rate of $1\mathrm{e}{-3}$ and optimize the fitting MLPs $\mathbf{F}_{T}$ and $\mathbf{F}_{W}$ for $1\mathrm{e}{4}$ iterations. This process takes 1 second per frame on a single Nvidia RTX 3090.

We implement all our MLPs in PyTorch [36]. In our supplementary material we discuss the performance implications of replacing our current python backend with the well engineered fused MLP implementation [32].

5 Results

We now present several results, applications and evaluations of our Anatomical Implicit face Model (AIM).

5.1 Learning Actor Specific Anatomy

We begin by showing the reconstruction accuracy of our AIM on facial blendshapes of multiple actors. As seen in Fig. 4 on 2 different actors, our method can faithfully represent facial shapes with high fidelity while capturing both the low and high frequency features of facial shape and expression. We also show the anatomic features recovered by our new formulation which includes the dense underlying facial anatomy (shown in red), the soft tissue thickness at every point on the anatomy (visualized as heatmap), and the optimized subject specific skinning weights. These results highlight the new abilities introduced by our method in recovering plausible anatomy features while jointly learning to model surface deformations.

5.2 Anatomy Manipulation

Our ability to estimate the underlying anatomy that densely constrains the skin surface opens up new, yet computationally inexpensive ways to edit facial geometry using our learned anatomic properties. For example, as illustrated in Fig. 5, by simply scaling the learned soft tissue thickness $d$ in desired regions of the face (denoted by the hand drawn mask), an artist can interactively sculpt/deform an actor’s face shape to match their requirements.

5.3 Expression Reconstruction

We next evaluate the expressiveness of our model by fitting it to unseen 3D performances of multiple actors. Given a sequence of $J$ dynamic 3D shapes from a studio scanner [2], we first deform our template mesh $C$ to match the scanned shapes using standard mesh registration techniques such that the dynamic 3D scans are in full vertex correspondence with our AIM. We then follow the fitting procedure described in Section 4.2 and obtain per-frame transformations $[\mathbf{T_{g}}^{j},\mathbf{T_{b}}^{j}]$ and shape coefficients $\mathbf{w}^{j}$ that explain the captured ground truth shape. For this experiment, we use the 3D position constraint from Eq. (24) and set $\mathbf{L_{2D}}$ to 0. We densely constrain the fitting procedure at every vertex of the ground truth shape. In Fig. 6 we provide both a qualitative and quantitative comparison of fitting to novel performance from an actor against global blendshapes (GBS) [27], a patch blendshape model (PBS) [13], and the anatomical local model (ALM) [47]. In this experiment, we use 20 ground truth actor blendshapes to build the GBS, PBS, and ALM models, and the anatomically reconstructed blendshapes for our method. Even under this slight disadvantage, our method outperforms both GBS, and PBS and provides visually comparable results to the ALM model. Table 1 shows the average fitting error of each method across 819 frames from 5 sequences of 5 different actors. Our method converges in a few seconds for each frame, while the ALM algorithm consistently requires several minutes per frame. While the continuous nature of AIM enables us to evaluate it with coefficients of arbitrary locality, it could result in situations where our fitting is underconstrained in the absence of dense constraints leading to broken shapes. To illustrate that this does not happen in our reparameterized optimization, we show the result of fitting the AIM to sparse constraints in Fig. 7. While increasing the density of constraints improves the fitting accuracy, fitting our model to sparse landmarks also provides plausible results. Note that we do not compare fitting accuracy against generic morphable models like FLAME [29] or NPHM [21] as ours is actor specific and therefore a quantitative comparison might be unfair to the other methods. However we present some qualitative comparison to generic models in our supplemental material.

Table 1: Average fitting error across 819 frames from 5 sequences of 5 different actors. See supplemental material for details.

GBS [27]	PBS [13]	ALM [47]	Ours
0.834 mm	0.51 mm	0.095 mm	0.312 mm

5.4 3D Performance Retargeting

Another important application of our method is in the area of 3D performance retargeting, where the goal is to transfer a facial animation from a source to a target character while respecting the identity and anatomic characteristics of the target character. To accomplish this using our model, we learn two separate instances of our model for the source and target character respectively from a sparse set of 20 blendshapes in correspondence. We then fit our source model to the facial animation of the source target character to obtain per-frame transformations $[\mathbf{T_{g}}^{j},\mathbf{T_{b}}^{j}]$ and shape coefficients $\mathbf{w}^{j}$ . These coefficients can simply be played back on the target model to achieve facial performance retargeting. In Fig. 8, we provide a qualitative comparison to the state-of-the-art 3D retargeting algorithm of Chandran et al. [10] by retargeting the performance from a source to a target character. Our method provides competitive results to state of the art, while also allowing users to disentangle the rigid jaw motion and the soft tissue deformations of the skin surface. Our method additionally provides a substantial runtime benefit here and retargets each frame in a few (2-3) seconds, while the method of Chandran et al. requires several minutes per frame due to a costly anatomic solve. Finally unlike the approach of Chandran et al., our method provides all of above benefits without having to manually choose design parameters such as the patch layout, number of overlaps etc.

5.5 Ablations

Table 2: Average error in mm on a sequence of 100 frames using different types of activation functions in our MLPs.

gelu	relu	siren
0.71 mm	0.62 mm	0.21 mm

Table 3: Average error in mm on a sequence of 80 frames using variation of our loss functions during the model learning stage.

no $\mathbf{L}_{\text{A}}$	no $\mathbf{L}_{\text{K}}$	no $\mathbf{L}_{\text{Sym}}$	no $\mathbf{L}_{\text{D}}$	$\mathbf{L}_{\text{Model}}$ (Ours)
0.29	0.22	0.24	0.21	0.19

Finally Table 2 shows an ablation study on our choice of activation in our MLPs and Table 3 shows an ablation on the effect of several of our loss functions on the recovered geometry. Additional ablations are provided in the supplemental material.

6 Conclusion

In this paper we propose a new anatomically constrained implicit face model which provides a holistic representation of both facial anatomy and the enclosing skin surface using an ensemble of coordinate neural networks. Given an arbtrary set of skin surface meshes and only a neutral shape with estimated skull and jaw bones, our method recovers a dense anatomical substructure to constrain each point on the skin surface, and can model complex skin deformations with high fidelity. While we have explored the use of such a model in the context of actor specific blendshape models, future work could analyze it’s implications as a generic morphable model, by extending our formulation to handle multiple identities at once. Our new Anatomical Implicit face Model (AIM) has applications in shape representation and manipulation, retargeting and more, and we hope that our method encourages exciting future research.

References

Beeler and Bradley [2014] Thabo Beeler and Derek Bradley. Rigid stabilization of facial expressions. ACM TOG, 33(4), 2014.
Beeler et al. [2011] Thabo Beeler, Fabian Hahn, Derek Bradley, Bernd Bickel, Paul Beardsley, Craig Gotsman, Robert W. Sumner, and Markus Gross. High-quality passive facial performance capture using anchor frames. ACM Trans. Graphics Proc SIGGRAPH, 30, 2011.
Biswas et al. [2021] Sourav Biswas, Kangxue Yin, Maria Shugrina, Sanja Fidler, and Sameh Khamis. Hierarchical neural implicit pose network for animation and motion retargeting. CoRR, abs/2112.00958, 2021.
Blanz and Vetter [1999] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Siggraph, 1999.
Bouritsas et al. [2019] G. Bouritsas, S. Bokhnyak, S. Ploumpis, S. Zafeiriou, and M. Bronstein. Neural 3d morphable models: Spiral convolutional networks for 3d shape representation learning and generation. In ICCV, 2019.
Bühler et al. [2023] Marcel C Bühler, Kripasindhu Sarkar, Tanmay Shah, Gengyan Li, Daoye Wang, Leonhard Helminger, Sergio Orts-Escolano, Dmitry Lagun, Otmar Hilliges, Thabo Beeler, et al. Preface: A data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Cao et al. [2014] Chen Cao, Yanlin Weng, Shun Zhou, Y. Tong, and Kun Zhou. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics, 20, 2014.
Chai et al. [2022] Zenghao Chai, Haoxian Zhang, Jing Ren, Di Kang, Zhengzhuo Xu, Xuefei Zhe, Chun Yuan, and Linchao Bao. Realy: Rethinking the evaluation of 3d face reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Chandran et al. [2020] Prashanth Chandran, Derek Bradley, Markus Gross, and Thabo Beeler. Semantic deep face models. In TDV, 2020.
Chandran et al. [2022a] Prashanth Chandran, Loïc Ciccone, Markus Gross, and Derek Bradley. Local anatomically-constrained facial performance retargeting. ACM Trans. Graph., 41(4), 2022a.
Chandran et al. [2022b] Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. Facial Animation with Disentangled Identity and Motion using Transformers. Computer Graphics Forum, 2022b.
Chandran et al. [2022c] Prashanth Chandran, Gaspard Zoss, Markus Gross, Paulo Gotardo, and Derek Bradley. Shape transformers: Topology-independent 3d shape models using transformers. 41(2), 2022c.
Chandran et al. [2023] P. Chandran, G. Zoss, P. Gotardo, and D. Bradley. Continuous landmark detection with 3d queries. In CVPR, Los Alamitos, CA, USA, 2023. IEEE Computer Society.
Chen and Kim [2021] Zhixiang Chen and Tae-Kyun Kim. Learning feature aggregation for deep 3d morphable models. In CVPR, 2021.
Choi et al. [2022] Byungkuk Choi, Haekwang Eom, Benjamin Mouscadet, Stephen Cullingford, Kurt Ma, Stefanie Gassel, Suzi Kim, Andrew Moffat, Millicent Maier, Marco Revelant, Joe Letteri, and Karan Singh. Animatomy: An animator-centric, anatomically inspired system for 3d facial modeling, animation and transfer. In SIGGRAPH Asia 2022 Conference Papers, 2022.
Deng et al. [2020] Boyang Deng, J. P. Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Nasa neural articulated shape approximation. In ECCV, 2020.
Egger et al. [2020] Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 3d morphable face models - past, present and future. ACM TOG, 39(5), 2020.
Ekman and Friesen [1978] Paul Ekman and Wallace V. Friesen. Facial action coding system: a technique for the measurement of facial movement. 1978.
Feng et al. [2021] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. Learning an animatable detailed 3D face model from in-the-wild images. TOG, 40(4), 2021.
Fratarcangeli et al. [2020] Marco Fratarcangeli, Derek Bradley, Aurel Gruber, Gaspard Zoss, and Thabo Beeler. Fast Nonlinear Least Squares Optimization of Large-Scale Semi-Sparse Problems. Computer Graphics Forum, 2020.
Giebenhain et al. [2023] Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner. Learning neural parametric head models. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.
Gong et al. [2019] S. Gong, L. Chen, M. Bronstein, and S. Zafeiriou. Spiralnet++: A fast and highly efficient mesh convolution operator. In ICCV, 2019.
Guo et al. [2023] Chen Guo, Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016.
Hoyer et al. [2019] Stephan Hoyer, Jascha Sohl-Dickstein, and Sam Greydanus. Neural reparameterization improves structural optimization. CoRR, abs/1909.04240, 2019.
Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Lewis et al. [2014] J. P. Lewis, K. Anjyo, Taehyun Rhee, M. Zhang, Frédéric H. Pighin, and Z. Deng. Practice and theory of blendshape facial models. In Computer Graphics Forum (Proc. Eurographics, 2014.
Li et al. [2020] Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, and Hao Li. Learning formation of physically-based face attributes. CoRR, abs/2004.03458, 2020.
Li et al. [2017] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM, 2017.
Martel et al. [2021] Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Monteiro, and Gordon Wetzstein. ACORN: adaptive coordinate networks for neural scene representation. CoRR, abs/2105.02788, 2021.
Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Müller [2021] Thomas Müller. tiny-cuda-nn, 2021.
Orvalho et al. [2012] Verónica Orvalho, Pedro Bastos, Frederic Parke, Bruno Oliveira, and Xenxo Alvarez. A Facial Rigging Survey. In Eurographics 2012 - State of the Art Reports. The Eurographics Association, 2012.
Palafox et al. [2021] Pablo R. Palafox, Aljaz Bozic, Justus Thies, Matthias Nießner, and Angela Dai. Npms: Neural parametric models for 3d deformable shapes. CoRR, abs/2104.00702, 2021.
Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. 2019.
Qiu et al. [2022] Zesong Qiu, Yuwei Li, Dongming He, Qixuan Zhang, Longwen Zhang, Yinghao Zhang, Jingya Wang, Lan Xu, Xudong Wang, Yuyao Zhang, and Jingyi Yu. Sculptor: Skeleton-consistent face creation using a learned parametric generator. ACM Trans. Graph., 41(6), 2022.
Ranjan et al. [2018] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J. Black. Generating 3d faces using convolutional mesh autoencoders. In ECCV, 2018.
Sifakis et al. [2006] Eftychios Sifakis, Andrew Selle, Avram Robinson-Mosher, and Ronald Fedkiw. Simulating speech with a physics-based facial muscle model. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Goslar, DEU, 2006. Eurographics Association.
Sitzmann et al. [2020] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020.
Srinivasan et al. [2021] Sangeetha Grama Srinivasan, Qisi Wang, Junior Rojas, Gergely Klár, Ladislav Kavan, and Eftychios Sifakis. Learning active quasistatic physics-based models from data. ACM Trans. Graph., 40(4), 2021.
Tena et al. [2011] J. Rafael Tena, Fernando De la Torre, and Iain Matthews. Interactive region-based linear 3d face models. ACM Trans. Graphics Proc SIGGRAPH, 30(4), 2011.
Vlasic et al. [2005] Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. Face transfer with multilinear models. ACM TOG, 24(3), 2005.
Wagner et al. [2023] Nicolas Wagner, Mario Botsch, and Ulrich Schwanecke. Softdeca: Computationally efficient physics-based facial animations. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games, 2023.
Wang et al. [2022] Daoye Wang, Prashanth Chandran, Gaspard Zoss, Derek Bradley, and Paulo Gotardo. Morf: Morphable radiance fields for multiview neural head modeling. In ACM SIGGRAPH 2022 Conference Proceedings, 2022.
Wood et al. [2022] Erroll Wood, Tadas Baltrušaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen, Nikola Milosavljevic, Daniel Wilde, Stephan Garbin, Chirag Raman, Jamie Shotton, Toby Sharp, Ivan Stojiljkovic, Tom Cashman, and Julien Valentin. 3d face reconstruction with dense landmarks, 2022.
Wu et al. [2016] Chenglei Wu, Derek Bradley, Markus Gross, and Thabo Beeler. An anatomically-constrained local deformation model for monocular face capture. ACM TOG, 35(4), 2016.
Yang et al. [2022] Lingchen Yang, Byungsoo Kim, Gaspard Zoss, Baran Gözcü, Markus Gross, and Barbara Solenthaler. Implicit neural representation for physics-driven actuated soft bodies. ACM Trans. Graph., 41(4), 2022.
Yenamandra et al. [2021] Tarun Yenamandra, Ayush Tewari, Florian Bernard, Hans-Peter Seidel, Mohamed Elgharib, Daniel Cremers, and Christian Theobalt. i3dmm: Deep implicit 3d morphable model of human heads. In CVPR, 2021.
Youwang et al. [2023] Kim Youwang, Lee Hyun, Kim Sung-Bin, Suekyeong Nam, Janghoon Ju, and Tae-Hyun Oh. A large-scale 3d face mesh video dataset via neural re-parameterized optimization. arXiv preprint, arXiv:2310.03205, 2023.
Zheng et al. [2022a] Mingwu Zheng, Hongyu Yang, Di Huang, and Liming Chen. Imface: A nonlinear 3d morphable face model with implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022a.
Zheng et al. [2022b] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C. Bühler, Xu Chen, Michael J. Black, and Otmar Hilliges. Im avatar: Implicit morphable head avatars from videos. In Computer Vision and Pattern Recognition (CVPR), 2022b.
Zheng et al. [2023] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. Pointavatar: Deformable point-based head avatars from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Zhou et al. [2020] Yi Zhou, Chenglei Wu, Zimo Li, Chen Cao, Yuting Ye, Jason Saragih, Hao Li, and Yaser Sheikh. Fully convolutional mesh autoencoder using efficient spatially varying kernels. In NeurIPS, 2020.
Zoss et al. [2018] Gaspard Zoss, Derek Bradley, Pascal Bérard, and Thabo Beeler. An empirical rig for jaw animation. ACM TOG, 37(4), 2018.
Zoss et al. [2019] Gaspard Zoss, Thabo Beeler, Markus Gross, and Derek Bradley. Accurate markerless jaw tracking for facial performance capture. ACM TOG, 38(4), 2019.

\thetitle

Supplementary Material

Appendix A Additional Details

A.1 Anatomy Constraints

We loosely regularize the skull and mandible geometries using sparse anatomical constraints. We compute these sparse constraints by fitting a template skull and mandible meshes to the neutral geometry following the method of Zoss et al. [56]. For any given skin point inside a hand-painted trusted region of the bone fitting process, we trace a ray along the inverse direction of the skin normal and store the bone intersection point only if the bone faces the same direction as the skin. We then trace another ray following now the bone normal, intersecting the skin again (potentially at a different point) and store the thickness and bone normal for the intersected skin point. Overall our sparse anatomical constraints exist only for 5 to 10% of the skin query points. We then use those bone points and thicknesses inside our losses $\mathbf{L}_{\text{A}}$ and $\mathbf{L}_{\text{D}}$ respectively. We show a visualization of the anatomical constraints and learned anatomies and thicknesses on Fig. 10. A visual depiction of the full set of 20 shapes used in our work is shown in Fig. 9.

A.2 Network Architecture

In Fig. 11 and Fig. 12, we show a detailed breakdown of our memorization and fitting networks.

Appendix B Additional Results

B.1 Face Reconstruction from 2D Landmarks

In the main paper, we describe how to formulate a 2D position constraint to fit our anatomical implicit face model to landmarks obtained from a pre-trained landmark detector. In Fig. 13, we show qualitative results of fitting our trained anatomical implicit model to 10,000 dense landmarks predicted by a 2D landmark detector [13] on an input monocular video.

B.2 Learning Actor Specific Anatomical Properties

In Fig. 18, we show additional results of the recovered dense anatomical properties on a number of actors with varying face shapes spanning different ethnicities, and age groups.

B.3 Runtime Analysis

Our model fitting stage, which involves the training of the fitting MLPs $F_{W}$ and $F_{T}$ (see the main text), takes atmost a few seconds per-frame to converge on a Nvidia RTX 3090. As an engineering update to our system, we experimented with the tinycuda framework of Muller et al. [32] and found that it provided a 2x performance improvement in model fitting, without any adverse effects on fitting accuracy. We leave a more thorough performance optimization of our pipeline to future work, which could also include exploring fused MLPs for the model learning stage.

B.4 3D Performance Retargeting

We kindly refer you to our supplemental video for additional retargeting results and qualitative comparisons.

B.5 Ablations

We provide visual results for the several ablations we performed in our work, which include the effect of removing certain regularizers used during the model learning stage (see section in the main text) in Fig. 14, the effect of different activation functions in Fig. 15, and the size of the hidden layers used during model learning in Fig. 16.

B.6 Generic Model Comparison

As discussed in the main text, a quantitative comparison of our actor specific model against a generic 3D morphable model would be unfair to general 3DMMs as they serve a more diverse purpose. However in Fig. 17 we show a visual comparison of 2 expressions fitted using 3D positions as constraints with our model and the FLAME model [29] for 2 different actors.