Beyond Alignment: Blind Video Face Restoration via Parsing-Guided
Temporal-Coherent Transformer

Current methods based on generator priors can recover high-quality texture details. For example, Gu et al. (2020); Menon et al. (2020) use the face generation prior from StyleGAN for blind face restoration, but the degraded LQ face is ill-posed when converting to HQ face, and the fidelity of these methods is difficult to be guaranteed. In order to solve the problem of too low fidelity, GLEANChan et al. (2021a) and GFPGANWang et al. (2021) directly introduce the generator prior into the decoder, which can improve the fidelity to a certain extent. Despite the increased fidelity, when the input is heavily corrupted, artifact artifacts can be produced.

Another method, VQFRGu et al. (2022) and CodeFormerZhou et al. (2022), directly uses the codebook dictionary information in the VQGAN series of methods to model face restoration as a codebook selection problem, and improves the fidelity through encoder information fusion. These methods all use the face prior information of the generator, but highly aligned face data is required to train the generator, and only the aligned face can be restored during use. If the input face is not pre-aligned, the resulting face image will have unacceptable artifacts.

In the video face, there will be structural changes in the faces of adjacent frames, and this structural change is more prominent in low-quality faces. The difference between two adjacent frames of faces is detected by key points, pre-aligned, face restoration, and objection After aligning the four passes were zoomed in, which caused major problems with discontinuity between frames.

Therefore, although these methods based on generative priors have achieved good performance in image face restoration tasks, they cannot effectively restore low-quality faces in videos.

Video restoration is to reduce or eliminate the quality loss of video in the process of lossy compression, so as to generate a high-quality image video that is closer to lossless. MFQEGuan et al. (2021); Yang et al. (2018) adds multi-scale information extraction and dense connection mapping structure to the quality enhancement network to improve the enhancement effect. In EDVRWang et al. (2019a), time information is gathered through space-time deformable convolution. According to different image frames, the space-time sampling position of convolution can be deformed adaptively, expanding the receptive field to a certain extent, and obtaining the most relevant context. information, exclude noisy content and improve the quality of image frames. The BasicVSRChan et al. (2021b, 2022) series methods restore high-quality videos through feature propagation and alignment.

To obtain a high-quality representation of video faces, we adopt a pre-training strategy using the temporal-spatial quantized autoencoder (TS-VQGAN). This approach aims to learn a codebook that encompasses spatio-temporal context, thereby enhancing the network’s expressiveness and robustness against degradation.

As depicted in Fig. 3 (a), the input consists of a sequence of high-quality face video frames denoted as $x_{h}\in\mathbb{R}^{T\times H\times W\times 3}$. These frames are fed into the spatio-temporal interactive encoder $E_{h}$, which generates a potential feature representation $z_{h}\in\mathbb{R}^{T\times H\times W\times D}$. Subsequently, the nearest neighbor matching technique is used to find the closest codebook vector $C_{k}$ in codebook $C$ for each vector in $z_{h}$, and the index $S$ of the codebook is obtained. Get the quantized representation latent $z_{q}$ through $S$. For the formal expression, please refer to Equation 1. Finally, the quantized representation $z_{q}$ is input into the decoder $D_{h}$, yielding the reconstructed video sequence $x_{r}$.

$\theta_{E_{h}}$, $\theta_{D_{h}}$ are the parameters of the encoder and decoder, respectively. Here, our encoder $E_{h}$ and decoder $D_{h}$ are comprised of 8 residual blocks and 4 resizing layers for downsampling and upsampling. To ensure accurate face reconstruction, we set the number of codebooks $N$ to 1024, which provides ample capacity. Furthermore, we set the code dimension $d$ to 512. To extract video sequence features effectively, we integrate Swin3D Transformer layers during the 2nd, 3rd, and 4th downsampling and upsampling stages.

Utilizing the trained TS-VQGAN’s codebook and decoder to generate high-quality faces, we formulate the face restoration problem as a controllable face generation process.

During the face restoration process, we input the low-quality face $x_{l}$ into the low-quality encoder $E_{l}$, generating a low-quality latent representation $z_{l}$. Our goal is to use the function $T$ to predict the codebook index $\hat{S}$ based on $z_{l}$, and obtain the corresponding high-quality features$\hat{z_{q}}$ from Codebook $C$ according to the index $\hat{S}$. By inputting $\hat{z_{q}}$ into the decoder $D_{h}$, we obtain the restored output $x_{o}$.

This process can be framed as a maximum likelihood estimation, where we maximize the conditional likelihood of the predicted codebook $\hat{z_{q}}$ with respect to the low-quality input image $x_{l}$, enabling optimal parameter estimation for $E_{l}$ and $T$ to achieve accurate code prediction. Due to the loss of face texture details in the low-quality input, the accuracy of codebook selection using the nearest neighbor method is limited. To address this, previous methods employ a simple Transformer to predict a single frame codebook.

The process can be formally defined as follows.

However, for video face restoration, straightforward using the Transformer prediction codebook cannot effectively utilize temporal context face information, and artifacts are prone to appear on unaligned low-quality video faces (such as the Fig. 1).

Based on this, we design a temporal parse-guided Non-Autoregressive Transformer for codebook prediction. This design pattern can predict the codebook more accurately and restore high-quality face videos. Specifically, we input the video sequence into the face parsing model $E_{p}$, and embed the obtained intermediate feature $x_{p}$ into the Transformer as a relative position code. The network structure of the face parsing model $E_{p}$ is consistent with Yu et al. (2021), and we use the CelebAMask-HQ Lee et al. (2020) dataset to train $E_{p}$. At this time, the process of our temporal parse-guided codebook predictor can be defined in the Eq. (3).

Compared with previous codebook predictior, our proposed codebook predictor TPCP has two advantages: 1) It can utilize multi-frame information for more accurate codebook prediction. 2) Face parsing is used as position coding to realize codebook selection for face parsing perception.

Although it is already possible to obtain high-quality face videos using codebook prediction, the fidelity and coherence of faces are also important when recovering faces from videos. To this end, this paper designs a Temporal Fidelity Regulator (TFR) that uses the encoder feature $x_{e}$ to modulate the decoder feature $x_{d}$. Specifically, our TFR is divided into two processes. The first is time feature compression, which performs dimensionality reduction (squeeze) on the input $x_{e}$ and $x_{d}$. Followed by time feature aggregation, temporal fusion is performed on $x_{e}$ and $x_{d}$ along the time dimension to get $x^{T}_{e}$ and $x^{T}_{d}$ through convolution. Next, concatenate the interacted $x^{T}_{e}$, $x^{T}_{d}$ with $x_{e}$ and $x_{d}$ to get $x_{concat}$. After two convolutions respectively, affine parameters $\alpha$ and $\beta$ are obtained, and $\alpha$ and $\beta$ are used to modulate the multi-level decoder feature $x_{d}$ to obtain the output feature $\hat{x_{d}}$. The TFR will be adopted in each decoder level to enhance the final temporal refined output $x_{o}$. The detail of TFR is shown in Eq.4.

TFR performs feature aggregation in the time dimension during the modulation process, thereby improving the consistency of restored video faces and reducing jitter artifacts.

Based on our sophisticated architecture, the training of the model is divided into four stages.

Stage I. In the training process of the quantized autoencoder, three loss functions ($\mathcal{L}_{1}$, perceptual loss, adversarial loss) are used to jointly optimize TS-VQGAN, and the specific form is shown in the Eq. 5, $\Phi$ is the VGG19 network. The weights of the three losses are set to 1.0, 1.0, and 0.75, respectively.

Stage II. The goal of the second stage is to fine-tune the encoder $E_{l}$ and train the parsing-guided codebook predictor (TPCP). Only code-level loss is needed at this stage. The codebook prediction loss comprises two distinct components. The primary component, denoted as $L_{\text{code}}$, involves computing the cross-entropy loss between the predicted and actual codebook indices. The secondary component pertains to the codebook’s feature loss, where we measure the L2 loss between the encoder’s output feature $z_{l}$ and the ground truth codebook feature $z_{q}$.

Stage III. The third stage is fidelity training for PGTFormer. To optimize the TFR module and fine-tune the encoder $E_{l}$, we use code loss $\mathcal{L}_{code}$, pixel reconstruction loss $\mathcal{L}_{1}$, perceptual loss $\mathcal{L}_{per}$, adversarial loss $\mathcal{L}_{adv}$, and set the weights to 0.5, 1.0, 1.0, 1.0, and 0.75, respectively.

Stage IV. The fourth stage is complex pose face fine-tuning training, here we use unaligned face video sequences to fine-tune the model, and the loss is consistent with the third stage. We added $\mathcal{L}_{MSRL}$Dai et al. (2022) as a loss to increase video coherence, defined in Eq. (7).

Datasets. We train the model on the VFHQ Xie et al. (2022) dataset, which contains 16,000 video sequences, the resolution of the video is 512 $\times$ 512. To make training pairs, we follow Xie et al. (2022) synthesize LQ videos $x$ from HQ video counterparts $y$ with the following degradation model (in Eq.8):

Metrics. We first use PSNR, SSIM, LPIPS Zhang et al. (2018) to evaluate the quality of generated faces. In addition, we also evaluate identity preservation by the cosine similarity of the features of ArcFace Deng et al. (2021); Sun et al. (2020), denote as Deg. To better measure the fidelity of accurate facial expressions and expressions, we further employ Landmark Distance (LMD) as a fidelity metric. Specifically, we use the method proposed by Wang et al. (2019b) to obtain 98 landmarks from recovered faces and real faces. We then compute the L2 distance for each landmark and average the distances into the final score for the LMD metric. Since we are a video face restoration task, it is necessary to evaluate the continuity of the face, we calculate the L2 distance between the real face landmark displacement and the recovered face landmark displacement, which can evaluate the continuity of the face components in the video. We further calculate the peak signal-to-noise ratio of MSRL Dai et al. (2022) to evaluate the continuity of the restored face video.