ConRF: Zero-shot Stylization of 3D Scenes with Conditioned Radiation Fields

Artistic image stylization is a long-term problem for computer vision. The primary objective is to naturally blend the content features from one image with the style features from another, resulting in a unique image that seamlessly incorporates both elements. The early traditional method is to use handmade methods to simulate style [15, 16]. With the development of deep learning, [12] is the first work that uses CNN to separate and manipulate the content and style of images for style transfer. Since then, style transfer based on neural networks has gradually evolved from single-style transfer to multi-style transfer, and now to arbitrary style transfer [7, 19, 25, 26, 29, 38, 42, 48]. In terms of video style transfer, enforcing the temporal smoothness constraints defined on the optical flow can successfully transfer the style of the video [2, 17, 41, 47]. Nevertheless, all stylization methods, whether for images or videos, currently rely on a predetermined reference style, imposing substantial constraints on the creative possibilities.

Recently, neural implicit representation methods, such as NeRF [28, 33, 50, 35, 51, 52, 53, 1, 31, 8, 32], have shown great potential for high-quality rendering. NeRF leverages multi-layer perceptrons (MLPs) to implicitly model continuous scenes, leading to impressive results in view synthesis when rendering scenes. The emergence of NeRF has made significant progress in the neural network representation of 3D scenes. More recently, several works [5, 20, 37, 54, 57, 27, 4, 10] combine NeRF with neural style transfer [13] to handle artistic 3D scene stylization, [22, 14] are based on the diffusion model and can use text to stylize 3D scenes. In particular, [5, 37, 54] incorporate style loss functions from 2D images to fine-tune pre-trained NeRF models. Meanwhile, [27, 20] refine pre-trained NeRF models using a 2D style transfer model based on previous work. The approach presented in [4] employs a pre-trained 2D realistic network to constrain the realistic styles of different views and style images within a 3D scene. It’s worth noting, however, that the primary focus of this method is on transferring only the color tone of the style image. SNeRF [37] addresses the memory limitations associated with whole-image training in NeRF and enhances visual quality by employing a cyclic process of stylization and NeRF training. On the other hand, ARF [54] and Ref-NPR [56] aim to transfer detailed style features through matching features between style images and scenes. However, both SNeRF, ARF, and Ref-NPR entail a time-intensive optimization process for each reference style. While [10] and [20] leverage latent coding to achieve commendable results in style migration, their methods are limited to known styles and cannot adapt to unseen ones. [5] can achieve arbitrary style transfer by implicitly injecting style information into MLP parameters. Similar to [4], it can only transfer the color tone of the style image. Most of the neural style transfer-based methods mentioned above necessitate the use of style reference images to steer the style transfer, whereas our approach can leverage text or image for style transfer. To accomplish text style transfer, we utilize CLIP-encoded text embeddings to manipulate the scene for style transfer, leveraging only the already aligned text images in the pre-trained CLIP latent space. Furthermore, our work only focuses on artistic style transfer and does not have the creative capabilities based on generative models.

Following previous work [27], we leverage the featured NeRF model to represent 3D scenes. In contrast to the original NeRF framework [35], for every queried 3D position $x\in\mathbb{R}^{3}$, not only an RGB color $c$ but also furnishes a volume density $\sigma(x)$ and a multi-channel feature vector $F(x)\in\mathbb{R}^{C}$, where $C$ signifies the number of feature channels. Subsequently, we compute the feature representation for any rays r intersecting the volume by performing integration over sampled points along the ray, utilizing an approximated volume rendering technique [35]:

where $\sigma_{i}$ and $\delta_{i}$ refer to the volume density and feature attributes of the sampled point $i$, with $w_{i}$ indicating the weighting of $F_{i}$ within the ray r, and $\delta_{i}$ denoting the distance between adjacent samples. In addition, we use the same style transfer module to transfer the content style of pre-trained feature NeRF, which can be expressed as:

where $\sigma_{I}$ denotes standard-deviation and $\mu_{I}$ mean of style image ,$w_{i}$ denotes the weight assigned to sampled point $i$, $F_{i}$ stands for the feature associated with sample $i$, and $\mathcal{R}$ denotes the set of rays within each training batch.

CLIP [39] represents a recent breakthrough in the realm of language-image pre-training. It employs two encoders, extensively trained on textual language, to construct a text-image embedding space. This space serves as a bridge connecting textual and image features, encompassing a diverse array of visual concepts. Given its ability to establish meaningful connections between text and image attributes, a natural question arises: Can this feature space be harnessed to facilitate text-based style transfer? On the one hand, because CLIP is not specifically designed for style transfer tasks, it poses challenges in directly decoupling content and style features. On the other hand, directly fine-tuning CLIP for style transfer tasks is impractical due to the highly subjective nature of style, and the absence of corresponding fine-tuning data. To this end, we propose a novel method to project text-image space onto a subspace of style features to facilitate the effective exchange of text or images with style features, so as to achieve the purpose of using text or image to control style transfer.

Prior work [27] allowed different styles on different objects. However, this approach necessitated the provision of precise masks in advance to regulate the styles for achieving the desired combinations. Taking inspiration from recent promising prompt-based methods, we introduce a simple approach using text prompts to apply styles directly to specific 3D regions.

As shown in Fig. 4, once training is completed, we can use text to perform style transfer on 3D scenes. Specifically, since text and image features are shared, we can get the mean and variance of the style when we input text, that is,

where $T_{s}$ denotes the text input, $E_{\mathcal{T}}$ denotes the CLIP text encoder and $(\sigma_{T},\mu_{T})$ is the style representation obtained from the text description. Thus, using CLIP for style transfer can be expressed as:

Eq. 18 can be interpreted as the individual transfer of style to every sampling point along the ray prior to volume rendering. Ultimately, we employ a decoder to reproject these stylized features into RGB space, thereby yielding stylized novel views. In addition, our local style transfer feature is optional and becomes active only once a content text prompt is entered. The local style transfer branch is activated, we can use a content text prompt and $F_{c}$ calculate the similarity $z$:

where $cos\left\langle,\right\rangle$ is the cosine similarities. We can then get the mask $M$:

$t$ is the threshold. Thuse Eq. 18 can be replace:

Our method supports both reference image style transfer and text prompt style transfer, enabling versatile multi-combination style transfers. After inputting content text prompts, our method allows for the use of various prompt combinations to transform different aspects. It supports combinations such as text-text, image-image, and text-image for style transfers.

We conducted a comprehensive evaluation of ConRF using two publicly available datasets: LLFF [34], which comprises real scenes with intricate geometry, and Synthetic NeRF [33], featuring $360^{\circ}$ views of objects. For methods that rely on reference-style images, we benchmarked against the state-of-the-art StyleRF [27], Ref-NPR [56], SNeRF [37], and ARF [54]. In the case of methods employing text prompts, ConRF was evaluated against the 2D single-text condition method, CLIPStyler [24], and a 3D text control editing method, CLIP-NeRF [46].

The qualitative comparisons are shown in Fig. 5 and Fig. 6. For methods using reference-style images, we can see that compared with other methods, our method transfers the color and artistic style of the reference-style image while maintaining relatively complete original content. Although the single style transfer methods SNeRF [37] and ARF [54] show better transfer results compared to our method (as shown in the Trex scenario), we transfer style features in localized regions as well and can perform arbitrary style transfers without retraining. In addition, for the Ref-NPR [56], which is highly dependent on the existing 2D transmission method. In order to make the comparison fair, we directly use the style image as the reference style for transfer. In terms of methods using text prompts, compared with CLIPStyler [24] and CLIP-NeRF [46], our method is closer to text description. Although our method can achieve excellent results, we observe that our method can be further improved in some aspects, which are described in the limitations of supplementary material. We present the locally transferred image in Fig. 7. Our method presents the ability to utilize text prompts effectively, allowing for direct and precise control over specific sections of content. The selection volume empowers users to effortlessly convey a wide range of distinct styles within localized areas. At present, our approach serves as an initial exploration, but it holds great promise.

At present, 3D style transfer stands as a nascent and scarcely explored domain, with limited metrics available for quantitatively evaluating stylization quality. Therefore, following [5, 4] we compared the consistency of multiple views, we compared the short-range and long-range consistency scores of adjacent views and distant views respectively. In our experiments, we warp one view to another based on optical flow [44] and then compute masked SSIM and LPIPS scores [55] to measure the stylization consistency.

As shown in Tab. 1, we compared with six methods. Compared to our baseline StyleRF [27], our short-range LPIPS score is improved by 31%, and long-range LPIPS score is improved by 59.4%. Please note that SNeRF [37] and Ref-NPR [56] outperform our method. This is primarily due to the fact that in certain scenarios, the style transfer effect of SNeRF [37] may not be as pronounced, whereas Ref-NPR [56] directly transfers the style image’s characteristics into the content. For methods using text prompts, we are completely superior to both methods.

To evaluate the effectiveness of our approach, we perform ablation experiments separately for global style transfer and local style transfer functions.

Our model benefits from CLIP, while also facing certain limitations imposed by it. For example, as shown in Figure 10, we were not able to migrate the blue text style well because we used CLIP directly without making any changes to it. Our work currently only focuses on artistic style transfer and does not have creative capabilities based on generative models. We expect future work to successfully address them. Additionally, our local style transfer currently only works in face-forwarding scenes.

Our model is trained on a single RTX 3090 GPU using approximately 80,000 samples from the WiKiArt dataset [9] as style images. During the training phase, the Adam optimizer [23] is employed with a learning rate of 1e-4. We assign the loss weights for style and content as 20 and 1, respectively, while all other loss weights are maintained at 1. For incorporating CLIP [39], we utilize the pre-trained ViT-B/32 model.

We conduct a user study, following previous research [27], where we compare our approach to a 3D style transfer baseline. We engage 25 participants with diverse demographic backgrounds, including different professions, age ranges, and ethnicities, to ensure a broad perspective. Each participant is shown a series of stylizations that include original scene videos, their stylized counterparts, and videos processed using both our ConRF method and the baseline technique.

Participants are tasked with two evaluations: identifying the stylized video that most accurately reflects the style of a given image, and selecting the video that exhibits the best multi-view consistency. For a comprehensive analysis, we prepare 24 unique scene-style pairings, which we then randomly distribute into 6 categories. Each participant evaluates one category, providing us with a diverse set of opinions on the effectiveness of our approach. The aggregated findings from this study are detailed in Table 2, as referenced in our report.

We present more qualitative results here and video results in the supplementary material.