AnyMaker: Zero-shot General Object Customization via Decoupled Dual-Level ID Injection

Global ID injection. In text-to-image generation, text prompts are leveraged to guide diffusion models to generate diverse images. Nevertheless, diffusion models are incapable of accurately generating images with a specific object ID solely relying on the text description. To inject the ID information of the interested objects into the text, while simultaneously maintaining the editing capability of text prompts for other elements such as the background and motions, we design the global ID injection mechanism. Specifically, we merge the semantic-level class tokens $(f_{dino}^{C},f_{mae}^{C})$ with the class word in the text embedding, such as "dog" or "cat",

$$f_{fuse}^{C}=MLP(Concat(f_{text}^{C},f_{dino}^{C},f_{mae}^{C})),$$

(3)

where $f_{fuse}^{C}$ is the fused class token, and $f_{text}^{C}$ represents the class word embedding of the interested object. Then we insert the fused class token $f_{fuse}^{C}$ to the original position of the class word in the text embeddings, obtaining the global-level condition $c_{g}$ with ID-related information of the object of interest, as shown in Fig. 2. Following this, the global-level condition $c_{g}$ engages with the cross-attention module in the UNet, like to standard text-to-image models [10], as follows,

$$Z_{g}=Attention(Q_{g},K_{g},V_{g}),$$

(4)

where $Q_{g}=Z_{g}W_{q}$, $K_{g}=c_{g}W_{k}$, $V_{g}=c_{g}W_{v}$ are the query, key, and value of cross-attention module, in which $W_{q}$, $W_{k}$, $W_{v}$ are the weight matrices of the trainable linear projection layers, and $Z_{g}$ is the query feature containing the latent input information.

Local ID injection. Semantic-level information in the global-level condition $c_{g}$ contains relatively fewer details of the objects, and thus the model cannot handle complex customization tasks for general objects effectively merely relying on global ID injection. To supply the model with sufficient ID-related details, we propose the local ID injection mechanism. Specifically, we first fuse the extracted patch tokens $f_{dino}^{P}$ and $f_{mae}^{P}$ with a two-layer MLP to densify the contained information to a unified representation $c_{l}$, as follows,

$$c_{l}=MLP(f_{dino}^{P})+MLP(f_{mae}^{P}),$$

(5)

where $c_{l}$ serves as the local-level condition for the diffusion UNet. Then, we add one cross-attention module in each upblock of the diffusion UNet in order to inject the ID-related details into the model without compressing $c_{l}$ dimensions, feeding the model with as much ID information as possible.

$$Z_{l}=Attention(Q_{l},K_{l},V_{l}),$$

(6)

where $Q_{l}=Z_{l}W_{q}$, $K_{l}=c_{l}W_{k}$, $V_{l}=c_{l}W_{v}$ are the query, key, and value of cross-attention module, in which $W_{l}$, $W_{l}$, $W_{l}$ are the weight matrices of the trainable linear projection layers, and $Z_{l}$ is the query feature containing the latent input information.

Decoupling branch and normal branch. In the decoupling branch, we first extract the target image features $f_{tar}$ through CLIP [24] to make an image embedding prior [11]. Next, we mask out the ID information contained in the image embedding by a trainable feature mask $m_{id}$, and thus the masked image feature $f_{msk}=f_{tar}\odot m_{id}$ contains only non-ID information of the target image, where $\odot$ represents element-wise multiplication. Then, we add the masked image feature $f_{msk}$ to the fused class token $f_{fuse}^{C}$ and then feed it to the diffusion UNet for generation following the injection way mentioned in the Sec. 3.3. Yet in the normal branch, we inject $f_{fuse}^{C}$ to the model without $f_{msk}$. The generation process of two branches can be illustrated as follows,

$$f_{msk}\oplus f_{fuse}^{C}\Rightarrow I_{tar},\hskip 14.22636ptf_{fuse}^{C}\Rightarrow I_{tar},\hskip 14.22636ptf_{msk}\perp f_{fuse}^{C},$$

(7)

where $\oplus$ represents element-wise addition and $I_{tar}$ is the target image, and the symbol $\perp$ signifies that the features it connects are independent of each other and have no obvious correlation. Through training in the two branches, we push the non-ID information into the masked image feature $f_{msk}$ and only retain the ID-related information in the fused class token $f_{fuse}^{C}$.

Training strategy. We use three losses in our training stage to ensure that the ID-aware decoupling module takes effect. Specifically, We utilize the following denoising losses to train each branch,

$$\mathcal{L}_{decouple}=\|\epsilon-\epsilon_{\theta}\big{(}x_{t},c_{g},f_{msk},c_{l},t\big{)}\|^{2},$$

(8)

$$\mathcal{L}_{normal}=\|\epsilon-\epsilon_{\theta}\big{(}x_{t},c_{g},c_{l},t\big{)}\|^{2},$$

(9)

where $t$ is the randomly sampled time step and $x_{t}$ represents the noisy latent of the target image, $\epsilon$ is the ground truth noise and $\epsilon_{\theta}$ is the predicted noise. $c_{g}$ and $c_{l}$ represent the global-level and local-level conditions acquired in Sec. 3.3. Additionally, we design a contrastive loss, ensuring that the masked image feature $f_{msk}$ captures non-ID information and the fused class token $f_{fuse}^{C}$ remains free of it, as below,

$$\mathcal{L}_{contrast}=Sim(f_{fuse}^{C},f_{msk}),$$

(10)

where $Sim$ is instantiated as cosine similarity. In summary, the training loss is the sum of the three loss functions mentioned above,

$$\mathcal{L}=\alpha_{1}\mathcal{L}_{normal}+\alpha_{2}\mathcal{L}_{decouple}+\alpha_{3}\mathcal{L}_{contrast},$$

(11)

where $\alpha_{1},\alpha_{2},\alpha_{3}$ represent hyperparameters to balance the three losses.

Large-scale dataset. Our dataset consists of approximately 315,000 samples in total with more than 10,000 categories, covering various types such as human faces, animals, clothes, human-made tools, etc. Adequate training samples and diverse categories endow the model with general ID understanding. We introduce the main categories of the dataset and their quantities in the Appendix.

High quality. The average resolution of the images in our dataset is 1039$\times$950, which is helpful for the high-quality generation of our model in both training and inference stage.

Reference-target image pair. Each sample consists of a reference image, a segmentation mask of the object of interest in the reference image, a target image, and a text caption of the target image. During training, the reference image with its segmentation mask provides the model with ID information, the text caption of the target image offers semantic-level guidance for generation, and the target image serves as the ground truth. In each sample, the reference image and the target image depict the same object in different states, which means that each reference-target image pair shares the same ID of the object of interest and has different non-ID elements such as motions and directions. Training with reference-target image pairs endows the model with the ID processing ability while not damaging the diversity of generation.

Data collection. MC-IDC dataset contains diverse sources including web-crawled images, movies and publicly available datasets. To facilitate the subsequent generation of image pairs, most of the data we collect are video datas or multi-view datas. To ensure the high quality of our dataset, we delete the images with a resolution less than 300$\times$300.

Instance detection and segmentation. For some publicly available datasets or web-crawled images which do not provide segmentation annotations, we use the advanced instance segmentation model [25] for instance detection and segmentation. For movie dataset, we extract the frames with 10fps and do object tracking with  [26], aiming to establish the ID connection across frames.

Image pair generation. For most data samples which are from a clip of video or a group of multi-view data, we randomly select two frames containing the same object and perform a random crop around the object to form the reference-target image pair. For the other data samples from the single image dataset, we perform random data augmentations on the single image two times, and take them as the reference image and the target image respectively.

Text prompt generation. In each image pair, the target image is densely captioned with the state-of-the-art large vision language model [27], serving as the text prompt to guide the generation.

Implementation details. To be compatible with open-community as Civitai.com, we employ SD1.5 [28], the most used foundational model in the public, as the backbone model in our experiments. We use the multi-scale training mode proposed in [1] considering the resolutions of images from different sources are diverse. We employ the same CLIP image encoder as in [28] in the ID-aware decoupling module. We use MAE [23] and DINOv2 [22] for extracting ID features. During training, we set the learning rate as 1e-5 with a batch size of 32. We conduct the training process about 6 epochs with 32 V100 GPUs, consuming about 30 hours. We adopt dropout with a ratio of 0.1 for the global-level condition $c_{g}$ and the local-level condition $c_{l}$ respectively. During inference, we utilize 50 steps denoising process for generation and set the classifier-free guidance scale as 7.

Competitors. To demonstrate the generality of our AnyMaker, we not only make a comparison with previous works on general object customization, but also compare it with task-specialized methods in two popular specific customization domains, that is, human customization and virtual try-on. Specifically, we compare AnyMaker with IP-Adapter [7] on zero-shot general object customization. On human customization, we further choose human specialized method PhotoMaker [6] and InstantID [5] for comparison. On virtual try-on, we conduct a comparison with the recent state-of-the-art MagicClothing  [21]. It is worth noting that all the methods mentioned above are zero-shot methods and are capable of customizing images based solely on one single reference image. Therefore, in this paper, we do not compare object-specific methods such as DreamBooth [8] and DisenBooth [9] which typically require 3 to 5 reference images and additional fine-tuning.

Evaluation dataset. Our evaluation dataset contains 1000 text-image samples, including general objects, human data, and virtual try-on data with a ratio of 4:3:3. All samples in the evaluation dataset are randomly selected from the MC-IDC and do not appear during the training stage.

Evaluation metrics. Following PhotoMaker [6], we utilize DINO-i [22] and CLIP-i [29] to measure the ID fidelity and use CLIP-t to measure the prompt fidelity. We leverage FID [30] to assess the generation quality. For human customization, we additionally calculate the face similarity (FaceSim) with FaceNet [31] as commonly done in [6, 5]. Furthermore, we design a novel metric DiverSim-i, which calculates the average DINO similarity among the images generated according to diverse text prompts containing different scenarios. The lower the value of DiverSim-i, the stronger the ability of the model to generate diverse images in line with text prompts.

Effectiveness of general ID Extraction. We compare our proposed general ID extractor with DINOv2 [22], MAE [23] and CLIP [29]. As in Tab. 2, our method performs the best, whether in terms of FID, which measures the quality of generation, or in terms of CLIP-i and DINO-i, which measure ID fidelity. As in Fig. 4, merely using DINOv2 as the extractor fails to maintain color consistency, while solely using MAE as the extractor lacks sufficient details. In contrast, our method combines the advantages of both DINOv2 and MAE, which enables our model to customize images with rich details and the consistent color of the object in reference images.

Benefits of dual-level ID injection. We explore the effectiveness of our dual-level ID injection module compared to only global-level injection and only local-level injection separately. As shown in Tab. 3, by combining global and local injection, AnyMaker achieves the best generation results both in quality as measured by FID and ID-fidelity as measured by CLIP-i and DINO-i.

Benefits of ID-aware decoupling. We conduct a comparative experiment on whether to add the ID decoupling module to the model during training, aiming to verify its effectiveness. As in Tab.4, the model trained with the ID-aware decoupling module achieves higher ID-fidelity scores in terms of CLIP-i and DINO-i, which means that the decoupling module can help the model to discern ID information embedded in object representations, thereby generating results with better ID consistency. Further, we visualize the results by models trained with and without decoupling in Fig.5. The model trained with decoupling exhibit enhanced capabilities in distinguishing ID information from non-ID elements such as motions (open or shut the mouth) and directions (front-facing or side-facing) of objects of interest. The enhanced discrimination ability allows the model to mitigate the influence of non-ID information during generation, thereby better preserving the text editing capabilities.