Training-Free Large Model Priors for Multiple-in-One Image Restoration

Unlike previous methods that require fine-tuning on large models or two-stage training, we obtain content and degradation embeddings in a training-free manner. We leverage prompt engineering to query a multimodal language model, which outputs degradation information present in the image as well as content information unrelated to degradation. We then obtain the corresponding embeddings of the generated text using the CLIP encoder, serving as global degradation and content embeddings to guide the model’s training. We utilize the GPT4o model [1], which has demonstrated strong performance in low-level tasks, to generate this global degradation priors $\mathbf{e}_{d}$ and content text embedding $\mathbf{e}_{c}$, shown in the top of Figure 2.

In addition to the global prior knowledge provided by text, we also utilize images generated by the diffusion model as fine-grained content priors, providing detailed texture and feature references for image restoration models. Specifically, we input the content text output by the multimodal large language model into the SDXL [27] model as a prompt and use the degraded text as a negative prompt, ensuring that the generated image shares similar content with the ground truth.

The degradation embedding $\mathbf{e}_{d}$ extracted directly from the CLIP encoder cannot be directly applied as a prior for the image restoration network due to two reasons: 1) The CLIP encoder lacks awareness of specific degradation details such as rain streaks and noise distribution, providing only global classification knowledge. 2) The textual description generated by the multi-modal language model may not be entirely reliable. Therefore, we design a query-based prompt encoder to refine $\mathbf{e}_{d}$ into a more fine-grained degradation representation $\mathbf{Z}_{d}$ that can effectively guide the restoration network, while incorporating degradation information from the image itself. Specifically, given a learnable query embedding $\mathbf{E}_{p}\in\mathbb{R}^{\rm\hat{N}\times C}$, the degradation text embedding $\mathbf{e}_{d}$ from CLIP, and the degraded image representation $\mathbf{I}_{d}$, the query-based prompt encoder computes the refined degradation representation $\mathbf{Z}_{d}$. In detail, $\mathbf{E}_{p}$ attends to itself via $\textrm{SA}(.)$ to obtain $\mathbf{E}_{p}^{\prime}$, which is projected to queries $\mathbf{Q}_{E_{p}}$. Then, cross-attention is performed between $\mathbf{Q}_{E_{p}}$ and keys/values from $\mathbf{e}_{d}$ to obtain $\mathbf{Z}_{\text{text}}$ encoding degradation information from text, and with $\mathbf{I}_{d}$ to obtain $\mathbf{Z}_{\text{image}}$ encoding image degradation information as:

where $\textrm{SA}(.)$ and $\textrm{CA}(.)$ denote self-attention and cross-attention, respectively. Finally, $\mathbf{Z}_{\text{text}}$ and $\mathbf{Z}_{\text{image}}$ are fused and processed by a feed-forward network (FFN) [11] to yield the refined degradation representation $Z_{d}$ as

This representation $\mathbf{Z}_{d}$ combines the information from text prior and the image feature, can provide restoration network with a better degradation presentation.

In the encoder part of our image restoration model, we employ degradation-aware transformer blocks to inject degradation information and enable dynamic feature processing based on the degradation type. Specifically, each degradation-aware transformer block consists of three components: transposed self-attention [43], gated feed-forward network [43], and a degradation embedding adapter, as shown in Figure. 3. Given the input feature map $\mathbf{F_{i}}$ and degradation embedding $\mathbf{Z}_{d}$, the operations are defined as follows:

where transposed self-attention $\textrm{TSA}(.)$ captures long-range dependencies in $F_{i}$, Gated forward network $\textrm{GFN}(.)$ refines the local feature. The degradation embedding adapter $\textrm{DEA}(.)$ projects $\mathbf{Z}_{d}$ to the same channel dimension as $\hat{F}_{i}$ and further generating degradation-aware parameters. Specifically, degradation adapter generate the degradation-aware parameters using:

where $\gamma$ and $\beta$ and $\mathcal{G}$ are scale, shift and gate parameters modulated by $\mathbf{Z}_{d}$. $Split(.)$ is the split operator along channel dimension. By integrating $\mathbf{Z}_{d}$ into the transformer blocks, the model can dynamically adapt its feature processing based on specific degradation representation, enabling effective restoration for diverse degradations using a single model.

In the bottleneck parts of our image restoration network, we designed content-aware transformer blocks to incorporate local content features and enhance restoration performance. In the bottleneck, we utilize the content text embedding $\mathbf{e}_{c}$ as a reference. As shown in Figure 3. We first project $\mathbf{e}_{c}$ to the same dimension as the feature map $\mathbf{F}_{i}$ using a multi-layer perceptron. Then, we perform self-attention on $\mathbf{F}_{i}$ and calculate the similarity between $\mathbf{F}_{i}$ and the projected $\mathbf{e}_{c}$. Based on this similarity, we adaptively select and integrate useful features from $\mathbf{e}_{c}$ into $\mathbf{F}_{i}$, followed by a gated FFN for local feature processing. This operation injects global content priors from the text embedding into the network. The content-aware transformer block can be formulated as:

Here we utilized the reference-attention $RA(.)$ to inject the reference feature, due to the token length of $\mathbf{e}_{c}$ is a fixed number. The integrated features $\hat{\mathbf{F}}_{i}$ are further processed by a gated FFN to produce the output $\mathbf{F}_{i+1}$. Given the input feature $\tilde{\mathbf{F}}_{i}$ and the reference feature denoted as $\mathbf{e}_{c}$, the operation of $\textrm{RA}(.)$ can be defined as follows:

In the decoder parts of our image restoration network, we introduce reference-based transformer blocks to integrate fine-grained reference features 4. Specifically, we leverage the reference image $\mathbf{I}_{r}$, generated by the diffusion model, as our reference. This block is designed to extract both global and local similar features from the reference image. To achieve this, we employ a hybrid approach that combines global and local attention mechanisms. The global reference attention utilizes a transposed cross-attention mechanism to compute the similarity between the two images along the channel dimension. In contrast, the local reference attention employs convolution to fuse similarity features along the spatial dimension. Given the input feature $\mathbf{F}_{i}$ and reference image $\mathbf{I}_{r}$, this process can be described as follows:

Here, $\phi(.)$ is the convolution operator that projects $\mathbf{I}_{r}$ to $\mathbf{F}_{ref}$ for dimension alignment. The $\textrm{TSA}(.)$ and $\textrm{split}(.)$ are the transposed self-attention and channel split operators, respectively. After generating the outputs from local reference attention $\textrm{LRA}(.)$ and global reference attention $\textrm{GRA}(.)$, the two features are concatenated and fused through the linear projection $\Theta(.)$. Finally, a gated forward network, $\textrm{GFN}(.)$, is utilized to enhance the locality of the features. The $\textrm{GRA}(.)$ is the cross attention version of $\textrm{TSA}(.)$, where $Q$ is derived from $\tilde{\mathbf{F}}_{i}$ and $K,V$ are generated from $\mathbf{F}_{ref}$. The local reference attention can be described as below:

where ($\mathbf{W}_{1}$), ($\mathbf{W}_{2}$), and ($\mathbf{W}_{a}$) are convolutional filters, ( $*$ ) denotes convolution operation, ($\odot$) denotes element-wise multiplication, and (ReLU) and (Softmax) are the activation function and softmax function, respectively.

In Table I, we present a comparison between our proposed LMDIR approach and existing state-of-the-art methods, demonstrating substantial enhancements across various tasks. Notably, in comparison to PromptIR, our method achieved an average improvement of 2.3 dB in PSNR of the image deraining task. Additionally, the denoising and low-light image enhancement tasks exhibited marked progress. PromptIR’s inability to produce accurate restoration outcomes can be attributed to its implicit degenerate feature learning method. It is noteworthy to highlight that DA-CLIP necessitates an extensive volume of data for training due to its reliance on diffusion models and fails to yield satisfactory results within our settings. Contrasting with these methods, our approach leverages the prior knowledge provided by the large model and the information intrinsically present in the degraded image, resulting in superior performance.

The comparative visualization of different methods is shown in Figures 5 and 6. For each task, we opted for two representive images for visual comparison. Within the the denoising task, we set the noise level $\sigma$=50 for comparison. As the figure depicted, our methods outperforms others in achieving superior restoration outcomes. In the context of denoising, DA-CLIP falls short in complete noise reducing, whereas PromptIR induces a loss of high-frequency details within the image. In the low-light image enhancement task, the color accuracy of our method closely aligns with the ground truth, while the results generated by AirNet manifest a dark texture. Evaluating the image deraining results, residual rain streaks are discernible in the images produced by AirNet and PromptIR. In contrast, our method exhibits the highest quality in rain removal results.

Furthermore, we conducted an additional evaluation of the generalization capabilities of various multi-in-one restoration models on out-of-distribution (OOD) data, thereby evaluating their practical performance in real-world applications. We analyzed the impact of varying noise and rain streak intensities on image restoration tasks. More specifically, for the denoising task, we selected two distinct noise level, 60 and 75. In the context of image deraining, we opted for the Rain100H [41] and Test100 [41] datasets as our testing datasets, both of which differ substantially from Rain100L and feature more intense rainfall conditions. The results of these experiments are presented in Table II and Figure 7. Our results indicate that the performance of both AirNet and DA-CLIP, whose degradation knowledge is solely based on limited classification knowledge, significantly diminishes when confronted with OOD data. In contrast, the implicit degradation representation of PromptIR exhibits a certain degree of adaptability to OOD data, thereby outperforming AirNet in OOD dataset significantly. Our proposed methods, which combines the prior knowledge of large-scale models with the inherited information present in degraded images, demonstrates an enhanced performance in the presence of OOD data.

In this study, we establish a baseline model based on Restormer. This baseline model operates without any reliance on prior information. Leveraging the self-attention inherent capacity for input-adaptive feature extraction, our baseline model demonstrates a Preliminary ability for dealing with diverse degradation, as shown in the first row of Table III.

Our query-based prompt encoder is designed to enhance the model with a comprehensive degradation prior. In the second set of the ablation study, we replace the transformer block in encoder part of the baseline model with our degradation-aware transform block, effectively providing the network with degradation-specific information. As evidenced by the second row in the table III, the introduction of such degradation knowledge led to marked enhancements across multiple task performances, demonstrating the efficacy of the query-based prompt encoder in providing the model with degradation knowledge.

Beyond generating degradation descriptions, our MLLM also produces context description text after receiving visual inputs. These descriptions serve as a source of global prior information, bolstering the model’s ability for scene understanding. Building upon Model (I), we replaced the transformer block of bottleneck component within the UNet with our reference-based transformer block. The results presented in the third row of the table reveals that, following the integration of contextual information, the model exhibits marked advancements across a various tasks, thereby affirming the pivotal role of content priors in enhancing overall model performance.

Building upon the Model (II), we incorporate fine-grained image priors into our network. We replace the Transformer block in the decoder part of Model(II) with our reference-based Transformer block. We harness a MLLM to generate contextually rich descriptions which serve as prompts for SDXL. This process yields high-fidelity images that retain the identical semantic content as the input image, yet are devoid of any degradation. The integration of such high-quality images plays a pivotal role in offering detailed guidance to the model. As evidenced by the results in the fourth row of the table III, the integration of fine-grained prior knowledge has led to an improvement in the overall performance of the model. Furthermore, we conduct a visual analysis of the similarity map between the CLIP features of the ground truth image and the reference image using the Test100 dataset. As illustrated in Fig. 12, the reference image demonstrates a high degree of similarity in the CLIP feature space. This observation provides strong evidence for the efficacy of our generated reference image and the associated content description text.

Our query-based prompt encoder is motivated by the observation that neither text embeddings nor image embeddings alone can fully capture the information necessary to discriminate between different degradations, as illustrated in Fig. 10. We employ t-SNE [36] to visualize the distributions of $e_{d}$ and $I_{d}$. The results clearly demonstrate that both $I_{d}$ and $e_{d}$ individually fail to effectively distinguish between various degradations. However, after processing through our query-based prompt encoder, the boundaries between degradation types become distinctly delineated. This transformation provides strong evidence for the efficacy of our proposed query-based prompt encoder.