Technique Report of CVPR 2024 PBDL Challenges

In addition, we attempt to introduce some advanced low-light image enhancement methods, such as CIDNet [30], GlobalDiff [45], and Retinexformer [9], to enhance the challenge data, and perform detection algorithm on the enhanced images. Unfortunately, the performance has not been improved or even decreased. We argue that since the challenge dataset does not have pairs of low-light and normal scene images, this leads us to use these image enhancement methods for cross-domain inference, which corrupts the distributional information in the data itself, and ultimately leads to a degradation of detection performance.

Some visual results of our method on the testing set are shown in Figure 4.

As shown in Figure 7, we trained several detectors, including RTMDet [71], YOLOX [38], Dino [128] and Co-DETR [132] on the challenge datasets, and then ensemble the predictions from those models to achieve better results. We employed Weighted Box Fusion [91] as our ensemble method.

RTMDet [71] is an efficient real-time object detector that surpasses the YOLO series. Apart from adjusting the number of output classes, we made no modifications to RTMDet. RTMDet-x and RTMDet-l models were chosen due to their high mAP on the COCO dataset.

YOLOX [38] is a highly advanced detector that represents a significant improvement upon the YOLO series. Apart from adjusting the number of output classes, we made no modifications to YOLOX. Taking into account both performance and training costs, we opted for YOLOX-l.

DINO [128] is an advanced end-to-end object detector. Apart from adjusting the number of output classes, we made no modifications to DINO. DINO-Swin-L model was chosen due to its high mAP on the COCO dataset.

Co-DETR [132] is a novel training scheme aimed at improving the efficiency and effectiveness of DETR-based detectors. Apart from adjusting the number of output classes, we made no modifications

During the initial phase of the challenge, only annotations for the training set were available. Initially, we randomly divided the training set into a proxy training set and a validation set using an 8:2 ratio. Subsequently, we trained the models and optimized the training settings to enhance performance. These settings were then uniformly applied for training on the original complete training dataset, ensuring full utilization of the available data.

During the validation and test phases, we retained the weights from the last epoch for evaluation on the official validation and test sets.

We utilized the MMDetection framework [14] to conduct all experiments on 4 machines, each equipped with 8 NVIDIA RTX 3090/4090 GPUs.

Due to the extensive nature of our training process, which involved training over 18 models for ensemble, providing detailed training configurations in this paper may not be feasible. We recommend referring to the config files in our code repository for more comprehensive information.

These predictions were then utilized in the weighted box fusion to ensemble predictions. The weight of each prediction was determined using a grid search algorithm on the proxy validation set described in 2.3.2.

For more details about the ensemble process, please refer to the configuration files in our code project.

As shown in Table 4, our ensemble method for the Object Detection track attained a mean Average Precision (mAP) of 0.76. Additionally, our RTMDet model for the Instance Segmentation track achieved an mAP of 0.58.

One of the critical advantages of Mask DINO is its ability to enable task cooperation, demonstrating that detection and segmentation can mutually enhance each other within query-based models. Additionally, Mask DINO leverages better visual representations pre-trained on large-scale detection datasets to improve semantic and panoptic segmentation. This synergistic approach not only enhances the performance but also provides a robust and versatile framework capable of handling multiple vision tasks effectively. By employing Mask DINO, we aim to leverage these strengths to achieve superior results in the “Low-light Object Detection and Instance Segmentation” competition.

We chose FaPN for our competition due to its demonstrated ability to improve multi-scale feature generation. Its integration into our network aims to leverage these strengths, thereby enhancing our model’s accuracy in the competition.

The total loss for learning is defined as:

where $x$ is the clean synthetic RAW image, $x^{\prime}$ is its noisy version, and $\alpha$ and $\beta$ are the weights of the respective losses. We empirically set $\alpha=1$ and $\beta=0.01$.

The loss $L_{\text{IS}}$ is the task loss, e.g., instance segmentation loss, which consists of classification loss, bounding box regression loss, and segmentation (per-pixel classification) loss. The specific formula for $L_{\text{IS}}$ is related to the model, we employ the same loss as the origianl model. This loss is applied to both the clean image $x$ and the noisy image $x^{\prime}$ to ensure the model performs consistently regardless of noise.

The loss $L_{\text{DS}}$ is the feature disturbance suppression loss, defined as:

where $f^{(i)}(x;\theta)$ represents the $i$-th stage of feature maps of the model. By minimizing the Euclidean distance between the clean features $f^{(i)}(x;\theta)$ and the noisy features $f^{(i)}(x^{\prime};\theta)$, the disturbance suppression loss encourages the model to learn disturbance-invariant features. This reduces feature disturbance caused by image noise and improves the model’s robustness to corrupted low-light images.

Unlike perceptual loss [39], our approach does not require pretraining a teacher model, making our training process simpler and faster. With $L_{\text{IS}}(x;\theta)$ and $L_{\text{IS}}(x^{\prime};\theta)$, our model can learn discriminative features from both clean and noisy images, maintaining stable accuracy regardless of noise. In contrast, the “student” model in perceptual loss [39] only sees noisy images, which can degrade performance on clean images and limit robustness. Additionally, the domain gap between the feature distributions of the teacher and student models can harm the learning process. By minimizing the distance between clean and noisy features predicted by the same model, we avoid this problem.

Besides, most existing methods use convolution for multi-frame features fusion, where the lacking of long-range modeling ability may decline their recovery result. Some methods utilize spatial self-similarity, e.g. [122], while the abundant temporal-spatial self-similarity in the extra temporal dimension is not fully exploited.

Self-attention is suited for aggregating self-similarity since it can dynamically allocate weight for each pixel. To this end, we combine 3D temporal-spatial attention with the hierarchical design of U-net. Nonetheless, Transformer may suffer from the deficiency of local feature extraction, which is indispensable for recovering image details. Thus, we combine the locality of convolution with the long-range interaction of self-attention in each Transformer block.

The overview architecture of our network is shown in Fig. 11. We focus on the Raw2Raw video denoising task, where the input and output are all in the raw domain. The input is of size $T\times H\times W\times 4$. $T$ represents the number of input frames, with each frame containing $H\times W\times 4$ pixels in the Bayer pattern. The output frame is of size $H\times W\times 4$. To embed the pixels in images as tokens, we first apply a $3\times 3$ convolution. After embedding, all the tokens pass through $K$ encoders and patch merging layers. Each encoder contains $M$ Shifted Window Transformer blocks. For downsampling, we use the $4\times 4$ convolution and double the dimensions. Symmetrically, the decoder includes $K$ Transformer blocks and patch expanding layers. The output of decoder layers is then projected back to image patches. Finally, the extracted multi-frame features are temporally fused to handle the misalignment.

Two consecutive 3D shifted window-based Transformer blocks are computed as:

where $\hat{\mathbf{z}}^{l}$ and $\mathbf{z}^{l}$ represent the output features of the 3DW-MSA and 3DSW-MSA for $l^{th}$ block. A LayerNorm (LN) is added before MSA and after the FeedForward layer (FFN). Following the previous studies, we add the relative position encoding $B\in\mathbb{R}^{T^{2}\times M^{2}\times M^{2}}$ to the 3D attention block. The self-attention is computed as:

where $Q,K,V\in\mathbb{R}^{TM^{2}\times d}$ are the query, key and value matrices. $d$ is the dimension of the query and key features. $TM^{2}$ is the number of tokens per window. And, the values of $B$ are taken from the 3D bias matrix $\hat{B}\in\mathbb{R}^{(2T-1)\times(2M-1)\times(2M-1)}$, corresponding to the temporal range of $[-T+1,T-1]$ and the spatial range of $[-M+1,M-1]$.

As shown in Fig. 13, the framework of this model adopts a three-stage design: 1) feature extraction, 2) multi-frame feature fusion with grouped spatiotemporal offsets, 3) final restoration.

In multi-frame fusion, frame features are aggregated with adjacent features to obtain temporally fused features. We adopt a two-dimensional U-Net structure for multi-frame fusion, maintaining skip connections within the U-Net. Instead of multiple 2D convolutional blocks, we replace them with stacked Grouped Spatiotemporal Shift (GSTS) blocks to effectively establish temporal correspondence and perform multi-frame fusion. GSTS blocks are not applied at the finest scale to save computational costs. The GSTS block consists of three parts: 1) temporal displacement, 2) spatial shift, 3) lightweight fusion layer, as illustrated in Fig. 14.

We employed a multi-scale approach that preserves the original high-resolution features through the network hierarchy, thereby minimizing the loss of precise spatial details. Simultaneously, it encodes multi-scale context by using parallel convolution streams to process features at lower spatial resolutions. The multi-resolution parallel branches operate complementarily to the main high-resolution branch, providing more accurate and contextually enriched feature representations.

One major distinction between MIRNetv2 and other multi-scale image processing methods lies in how contextual information is aggregated. While other methods focus on processing each scale separately, MIRNetv2 progressively exchanges and fuses information from coarse to fine resolution levels. Additionally, unlike methods that use simple concatenation or averaging of features from multi-resolution branches, MIRNetv2 introduces a new selective kernel fusion approach. This approach dynamically selects the useful set of kernels from each branch representation using a self-attention mechanism. Moreover, the proposed fusion block combines features with varying receptive fields while preserving their distinctive complementary characteristics.

The MIRNetv2 network is divided into four modules: Dual-Pixel Defocus Deblurring, Image Denoising, Image Super-Resolution, and Image Enhancement. Among them, I employed the Dual-Pixel Defocus Deblurring module for my task.

Training dataset contains a total of 72 scenes. The noisy dataset for each scene contains 10 consecutive images with three different noise levels. The validation set consists of scenes 3 and 4, with images containing noise levels of 125, 160, and 200. The test set includes scenes 16, 22, 36, and 66, each with ten images at noise levels of 100, 125, 160, 200, 250, and 320. Image size is (1300, 2700, 1), formatted as TIFF images arranged in RGGB order. Black level is 512, and white level is 15360. Specific parameter settings for the training process:

With these training strategies, I trained on the training set for 300,000 iterations and achieved good performance.showing in Table 6:

From the perspective of input image types, directly training with raw images yields lower scores, as shown in Fig. 16. This could be due to the resulting test images having blurry details, unclear textures, and a greenish hue. However, when input images are RGB images, the model MIRNetv2 is used for deblurring tasks. When our input images are not normalized, the test results tend to be darker with heavier colors. Then, after normalization, the color changes in the resulting test images are smaller, and the details and textures become clearer and more visible.

During the early stages of training, we also utilized the RViDeNet model, undergoing training in three phases: predenoising, pretraining, and finetuning. However, as training required four different noise levels while each scene in our training set only had three noise levels, our strategy was to directly duplicate the highest noise level from the training set. This led to poor generalization of our model during training, resulting in unsatisfactory performance during the validation phase. Therefore, we did not continue using this model during the testing phase. Nevertheless, we still believe that with a sufficient dataset and training time, this model has the potential to achieve better results.

To solve UHD low-light image enhancement, several recent state-of-the-art methods have been proposed, for example LLFormer [100], UHDFour[57], and MixNet [111]. We first conduct cross-domain generalization analysis on these methods, and we find that MixNet can better generalize to unseen real images. Thus, MixNet [111] is employed as the network backbone for low-light image enhancement.
Fig. 25 shows the overview of the network architecture. It aims to map an UHD low-light input image $x\in\mathbb{R}^{H\times W\times C}$ to its corresponding normal-clear version $y\in\mathbb{R}^{H\times W\times C}$, where $H$, $W$, and $C$ represent height, width, and channel, respectively. To reduce computational complexity, it downsample the input to 1/4 of the original resolution by PixelUnshuffle. Subsequently, the shallow features go through multiple deep feature mixer blocks. Each feature mixer block mainly consists of a feature modulation network and a feed forward network. To better capture long-range pixel dependencies in UHD images, feature modulation network combines spatial and channel dimensions for joint feature modeling. Finally, we use PixelShuffle upsampling to reconstruct the final image.

We conduct model training in PyTorch framework on 8 NVIDIA GeForce RTX 4090 GPUs. Furthermore, we incorporate other public UHD low-light image enhancement datasets (UHD-LL [57] and UHD-LOL [100]) into the network training. Similar to [65], patches at the size of $2000\times 2000$ are randomly cropped from the image pairs as training samples. The training data is augmented with random rotation and flipping. To optimize the network, we adopt L1 loss as the optimization objective, and we employ the Adam optimizer with a learning rate $2\times 10^{-4}$. In total, we perform 600k iterations. During the testing phase, we perform full-resolution inference using one NVIDIA GeForce RTX 4090 GPU. Note that we employ a self-ensemble strategy to further improve performance. The code and model are released at code.

Fig. 26 illustrates the overall architecture of our method. Specifically, the input $x$ is first reshaped to feature tensor via PixelUnshuffle ($4\times\downarrow$) to preserve original information, and then fed to 8 feature extraction modules. Finally, the output feature $y$ is reshaped to the original height and width of input $x$ via Pixelshuffle ($4\times\uparrow$). The feature extraction module mainly contains a feature rearrangement block (FRB), a feature enhancement block (FEB), and a feed-forward network (FFN). Here, FRB adopts MLP-based tensor dimensional transformations [131], while FEB employs CNN-based local operators [123]. The overall process can be represented as follows:

where $F_{0}$ denote the input features, $F_{1}$ denote the intermediate features and $F_{2}$ denote the output features. LN refers to the layer normalization.

To supervise the training process, we employ the L1 loss as the objective function. We conduct model training on 4 NVIDIA TESLA V100s with 32GB memory. In total, we perform 500 epochs of training. During the training, we adopt the Adam optimizer with a learning rate of $2\times 10^{-4}$. The patch size is set to be $768\times 768$ pixels and the batch size is set to be 16. To augment the training data, we apply random horizontal and vertical flips. For testing images, we use one NVIDIA GeForce RTX 4090 GPU with 24GB memory. The source code and pre-trained model are available at model.

As shown in Fig. 28(b), we designed the SimPF block with the idea of retaining the necessary convolution and normalisation layers and using less computationally intensive components to reduce multiplication operations across feature maps. We use $1\times 1$ convolutions and $3\times 3$ depth-wise separable convolutions for feature extraction, both convolution types have been applied and proven effective in a variety of image restoration tasks. Specifically, the feature map first undergoes a $1\times 1$ convolution to expand the number of channels while preserving spatial information. Subsequently, a $3\times 3$ depth-wise separable convolution is employed to encode features from spatially adjacent pixel positions, facilitating the learning of local image structures.

Our method is implemented using the PyTorch framework. The diffusion time step $T$ is established at $100$. A cosine scheduling scheme is utilized for noise scheduling. The optimization is carried out using the LION optimizer. The batch size is set to $6$. The initial learning rate is set at $4\times 10^{-5}$, and the Cosine Annealing strategy is employed for learning rate scheduling. The model is trained on a single NVIDIA GeForce RTX 3090 GPU and converged after 300,000 iterations.
During the training phase, we first attempted to crop or randomly crop the center of the training set to 256 x 256, but did not achieve good results. Finally, we resized the training set to 256 x 256 and achieved good results. During testing, due to the large size of 6720 x 4480, which exceeded the maximum range that the model could handle, we first attempted to crop the image into 2240 x 2240 and merge it, but the effect was not good. Finally, we resized the image to 480 x 320, and after using model enhancement, we resized it to 6720 x 4480, achieving good results. In addition, due to the unknown GT, we used the lpips metric to preliminarily evaluate the enhancement results of the model. We found that our proposed SimPF block performed better on the three images 1162, 496, 735, while the model trained on the original NAF block performed better on the other test images. Therefore, we combined the results of the two models to obtain the final version for submission.

To systematically study the generality of the proposed noise formation model, we collect an extreme low-light denoising (ELD) dataset [108] that covers 10 indoor scenes and 4 camera devices from multiple brands (SonyA7S2, NikonD850, CanonEOS70D, CanonEOS700D). We also record bias and flat field frames for each camera to calibrate our noise model. The data capture setup is shown in Fig. 29 For each scene and each camera, a reference image at the base ISO was firstly taken, followed by noisy images whose exposure time was deliberately decreased by low light factors f to simulate extreme low light conditions. Another reference image then was taken akin to the first one, to ensure no accidental error (e.g. drastic illumination change or accidental camera/scene motion) occurred. We choose three ISO levels (800, 1600, 3200) and two low light factors (100, 200) for noisy images to capture our dataset, resulting in 240 (3×2×10×4) raw image pairs in total. The hardest example in our dataset resembles the image captured at a “pseudo” ISO up to 640000 (3200×200).

The quantitative evaluation metrics include the standard Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), which are commonly used to assess image quality. To ensure fairness and accuracy in testing, we use the Codalab platform (https://codalab.lisn.upsaclay.fr/competitions/17787) for result evaluation. Due to the storage limitations of the Codalab platform, we have cropped the original raw images to a size of 1024×1024 and saved the cropped images along with relevant camera parameters in mat files provided to participants. During testing, participants are also required to save the denoised images in mat files for submission. The Codalab platform will then compute the evaluation metrics based on the ground truth. The final competition score is :

where k=1.2.

As shown in Tab. 8, in the extremely low-light detection track, jly724215288 achieved first place with a score of 43.80, a PSNR of 43.89, and an SSIM of 0.99, demonstrating their exceptional denoising capabilities for extremely low-light images. Yuxiaoxi secured second place with a score of 43.07, a PSNR of 43.15, and an SSIM of 0.99. Both teams performed excellently in low-light instance segmentation, further highlighting the significance of their contributions.

These results highlight the remarkable progress made by the participating teams in addressing the challenges of noise in extremely low-light images. The top-ranked teams showcased their expertise and innovation in developing robust algorithms adapted to low-light conditions, paving the way for future advancements in computer vision research.

The authors’ method first employs accurate patch-based registration while the images are captured on a tripod.

The registration is performed using phase-correlation Fig.30, which is robust against strong noise and brightness changes. First, the images are transformed into the frequency domain using Fast Fourier Transform, then the cross-power spectrum is calculated by taking the complex conjugate multiplication with element-wise normalization. $\mathbf{G}_{a}=\mathcal{F}\{I_{a}\},\;\mathbf{G}_{b}=\mathcal{F}\{I_{b}\}$

Then they find the maximum response phase as the image patch offset $(\Delta x,\Delta y)$ in the inverse Fast-Fourier transform result.

The authors acknowledge that due to the settings and exposure time of different brands of sensors, as well as the sensitivity of the expected exposure value (EV) to the final result, they introduce a variable $\lambda$, ranging from 0.1 to 10, to reduce reliance on precise exposure accuracy. The digital gain (ISO) and exposure time in seconds are extracted from EXIF metadata and calculated into the exposure value (EV) for ratio estimation.

The authors carefully perform data augmentation through random size cropping and rotation while maintaining Bayer pixel alignment. They avoid any scale-like resampling augmentation to preserve the sensor noise properties.

The authors first train their model on raw files from various camera brands. Each training image patch pair undergoes extensive augmentation, including cropping, phase alignment with the ground truth, rotation, and scaling by the $\lambda$ ratio.

Following the initial pretraining, the authors fine-tune the model using the specific camera brand intended for testing to improve noise model estimation.

In practice, different sensors exhibit unequal black levels and white points per channel. Another contribution by the authors is the development of a method to normalize raw Bayer data values $I_{n}$ across images taken by different camera brands and exposure settings. When an image is captured, the raw file also records the black levels $L_{b}$ and white points per channel $L_{w}$. After multiplying by the EV ratio (which is approximately 10 times larger than the camera EV change) to achieve normal brightness, the noisy input might be clipped by the sensor’s maximum bit value (usually 14 bits), leading to signal loss. The authors address this problem by normalizing using a large denominator $L_{m}$ (above 16 bits) after subtracting the black levels. Replacing $L_{w}$ with $L_{m}$ helps preserve image detail. Additionally, the variable $\lambda$ in training results in better denoised images while maintaining GPU-capable precision.

The following equation shows a normalized Bayer raw multiply exposure ratio, then $f(.)$ donates our network function, finally de-normalize to the expect denoised bright image $I_{nr}$.

The authors adopt the classic single-stage U-shaped architecture with skip-connections, as shown in Fig. 32, to reduce inter-block complexity. The neural networks are constructed by stacking blocks. They start with a plain block containing the most common components: convolution, ReLU, and shortcut. Additionally, they find that vanilla channel attention meets the requirements for computational efficiency and brings global information to the feature map. Given the proven effectiveness of channel attention in image restoration tasks, the authors incorporate channel attention into the plain block.

The authors use convolution with a kernel size of 2 and a stride of 2 for the downsample layer. For the upsample layer, they double the channel width using a pointwise convolution followed by a pixel shuffle module. There are skip connections from the encoder block to the decoder block, and the authors simply add the encoder and decoder features element-wise for feature fusion. The default width and number of blocks are 64 and 36, respectively, and their network architecture consists of a total of 5 layers. The encoder has 4 layers with block quantities of 2, 2, 4, and 8, respectively. The intermediate connection layer has 12 blocks. The decoder also has 4 layers, with each layer containing 2 blocks.

The authors utilize the $L_{1}$ loss function as the training objective, similar to most denoising methods. They employ the same data preprocessing and optimization strategy as ELD during pre-training. The raw images with long exposure times in the SID train subset are used for noise synthesis. For data preprocessing, they pack the Bayer images into 4 channels, then crop the long exposure data into patches of size 512×512 with a non-overlapping step of 256. The models are trained for 300 epochs using the Adam optimizer with $\beta_{1}$ = 0.9 and $\beta_{2}$ = 0.999, without applying weight decay. The initial learning rate is set to $10^{-4}$, halved at the 150th epoch, and further reduced to $10^{-5}$ at the 220th epoch.

The inference code and the pre-trained models are released at here.

The Miers proposed a multi-scale, light-weight transformer model for low-light raw image enhancement. Unlike previous Retinex-based methods [10, 60, 119] that generally decompose the input image into illumination components and reflection components, the proposed method adaptively aligns brightness-induced differences by introducing a learnable guidance vector in the self-attention mechanism. The network architecture called SANet is shown in Fig. 34. The SANet extracts features at different scales sequentially and performs feature fusion through long connections, which can effectively reduce the calculation. At each scale, the proposed method uses concatenated residual blocks and the SABlock as basic modules to obtain non-local views.

The self-attention mechanism in Transformer structure has been proven to have great advantages in low-level image enhancement, and the proposed SABlock (shown in Fig. 35) is also based on this. The SABlock captures global dependency information by building key-value pairs on feature blocks. In the low-light enhancement task, due to the large differences in input image distribution caused by illumination, the model is not easy to fit for natural state. This team introduced a learnable adaptive vector in SABlock to control the gap between the input RAW and the target. This allows the model to be effectively fitted to the direction that contributes to the correct output.

It is also worth noting that downsampling in SANet is implemented using $4\times 4$ convolution with $stride=2$ and the UP Block consists of a $3\times 3$ depthwise separable convolution and pixelshuffle.

The code is based on BasicSR [104] and EFNet [93]. During the training stage, only the data provided by the competition is used. First, the input image is cropped to 128x128, rotation and flipping are added as data augmentation. It should be noted that due to the large difference in image brightness under different ISOs, the input RAW image subtracts the black level and divides by the difference between the white level and the black level, and then multiplies by the ratio for normalization. The ratio is calculated by

In the training process, the batch size is 4, total iterations is set to 500,000. This team uses L1 loss as the training loss and MultiStepLR for learning rate decay. In addition, the model weight uses exponential moving average (EMA), and the model with the highest PSNR on the validation set is finally selected for testing.

The ISS used the algorithm proposed in the Lighting Every Darkness in Two Pairs: A Calibration-Free Pipeline for RAW Denoising [51], which can adapt to the target camera without calibrating noise parameters and repeated training, requiring only a small amount of lens pairing data and fine-tuning, eliminating the complicated calibration steps, and achieved good performance.

As shown in Fig. 36, the whole network adopts the macro architecture of Unet [83], in which the convolution blocks of the Unet network itself is replaced with the reparameterized noise removal (RepNR) block [51]. In the Pre-train stage, In RepNR Block has k branches of Camera-Specific Alignment (CSA) module [51], Each of these branches is fitted to a class of camera noise, In the Fine-tune phase, By averaging the k CSA module, Equivalent to the model integration of noise from multiple classes of cameras, At this time the RepNR block consists of two branches, Where the upper 3x3 convolution is designed to fit the out-of-model noise, Lower Camera-Specific Alignment (CSA) module, The main role is to adjust the distribution of the input features.

By utilizing LED methods [51], we ultimately reduced the number of scenes to just four groups, randomly selecting three distinct images for each scene to pair with their respective ground truth images. These image pairs were rapidly deployed to a new camera using the provided pre-trained weights.

Subsequently, fine-tuning was conducted using a small amount of real data, with the RepNR block replacing the convolutional layers in the UNet architecture. During the fine-tuning process, we initially iterated the CSA from the pre-trained model for 5000 iterations until convergence. The optimizer used was Adam with a learning rate of 0.0001, and the training strategy employed a cosine annealing approach. An additional branch was then fine-tuned for an additional 3000 iterations, with the optimizer and training strategy consistent with the main branch. The loss function chosen was L1 loss.

We introduces MHDRUNet, a model that utilizes the emphasis on different channel information in Raw images for exposure guidance, subsequently used for HDR reconstruction of single-frame Raw images. The model framework is shown in Fig. 38. Inspired by RawHDR[136], Raw images have higher intensity values in green channels compared to red and blue. Therefore, we split $I_{RGBG}$ into $I_{RB}$ and $I_{G}$, using the RB channels for exposure estimation to derive an underexposure mask $M_{under}$, and then reconstruct the underexposed areas based on the G channel to get $Y_{G}$. Similarly, we use the G channel for exposure estimation to obtain an overexposure mask $M_{over}$, and then reconstruct the overexposed regions using the RB channels to get $Y_{RB}$. We combine $Y_{G}$ and $Y_{RB}$ through a weighted sum. To ensure smoothness in HDR reconstruction across the global range, we utilize the original Raw data for global exposure-guided reconstruction. The exposure reconstruction network is comprised of the complete HDRUNet[19], with inputs including the Raw image and a condition image, which by default matches the Raw image. The exposure estimation mask module consists of a CNN with residual connections, and the exposure reconstruction is carried out by the complete HDRUNet network. Secondly, we propose the method of Refine Exposure Adjustment. By analyzing the distribution of values in the input Raw image, we can estimate the areas of exposure and underexposure. For images where the area of the underexposure mask is greater than that of the overexposure mask, we consider it to be underexposed; conversely, if larger for the overexposure mask, it is considered overexposed. Based on this, we make appropriate exposure adjustments on the original Raw data, bringing underexposed images to a slightly underexposed state and overexposed images to a slightly overexposed state. These are then used as the condition images inputted into the HDRUNet network, thereby achieving better reconstruction results.

We use the dataset proposed by HDR Reconstruction from a Single Raw Image challenge. Before training, we pre-process the data by cropping images into $768\times 768$. During training, the mini-batch size is set to 1 and Adam [52] optimizer and Kaiming-initialization [42] are adopted for training. The initial learning rate is set to $1e-4$ and all models are built on the PyTorch framework and trained with NVIDIA 3090 GPU. It’s noteworthy that we find the HDR reconstruction task for overexposed images to be more challenging compared to that for underexposed images. Therefore, we propose the training strategy of Random Overexposure Adjustment. Specifically, during training, we randomly apply varying degrees of overexposure adjustments to the input underexposed images to generate pseudo-overexposed images for data augmentation, thereby enhancing the model’s robustness. During training, we use tanh L2 loss[19] and SSIM loss [106] to achieve better training effects. Additionally, we employ a constraint loss $L_{mask}$ [136], which guides the learning of the mask.

Overall Pipeline. Inspired by Restormer [124], our network architecture is shown in Fig. 39. It overcomes the limitations of traditional Convolutional Neural Networks (CNNs) by utilizing Transformers’ ability to capture long-range pixel interactions, which is crucial for High Dynamic Range (HDR) image reconstruction. We introduce the two modules of the Transformer block: (a) multi-Dconv head transposed attention (MDTA) and (b) gated-Dconv feed-forward network (GDFN).

Multi-Dconv Head Transposed Attention This module replaces the standard multi-head self-attention mechanism. It operates across feature dimensions instead of spatial dimensions, reducing complexity. It uses $1\times 1$ convolutions for pixel-wise cross-channel context aggregation and $3\times 3$ depth-wise convolutions for channel-wise spatial context encoding.

where $\hat{I}$ is the predicted HDR Raw image, $I$ is the ground truth, and $\epsilon$ is set to 0.0001 as default.

In addition to the content loss, we leverage frequency domain information to introduce auxiliary loss to our network, which is defined as follows:

where $\mathcal{F}\left(\cdot\right)$ indicates the Fast Fourier Transform (FFT). Finally, the total loss could be defined as:

where $\lambda$ denotes the balanced weight, and we empirically set $\lambda$ to 0.5 as default.

DERNet adopts an encoder-decoder network with a recursive design to process dual-stream event voxels to estimate high-speed HDR videos. Specifically, when reconstructing the $t$-th frame of the HDR video, considering that the long-time event stream around frame $t$ can help reconstruct the low-frequency brightness, DERNet voxelizes the event data from frame $t-T_{l}$ to frame $t+T_{l}$ into a $b$-bins event voxel $V_{l,t}$. Simultaneously, considering that the short-time event stream around frame $t$ can help reconstruct high-frequency texture, DERNet voxelizes the event data from frame $t-T_{s}$ to frame $t+T_{s}$ into a $b$-bins event voxel $V_{s,t}$. Subsequently, the event voxels $V_{l,t}$ and $V_{s,t}$ are concatenated and input into the network. To fuse the features of the two event voxels, DERNet utilizes convolutional layers to generate fused features from the event voxels. The network then adopts a two-branch encoder. This structure includes a complex branch that extracts high-level semantic information from the fused features, leveraging Swin Transformer [67] blocks to capture spatial context and Conv-GRU blocks to capture temporal context by integrating historical states. It also includes a simple branch that utilizes convolutional layers to capture detailed information from the fused features. Next, the decoder of DERNet adopts multiple Swin Transformer blocks to fuse and upsample the features extracted by the two-branch encoder, finally using convolutional networks to predict the $t$-th frame HDR image $I_{t}$.

To train DERNet, a reconstruction loss $\mathcal{L}_{r}$ is designed for the estimated HDR image $\mathcal{L}_{t}$ as

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, and $\lambda_{4}$ coefficients balancing the loss terms, ${\rm L_{1}}(\cdot,\cdot)$ is the absolute loss function, ${\rm L_{2}}(\cdot,\cdot)$ is the mean squared error loss function, $I_{t}^{gt}$ is the ground truth $t$-th frame HDR image, and ${\rm M}(\cdot)$ is the HDR to SDR function defined as ${\rm M}(x)=\frac{\log(1+5000x)}{\log(5001)}$.

DERNet is implemented using PyTorch. During training, a batch size of 2 is utilized, with a video sequence length of 10 and a data size of $224\times 224$. An AdamW optimizer [68] is adopted with a learning rate of $4\times 10^{-5}$ and weight decay of $10^{-6}$ to optimize the network weights for 60 epochs. A cosine annealing scheduler is adopted to decay the learning rate. To prevent overfitting, random flipping, rotation, and cropping are applied to the event voxels for data augmentation. The coefficients are defined as $b=6$, $T_{l}=16$, $T_{s}=5$, $\lambda_{1}=1$, $\lambda_{2}=0.1$,$\lambda_{3}=500$, and $\lambda_{4}=10$.

Our method employs a convolutional recurrent neural network designed to reconstruct HDR videos from event streams [134]. As shown in Figure 42, the network processes $T=2N+1$ consecutive event voxel grids $\{\mathbf{E}_{t-N},\ldots,\mathbf{E}_{t+N}\}$ to generate the HDR frame $\mathbf{H}_{t}$ at timestamp $t$. The architecture includes several key modules.

Firstly, the shared feature extractor downsamples event frames to a low spatial resolution feature space using strided convolution layers, producing $2N+1$ output feature maps, $\{\mathbf{F}_{t-N},\ldots,\mathbf{F}_{t+N}\}$. This shared encoding facilitates subsequent alignment by transforming the input data into a consistent feature space.

We then employ a deformable convolution-based alignment module [102], which uses pyramidal deformable convolutions to align features of different event frames with the central frame feature $\mathbf{F}_{t}$. This approach predicts offsets for the convolution kernels through a pyramidal processing structure, allowing the network to handle larger movements and align features accurately, thereby avoiding the pitfalls of inaccurate optical flow estimation.

The aligned features are combined in the attentive fusion and reconstruction module. Here, the features are stacked and processed by attention mechanisms that independently focus on height, width, and temporal/channel correlations. The fused features are passed through a recurrent residual network and a ConvLSTM [90] module, which help maintain temporal continuity by remembering information from successive sequences.

To enhance temporal consistency, we introduce a novel temporal consistency loss based on the integral relationship between consecutive frames and events, modeled using a pre-trained UNet-like [84] network. This loss ensures smooth transitions between frames, mitigating issues related to temporal discontinuity.

The training strategy involves a combination of losses to optimize HDR video reconstruction and maintain temporal consistency. Given the reconstructed video sequence $\mathbf{H}_{i}$ and the corresponding ground truth frames $\hat{\mathbf{H}}_{i}$, we employ several loss functions.

We use the $l_{1}$ loss to measure the pixel-wise difference between the reconstructed and ground truth frames:

To enhance perceptual quality, we introduce the Learned Perceptual Image Patch Similarity (LPIPS) loss [130], which focuses on high-level and structural similarity. Additionally, the temporal consistency loss $\mathcal{L}_{C}$, derived from the pre-trained network, is defined as:

This loss ensures that the reconstructed frames maintain smooth transitions.

The overall loss function used to train our model is:

where $\tau_{1}$ and $\tau_{2}$ are empirically set to $2$ and $0.2$, respectively.

The network is initialized using Kaiming initialization [42], and trained with the Adam optimizer (momentum set to 0.9). The initial learning rate is $10^{-4}$, reduced by a factor of 10 every 50 epochs. We set the batch size to 4, and train the model for 100 epochs. The implementation uses the PyTorch framework, and training is performed on NVIDIA TITAN V GPUs.

In summary, our network architecture and training strategy effectively reconstruct high-quality HDR videos from event data, ensuring both spatial and temporal coherence.

As shown in the figure 43, we introduce three types of inputs at the input end. The first part is data generated from the voxel structure of raw event camera data, with the time step of 8. This part of the data represents the most primitive detail information of the image. The second part is 2D image data generated by E2VID [81], which serves as the reference frame for LDR 2D images. The third part is HDR image data generated by the decoder, representing the HDR data of historical frames, which can help smooth the entire video and ultimately generate HDR image data for the current frame.

According to the aforementioned network architecture, this method is trained using the following approach. Firstly, the training data is converted into voxels based on the original timestamps and stored in the npy format. For the purpose of facilitating temporal training, each npy file is stored with a length of T=16. Subsequently, e2vid is employed to generate reference frames corresponding to each npy file. It is noteworthy that the reference frames generated by E2VID are often affected by the actual data distribution density, leading to occurrences of blank spaces or excessive noise.

Regarding the network itself, data augmentation techniques such as random horizontal flipping and the addition of Gaussian noise with a standard deviation of 0.001 are applied to the input data. Additionally, to better align with the evaluation metrics of this challenge, four types of noise are introduced, including KL divergence noise (to ensure alignment between HDR ground truth images and generated images), L1 noise, L2 noise, and SSIM noise. Due to time constraints, rigorous ablation experiments were not conducted. However, from a holistic analysis of the results, KL divergence noise yielded relatively favorable gains.

Other training parameters include the Adam optimizer with a learning rate of 0.001, cosine annealing for learning rate scheduling, a batch size of 48, and iterative training conducted using four NVIDIA 4090 GPUs. Training is performed for 1000 epochs, with the overall training and testing dataset split in an 8:2 ratio. To address challenging samples, this method manually removes data with poor distributions (some data lack original event information due to bandwidth congestion, rendering them unsuitable for training). Finally, inference can be performed, followed by truncating and normalizing the image with a maximum value of 65535, resulting in the HDR image of the current moment in the corresponding distribution domain.