-
Burst Super-Resolution with Diffusion Models for Improving Perceptual Quality
Authors:
Kyotaro Tokoro,
Kazutoshi Akita,
Norimichi Ukita
Abstract:
While burst LR images are useful for improving the SR image quality compared with a single LR image, prior SR networks accepting the burst LR images are trained in a deterministic manner, which is known to produce a blurry SR image. In addition, it is difficult to perfectly align the burst LR images, making the SR image more blurry. Since such blurry images are perceptually degraded, we aim to rec…
▽ More
While burst LR images are useful for improving the SR image quality compared with a single LR image, prior SR networks accepting the burst LR images are trained in a deterministic manner, which is known to produce a blurry SR image. In addition, it is difficult to perfectly align the burst LR images, making the SR image more blurry. Since such blurry images are perceptually degraded, we aim to reconstruct the sharp high-fidelity boundaries. Such high-fidelity images can be reconstructed by diffusion models. However, prior SR methods using the diffusion model are not properly optimized for the burst SR task. Specifically, the reverse process starting from a random sample is not optimized for image enhancement and restoration methods, including burst SR. In our proposed method, on the other hand, burst LR features are used to reconstruct the initial burst SR image that is fed into an intermediate step in the diffusion model. This reverse process from the intermediate step 1) skips diffusion steps for reconstructing the global structure of the image and 2) focuses on steps for refining detailed textures. Our experimental results demonstrate that our method can improve the scores of the perceptual quality metrics. Code: https://github.com/placerkyo/BSRD
△ Less
Submitted 8 April, 2024; v1 submitted 28 March, 2024;
originally announced March 2024.
-
Kernelized Back-Projection Networks for Blind Super Resolution
Authors:
Tomoki Yoshida,
Yuki Kondo,
Takahiro Maeda,
Kazutoshi Akita,
Norimichi Ukita
Abstract:
Since non-blind Super Resolution (SR) fails to super-resolve Low-Resolution (LR) images degraded by arbitrary degradations, SR with the degradation model is required. However, this paper reveals that non-blind SR that is trained simply with various blur kernels exhibits comparable performance as those with the degradation model for blind SR. This result motivates us to revisit high-performance non…
▽ More
Since non-blind Super Resolution (SR) fails to super-resolve Low-Resolution (LR) images degraded by arbitrary degradations, SR with the degradation model is required. However, this paper reveals that non-blind SR that is trained simply with various blur kernels exhibits comparable performance as those with the degradation model for blind SR. This result motivates us to revisit high-performance non-blind SR and extend it to blind SR with blur kernels. This paper proposes two SR networks by integrating kernel estimation and SR branches in an iterative end-to-end manner. In the first model, which is called the Kernel Conditioned Back-Projection Network (KCBPN), the low-dimensional kernel representations are estimated for conditioning the SR branch. In our second model, the Kernelized BackProjection Network (KBPN), a raw kernel is estimated and directly employed for modeling the image degradation. The estimated kernel is employed not only for back-propagating its residual but also for forward-propagating the residual to iterative stages. This forward-propagation encourages these stages to learn a variety of different features in different stages by focusing on pixels with large residuals in each stage. Experimental results validate the effectiveness of our proposed networks for kernel estimation and SR. We will release the code for this work.
△ Less
Submitted 27 October, 2023; v1 submitted 16 February, 2023;
originally announced February 2023.
-
Composing a surrogate observation operator for sequential data assimilation
Authors:
Kosuke Akita,
Yuto Miyatake,
Daisuke Furihata
Abstract:
In data assimilation, state estimation is not straightforward when the observation operator is unknown. This study proposes a method for composing a surrogate operator when the true operator is unknown. A neural network is used to improve the surrogate model iteratively to decrease the difference between the observations and the results of the surrogate model. A twin experiment suggests that the p…
▽ More
In data assimilation, state estimation is not straightforward when the observation operator is unknown. This study proposes a method for composing a surrogate operator when the true operator is unknown. A neural network is used to improve the surrogate model iteratively to decrease the difference between the observations and the results of the surrogate model. A twin experiment suggests that the proposed method outperforms approaches that tentatively use a specific operator throughout the data assimilation process.
△ Less
Submitted 1 June, 2022; v1 submitted 29 January, 2022;
originally announced January 2022.
-
NTIRE 2021 Challenge on Burst Super-Resolution: Methods and Results
Authors:
Goutam Bhat,
Martin Danelljan,
Radu Timofte,
Kazutoshi Akita,
Wooyeong Cho,
Haoqiang Fan,
Lanpeng Jia,
Daeshik Kim,
Bruno Lecouat,
Youwei Li,
Shuaicheng Liu,
Ziluan Liu,
Ziwei Luo,
Takahiro Maeda,
Julien Mairal,
Christian Micheloni,
Xuan Mo,
Takeru Oba,
Pavel Ostyakov,
Jean Ponce,
Sanghyeok Son,
Jian Sun,
Norimichi Ukita,
Rao Muhammad Umer,
Youliang Yan
, et al. (3 additional authors not shown)
Abstract:
This paper reviews the NTIRE2021 challenge on burst super-resolution. Given a RAW noisy burst as input, the task in the challenge was to generate a clean RGB image with 4 times higher resolution. The challenge contained two tracks; Track 1 evaluating on synthetically generated data, and Track 2 using real-world bursts from mobile camera. In the final testing phase, 6 teams submitted results using…
▽ More
This paper reviews the NTIRE2021 challenge on burst super-resolution. Given a RAW noisy burst as input, the task in the challenge was to generate a clean RGB image with 4 times higher resolution. The challenge contained two tracks; Track 1 evaluating on synthetically generated data, and Track 2 using real-world bursts from mobile camera. In the final testing phase, 6 teams submitted results using a diverse set of solutions. The top-performing methods set a new state-of-the-art for the burst super-resolution task.
△ Less
Submitted 7 June, 2021;
originally announced June 2021.
-
AIM 2020 Challenge on Video Extreme Super-Resolution: Methods and Results
Authors:
Dario Fuoli,
Zhiwu Huang,
Shuhang Gu,
Radu Timofte,
Arnau Raventos,
Aryan Esfandiari,
Salah Karout,
Xuan Xu,
Xin Li,
Xin Xiong,
Jinge Wang,
Pablo Navarrete Michelini,
Wenhao Zhang,
Dongyang Zhang,
Hanwei Zhu,
Dan Xia,
Haoyu Chen,
Jinjin Gu,
Zhi Zhang,
Tongtong Zhao,
Shanshan Zhao,
Kazutoshi Akita,
Norimichi Ukita,
Hrishikesh P S,
Densen Puthussery
, et al. (1 additional authors not shown)
Abstract:
This paper reviews the video extreme super-resolution challenge associated with the AIM 2020 workshop at ECCV 2020. Common scaling factors for learned video super-resolution (VSR) do not go beyond factor 4. Missing information can be restored well in this region, especially in HR videos, where the high-frequency content mostly consists of texture details. The task in this challenge is to upscale v…
▽ More
This paper reviews the video extreme super-resolution challenge associated with the AIM 2020 workshop at ECCV 2020. Common scaling factors for learned video super-resolution (VSR) do not go beyond factor 4. Missing information can be restored well in this region, especially in HR videos, where the high-frequency content mostly consists of texture details. The task in this challenge is to upscale videos with an extreme factor of 16, which results in more serious degradations that also affect the structural integrity of the videos. A single pixel in the low-resolution (LR) domain corresponds to 256 pixels in the high-resolution (HR) domain. Due to this massive information loss, it is hard to accurately restore the missing information. Track 1 is set up to gauge the state-of-the-art for such a demanding task, where fidelity to the ground truth is measured by PSNR and SSIM. Perceptually higher quality can be achieved in trade-off for fidelity by generating plausible high-frequency content. Track 2 therefore aims at generating visually pleasing results, which are ranked according to human perception, evaluated by a user study. In contrast to single image super-resolution (SISR), VSR can benefit from additional information in the temporal domain. However, this also imposes an additional requirement, as the generated frames need to be consistent along time.
△ Less
Submitted 14 September, 2020;
originally announced September 2020.
-
Image Super-Resolution using Explicit Perceptual Loss
Authors:
Tomoki Yoshida,
Kazutoshi Akita,
Muhammad Haris,
Norimichi Ukita
Abstract:
This paper proposes an explicit way to optimize the super-resolution network for generating visually pleasing images. The previous approaches use several loss functions which is hard to interpret and has the implicit relationships to improve the perceptual score. We show how to exploit the machine learning based model which is directly trained to provide the perceptual score on generated images. I…
▽ More
This paper proposes an explicit way to optimize the super-resolution network for generating visually pleasing images. The previous approaches use several loss functions which is hard to interpret and has the implicit relationships to improve the perceptual score. We show how to exploit the machine learning based model which is directly trained to provide the perceptual score on generated images. It is believed that these models can be used to optimizes the super-resolution network which is easier to interpret. We further analyze the characteristic of the existing loss and our proposed explicit perceptual loss for better interpretation. The experimental results show the explicit approach has a higher perceptual score than other approaches. Finally, we demonstrate the relation of explicit perceptual loss and visually pleasing images using subjective evaluation.
△ Less
Submitted 1 September, 2020;
originally announced September 2020.
-
NTIRE 2020 Challenge on Perceptual Extreme Super-Resolution: Methods and Results
Authors:
Kai Zhang,
Shuhang Gu,
Radu Timofte,
Taizhang Shang,
Qiuju Dai,
Shengchen Zhu,
Tong Yang,
Yandong Guo,
Younghyun Jo,
Sejong Yang,
Seon Joo Kim,
Lin Zha,
Jiande Jiang,
Xinbo Gao,
Wen Lu,
Jing Liu,
Kwangjin Yoon,
Taegyun Jeon,
Kazutoshi Akita,
Takeru Ooba,
Norimichi Ukita,
Zhipeng Luo,
Yuehan Yao,
Zhenyu Xu,
Dongliang He
, et al. (38 additional authors not shown)
Abstract:
This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor 16 based on a set of prior examples of low and corresponding high resolution images. The goal is to obtain a network design capable to produce high resolution results with the best percept…
▽ More
This paper reviews the NTIRE 2020 challenge on perceptual extreme super-resolution with focus on proposed solutions and results. The challenge task was to super-resolve an input image with a magnification factor 16 based on a set of prior examples of low and corresponding high resolution images. The goal is to obtain a network design capable to produce high resolution results with the best perceptual quality and similar to the ground truth. The track had 280 registered participants, and 19 teams submitted the final results. They gauge the state-of-the-art in single image super-resolution.
△ Less
Submitted 3 May, 2020;
originally announced May 2020.
