-
TrackGo: A Flexible and Efficient Method for Controllable Video Generation
Authors:
Haitao Zhou,
Chuang Wang,
Rui Nie,
Jinxiao Lin,
Dongdong Yu,
Qian Yu,
Changhu Wang
Abstract:
Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video gene…
▽ More
Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo can be found at: https://zhtjtcz.github.io/TrackGo-Page/
△ Less
Submitted 21 August, 2024;
originally announced August 2024.
-
Toward Real World Stereo Image Super-Resolution via Hybrid Degradation Model and Discriminator for Implied Stereo Image Information
Authors:
Yuanbo Zhou,
Yuyang Xue,
Jiang Bi,
Wenlin He,
Xinlin Zhang,
Jiajun Zhang,
Wei Deng,
Ruofeng Nie,
Junlin Lan,
Qinquan Gao,
Tong Tong
Abstract:
Real-world stereo image super-resolution has a significant influence on enhancing the performance of computer vision systems. Although existing methods for single-image super-resolution can be applied to improve stereo images, these methods often introduce notable modifications to the inherent disparity, resulting in a loss in the consistency of disparity between the original and the enhanced ster…
▽ More
Real-world stereo image super-resolution has a significant influence on enhancing the performance of computer vision systems. Although existing methods for single-image super-resolution can be applied to improve stereo images, these methods often introduce notable modifications to the inherent disparity, resulting in a loss in the consistency of disparity between the original and the enhanced stereo images. To overcome this limitation, this paper proposes a novel approach that integrates a implicit stereo information discriminator and a hybrid degradation model. This combination ensures effective enhancement while preserving disparity consistency. The proposed method bridges the gap between the complex degradations in real-world stereo domain and the simpler degradations in real-world single-image super-resolution domain. Our results demonstrate impressive performance on synthetic and real datasets, enhancing visual perception while maintaining disparity consistency. The complete code is available at the following \href{https://github.com/fzuzyb/SCGLANet}{link}.
△ Less
Submitted 13 December, 2023;
originally announced December 2023.
-
LP-BFGS attack: An adversarial attack based on the Hessian with limited pixels
Authors:
Jiebao Zhang,
Wenhua Qian,
Rencan Nie,
Jinde Cao,
Dan Xu
Abstract:
Deep neural networks are vulnerable to adversarial attacks. Most $L_{0}$-norm based white-box attacks craft perturbations by the gradient of models to the input. Since the computation cost and memory limitation of calculating the Hessian matrix, the application of Hessian or approximate Hessian in white-box attacks is gradually shelved. In this work, we note that the sparsity requirement on pertur…
▽ More
Deep neural networks are vulnerable to adversarial attacks. Most $L_{0}$-norm based white-box attacks craft perturbations by the gradient of models to the input. Since the computation cost and memory limitation of calculating the Hessian matrix, the application of Hessian or approximate Hessian in white-box attacks is gradually shelved. In this work, we note that the sparsity requirement on perturbations naturally lends itself to the usage of Hessian information. We study the attack performance and computation cost of the attack method based on the Hessian with a limited number of perturbation pixels. Specifically, we propose the Limited Pixel BFGS (LP-BFGS) attack method by incorporating the perturbation pixel selection strategy and the BFGS algorithm. Pixels with top-k attribution scores calculated by the Integrated Gradient method are regarded as optimization variables of the LP-BFGS attack. Experimental results across different networks and datasets demonstrate that our approach has comparable attack ability with reasonable computation in different numbers of perturbation pixels compared with existing solutions.
△ Less
Submitted 6 April, 2023; v1 submitted 25 October, 2022;
originally announced October 2022.
-
Exploring Adversarial Examples and Adversarial Robustness of Convolutional Neural Networks by Mutual Information
Authors:
Jiebao Zhang,
Wenhua Qian,
Rencan Nie,
Jinde Cao,
Dan Xu
Abstract:
A counter-intuitive property of convolutional neural networks (CNNs) is their inherent susceptibility to adversarial examples, which severely hinders the application of CNNs in security-critical fields. Adversarial examples are similar to original examples but contain malicious perturbations. Adversarial training is a simple and effective defense method to improve the robustness of CNNs to adversa…
▽ More
A counter-intuitive property of convolutional neural networks (CNNs) is their inherent susceptibility to adversarial examples, which severely hinders the application of CNNs in security-critical fields. Adversarial examples are similar to original examples but contain malicious perturbations. Adversarial training is a simple and effective defense method to improve the robustness of CNNs to adversarial examples. The mechanisms behind adversarial examples and adversarial training are worth exploring. Therefore, this work investigates similarities and differences between normally trained CNNs (NT-CNNs) and adversarially trained CNNs (AT-CNNs) in information extraction from the mutual information perspective. We show that 1) whether NT-CNNs or AT-CNNs, for original and adversarial examples, the trends towards mutual information are almost similar throughout training; 2) compared with normal training, adversarial training is more difficult and the amount of information that AT-CNNs extract from the input is less; 3) the CNNs trained with different methods have different preferences for certain types of information; NT-CNNs tend to extract texture-based information from the input, while AT-CNNs prefer to shape-based information. The reason why adversarial examples mislead CNNs may be that they contain more texture-based information about other classes. Furthermore, we also analyze the mutual information estimators used in this work and find that they outline the geometric properties of the middle layer's output.
△ Less
Submitted 26 July, 2022; v1 submitted 12 July, 2022;
originally announced July 2022.
-
UFA-FUSE: A novel deep supervised and hybrid model for multi-focus image fusion
Authors:
Yongsheng Zang,
Dongming Zhou,
Changcheng Wang,
Rencan Nie,
Yanbu Guo
Abstract:
Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion n…
▽ More
Traditional and deep learning-based fusion methods generated the intermediate decision map to obtain the fusion image through a series of post-processing procedures. However, the fusion results generated by these methods are easy to lose some source image details or results in artifacts. Inspired by the image reconstruction techniques based on deep learning, we propose a multi-focus image fusion network framework without any post-processing to solve these problems in the end-to-end and supervised learning way. To sufficiently train the fusion model, we have generated a large-scale multi-focus image dataset with ground-truth fusion images. What's more, to obtain a more informative fusion image, we further designed a novel fusion strategy based on unity fusion attention, which is composed of a channel attention module and a spatial attention module. Specifically, the proposed fusion approach mainly comprises three key components: feature extraction, feature fusion and image reconstruction. We firstly utilize seven convolutional blocks to extract the image features from source images. Then, the extracted convolutional features are fused by the proposed fusion strategy in the feature fusion layer. Finally, the fused image features are reconstructed by four convolutional blocks. Experimental results demonstrate that the proposed approach for multi-focus image fusion achieves remarkable fusion performance compared to 19 state-of-the-art fusion methods.
△ Less
Submitted 20 April, 2021; v1 submitted 12 January, 2021;
originally announced January 2021.
-
Impact and Calibration of Nonlinear Reciprocity Mismatch in Massive MIMO
Authors:
Rongjiang Nie,
Li Chen,
Nan Zhao,
Yunfei Chen,
Weidong Wang,
Xianbin Wang
Abstract:
Time-division-duplexing massive multiple-input multiple-output (MIMO) systems estimate the channel state information (CSI) by leveraging the uplink-downlink channel reciprocity, which is no longer valid when the mismatch arises from the asymmetric uplink and downlink radio frequency (RF) chains. Existing works treat the reciprocity mismatch as constant for simplicity. However, the practical RF cha…
▽ More
Time-division-duplexing massive multiple-input multiple-output (MIMO) systems estimate the channel state information (CSI) by leveraging the uplink-downlink channel reciprocity, which is no longer valid when the mismatch arises from the asymmetric uplink and downlink radio frequency (RF) chains. Existing works treat the reciprocity mismatch as constant for simplicity. However, the practical RF chain consists of nonlinear components, which leads to nonlinear reciprocity mismatch. In this work, we examine the impact and the calibration approach of the nonlinear reciprocity mismatch in massive MIMO systems. To evaluate the impact of the nonlinear mismatch, we first derive the closed-form expression of the ergodic achievable rate. Then, we analyze the performance loss caused by the nonlinear mismatch to show that the impact of the mismatch at the base station (BS) side is much larger than that at the user equipment side. Therefore, we propose a calibration method for the BS. During the calibration, polynomial function is applied to approximate the nonlinear mismatch factor, and over-the-air training is employed to estimate the polynomial coefficients. After that, the calibration coefficients are computed by maximizing the downlink achievable rate. Simulation results are presented to verify the analytical results and to show the performance of the proposed calibration approach.
△ Less
Submitted 30 September, 2020;
originally announced September 2020.
-
Application of Structural Similarity Analysis of Visually Salient Areas and Hierarchical Clustering in the Screening of Similar Wireless Capsule Endoscopic Images
Authors:
Rui Nie,
Huan Yang,
Hejuan Peng,
Wenbin Luo,
Weiya Fan,
Jie Zhang,
Jing Liao,
Fang Huang,
Yufeng Xiao
Abstract:
Small intestinal capsule endoscopy is the mainstream method for inspecting small intestinal lesions,but a single small intestinal capsule endoscopy will produce 60,000 - 120,000 images, the majority of which are similar and have no diagnostic value. It takes 2 - 3 hours for doctors to identify lesions from these images. This is time-consuming and increase the probability of misdiagnosis and missed…
▽ More
Small intestinal capsule endoscopy is the mainstream method for inspecting small intestinal lesions,but a single small intestinal capsule endoscopy will produce 60,000 - 120,000 images, the majority of which are similar and have no diagnostic value. It takes 2 - 3 hours for doctors to identify lesions from these images. This is time-consuming and increase the probability of misdiagnosis and missed diagnosis since doctors are likely to experience visual fatigue while focusing on a large number of similar images for an extended period of time.In order to solve these problems, we proposed a similar wireless capsule endoscope (WCE) image screening method based on structural similarity analysis and the hierarchical clustering of visually salient sub-image blocks. The similarity clustering of images was automatically identified by hierarchical clustering based on the hue,saturation,value (HSV) spatial color characteristics of the images,and the keyframe images were extracted based on the structural similarity of the visually salient sub-image blocks, in order to accurately identify and screen out similar small intestinal capsule endoscopic images. Subsequently, the proposed method was applied to the capsule endoscope imaging workstation. After screening out similar images in the complete data gathered by the Type I OMOM Small Intestinal Capsule Endoscope from 52 cases covering 17 common types of small intestinal lesions, we obtained a lesion recall of 100% and an average similar image reduction ratio of 76%. With similar images screened out, the average play time of the OMOM image workstation was 18 minutes, which greatly reduced the time spent by doctors viewing the images.
△ Less
Submitted 1 April, 2020;
originally announced April 2020.
