Showing 1–50 of 546 results for author: Deng, W

Differentiating Choices via Commonality for Multiple-Choice Question Answering

Authors: Wenqing Deng, Zhe Wang, Kewen Wang, Shirui Pan, Xiaowang Zhang, Zhiyong Feng

Abstract: Multiple-choice question answering (MCQA) becomes particularly challenging when all choices are relevant to the question and are semantically similar. Yet this setting of MCQA can potentially provide valuable clues for choosing the right answer. Existing models often rank each choice separately, overlooking the context provided by other choices. Specifically, they fail to leverage the semantic com… ▽ More Multiple-choice question answering (MCQA) becomes particularly challenging when all choices are relevant to the question and are semantically similar. Yet this setting of MCQA can potentially provide valuable clues for choosing the right answer. Existing models often rank each choice separately, overlooking the context provided by other choices. Specifically, they fail to leverage the semantic commonalities and nuances among the choices for reasoning. In this paper, we propose a novel MCQA model by differentiating choices through identifying and eliminating their commonality, called DCQA. Our model captures token-level attention of each choice to the question, and separates tokens of the question attended to by all the choices (i.e., commonalities) from those by individual choices (i.e., nuances). Using the nuances as refined contexts for the choices, our model can effectively differentiate choices with subtle differences and provide justifications for choosing the correct answer. We conduct comprehensive experiments across five commonly used MCQA benchmarks, demonstrating that DCQA consistently outperforms baseline models. Furthermore, our case study illustrates the effectiveness of the approach in directing the attention of the model to more differentiating features. △ Less

Submitted 21 August, 2024; originally announced August 2024.

Comments: 9 pages, accepted to ECAI 2024

Generalizable Facial Expression Recognition

Authors: Yuhang Zhang, Xiuqi Zheng, Chenyi Liang, Jiani Hu, Weihong Deng

Abstract: SOTA facial expression recognition (FER) methods fail on test sets that have domain gaps with the train set. Recent domain adaptation FER methods need to acquire labeled or unlabeled samples of target domains to fine-tune the FER model, which might be infeasible in real-world deployment. In this paper, we aim to improve the zero-shot generalization ability of FER methods on different unseen test s… ▽ More SOTA facial expression recognition (FER) methods fail on test sets that have domain gaps with the train set. Recent domain adaptation FER methods need to acquire labeled or unlabeled samples of target domains to fine-tune the FER model, which might be infeasible in real-world deployment. In this paper, we aim to improve the zero-shot generalization ability of FER methods on different unseen test sets using only one train set. Inspired by how humans first detect faces and then select expression features, we propose a novel FER pipeline to extract expression-related features from any given face images. Our method is based on the generalizable face features extracted by large models like CLIP. However, it is non-trivial to adapt the general features of CLIP for specific tasks like FER. To preserve the generalization ability of CLIP and the high precision of the FER model, we design a novel approach that learns sigmoid masks based on the fixed CLIP face features to extract expression features. To further improve the generalization ability on unseen test sets, we separate the channels of the learned masked features according to the expression classes to directly generate logits and avoid using the FC layer to reduce overfitting. We also introduce a channel-diverse loss to make the learned masks separated. Extensive experiments on five different FER datasets verify that our method outperforms SOTA FER methods by large margins. Code is available in https://github.com/zyh-uaiaaaa/Generalizable-FER. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: Accepted by ECCV2024

Probing Non-Thermal Gravitinos Through Large-Scale Structure Observations

Authors: Weiyi Deng, Liyuan Guo, Chengcheng Han, Zhanhong Lei

Abstract: We investigate the effects of non-thermally produced dark matter on large-scale structure formation, focusing on the gravitino dark matter. Our analysis shows that large-scale structure measurements from the SDSS LRG data offer a promising approach to probing non-thermally produced gravitinos. Specifically, if gravitinos resulting from neutralino decay constitute a fraction… ▽ More We investigate the effects of non-thermally produced dark matter on large-scale structure formation, focusing on the gravitino dark matter. Our analysis shows that large-scale structure measurements from the SDSS LRG data offer a promising approach to probing non-thermally produced gravitinos. Specifically, if gravitinos resulting from neutralino decay constitute a fraction $F_\text{dec}=0.35~(0.1)$ of the dark matter component, the data exclude bino-like neutralino with a mass $m_{χ_1^0}\lesssim25~(11)\text{ GeV}$. Furthermore, this constraint appears to be largely independent of the gravitino mass. △ Less

Submitted 21 August, 2024; v1 submitted 20 August, 2024; originally announced August 2024.

Comments: 7 pages, 3 figures

DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution

Authors: Yuanbo Zhou, Xinlin Zhang, Wei Deng, Tao Wang, Tao Tan, Qinquan Gao, Tong Tong

Abstract: We introduce DiffSteISR, a pioneering framework for reconstructing real-world stereo images. DiffSteISR utilizes the powerful prior knowledge embedded in pre-trained text-to-image model to efficiently recover the lost texture details in low-resolution stereo images. Specifically, DiffSteISR implements a time-aware stereo cross attention with temperature adapter (TASCATA) to guide the diffusion pro… ▽ More We introduce DiffSteISR, a pioneering framework for reconstructing real-world stereo images. DiffSteISR utilizes the powerful prior knowledge embedded in pre-trained text-to-image model to efficiently recover the lost texture details in low-resolution stereo images. Specifically, DiffSteISR implements a time-aware stereo cross attention with temperature adapter (TASCATA) to guide the diffusion process, ensuring that the generated left and right views exhibit high texture consistency thereby reducing disparity error between the super-resolved images and the ground truth (GT) images. Additionally, a stereo omni attention control network (SOA ControlNet) is proposed to enhance the consistency of super-resolved images with GT images in the pixel, perceptual, and distribution space. Finally, DiffSteISR incorporates a stereo semantic extractor (SSE) to capture unique viewpoint soft semantic information and shared hard tag semantic information, thereby effectively improving the semantic accuracy and consistency of the generated left and right images. Extensive experimental results demonstrate that DiffSteISR accurately reconstructs natural and precise textures from low-resolution stereo images while maintaining a high consistency of semantic and texture between the left and right views. △ Less

Submitted 14 August, 2024; v1 submitted 14 August, 2024; originally announced August 2024.

R-adaptive DeepONet: Learning Solution Operators for PDEs with Discontinuous Solutions Using an R-adaptive Strategy

Authors: Yameng Zhu, Jingrun Chen, Weibing Deng

Abstract: DeepONet has recently been proposed as a representative framework for learning nonlinear mappings between function spaces. However, when it comes to approximating solution operators of partial differential equations (PDEs) with discontinuous solutions, DeepONet poses a foundational approximation lower bound due to its linear reconstruction property. Inspired by the moving mesh (R-adaptive) method,… ▽ More DeepONet has recently been proposed as a representative framework for learning nonlinear mappings between function spaces. However, when it comes to approximating solution operators of partial differential equations (PDEs) with discontinuous solutions, DeepONet poses a foundational approximation lower bound due to its linear reconstruction property. Inspired by the moving mesh (R-adaptive) method, we propose an R-adaptive DeepONet method, which contains the following components: (1) the output data representation is transformed from the physical domain to the computational domain using the equidistribution principle; (2) the maps from input parameters to the solution and the coordinate transformation function over the computational domain are learned using DeepONets separately; (3) the solution over the physical domain is obtained via post-processing methods such as the (linear) interpolation method. Additionally, we introduce a solution-dependent weighting strategy in the training process to reduce the final error. We establish an upper bound for the reconstruction error based on piecewise linear interpolation and show that the introduced R-adaptive DeepONet can reduce this bound. Moreover, for two prototypical PDEs with sharp gradients or discontinuities, we prove that the approximation error decays at a superlinear rate with respect to the trunk basis size, unlike the linear decay observed in vanilla DeepONets. Therefore, the R-adaptive DeepONet overcomes the limitations of DeepONet, and can reduce the approximation error for problems with discontinuous solutions. Numerical experiments on PDEs with discontinuous solutions, including the linear advection equation, the Burgers' equation with low viscosity, and the compressible Euler equations of gas dynamics, are conducted to verify the advantages of the R-adaptive DeepONet over available variants of DeepONet. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Comments: arXiv admin note: text overlap with arXiv:2210.01074 by other authors

A new method to clarify contribution of chiral magnetic effect in small collision system $p^{\uparrow} + A$ involving a transversely polarized proton

Authors: Gui-Zhen Wu, Zong-Wei Zhang, Chen Gao, Yi Xu, Wei-Tian Deng

Abstract: With experimental data of DIS involving transversely polarized proton, we have calculated the 3-D charge density inside the polarized proton, which is found to have a significant non-spherical symmetry. Then we have calculated the property of electromagnetic field (E-M field) generated by a single transversely polarized proton ($p^{\uparrow}$). Based on them, the E-M field generated in small colli… ▽ More With experimental data of DIS involving transversely polarized proton, we have calculated the 3-D charge density inside the polarized proton, which is found to have a significant non-spherical symmetry. Then we have calculated the property of electromagnetic field (E-M field) generated by a single transversely polarized proton ($p^{\uparrow}$). Based on them, the E-M field generated in small collision system $p^{\uparrow}+A$ are studied. We find that the orientation of this E-M field has a significant dependence on the polarization direction of the proton, and the correlator ($Δγ$ ) has also significant dependence on the angle between reaction plane and polarization direction. This finding provides us a new method for probing the chiral magnetic effect (CME). △ Less

Submitted 5 August, 2024; originally announced August 2024.

Comments: 5 pages, 8 figures

Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work

Authors: Wesley Hanwen Deng, Solon Barocas, Jennifer Wortman Vaughan

Abstract: Recent years have witnessed increasing calls for computing researchers to grapple with the societal impacts of their work. Tools such as impact assessments have gained prominence as a method to uncover potential impacts, and a number of publication venues now encourage authors to include an impact statement in their submissions. Despite this push, little is known about the way researchers assess,… ▽ More Recent years have witnessed increasing calls for computing researchers to grapple with the societal impacts of their work. Tools such as impact assessments have gained prominence as a method to uncover potential impacts, and a number of publication venues now encourage authors to include an impact statement in their submissions. Despite this push, little is known about the way researchers assess, articulate, and address the potential negative societal impact of their work -- especially in industry settings, where research outcomes are often quickly integrated into products. In addition, while there are nascent efforts to support researchers in this task, there remains a dearth of empirically-informed tools and processes. Through interviews with 25 industry computing researchers across different companies and research areas, we first identify four key factors that influence how they grapple with (or choose not to grapple with) the societal impact of their research. To develop an effective impact assessment template tailored to industry computing researchers' needs, we conduct an iterative co-design process with these 25 industry researchers and an additional 16 researchers and practitioners with prior experience and expertise in reviewing and developing impact assessments or broad responsible computing practices. Through the co-design process, we develop 10 design considerations to facilitate the effective design, development, and adaptation of an impact assessment template for use in industry research settings and beyond, as well as our own ``Societal Impact Assessment'' template with concrete scaffolds. We explore the effectiveness of this template through a user study with 15 industry research interns, revealing both its strengths and limitations. Finally, we discuss the implications for future researchers and organizations seeking to foster more responsible research practices. △ Less

Submitted 12 August, 2024; v1 submitted 2 August, 2024; originally announced August 2024.

Two-Component gamma-ray Emission Spectrum and X-Ray Polarization of the Radio Galaxy Pictor A

Authors: Jia-Xuan Li, Xin-Ke Hu, Ji-Shun Lian, Yu-Wei Yu, Wei Deng, Kuan Liu, Hai-Ming Zhang, Liang Chen, Jin Zhang

Abstract: Pictor A is a $γ$-ray emitting radio galaxy and has a bright hotspot called WHS, located $\sim$4 arcmin away from the nucleus. In this letter, we present an analysis of its 16-year Fermi-LAT data and report the first Imaging X-ray Polarimetry Explorer (IXPE) observation for this source. Our analysis of the Fermi-LAT observations reveals evidence of two components in the average $γ$-ray spectrum of… ▽ More Pictor A is a $γ$-ray emitting radio galaxy and has a bright hotspot called WHS, located $\sim$4 arcmin away from the nucleus. In this letter, we present an analysis of its 16-year Fermi-LAT data and report the first Imaging X-ray Polarimetry Explorer (IXPE) observation for this source. Our analysis of the Fermi-LAT observations reveals evidence of two components in the average $γ$-ray spectrum of Pictor A, exhibiting a statistically significant hardening from $Γ^1_γ=3.25\pm0.15$ to $Γ^2_γ=1.81\pm0.07$ at a break energy of $2.46\pm0.09$ GeV. The evident variability of $γ$-rays is observed in Pictor A. Interestingly, the variability is dominated by the component below the break energy, and the component above the break energy shows no variability. Furthermore, we find that a power-law function can adequately fit the spectrum during high-flux states, whereas a broken power-law is still required to explain the spectrum during low-flux state. We suggest that the low-energy component originates from the nucleus, while the high-energy component primarily stems from WHS. The broadband spectral energy distributions of both nucleus and WHS can be well represented by a simple leptonic model, with both $γ$-ray components attributed to the synchrotron-self-Compton (SSC) process. The analysis of IXPE data on the nucleus yields an upper limit to the polarization degree $Π_{\rm X}<$8.9\% in the 2--8 keV band, agreeing with its X-ray emission originating from SSC. However, $Π_{\rm X}=23.5\%\pm5.6\%$ is observed at a confidence level of $>99\%$ in the 5--7 keV band, and the possible physical origin of this narrow-energy-band polarization signal is discussed. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Comments: 14 Pages, 4 Figures, 3 Tables, submitted, comments are welcome

FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representation

Authors: Hao Zhu, Zhen Liu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, Xun Cao

Abstract: Implicit Neural Representation (INR), which utilizes a neural network to map coordinate inputs to corresponding attributes, is causing a revolution in the field of signal processing. However, current INR techniques suffer from the "frequency"-specified spectral bias and capacity-convergence gap, resulting in imperfect performance when representing complex signals with multiple "frequencies". We ha… ▽ More Implicit Neural Representation (INR), which utilizes a neural network to map coordinate inputs to corresponding attributes, is causing a revolution in the field of signal processing. However, current INR techniques suffer from the "frequency"-specified spectral bias and capacity-convergence gap, resulting in imperfect performance when representing complex signals with multiple "frequencies". We have identified that both of these two characteristics could be handled by increasing the utilization of definition domain in current activation functions, for which we propose the FINER++ framework by extending existing periodic/non-periodic activation functions to variable-periodic ones. By initializing the bias of the neural network with different ranges, sub-functions with various frequencies in the variable-periodic function are selected for activation. Consequently, the supported frequency set can be flexibly tuned, leading to improved performance in signal representation. We demonstrate the generalization and capabilities of FINER++ with different activation function backbones (Sine, Gauss. and Wavelet) and various tasks (2D image fitting, 3D signed distance field representation, 5D neural radiance fields optimization and streamable INR transmission), and we show that it improves existing INRs. Project page: {https://liuzhen0212.github.io/finerpp/} △ Less

Submitted 28 July, 2024; originally announced July 2024.

Comments: Extension of previous CVPR paper "FINER: Flexible spectral-bias tuning in implicit neural representation by variable-periodic activation functions". arXiv admin note: substantial text overlap with arXiv:2312.02434

Predicting spatial curvature $Ω_K$ in globally $CPT$-symmetric universes

Authors: Wei-Ning Deng, Will Handley

Abstract: Boyle and Turok's $CPT$-symmetric universe model posits that the universe was symmetric at the Big Bang, addressing numerous problems in both cosmology and the Standard Model of particle physics. We extend this model by incorporating the symmetric conditions at the end of the Universe, which impose constraints on the allowed perturbation modes. These constrained modes conflict with the integer wav… ▽ More Boyle and Turok's $CPT$-symmetric universe model posits that the universe was symmetric at the Big Bang, addressing numerous problems in both cosmology and the Standard Model of particle physics. We extend this model by incorporating the symmetric conditions at the end of the Universe, which impose constraints on the allowed perturbation modes. These constrained modes conflict with the integer wave vectors required by the global spatial geometry in a closed universe. To resolve this conflict, only specific values of curvature are permissible, and in particular the curvature density is constrained to be $Ω_K \in \{-0.014, -0.009, -0.003, \ldots\}$, consistent with Planck observations. △ Less

Submitted 13 August, 2024; v1 submitted 25 July, 2024; originally announced July 2024.

Comments: 5 pages, 5 figures

Multi-Beam Symbol-Level Secure Communication for Hybrid Near- and Far-Field Communications

Authors: Wensheng Deng, Bin Qiu, Wenchi Cheng

Abstract: This paper introduces a multi-beam secure communication scheme for mixed near-field and far-field (hNF) scenarios, primarily designed to address the challenges faced by sixth-generation (6G) networks in simultaneously managing near-field and far-field communications. This method significantly reduces the signal quality at eavesdroppers(Eves) while ensuring high-quality reception for legitimate use… ▽ More This paper introduces a multi-beam secure communication scheme for mixed near-field and far-field (hNF) scenarios, primarily designed to address the challenges faced by sixth-generation (6G) networks in simultaneously managing near-field and far-field communications. This method significantly reduces the signal quality at eavesdroppers(Eves) while ensuring high-quality reception for legitimate users, effectively enhancing communication security. At the transmitter, this study employs multi-beam symbol-level directional modulation to ensure secure and reliable transmission in a mixed eavesdropping environment. Given the uncertainty about eavesdropper information, the transmission beamforming vectors are specially designed to meet specific symbol-level constraints, thereby ensuring effective reception for legitimate users. Experimental and simulation results demonstrate the effectiveness of our approach in improving secrecy performance, reducing transmission power, and enhancing energy efficiency, offering a practical solution to the security challenges faced by future wireless networks. △ Less

Submitted 23 July, 2024; originally announced July 2024.

XI-DeepONet: An operator learning method for elliptic interface problems

Authors: Ran Bi, Jingrun Chen, Weibing Deng

Abstract: Scientific computing has been an indispensable tool in applied sciences and engineering, where traditional numerical methods are often employed due to their superior accuracy guarantees. However, these methods often encounter challenges when dealing with problems involving complex geometries. Machine learning-based methods, on the other hand, are mesh-free, thus providing a promising alternative.… ▽ More Scientific computing has been an indispensable tool in applied sciences and engineering, where traditional numerical methods are often employed due to their superior accuracy guarantees. However, these methods often encounter challenges when dealing with problems involving complex geometries. Machine learning-based methods, on the other hand, are mesh-free, thus providing a promising alternative. In particular, operator learning methods have been proposed to learn the mapping from the input space to the solution space, enabling rapid inference of solutions to partial differential equations (PDEs) once trained. In this work, we address the parametric elliptic interface problem. Building upon the deep operator network (DeepONet), we propose an extended interface deep operator network (XI-DeepONet). XI-DeepONet exhibits three unique features: (1) The interface geometry is incorporated into the neural network as an additional input, enabling the network to infer solutions for new interface geometries once trained; (2) The level set function associated with the interface geometry is treated as the input, on which the solution mapping is continuous and can be effectively approximated by the deep operator network; (3) The network can be trained without any input-output data pairs, thus completely avoiding the need for meshes of any kind, directly or indirectly. We conduct a comprehensive series of numerical experiments to demonstrate the accuracy and robustness of the proposed method. △ Less

Submitted 20 July, 2024; originally announced July 2024.

How to quantify an examination? Evidence from physics examinations via complex networks

Authors: Min Xia, Zhu Su, Weibing Deng, Xiumei Feng, Benwei Zhang

Abstract: Given the untapped potential for continuous improvement of examinations, quantitative investigations of examinations could guide efforts to considerably improve learning efficiency and evaluation and thus greatly help both learners and educators. However, there is a general lack of quantitative methods for investigating examinations. To address this gap, we propose a new metric via complex network… ▽ More Given the untapped potential for continuous improvement of examinations, quantitative investigations of examinations could guide efforts to considerably improve learning efficiency and evaluation and thus greatly help both learners and educators. However, there is a general lack of quantitative methods for investigating examinations. To address this gap, we propose a new metric via complex networks; i.e., the knowledge point network (KPN) of an examination is constructed by representing the knowledge points (concepts, laws, etc.) as nodes and adding links when these points appear in the same question. Then, the topological quantities of KPNs, such as degree, centrality, and community, can be employed to systematically explore the structural properties and evolution of examinations. In this work, 35 physics examinations from the NCEE examination spanning from 2006 to 2020 were investigated as an evidence. We found that the constructed KPNs are scale-free networks that show strong assortativity and small-world effects in most cases. The communities within the KPNs are obvious, and the key nodes are mainly related to mechanics and electromagnetism. Different question types are related to specific knowledge points, leading to noticeable structural variations in KPNs. Moreover, changes in the KPN topology between examinations administered in different years may offer insights guiding college entrance examination reforms. Based on topological quantities such as the average degree, network density, average clustering coefficient, and network transitivity, the Fd is proposed to evaluate examination difficulty. All the above results show that our approach can comprehensively quantify the knowledge structures and examination characteristics. These networks may elucidate comprehensive examination knowledge graphs for educators and guide improvements in teaching. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Analysis of Crab X-ray Polarization using Deeper IXPE Observations

Authors: Josephine Wong, Tsunefumi Mizuno, Niccoló Bucciantini, Roger W. Romani, Yi-Jung Yang, Kuan Liu, Wei Deng, Kazuho Goya, Fei Xie, Maura Pilia, Philip Kaaret, Martin C. Weisskopf, Stefano Silvestri, C. -Y. Ng, Chien-Ting Chen, Iván Agudo, Lucio A. Antonelli, Matteo Bachetti, Luca Baldini, Wayne H. Baumgartner, Ronaldo Bellazzini, Stefano Bianchi, Stephen D. Bongiorno, Raffaella Bonino, Alessandro Brez , et al. (76 additional authors not shown)

Abstract: We present Crab X-ray polarization measurements using IXPE data with a total exposure of 300ks, three times more than the initial 2022 discovery paper. Polarization is detected in three times more pulsar phase bins, revealing an S-shaped $+40^\circ$ polarization angle sweep in the main pulse and ${>}1σ$ departures from the OPTIMA optical polarization in both pulses, suggesting different radiation… ▽ More We present Crab X-ray polarization measurements using IXPE data with a total exposure of 300ks, three times more than the initial 2022 discovery paper. Polarization is detected in three times more pulsar phase bins, revealing an S-shaped $+40^\circ$ polarization angle sweep in the main pulse and ${>}1σ$ departures from the OPTIMA optical polarization in both pulses, suggesting different radiation mechanisms or sites for the polarized emission at the two wavebands. Our polarization map of the inner nebula reveals a toroidal magnetic field, as seen in prior IXPE analyses. Along the southern jet, the magnetic field orientation relative to the jet axis changes from perpendicular to parallel and the polarization degree decreases by ${\sim}6\%$. These observations may be explained by kink instabilities along the jet or a collision with a dense, jet-deflecting medium at the tip. Using spectropolarimetric analysis, we find asymmetric polarization in the four quadrants of the inner nebula, as expected for a toroidal field geometry, and a spatial correlation between polarization degree and photon index. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Comments: 14 pages, 7 figures, 3 tables. Accepted by The Astrophysical Journal

E5-V: Universal Embeddings with Multimodal Large Language Models

Authors: Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang

Abstract: Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in rep… ▽ More Multimodal large language models (MLLMs) have shown promising advancements in general visual and language understanding. However, the representation of multimodal information using MLLMs remains largely unexplored. In this work, we introduce a new framework, E5-V, designed to adapt MLLMs for achieving universal multimodal embeddings. Our findings highlight the significant potential of MLLMs in representing multimodal inputs compared to previous approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the modality gap between different types of inputs, demonstrating strong performance in multimodal embeddings even without fine-tuning. We propose a single modality training approach for E5-V, where the model is trained exclusively on text pairs. This method demonstrates significant improvements over traditional multimodal training on image-text pairs, while reducing training costs by approximately 95%. Additionally, this approach eliminates the need for costly multimodal training data collection. Extensive experiments across four types of tasks demonstrate the effectiveness of E5-V. As a universal multimodal model, E5-V not only achieves but often surpasses state-of-the-art performance in each task, despite being trained on a single modality. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Comments: Code and models are available at https://github.com/kongds/E5-V

Cross Domain Object Detection via Multi-Granularity Confidence Alignment based Mean Teacher

Authors: Jiangming Chen, Li Liu, Wanxia Deng, Zhen Liu, Yu Liu, Yingmei Wei, Yongxiang Liu

Abstract: Cross domain object detection learns an object detector for an unlabeled target domain by transferring knowledge from an annotated source domain. Promising results have been achieved via Mean Teacher, however, pseudo labeling which is the bottleneck of mutual learning remains to be further explored. In this study, we find that confidence misalignment of the predictions, including category-level ov… ▽ More Cross domain object detection learns an object detector for an unlabeled target domain by transferring knowledge from an annotated source domain. Promising results have been achieved via Mean Teacher, however, pseudo labeling which is the bottleneck of mutual learning remains to be further explored. In this study, we find that confidence misalignment of the predictions, including category-level overconfidence, instance-level task confidence inconsistency, and image-level confidence misfocusing, leading to the injection of noisy pseudo label in the training process, will bring suboptimal performance on the target domain. To tackle this issue, we present a novel general framework termed Multi-Granularity Confidence Alignment Mean Teacher (MGCAMT) for cross domain object detection, which alleviates confidence misalignment across category-, instance-, and image-levels simultaneously to obtain high quality pseudo supervision for better teacher-student learning. Specifically, to align confidence with accuracy at category level, we propose Classification Confidence Alignment (CCA) to model category uncertainty based on Evidential Deep Learning (EDL) and filter out the category incorrect labels via an uncertainty-aware selection strategy. Furthermore, to mitigate the instance-level misalignment between classification and localization, we design Task Confidence Alignment (TCA) to enhance the interaction between the two task branches and allow each classification feature to adaptively locate the optimal feature for the regression. Finally, we develop imagery Focusing Confidence Alignment (FCA) adopting another way of pseudo label learning, i.e., we use the original outputs from the Mean Teacher network for supervised learning without label assignment to concentrate on holistic information in the target image. These three procedures benefit from each other from a cooperative learning perspective. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Bayesian Federated Learning with Hamiltonian Monte Carlo: Algorithm and Theory

Authors: Jiajun Liang, Qian Zhang, Wei Deng, Qifan Song, Guang Lin

Abstract: This work introduces a novel and efficient Bayesian federated learning algorithm, namely, the Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC), for parameter estimation and uncertainty quantification. We establish rigorous convergence guarantees of FA-HMC on non-iid distributed data sets, under the strong convexity and Hessian smoothness assumptions. Our analysis investigates the ef… ▽ More This work introduces a novel and efficient Bayesian federated learning algorithm, namely, the Federated Averaging stochastic Hamiltonian Monte Carlo (FA-HMC), for parameter estimation and uncertainty quantification. We establish rigorous convergence guarantees of FA-HMC on non-iid distributed data sets, under the strong convexity and Hessian smoothness assumptions. Our analysis investigates the effects of parameter space dimension, noise on gradients and momentum, and the frequency of communication (between the central node and local nodes) on the convergence and communication costs of FA-HMC. Beyond that, we establish the tightness of our analysis by showing that the convergence rate cannot be improved even for continuous FA-HMC process. Moreover, extensive empirical studies demonstrate that FA-HMC outperforms the existing Federated Averaging-Langevin Monte Carlo (FA-LD) algorithm. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Observation of exceptional line semimetal in three-dimensional non-Hermitian phononic crystals

Authors: Yejian Hu, Jien Wu, Peidong Ye, Weiyin Deng, Jiuyang Lu, Xueqin Huang, Ziyu Wang, Manzhu Ke, Zhengyou Liu

Abstract: Non-Hermitian topological phases, which exhibit unique features such as skin effect and exceptional points originated from nontrivial band topologies in complex plane, have attracted enormous attention in condensed-matter physics and metamaterials. Here we report the realization of an exceptional line semimetal in a three-dimensional non-Hermitian phononic crystal. A pair of exceptional rings with… ▽ More Non-Hermitian topological phases, which exhibit unique features such as skin effect and exceptional points originated from nontrivial band topologies in complex plane, have attracted enormous attention in condensed-matter physics and metamaterials. Here we report the realization of an exceptional line semimetal in a three-dimensional non-Hermitian phononic crystal. A pair of exceptional rings with opposite topologies are connected by the drumhead bulk states in the first Brillouin zone. The exceptional rings not only possess wave-function topology and thus result in the drumhead surface states, but also host spectral topology and thereby give rise to the hybrid-order geometry-dependent skin effect in three dimensions. Our experimental results evidence the complete non-Hermitian bulk-boundary correspondence of the three-dimensional exceptional line semimetal, and may pave the way for designing non-Hermitian acoustic devices. △ Less

Submitted 4 July, 2024; originally announced July 2024.

Comments: 5 figures

ASteISR: Adapting Single Image Super-resolution Pre-trained Model for Efficient Stereo Image Super-resolution

Authors: Yuanbo Zhou, Yuyang Xue, Wei Deng, Xinlin Zhang, Qinquan Gao, Tong Tong

Abstract: Despite advances in the paradigm of pre-training then fine-tuning in low-level vision tasks, significant challenges persist particularly regarding the increased size of pre-trained models such as memory usage and training time. Another concern often encountered is the unsatisfying results yielded when directly applying pre-trained single-image models to multi-image domain. In this paper, we propos… ▽ More Despite advances in the paradigm of pre-training then fine-tuning in low-level vision tasks, significant challenges persist particularly regarding the increased size of pre-trained models such as memory usage and training time. Another concern often encountered is the unsatisfying results yielded when directly applying pre-trained single-image models to multi-image domain. In this paper, we propose a efficient method for transferring a pre-trained single-image super-resolution (SISR) transformer network to the domain of stereo image super-resolution (SteISR) through a parameter-efficient fine-tuning (PEFT) method. Specifically, we introduce the concept of stereo adapters and spatial adapters which are incorporated into the pre-trained SISR transformer network. Subsequently, the pre-trained SISR model is frozen, enabling us to fine-tune the adapters using stereo datasets along. By adopting this training method, we enhance the ability of the SISR model to accurately infer stereo images by 0.79dB on the Flickr1024 dataset. This method allows us to train only 4.8% of the original model parameters, achieving state-of-the-art performance on four commonly used SteISR benchmarks. Compared to the more complicated full fine-tuning approach, our method reduces training time and memory consumption by 57% and 15%, respectively. △ Less

Submitted 3 July, 2024; originally announced July 2024.

A Wolf in Sheep's Clothing: Practical Black-box Adversarial Attacks for Evading Learning-based Windows Malware Detection in the Wild

Authors: Xiang Ling, Zhiyu Wu, Bin Wang, Wei Deng, Jingzheng Wu, Shouling Ji, Tianyue Luo, Yanjun Wu

Abstract: Given the remarkable achievements of existing learning-based malware detection in both academia and industry, this paper presents MalGuise, a practical black-box adversarial attack framework that evaluates the security risks of existing learning-based Windows malware detection systems under the black-box setting. MalGuise first employs a novel semantics-preserving transformation of call-based redi… ▽ More Given the remarkable achievements of existing learning-based malware detection in both academia and industry, this paper presents MalGuise, a practical black-box adversarial attack framework that evaluates the security risks of existing learning-based Windows malware detection systems under the black-box setting. MalGuise first employs a novel semantics-preserving transformation of call-based redividing to concurrently manipulate both nodes and edges of malware's control-flow graph, making it less noticeable. By employing a Monte-Carlo-tree-search-based optimization, MalGuise then searches for an optimized sequence of call-based redividing transformations to apply to the input Windows malware for evasions. Finally, it reconstructs the adversarial malware file based on the optimized transformation sequence while adhering to Windows executable format constraints, thereby maintaining the same semantics as the original. MalGuise is systematically evaluated against three state-of-the-art learning-based Windows malware detection systems under the black-box setting. Evaluation results demonstrate that MalGuise achieves a remarkably high attack success rate, mostly exceeding 95%, with over 91% of the generated adversarial malware files maintaining the same semantics. Furthermore, MalGuise achieves up to a 74.97% attack success rate against five anti-virus products, highlighting potential tangible security concerns to real-world users. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: This paper has been accepted by 33rd USENIX Security Symposium 2024

A Treatment of EIP-1559: Enhancing Transaction Fee Mechanism through Nth-Price Auction

Authors: Kun Li, Guangpeng Qi, Guangyong Shang, Wanli Deng, Minghui Xu, Xiuzhen Cheng

Abstract: With the widespread adoption of blockchain technology, the transaction fee mechanism (TFM) in blockchain systems has become a prominent research topic. An ideal TFM should satisfy user incentive compatibility (UIC), miner incentive compatibility (MIC), and miner-user side contract proofness ($c$-SCP). However, state-of-the-art works either fail to meet these three properties simultaneously or only… ▽ More With the widespread adoption of blockchain technology, the transaction fee mechanism (TFM) in blockchain systems has become a prominent research topic. An ideal TFM should satisfy user incentive compatibility (UIC), miner incentive compatibility (MIC), and miner-user side contract proofness ($c$-SCP). However, state-of-the-art works either fail to meet these three properties simultaneously or only satisfy them under certain conditions. In this paper, we propose a burning $N$-price auction TFM named BNP. This mechanism divides the transaction fee into a base fee, which is burned, and a priority fee, which is allocated to miners. Theoretical proofs and experimental analyses demonstrate that, even under conditions of significant transaction congestion, this mechanism satisfies UIC, MIC, and $c$-SCP simultaneously. Furthermore, the BNP mechanism is not constrained by the type of blockchain consensus, making it widely applicable. △ Less

Submitted 27 June, 2024; originally announced June 2024.

Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG

Authors: Xueying Du, Geng Zheng, Kaixin Wang, Jiayi Feng, Wentai Deng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, Yiling Lou

Abstract: Vulnerability detection is essential for software quality assurance. In recent years, deep learning models (especially large language models) have shown promise in vulnerability detection. In this work, we propose a novel LLM-based vulnerability detection technique Vul-RAG, which leverages knowledge-level retrieval-augmented generation (RAG) framework to detect vulnerability for the given code in… ▽ More Vulnerability detection is essential for software quality assurance. In recent years, deep learning models (especially large language models) have shown promise in vulnerability detection. In this work, we propose a novel LLM-based vulnerability detection technique Vul-RAG, which leverages knowledge-level retrieval-augmented generation (RAG) framework to detect vulnerability for the given code in three phases. First, Vul-RAG constructs a vulnerability knowledge base by extracting multi-dimension knowledge via LLMs from existing CVE instances; second, for a given code snippet, Vul-RAG} retrieves the relevant vulnerability knowledge from the constructed knowledge base based on functional semantics; third, Vul-RAG leverages LLMs to check the vulnerability of the given code snippet by reasoning the presence of vulnerability causes and fixing solutions of the retrieved vulnerability knowledge. Our evaluation of Vul-RAG on our constructed benchmark PairVul shows that Vul-RAG substantially outperforms all baselines by 12.96\%/110\% relative improvement in accuracy/pairwise-accuracy. In addition, our user study shows that the vulnerability knowledge generated by Vul-RAG can serve as high-quality explanations which can improve the manual detection accuracy from 0.60 to 0.77. △ Less

Submitted 19 June, 2024; v1 submitted 16 June, 2024; originally announced June 2024.

What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?

Authors: Weijie Tu, Weijian Deng, Liang Zheng, Tom Gedeon

Abstract: This work aims to develop a measure that can accurately rank the performance of various classifiers when they are tested on unlabeled data from out-of-distribution (OOD) distributions. We commence by demonstrating that conventional uncertainty metrics, notably the maximum Softmax prediction probability, possess inherent utility in forecasting model generalization across certain OOD contexts. Build… ▽ More This work aims to develop a measure that can accurately rank the performance of various classifiers when they are tested on unlabeled data from out-of-distribution (OOD) distributions. We commence by demonstrating that conventional uncertainty metrics, notably the maximum Softmax prediction probability, possess inherent utility in forecasting model generalization across certain OOD contexts. Building on this insight, we introduce a new measure called Softmax Correlation (SoftmaxCorr). It calculates the cosine similarity between a class-class correlation matrix, constructed from Softmax output vectors across an unlabeled test dataset, and a predefined reference matrix that embodies ideal class correlations. A high resemblance of predictions to the reference matrix signals that the model delivers confident and uniform predictions across all categories, reflecting minimal uncertainty and confusion. Through rigorous evaluation across a suite of datasets, including ImageNet, CIFAR-10, and WILDS, we affirm the predictive validity of SoftmaxCorr in accurately forecasting model performance within both in-distribution (ID) and OOD settings. Furthermore, we discuss the limitations of our proposed measure and suggest avenues for future research. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: TMLR 2024 (https://openreview.net/forum?id=vtiDUgGjyx)

MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs

Authors: Xuannan Liu, Zekun Li, Peipei Li, Shuhan Xia, Xing Cui, Linzhi Huang, Huaibo Huang, Weihong Deng, Zhaofeng He

Abstract: Current multimodal misinformation detection (MMD) methods often assume a single source and type of forgery for each sample, which is insufficient for real-world scenarios where multiple forgery sources coexist. The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, we introduce MMFakeBench, the first comprehensive benchmark for mixed-source MM… ▽ More Current multimodal misinformation detection (MMD) methods often assume a single source and type of forgery for each sample, which is insufficient for real-world scenarios where multiple forgery sources coexist. The lack of a benchmark for mixed-source misinformation has hindered progress in this field. To address this, we introduce MMFakeBench, the first comprehensive benchmark for mixed-source MMD. MMFakeBench includes 3 critical sources: textual veracity distortion, visual veracity distortion, and cross-modal consistency distortion, along with 12 sub-categories of misinformation forgery types. We further conduct an extensive evaluation of 6 prevalent detection methods and 15 large vision-language models (LVLMs) on MMFakeBench under a zero-shot setting. The results indicate that current methods struggle under this challenging and realistic mixed-source MMD setting. Additionally, we propose an innovative unified framework, which integrates rationales, actions, and tool-use capabilities of LVLM agents, significantly enhancing accuracy and generalization. We believe this study will catalyze future research into more realistic mixed-source multimodal misinformation and provide a fair evaluation of misinformation detection methods. △ Less

Submitted 21 August, 2024; v1 submitted 12 June, 2024; originally announced June 2024.

Comments: Project page: https://liuxuannan.github.io/MMFakeBench.github.io/

Heavy baryons in the relativized quark model with chromodynamics

Authors: Xin-Zhen Weng, Wei-Zhen Deng, Shi-Lin Zhu

Abstract: Following the work of Capstick and Isgur [\href{https://doi.org/10.1103/PhysRevD.34.2809}{Phys.~Rev.~D~34,~2809~(1986)}], we systematically study the mass spectrum of the heavy baryons in the relativized quark potential model with chromodynamics. Besides the original Godfrey-Isgur (GI) model, we also adopt a modified GI model which replaces the linear confinement by a screened one. The two models… ▽ More Following the work of Capstick and Isgur [\href{https://doi.org/10.1103/PhysRevD.34.2809}{Phys.~Rev.~D~34,~2809~(1986)}], we systematically study the mass spectrum of the heavy baryons in the relativized quark potential model with chromodynamics. Besides the original Godfrey-Isgur (GI) model, we also adopt a modified GI model which replaces the linear confinement by a screened one. The two models give similar results in our work. All heavy baryons observed so far can be explained as three-quark states. In particular, we identify the $Ω_{c}(3000)$/$Ω_{b}(6316)$, $Ω_{c}(3050)$/$Ω_{b}(6330)$, $Ω_{c}(3065)$/$Ω_{b}(6340)$ and $Ω_{c}(3090)$/$Ω_{b}(6350)$ states as the $p_λ$ excitations with quantum numbers $1/2^{-}$, $3/2^{-}$, $3/2^{-}$ and $5/2^{-}$. The $Ω_{c}(3120)$ is a $3/2^{-}$ state with the $p_ρ$ excitation, whose bottom partner is predicted to be $Ω_{b}(6446/6457,3/2^{-})$. The higher state $Ω_{c}(3188)$ is the $2s_λ$ excitation with quantum numbers $1/2^{+}$, and $Ω_{c}(3327)$ is a $d_λ$ excitation with quantum numbers $3/2^{+}$ or $5/2^{+}$. In addition, the $Λ_{c}(2940)$ with quantum numbers $J^{P}=3/2^{-}$ could be explained as the $p_ρ$ excitation. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Comments: 20 pages, 10 figures

MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts

Authors: Renchunzi Xie, Ambroise Odonnat, Vasilii Feofanov, Weijian Deng, Jianfeng Zhang, Bo An

Abstract: Leveraging the models' outputs, specifically the logits, is a common approach to estimating the test accuracy of a pre-trained neural network on out-of-distribution (OOD) samples without requiring access to the corresponding ground truth labels. Despite their ease of implementation and computational efficiency, current logit-based methods are vulnerable to overconfidence issues, leading to predict… ▽ More Leveraging the models' outputs, specifically the logits, is a common approach to estimating the test accuracy of a pre-trained neural network on out-of-distribution (OOD) samples without requiring access to the corresponding ground truth labels. Despite their ease of implementation and computational efficiency, current logit-based methods are vulnerable to overconfidence issues, leading to prediction bias, especially under the natural shift. In this work, we first study the relationship between logits and generalization performance from the view of low-density separation assumption. Our findings motivate our proposed method MaNo which (1) applies a data-dependent normalization on the logits to reduce prediction bias, and (2) takes the $L_p$ norm of the matrix of normalized logits as the estimation score. Our theoretical analysis highlights the connection between the provided score and the model's uncertainty. We conduct an extensive empirical study on common unsupervised accuracy estimation benchmarks and demonstrate that MaNo achieves state-of-the-art performance across various architectures in the presence of synthetic, natural, or subpopulation shifts. △ Less

Submitted 24 June, 2024; v1 submitted 29 May, 2024; originally announced May 2024.

Comments: The three first authors contributed equally

Energy flux and waveform of gravitational wave generated by coalescing slow-spinning binary system in effective one-body theory

Authors: Weike Deng, Sheng Long, Jiliang Jing

Abstract: We extend our research on the energy flux and waveform characteristics of gravitational waves generated by merging nonspinning binary black holes through self-consistent effective one-body theory \cite{L2023} to include binary systems with slowly spinning black holes. Initially, we decompose the equation for the null tetrad component of the gravitationally perturbed Weyl tensor $ψ^B_{4}$ into radi… ▽ More We extend our research on the energy flux and waveform characteristics of gravitational waves generated by merging nonspinning binary black holes through self-consistent effective one-body theory \cite{L2023} to include binary systems with slowly spinning black holes. Initially, we decompose the equation for the null tetrad component of the gravitationally perturbed Weyl tensor $ψ^B_{4}$ into radial and angular parts, leveraging the second-order approximation of the rotation parameter $a$. Subsequently, we derive an analytical solution for the radial equation and observe that our results are contingent upon the parameters $a_2$, $a_3$ and $a$, which represent the second- and third-order correction parameters, respectively. Ultimately, we calculate the energy flux, the radiation-reaction force and the waveform for the ``plus" and ``cross" modes of the gravitational waves generated by merging slowly spinning binary black holes. △ Less

Submitted 26 May, 2024; originally announced May 2024.

Comments: 38 pages, 2 figures

ASI++: Towards Distributionally Balanced End-to-End Generative Retrieval

Authors: Yuxuan Liu, Tianchi Yang, Zihan Zhang, Minghui Song, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang

Abstract: Generative retrieval, a promising new paradigm in information retrieval, employs a seq2seq model to encode document features into parameters and decode relevant document identifiers (IDs) based on search queries. Existing generative retrieval solutions typically rely on a preprocessing stage to pre-define document IDs, which can suffer from a semantic gap between these IDs and the retrieval task.… ▽ More Generative retrieval, a promising new paradigm in information retrieval, employs a seq2seq model to encode document features into parameters and decode relevant document identifiers (IDs) based on search queries. Existing generative retrieval solutions typically rely on a preprocessing stage to pre-define document IDs, which can suffer from a semantic gap between these IDs and the retrieval task. However, end-to-end training for both ID assignments and retrieval tasks is challenging due to the long-tailed distribution characteristics of real-world data, resulting in inefficient and unbalanced ID space utilization. To address these issues, we propose ASI++, a novel fully end-to-end generative retrieval method that aims to simultaneously learn balanced ID assignments and improve retrieval performance. ASI++ builds on the fully end-to-end training framework of vanilla ASI and introduces several key innovations. First, a distributionally balanced criterion addresses the imbalance in ID assignments, promoting more efficient utilization of the ID space. Next, a representation bottleneck criterion enhances dense representations to alleviate bottlenecks in learning ID assignments. Finally, an information consistency criterion integrates these processes into a joint optimization framework grounded in information theory. We further explore various module structures for learning ID assignments, including neural quantization, differentiable product quantization, and residual quantization. Extensive experiments on both public and industrial datasets demonstrate the effectiveness of ASI++ in improving retrieval performance and achieving balanced ID assignments. △ Less

Submitted 23 May, 2024; originally announced May 2024.

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

Authors: Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang

Abstract: Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs… ▽ More Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks. △ Less

Submitted 20 May, 2024; originally announced May 2024.

Comments: Work in Progress

Constrained Exploration via Reflected Replica Exchange Stochastic Gradient Langevin Dynamics

Authors: Haoyang Zheng, Hengrong Du, Qi Feng, Wei Deng, Guang Lin

Abstract: Replica exchange stochastic gradient Langevin dynamics (reSGLD) is an effective sampler for non-convex learning in large-scale datasets. However, the simulation may encounter stagnation issues when the high-temperature chain delves too deeply into the distribution tails. To tackle this issue, we propose reflected reSGLD (r2SGLD): an algorithm tailored for constrained non-convex exploration by util… ▽ More Replica exchange stochastic gradient Langevin dynamics (reSGLD) is an effective sampler for non-convex learning in large-scale datasets. However, the simulation may encounter stagnation issues when the high-temperature chain delves too deeply into the distribution tails. To tackle this issue, we propose reflected reSGLD (r2SGLD): an algorithm tailored for constrained non-convex exploration by utilizing reflection steps within a bounded domain. Theoretically, we observe that reducing the diameter of the domain enhances mixing rates, exhibiting a $\textit{quadratic}$ behavior. Empirically, we test its performance through extensive experiments, including identifying dynamical systems with physical constraints, simulations of constrained multi-modal distributions, and image classification tasks. The theoretical and empirical findings highlight the crucial role of constrained exploration in improving the simulation efficiency. △ Less

Submitted 3 June, 2024; v1 submitted 13 May, 2024; originally announced May 2024.

Comments: 28 pages, 13 figures

Variational Schrödinger Diffusion Models

Authors: Wei Deng, Weijian Luo, Yixin Tan, Marin Biloš, Yu Chen, Yuriy Nevmyvaka, Ricky T. Q. Chen

Abstract: Schrödinger bridge (SB) has emerged as the go-to method for optimizing transportation plans in diffusion models. However, SB requires estimating the intractable forward score functions, inevitably resulting in the costly implicit training loss based on simulated trajectories. To improve the scalability while preserving efficient transportation plans, we leverage variational inference to linearize… ▽ More Schrödinger bridge (SB) has emerged as the go-to method for optimizing transportation plans in diffusion models. However, SB requires estimating the intractable forward score functions, inevitably resulting in the costly implicit training loss based on simulated trajectories. To improve the scalability while preserving efficient transportation plans, we leverage variational inference to linearize the forward score functions (variational scores) of SB and restore simulation-free properties in training backward scores. We propose the variational Schrödinger diffusion model (VSDM), where the forward process is a multivariate diffusion and the variational scores are adaptively optimized for efficient transport. Theoretically, we use stochastic approximation to prove the convergence of the variational scores and show the convergence of the adaptively generated samples based on the optimal variational scores. Empirically, we test the algorithm in simulated examples and observe that VSDM is efficient in generations of anisotropic shapes and yields straighter sample trajectories compared to the single-variate diffusion. We also verify the scalability of the algorithm in real-world data and achieve competitive unconditional generation performance in CIFAR10 and conditional generation in time series modeling. Notably, VSDM no longer depends on warm-up initializations and has become tuning-friendly in training large-scale experiments. △ Less

Submitted 19 June, 2024; v1 submitted 8 May, 2024; originally announced May 2024.

Comments: ICML 2024

WeightedPose: Generalizable Cross-Pose Estimation via Weighted SVD

Authors: Xuxin Cheng, Heng Yu, Harry Zhang, Wenxing Deng

Abstract: We introduce a new approach for robotic manipulation tasks in human settings that necessitates understanding the 3D geometric connections between a pair of objects. Conventional end-to-end training approaches, which convert pixel observations directly into robot actions, often fail to effectively understand complex pose relationships and do not easily adapt to new object configurations. To overcom… ▽ More We introduce a new approach for robotic manipulation tasks in human settings that necessitates understanding the 3D geometric connections between a pair of objects. Conventional end-to-end training approaches, which convert pixel observations directly into robot actions, often fail to effectively understand complex pose relationships and do not easily adapt to new object configurations. To overcome these issues, our method focuses on learning the 3D geometric relationships, particularly how critical parts of one object relate to those of another. We employ Weighted SVD in our standalone model to analyze pose relationships both in articulated parts and in free-floating objects. For instance, our model can comprehend the spatial relationship between an oven door and the oven body, as well as between a lasagna plate and the oven. By concentrating on the 3D geometric connections, our strategy empowers robots to carry out intricate manipulation tasks based on object-centric perspectives △ Less

Submitted 21 May, 2024; v1 submitted 3 May, 2024; originally announced May 2024.

Comments: arXiv admin note: text overlap with arXiv:2211.09325

Trust Dynamics and Market Behavior in Cryptocurrency: A Comparative Study of Centralized and Decentralized Exchanges

Authors: Xintong Wu, Wanling Deng, Yuotng Quan, Luyao Zhang

Abstract: In the evolving landscape of digital finance, the transition from centralized to decentralized trust mechanisms, primarily driven by blockchain technology, plays a critical role in shaping the cryptocurrency ecosystem. This paradigm shift raises questions about the traditional reliance on centralized trust and introduces a novel, decentralized trust framework built upon distributed networks. Our r… ▽ More In the evolving landscape of digital finance, the transition from centralized to decentralized trust mechanisms, primarily driven by blockchain technology, plays a critical role in shaping the cryptocurrency ecosystem. This paradigm shift raises questions about the traditional reliance on centralized trust and introduces a novel, decentralized trust framework built upon distributed networks. Our research delves into the consequences of this shift, particularly focusing on how incidents influence trust within cryptocurrency markets, thereby affecting trade behaviors in centralized (CEXs) and decentralized exchanges (DEXs). We conduct a comprehensive analysis of various events, assessing their effects on market dynamics, including token valuation and trading volumes in both CEXs and DEXs. Our findings highlight the pivotal role of trust in directing user preferences and the fluidity of trust transfer between centralized and decentralized platforms. Despite certain anomalies, the results largely align with our initial hypotheses, revealing the intricate nature of user trust in cryptocurrency markets. This study contributes significantly to interdisciplinary research, bridging distributed systems, behavioral finance, and Decentralized Finance (DeFi). It offers valuable insights for the distributed computing community, particularly in understanding and applying distributed trust mechanisms in digital economies, paving the way for future research that could further explore the socio-economic dimensions and leverage blockchain data in this dynamic domain. △ Less

Submitted 26 April, 2024; originally announced April 2024.

Real-Time 4K Super-Resolution of Compressed AVIF Images. AIS 2024 Challenge Survey

Authors: Marcos V. Conde, Zhijun Lei, Wen Li, Cosmin Stejerean, Ioannis Katsavounidis, Radu Timofte, Kihwan Yoon, Ganzorig Gankhuyag, Jiangtao Lv, Long Sun, Jinshan Pan, Jiangxin Dong, Jinhui Tang, Zhiyuan Li, Hao Wei, Chenyang Ge, Dongyang Zhang, Tianle Liu, Huaian Chen, Yi Jin, Menghan Zhou, Yiqiang Yan, Si Gao, Biao Wu, Shaoli Liu , et al. (50 additional authors not shown)

Abstract: This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF cod… ▽ More This paper introduces a novel benchmark as part of the AIS 2024 Real-Time Image Super-Resolution (RTSR) Challenge, which aims to upscale compressed images from 540p to 4K resolution (4x factor) in real-time on commercial GPUs. For this, we use a diverse test set containing a variety of 4K images ranging from digital art to gaming and photography. The images are compressed using the modern AVIF codec, instead of JPEG. All the proposed methods improve PSNR fidelity over Lanczos interpolation, and process images under 10ms. Out of the 160 participants, 25 teams submitted their code and models. The solutions present novel designs tailored for memory-efficiency and runtime on edge devices. This survey describes the best solutions for real-time SR of compressed high-resolution images. △ Less

Submitted 25 April, 2024; originally announced April 2024.

Comments: CVPR 2024, AI for Streaming (AIS) Workshop

NTIRE 2024 Challenge on Low Light Image Enhancement: Methods and Results

Authors: Xiaoning Liu, Zongwei Wu, Ao Li, Florin-Alexandru Vasluianu, Yulun Zhang, Shuhang Gu, Le Zhang, Ce Zhu, Radu Timofte, Zhi Jin, Hongjun Wu, Chenxi Wang, Haitao Ling, Yuanhao Cai, Hao Bian, Yuxin Zheng, Jing Lin, Alan Yuille, Ben Shao, Jin Guo, Tianli Liu, Mohao Wu, Yixu Feng, Shuo Hou, Haotian Lin , et al. (87 additional authors not shown)

Abstract: This paper reviews the NTIRE 2024 low light image enhancement challenge, highlighting the proposed solutions and results. The aim of this challenge is to discover an effective network design or solution capable of generating brighter, clearer, and visually appealing results when dealing with a variety of conditions, including ultra-high resolution (4K and beyond), non-uniform illumination, backlig… ▽ More This paper reviews the NTIRE 2024 low light image enhancement challenge, highlighting the proposed solutions and results. The aim of this challenge is to discover an effective network design or solution capable of generating brighter, clearer, and visually appealing results when dealing with a variety of conditions, including ultra-high resolution (4K and beyond), non-uniform illumination, backlighting, extreme darkness, and night scenes. A notable total of 428 participants registered for the challenge, with 22 teams ultimately making valid submissions. This paper meticulously evaluates the state-of-the-art advancements in enhancing low-light images, reflecting the significant progress and creativity in this field. △ Less

Submitted 22 April, 2024; originally announced April 2024.

Comments: NTIRE 2024 Challenge Report

Valley edge states as bound states in the continuum

Authors: Shunda Yin, Liping Ye, Hailong He, Xueqin Huang, Manzhu Ke, Weiyin Deng, Jiuyang Lu, Zhengyou Liu

Abstract: Bound states in the continuum (BICs) are spatially localized states with energy embedded in the continuum spectrum of extended states. The combination of BICs physics and nontrivial band topology theory giving rise to topological BICs, which are robust against disorders and meanwhile of the merit of conventional BICs, is attracting wide attention recently. Here, we report valley edge states as top… ▽ More Bound states in the continuum (BICs) are spatially localized states with energy embedded in the continuum spectrum of extended states. The combination of BICs physics and nontrivial band topology theory giving rise to topological BICs, which are robust against disorders and meanwhile of the merit of conventional BICs, is attracting wide attention recently. Here, we report valley edge states as topological BICs, which appear at domain wall between two distinct valley topological phases. The robustness of such BICs is demonstrated. The simulations and experiments show great agreement. Our findings of valley related topological BICs shed light on both BICs and valley physics, and may foster innovative applications of topological acoustic devices. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Comments: A revised version has been accepted by Science Bulletin

Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation

Authors: Wenxiao Deng, Wenbin Li, Tianyu Ding, Lei Wang, Hongguang Zhang, Kuihua Huang, Jing Huo, Yang Gao

Abstract: Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the… ▽ More Dataset distillation has emerged as a promising approach in deep learning, enabling efficient training with small synthetic datasets derived from larger real ones. Particularly, distribution matching-based distillation methods attract attention thanks to its effectiveness and low computational cost. However, these methods face two primary limitations: the dispersed feature distribution within the same class in synthetic datasets, reducing class discrimination, and an exclusive focus on mean feature consistency, lacking precision and comprehensiveness. To address these challenges, we introduce two novel constraints: a class centralization constraint and a covariance matching constraint. The class centralization constraint aims to enhance class discrimination by more closely clustering samples within classes. The covariance matching constraint seeks to achieve more accurate feature distribution matching between real and synthetic datasets through local feature covariance matrices, particularly beneficial when sample sizes are much smaller than the number of features. Experiments demonstrate notable improvements with these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9% on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the state-of-the-art relevant methods. In addition, our method maintains robust performance in cross-architecture settings, with a maximum performance drop of 1.7% on four architectures. Code is available at https://github.com/VincenDen/IID. △ Less

Submitted 31 March, 2024; originally announced April 2024.

Comments: Accepted to CVPR 2024

Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models

Authors: Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Weimo Deng, Ziyong Feng, Yongle Zhao, Yin Xie

Abstract: The rise of Multimodal Large Language Models (MLLMs), renowned for their advanced instruction-following and reasoning capabilities, has significantly propelled the field of visual reasoning. However, due to limitations in their image tokenization processes, most MLLMs struggle to capture fine details of text and objects in images, especially in high-resolution samples. To overcome this limitation,… ▽ More The rise of Multimodal Large Language Models (MLLMs), renowned for their advanced instruction-following and reasoning capabilities, has significantly propelled the field of visual reasoning. However, due to limitations in their image tokenization processes, most MLLMs struggle to capture fine details of text and objects in images, especially in high-resolution samples. To overcome this limitation, we introduce P2G, a novel framework for plug-and-play grounding in MLLMs. P2G utilizes the tool-usage potential of MLLMs to employ expert agents for on-the-fly grounding of reasoning into critical visual and textual elements in images, thereby enabling deliberate reasoning through multimodal prompting. Additionally, we develop P2GB, a benchmark designed to evaluate MLLMs' proficiency in understanding inter-object relationships and textual content in challenging high-resolution images. Extensive experiments on visual reasoning tasks demonstrate the superiority of P2G, achieving performance comparable to GPT-4V on P2GB with a 7B backbone. Our work underscores the potential of grounding reasoning with external agents in MLLMs, presenting a promising alternative to mere model scaling. △ Less

Submitted 18 June, 2024; v1 submitted 28 March, 2024; originally announced March 2024.

Comments: 15 pages, 8 figures

Can multiple-choice questions really be useful in detecting the abilities of LLMs?

Authors: Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, Noa Garcia

Abstract: Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoug… ▽ More Multiple-choice questions (MCQs) are widely used in the evaluation of large language models (LLMs) due to their simplicity and efficiency. However, there are concerns about whether MCQs can truly measure LLM's capabilities, particularly in knowledge-intensive scenarios where long-form generation (LFG) answers are required. The misalignment between the task and the evaluation method demands a thoughtful analysis of MCQ's efficacy, which we undertake in this paper by evaluating nine LLMs on four question-answering (QA) datasets in two languages: Chinese and English. We identify a significant issue: LLMs exhibit an order sensitivity in bilingual MCQs, favoring answers located at specific positions, i.e., the first position. We further quantify the gap between MCQs and long-form generation questions (LFGQs) by comparing their direct outputs, token logits, and embeddings. Our results reveal a relatively low correlation between answers from MCQs and LFGQs for identical questions. Additionally, we propose two methods to quantify the consistency and confidence of LLMs' output, which can be generalized to other QA evaluation benchmarks. Notably, our analysis challenges the idea that the higher the consistency, the greater the accuracy. We also find MCQs to be less reliable than LFGQs in terms of expected calibration error. Finally, the misalignment between MCQs and LFGQs is not only reflected in the evaluation performance but also in the embedding space. Our code and models can be accessed at https://github.com/Meetyou-AI-Lab/Can-MC-Evaluate-LLMs. △ Less

Submitted 23 May, 2024; v1 submitted 26 March, 2024; originally announced March 2024.

Comments: LREC-COLING 2024

Can 3D Vision-Language Models Truly Understand Natural Language?

Authors: Weipeng Deng, Jihan Yang, Runyu Ding, Jiahui Liu, Yijiang Li, Xiaojuan Qi, Edith Ngai

Abstract: Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new avenues for human interaction with embodied agents or robots using natural language. Despite this progress, we find a notable limitation: existing 3D-VL models exhibit sensitivity to the styles of language input, struggling to understand sentences with the same semantic meaning but written in different variants. This observa… ▽ More Rapid advancements in 3D vision-language (3D-VL) tasks have opened up new avenues for human interaction with embodied agents or robots using natural language. Despite this progress, we find a notable limitation: existing 3D-VL models exhibit sensitivity to the styles of language input, struggling to understand sentences with the same semantic meaning but written in different variants. This observation raises a critical question: Can 3D vision-language models truly understand natural language? To test the language understandability of 3D-VL models, we first propose a language robustness task for systematically assessing 3D-VL models across various tasks, benchmarking their performance when presented with different language style variants. Importantly, these variants are commonly encountered in applications requiring direct interaction with humans, such as embodied robotics, given the diversity and unpredictability of human language. We propose a 3D Language Robustness Dataset, designed based on the characteristics of human language, to facilitate the systematic study of robustness. Our comprehensive evaluation uncovers a significant drop in the performance of all existing models across various 3D-VL tasks. Even the state-of-the-art 3D-LLM fails to understand some variants of the same sentences. Further in-depth analysis suggests that the existing models have a fragile and biased fusion module, which stems from the low diversity of the existing dataset. Finally, we propose a training-free module driven by LLM, which improves language robustness. Datasets and code will be available at github. △ Less

Submitted 3 July, 2024; v1 submitted 21 March, 2024; originally announced March 2024.

Comments: https://github.com/VincentDENGP/3D-LR

CSI Transfer From Sub-6G to mmWave: Reduced-Overhead Multi-User Hybrid Beamforming

Authors: Weicao Deng, Min Li, Ming-Min Zhao, Min-Jian Zhao, Osvaldo Simeone

Abstract: Hybrid beamforming is vital in modern wireless systems, especially for massive MIMO and millimeter-wave deployments, offering efficient directional transmission with reduced hardware complexity. However, effective beamforming in multi-user scenarios relies heavily on accurate channel state information, the acquisition of which often incurs excessive pilot overhead, degrading system performance. To… ▽ More Hybrid beamforming is vital in modern wireless systems, especially for massive MIMO and millimeter-wave deployments, offering efficient directional transmission with reduced hardware complexity. However, effective beamforming in multi-user scenarios relies heavily on accurate channel state information, the acquisition of which often incurs excessive pilot overhead, degrading system performance. To address this and inspired by the spatial congruence between sub-6GHz (sub-6G) and mmWave channels, we propose a Sub-6G information Aided Multi-User Hybrid Beamforming (SA-MUHBF) framework, avoiding excessive use of pilots. SA-MUHBF employs a convolutional neural network to predict mmWave beamspace from sub-6G channel estimate, followed by a novel multi-layer graph neural network for analog beam selection and a linear minimum mean-square error algorithm for digital beamforming. Numerical results demonstrate that SA-MUHBF efficiently predicts the mmWave beamspace representation and achieves superior spectrum efficiency over state-of-the-art benchmarks. Moreover, SA-MUHBF demonstrates robust performance across varied sub-6G system configurations and exhibits strong generalization to unseen scenarios. △ Less

Submitted 16 March, 2024; originally announced March 2024.

Comments: 13 pages, 12 figures, submitted

Faceptor: A Generalist Model for Face Perception

Authors: Lixiong Qin, Mei Wang, Xuannan Liu, Yuhang Zhang, Wei Deng, Xiaoshuai Song, Weiran Xu, Weihong Deng

Abstract: With the comprehensive research conducted on various face analysis tasks, there is a growing interest among researchers to develop a unified approach to face perception. Existing methods mainly discuss unified representation and training, which lack task extensibility and application efficiency. To tackle this issue, we focus on the unified model structure, exploring a face generalist model. As an… ▽ More With the comprehensive research conducted on various face analysis tasks, there is a growing interest among researchers to develop a unified approach to face perception. Existing methods mainly discuss unified representation and training, which lack task extensibility and application efficiency. To tackle this issue, we focus on the unified model structure, exploring a face generalist model. As an intuitive design, Naive Faceptor enables tasks with the same output shape and granularity to share the structural design of the standardized output head, achieving improved task extensibility. Furthermore, Faceptor is proposed to adopt a well-designed single-encoder dual-decoder architecture, allowing task-specific queries to represent new-coming semantics. This design enhances the unification of model structure while improving application efficiency in terms of storage overhead. Additionally, we introduce Layer-Attention into Faceptor, enabling the model to adaptively select features from optimal layers to perform the desired tasks. Through joint training on 13 face perception datasets, Faceptor achieves exceptional performance in facial landmark localization, face parsing, age estimation, expression recognition, binary attribute classification, and face recognition, achieving or surpassing specialized methods in most tasks. Our training framework can also be applied to auxiliary supervised learning, significantly improving performance in data-sparse tasks such as age estimation and expression recognition. The code and models will be made publicly available at https://github.com/lxq1000/Faceptor. △ Less

Submitted 14 March, 2024; originally announced March 2024.

Energy flux and waveforms by coalescing spinless binary system in effective one-body theory

Authors: Sheng Long, Weike Deng, Jiliang Jing

Abstract: We present a study on the energy radiation rate and waveforms of the gravitational wave generated by coalescing spinless binary systems up to the third post-Minkowskian approximation in the effective one-body theory. To derive an analytical expansion of the null tetrad components of the gravitational perturbed Weyl tensor $\varPsi_{4}$ in the effective spacetime, we utilize the method proposed by… ▽ More We present a study on the energy radiation rate and waveforms of the gravitational wave generated by coalescing spinless binary systems up to the third post-Minkowskian approximation in the effective one-body theory. To derive an analytical expansion of the null tetrad components of the gravitational perturbed Weyl tensor $\varPsi_{4}$ in the effective spacetime, we utilize the method proposed by Sasaki $et$ $al.$ During this investigation, we discover more general integral formulas that provide a theoretical framework for computing the results in any order. Subsequently, we successfully compute the energy radiation rate and waveforms of the gravitational wave, which include the results of the Schwarzschild case and the correction terms resulting from the dimensionless parameters $a_{2}$ and $a_{3}$ in the effective metric. △ Less

Submitted 9 May, 2024; v1 submitted 14 March, 2024; originally announced March 2024.

Comments: 33 pages, 5 figures

Journal ref: SCIENCE CHINA Physics, Mechanics & Astronomy, 2024, V67, 260412

Viro's patchworking and the signed reduced A-discriminant

Authors: Weixun Deng, J. Maurice Rojas, Máté L. Telek

Abstract: Computing the isotopy type of a hypersurface, defined as the positive real zero set of a multivariate polynomial, is a challenging problem in real algebraic geometry. We focus on the case where the defining polynomial has combinatorially restricted exponent vectors and fixed coefficient signs, enabling faster computation of the isotopy type. In particular, Viro's patchworking provides a polyhedral… ▽ More Computing the isotopy type of a hypersurface, defined as the positive real zero set of a multivariate polynomial, is a challenging problem in real algebraic geometry. We focus on the case where the defining polynomial has combinatorially restricted exponent vectors and fixed coefficient signs, enabling faster computation of the isotopy type. In particular, Viro's patchworking provides a polyhedral complex that has the same isotopy type as the hypersurface, for certain choices of the coefficients. So we present properties of the signed support, focussing mainly on the case of n-variate (n+3)-nomials, that ensure all possible isotopy types can be obtained via patchworking. To prove this, we study the signed reduced A-discriminant and show that it has a simple structure if the signed support satisfies some combinatorial conditions. △ Less

Submitted 13 March, 2024; originally announced March 2024.

Confidence-Aware RGB-D Face Recognition via Virtual Depth Synthesis

Authors: Zijian Chen, Mei Wang, Weihong Deng, Hongzhi Shi, Dongchao Wen, Yingjie Zhang, Xingchen Cui, Jian Zhao

Abstract: 2D face recognition encounters challenges in unconstrained environments due to varying illumination, occlusion, and pose. Recent studies focus on RGB-D face recognition to improve robustness by incorporating depth information. However, collecting sufficient paired RGB-D training data is expensive and time-consuming, hindering wide deployment. In this work, we first construct a diverse depth datase… ▽ More 2D face recognition encounters challenges in unconstrained environments due to varying illumination, occlusion, and pose. Recent studies focus on RGB-D face recognition to improve robustness by incorporating depth information. However, collecting sufficient paired RGB-D training data is expensive and time-consuming, hindering wide deployment. In this work, we first construct a diverse depth dataset generated by 3D Morphable Models for depth model pre-training. Then, we propose a domain-independent pre-training framework that utilizes readily available pre-trained RGB and depth models to separately perform face recognition without needing additional paired data for retraining. To seamlessly integrate the two distinct networks and harness the complementary benefits of RGB and depth information for improved accuracy, we propose an innovative Adaptive Confidence Weighting (ACW). This mechanism is designed to learn confidence estimates for each modality to achieve modality fusion at the score level. Our method is simple and lightweight, only requiring ACW training beyond the backbone models. Experiments on multiple public RGB-D face recognition benchmarks demonstrate state-of-the-art performance surpassing previous methods based on depth estimation and feature fusion, validating the efficacy of our approach. △ Less

Submitted 16 March, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: 9 pages, 5 figures

Debiased Noise Editing on Foundation Models for Fair Medical Image Classification

Authors: Ruinan Jin, Wenlong Deng, Minghui Chen, Xiaoxiao Li

Abstract: In the era of Foundation Models' (FMs) rising prominence in AI, our study addresses the challenge of biases in medical images while the model operates in black-box (e.g., using FM API), particularly spurious correlations between pixels and sensitive attributes. Traditional methods for bias mitigation face limitations due to the restricted access to web-hosted FMs and difficulties in addressing the… ▽ More In the era of Foundation Models' (FMs) rising prominence in AI, our study addresses the challenge of biases in medical images while the model operates in black-box (e.g., using FM API), particularly spurious correlations between pixels and sensitive attributes. Traditional methods for bias mitigation face limitations due to the restricted access to web-hosted FMs and difficulties in addressing the underlying bias encoded within the FM API. We propose a D(ebiased) N(oise) E(diting) strategy, termed DNE, which generates DNE noise to mask such spurious correlation. DNE is capable of mitigating bias both within the FM API embedding and the images themselves. Furthermore, DNE is suitable for both white-box and black-box FM APIs, where we introduced G(reedy) (Z)eroth-O(rder) (GeZO) optimization for it when the gradient is inaccessible in black-box APIs. Our whole pipeline enables fairness-aware image editing that can be applied across various medical contexts without requiring direct model manipulation or significant computational resources. Our empirical results demonstrate the method's effectiveness in maintaining fairness and utility across different patient groups and diseases. In the era of AI-driven medicine, this work contributes to making healthcare diagnostics more equitable, showcasing a practical solution for bias mitigation in pre-trained image FMs. Our code is provided at https://github.com/ubc-tea/DNE-foundation-model-fairness. △ Less

Submitted 12 July, 2024; v1 submitted 10 March, 2024; originally announced March 2024.

Comments: 13 pages, 3 figures. Accepted by MICCAI 2024

Beyond Finite Data: Towards Data-free Out-of-distribution Generalization via Extrapolation

Authors: Yijiang Li, Sucheng Ren, Weipeng Deng, Yuzhi Xu, Ying Gao, Edith Ngai, Haohan Wang

Abstract: Out-of-distribution (OOD) generalization is a favorable yet challenging property for deep neural networks. The core challenges lie in the limited availability of source domains that help models learn an invariant representation from the spurious features. Various domain augmentation have been proposed but largely rely on interpolating existing domains and frequently face difficulties in creating t… ▽ More Out-of-distribution (OOD) generalization is a favorable yet challenging property for deep neural networks. The core challenges lie in the limited availability of source domains that help models learn an invariant representation from the spurious features. Various domain augmentation have been proposed but largely rely on interpolating existing domains and frequently face difficulties in creating truly "novel" domains. Humans, on the other hand, can easily extrapolate novel domains, thus, an intriguing question arises: How can neural networks extrapolate like humans and achieve OOD generalization? We introduce a novel approach to domain extrapolation that leverages reasoning ability and the extensive knowledge encapsulated within large language models (LLMs) to synthesize entirely new domains. Starting with the class of interest, we query the LLMs to extract relevant knowledge for these novel domains. We then bridge the gap between the text-centric knowledge derived from LLMs and the pixel input space of the model using text-to-image generation techniques. By augmenting the training set of domain generalization datasets with high-fidelity, photo-realistic images of these new domains, we achieve significant improvements over all existing methods, as demonstrated in both single and multi-domain generalization across various benchmarks. With the ability to extrapolate any domains for any class, our method has the potential to learn a generalized model for any task without any data. To illustrate, we put forth a much more difficult setting termed, data-free domain generalization, that aims to learn a generalized model in the absence of any collected data. Our empirical findings support the above argument and our methods exhibit commendable performance in this setting, even surpassing the supervised setting by approximately 1-2\% on datasets such as VLCS. △ Less

Submitted 11 March, 2024; v1 submitted 8 March, 2024; originally announced March 2024.

Comments: Preprint. Paper under review

FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs

Authors: Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, Zhaofeng He

Abstract: The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-Language Models (LVLMs) have encoded rich world knowledge, they are not inherently tailore… ▽ More The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-Language Models (LVLMs) have encoded rich world knowledge, they are not inherently tailored for combating fake news and struggle to comprehend local forgery details. In this paper, we propose FKA-Owl, a novel framework that leverages forgery-specific knowledge to augment LVLMs, enabling them to reason about manipulations effectively. The augmented forgery-specific knowledge includes semantic correlation between text and images, and artifact trace in image manipulation. To inject these two kinds of knowledge into the LVLM, we design two specialized modules to establish their representations, respectively. The encoded knowledge embeddings are then incorporated into LVLMs. Extensive experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods. Code is publicly available at https://liuxuannan.github.io/FKA_Owl.github.io/. △ Less

Submitted 6 August, 2024; v1 submitted 4 March, 2024; originally announced March 2024.

Comments: Accepted by ACM MM 2024.Project page: https://liuxuannan.github.io/FKA_Owl.github.io/

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

Authors: Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Abstract: As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual pa… ▽ More As one of the most popular parameter-efficient fine-tuning (PEFT) methods, low-rank adaptation (LoRA) is commonly applied to fine-tune large language models (LLMs). However, updating the weights of LoRA blocks effectively and expeditiously is challenging due to the long calculation path in the original model. To address this, we propose ResLoRA, an improved framework of LoRA. By adding residual paths during training and using merging approaches to eliminate these extra paths during inference, our method can achieve better results in fewer training steps without any extra trainable parameters or inference cost compared to LoRA. The experiments on NLG, NLU, and text-to-image tasks demonstrate the effectiveness of our method. To the best of our knowledge, ResLoRA is the first work that combines the residual path with LoRA. The code of our method is available at https://github.com/microsoft/LMOps/tree/main/reslora . △ Less

Submitted 27 February, 2024; originally announced February 2024.

Comments: 14 pages, 7 figures

HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition

Authors: Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang

Abstract: Large language models (LLMs) have emerged as a promising alternative to expensive human evaluations. However, the alignment and coverage of LLM-based evaluations are often limited by the scope and potential bias of the evaluation prompts and criteria. To address this challenge, we propose HD-Eval, a novel framework that iteratively aligns LLM-based evaluators with human preference via Hierarchical… ▽ More Large language models (LLMs) have emerged as a promising alternative to expensive human evaluations. However, the alignment and coverage of LLM-based evaluations are often limited by the scope and potential bias of the evaluation prompts and criteria. To address this challenge, we propose HD-Eval, a novel framework that iteratively aligns LLM-based evaluators with human preference via Hierarchical Criteria Decomposition. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators by decomposing a given evaluation task into finer-grained criteria, aggregating them according to estimated human preferences, pruning insignificant criteria with attribution, and further decomposing significant criteria. By integrating these steps within an iterative alignment training process, we obtain a hierarchical decomposition of criteria that comprehensively captures aspects of natural language at multiple levels of granularity. Implemented as a white box, the human preference-guided aggregator is efficient to train and more explainable than relying solely on prompting, and its independence from model parameters makes it applicable to closed-source LLMs. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators and providing deeper insights into the explanation of evaluation results and the task itself. △ Less

Submitted 24 February, 2024; originally announced February 2024.

Comments: 20 pages, 13 figures