Showing 51–100 of 508 results for author: Wen, Z

Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation

Authors: Zhiyuan Wen, Jiannong Cao, Yu Yang, Ruosong Yang, Shuaiqi Liu

Abstract: Personality Recognition in Conversation (PRC) aims to identify the personality traits of speakers through textual dialogue content. It is essential for providing personalized services in various applications of Human-Computer Interaction (HCI), such as AI-based mental therapy and companion robots for the elderly. Most recent studies analyze the dialog content for personality classification yet ove… ▽ More Personality Recognition in Conversation (PRC) aims to identify the personality traits of speakers through textual dialogue content. It is essential for providing personalized services in various applications of Human-Computer Interaction (HCI), such as AI-based mental therapy and companion robots for the elderly. Most recent studies analyze the dialog content for personality classification yet overlook two major concerns that hinder their performance. First, crucial implicit factors contained in conversation, such as emotions that reflect the speakers' personalities are ignored. Second, only focusing on the input dialog content disregards the semantic understanding of personality itself, which reduces the interpretability of the results. In this paper, we propose Affective Natural Language Inference (Affective-NLI) for accurate and interpretable PRC. To utilize affectivity within dialog content for accurate personality recognition, we fine-tuned a pre-trained language model specifically for emotion recognition in conversations, facilitating real-time affective annotations for utterances. For interpretability of recognition results, we formulate personality recognition as an NLI problem by determining whether the textual description of personality labels is entailed by the dialog content. Extensive experiments on two daily conversation datasets suggest that Affective-NLI significantly outperforms (by 6%-7%) state-of-the-art approaches. Additionally, our Flow experiment demonstrates that Affective-NLI can accurately recognize the speaker's personality in the early stages of conversations by surpassing state-of-the-art methods with 22%-34%. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Comments: Accepted by IEEE PerCom 2024

A catalog of 1.58 million clusters of galaxies identified from the DESI Legacy Imaging Surveys

Authors: Z. L. Wen, J. L. Han

Abstract: Based on the DESI Legacy Imaging Surveys released data and available spectroscopic redshifts, we identify 1.58 million clusters of galaxies by searching for the overdensity of stellar mass distribution of galaxies within redshift slices around pre-selected massive galaxies, among which 877,806 clusters are found for the first time. The identified clusters have an equivalent mass of M_{500}> 0.47*1… ▽ More Based on the DESI Legacy Imaging Surveys released data and available spectroscopic redshifts, we identify 1.58 million clusters of galaxies by searching for the overdensity of stellar mass distribution of galaxies within redshift slices around pre-selected massive galaxies, among which 877,806 clusters are found for the first time. The identified clusters have an equivalent mass of M_{500}> 0.47*10^{14}~Msolar with an uncertainty of 0.2 dex. The redshift distribution of clusters extends to z~1.5, and 338,841 clusters have spectroscopic redshifts. Our cluster sample includes most of the rich optical clusters in previous catalogs, more than 95% massive Sunyaev-Zeldovich clusters and 90% ROSAT and eROSITA X-ray clusters. From the light distributions of member galaxies, we derive the dynamical state parameters for 28,038 rich clusters and find no significant evolution of the dynamical state with redshift. We find that the stellar mass of the brightest cluster galaxies grows by a factor of 2 since z=1. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Comments: 17 pages, 14 figures, 2 tables, re-submitted to ApJS after referee's comments being incorporated

Intrinsic mass-richness relation of clusters from THE THREE HUNDRED hydrodynamic simulations

Authors: Mingjing Chen, Weiguang Cui, Wenjuan Fang, Zhonglue Wen

Abstract: The main systematics in cluster cosmology is the uncertainty in the mass-observable relation. In this paper, we focus on the most direct cluster observable in optical surveys, i.e. richness, and constrain the intrinsic mass-richness (MR) relation of clusters in THE THREE HUNDRED hydrodynamic simulations with two runs: GIZMO-SIMBA and GADGET-X. We find that modeling the richness at fixed halo mass… ▽ More The main systematics in cluster cosmology is the uncertainty in the mass-observable relation. In this paper, we focus on the most direct cluster observable in optical surveys, i.e. richness, and constrain the intrinsic mass-richness (MR) relation of clusters in THE THREE HUNDRED hydrodynamic simulations with two runs: GIZMO-SIMBA and GADGET-X. We find that modeling the richness at fixed halo mass with a skewed Gaussian distribution yields a simpler and smaller scatter compared to the commonly used log-normal distribution. Additionally, we observe that baryon models have a significant impact on the scatter, while exhibiting no influence on the mass dependence and a slight effect on the amplitude in the MR relation. We select member galaxies based on both stellar mass $M_\star$ and absolute magnitude $\mathscr{M}$. We demonstrate that the MR relation obtained from these two selections can be converted to each other by using the $M_\star-\mathscr{M}$ relation. Finally, we provide a 7-parameter fitting result comprehensively capturing the dependence of the MR relation on both stellar mass cutoff and redshift. △ Less

Submitted 2 April, 2024; v1 submitted 30 March, 2024; originally announced April 2024.

Comments: accepted to ApJ

Journal ref: The Astrophysical Journal (2024)

ChatGPT v.s. Media Bias: A Comparative Study of GPT-3.5 and Fine-tuned Language Models

Authors: Zehao Wen, Rabih Younes

Abstract: In our rapidly evolving digital sphere, the ability to discern media bias becomes crucial as it can shape public sentiment and influence pivotal decisions. The advent of large language models (LLMs), such as ChatGPT, noted for their broad utility in various natural language processing (NLP) tasks, invites exploration of their efficacy in media bias detection. Can ChatGPT detect media bias? This st… ▽ More In our rapidly evolving digital sphere, the ability to discern media bias becomes crucial as it can shape public sentiment and influence pivotal decisions. The advent of large language models (LLMs), such as ChatGPT, noted for their broad utility in various natural language processing (NLP) tasks, invites exploration of their efficacy in media bias detection. Can ChatGPT detect media bias? This study seeks to answer this question by leveraging the Media Bias Identification Benchmark (MBIB) to assess ChatGPT's competency in distinguishing six categories of media bias, juxtaposed against fine-tuned models such as BART, ConvBERT, and GPT-2. The findings present a dichotomy: ChatGPT performs at par with fine-tuned models in detecting hate speech and text-level context bias, yet faces difficulties with subtler elements of other bias detections, namely, fake news, racial, gender, and cognitive biases. △ Less

Submitted 29 March, 2024; originally announced March 2024.

Comments: 9 pages, 1 figure, published on Applied and Computational Engineering

Journal ref: ACE (2023) Vol. 21: 249-257.

Accelerating Federated Learning by Selecting Beneficial Herd of Local Gradients

Authors: Ping Luo, Xiaoge Deng, Ziqing Wen, Tao Sun, Dongsheng Li

Abstract: Federated Learning (FL) is a distributed machine learning framework in communication network systems. However, the systems' Non-Independent and Identically Distributed (Non-IID) data negatively affect the convergence efficiency of the global model, since only a subset of these data samples are beneficial for model convergence. In pursuit of this subset, a reliable approach involves determining a m… ▽ More Federated Learning (FL) is a distributed machine learning framework in communication network systems. However, the systems' Non-Independent and Identically Distributed (Non-IID) data negatively affect the convergence efficiency of the global model, since only a subset of these data samples are beneficial for model convergence. In pursuit of this subset, a reliable approach involves determining a measure of validity to rank the samples within the dataset. In this paper, We propose the BHerd strategy which selects a beneficial herd of local gradients to accelerate the convergence of the FL model. Specifically, we map the distribution of the local dataset to the local gradients and use the Herding strategy to obtain a permutation of the set of gradients, where the more advanced gradients in the permutation are closer to the average of the set of gradients. These top portion of the gradients will be selected and sent to the server for global aggregation. We conduct experiments on different datasets, models and scenarios by building a prototype system, and experimental results demonstrate that our BHerd strategy is effective in selecting beneficial local gradients to mitigate the effects brought by the Non-IID dataset, thus accelerating model convergence. △ Less

Submitted 25 March, 2024; originally announced March 2024.

Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild

Authors: Zhuofan Wen, Fengyu Zhang, Siyuan Zhang, Haiyang Sun, Mingyu Xu, Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao

Abstract: Multimodal fusion is a significant method for most multimodal tasks. With the recent surge in the number of large pre-trained models, combining both multimodal fusion methods and pre-trained model features can achieve outstanding performance in many multimodal tasks. In this paper, we present our approach, which leverages both advantages for addressing the task of Expression (Expr) Recognition and… ▽ More Multimodal fusion is a significant method for most multimodal tasks. With the recent surge in the number of large pre-trained models, combining both multimodal fusion methods and pre-trained model features can achieve outstanding performance in many multimodal tasks. In this paper, we present our approach, which leverages both advantages for addressing the task of Expression (Expr) Recognition and Valence-Arousal (VA) Estimation. We evaluate the Aff-Wild2 database using pre-trained models, then extract the final hidden layers of the models as features. Following preprocessing and interpolation or convolution to align the extracted features, different models are employed for modal fusion. Our code is available at GitHub - FulgenceWen/ABAW6th. △ Less

Submitted 22 March, 2024; originally announced March 2024.

Formalization of Complexity Analysis of the First-order Algorithms for Convex Optimization

Authors: Chenyi Li, Ziyu Wang, Wanyi He, Yuxuan Wu, Shengyang Xu, Zaiwen Wen

Abstract: The convergence rate of various first-order optimization algorithms is a pivotal concern within the numerical optimization community, as it directly reflects the efficiency of these algorithms across different optimization problems. Our goal is making a significant step forward in the formal mathematical representation of optimization techniques using the Lean4 theorem prover. We first formalize t… ▽ More The convergence rate of various first-order optimization algorithms is a pivotal concern within the numerical optimization community, as it directly reflects the efficiency of these algorithms across different optimization problems. Our goal is making a significant step forward in the formal mathematical representation of optimization techniques using the Lean4 theorem prover. We first formalize the gradient for smooth functions and the subgradient for convex functions on a Hilbert space, laying the groundwork for the accurate formalization of algorithmic structures. Then, we extend our contribution by proving several properties of differentiable convex functions that have not yet been formalized in Mathlib. Finally, a comprehensive formalization of these algorithms is presented. These developments are not only noteworthy on their own but also serve as essential precursors to the formalization of a broader spectrum of numerical algorithms and their applications in machine learning as well as many other areas. △ Less

Submitted 21 July, 2024; v1 submitted 17 March, 2024; originally announced March 2024.

ACM Class: G.1.6

Strength Lies in Differences! Towards Effective Non-collaborative Dialogues via Tailored Strategy Planning

Authors: Tong Zhang, Chen Huang, Yang Deng, Hongru Liang, Jia Liu, Zujie Wen, Wenqiang Lei, Tat-Seng Chua

Abstract: We investigate non-collaborative dialogue agents, which are expected to engage in strategic conversations with diverse users, for securing a mutual agreement that leans favorably towards the system's objectives. This poses two main challenges for existing dialogue agents: 1) The inability to integrate user-specific characteristics into the strategic planning, and 2) The difficulty of training stra… ▽ More We investigate non-collaborative dialogue agents, which are expected to engage in strategic conversations with diverse users, for securing a mutual agreement that leans favorably towards the system's objectives. This poses two main challenges for existing dialogue agents: 1) The inability to integrate user-specific characteristics into the strategic planning, and 2) The difficulty of training strategic planners that can be generalized to diverse users. To address these challenges, we propose Trip to enhance the capability in tailored strategic planning, incorporating a user-aware strategic planning module and a population-based training paradigm. Through experiments on benchmark non-collaborative dialogue tasks, we demonstrate the effectiveness of Trip in catering to diverse users. △ Less

Submitted 6 May, 2024; v1 submitted 11 March, 2024; originally announced March 2024.

Comments: V2: 20 pages, 8 figures, and 20 tables

Low-Resource Court Judgment Summarization for Common Law Systems

Authors: Shuaiqi Liu, Jiannong Cao, Yicong Li, Ruosong Yang, Zhiyuan Wen

Abstract: Common law courts need to refer to similar precedents' judgments to inform their current decisions. Generating high-quality summaries of court judgment documents can facilitate legal practitioners to efficiently review previous cases and assist the general public in accessing how the courts operate and how the law is applied. Previous court judgment summarization research focuses on civil law or a… ▽ More Common law courts need to refer to similar precedents' judgments to inform their current decisions. Generating high-quality summaries of court judgment documents can facilitate legal practitioners to efficiently review previous cases and assist the general public in accessing how the courts operate and how the law is applied. Previous court judgment summarization research focuses on civil law or a particular jurisdiction's judgments. However, judges can refer to the judgments from all common law jurisdictions. Current summarization datasets are insufficient to satisfy the demands of summarizing precedents across multiple jurisdictions, especially when labeled data are scarce for many jurisdictions. To address the lack of datasets, we present CLSum, the first dataset for summarizing multi-jurisdictional common law court judgment documents. Besides, this is the first court judgment summarization work adopting large language models (LLMs) in data augmentation, summary generation, and evaluation. Specifically, we design an LLM-based data augmentation method incorporating legal knowledge. We also propose a legal knowledge enhanced evaluation metric based on LLM to assess the quality of generated judgment summaries. Our experimental results verify that the LLM-based summarization methods can perform well in the few-shot and zero-shot settings. Our LLM-based data augmentation method can mitigate the impact of low data resources. Furthermore, we carry out comprehensive comparative experiments to find essential model components and settings that are capable of enhancing summarization performance. △ Less

Submitted 7 March, 2024; originally announced March 2024.

Comments: First submitted to Information Processing and Management on Oct. 29, 2023. Major Revision submitted on Mar.6, 2024

ACM Class: I.2.7; I.7

How Transformers Learn Diverse Attention Correlations in Masked Vision Pretraining

Authors: Yu Huang, Zixin Wen, Yuejie Chi, Yingbin Liang

Abstract: Masked reconstruction, which predicts randomly masked patches from unmasked ones, has emerged as an important approach in self-supervised pretraining. However, the theoretical understanding of masked pretraining is rather limited, especially for the foundational architecture of transformers. In this paper, to the best of our knowledge, we provide the first end-to-end theoretical guarantee of learn… ▽ More Masked reconstruction, which predicts randomly masked patches from unmasked ones, has emerged as an important approach in self-supervised pretraining. However, the theoretical understanding of masked pretraining is rather limited, especially for the foundational architecture of transformers. In this paper, to the best of our knowledge, we provide the first end-to-end theoretical guarantee of learning one-layer transformers in masked reconstruction self-supervised pretraining. On the conceptual side, we posit a mechanism of how transformers trained with masked vision pretraining objectives produce empirically observed local and diverse attention patterns, on data distributions with spatial structures that highlight feature-position correlations. On the technical side, our end-to-end characterization of training dynamics in softmax-attention models simultaneously accounts for input and position embeddings, which is developed based on a careful analysis tracking the interplay between feature-wise and position-wise attention correlations. △ Less

Submitted 4 June, 2024; v1 submitted 4 March, 2024; originally announced March 2024.

Comments: v2 polishes writing

Collision-Free Robot Navigation in Crowded Environments using Learning based Convex Model Predictive Control

Authors: Zhuanglei Wen, Mingze Dong, Xiai Chen

Abstract: Navigating robots safely and efficiently in crowded and complex environments remains a significant challenge. However, due to the dynamic and intricate nature of these settings, planning efficient and collision-free paths for robots to track is particularly difficult. In this paper, we uniquely bridge the robot's perception, decision-making and control processes by utilizing the convex obstacle-fr… ▽ More Navigating robots safely and efficiently in crowded and complex environments remains a significant challenge. However, due to the dynamic and intricate nature of these settings, planning efficient and collision-free paths for robots to track is particularly difficult. In this paper, we uniquely bridge the robot's perception, decision-making and control processes by utilizing the convex obstacle-free region computed from 2D LiDAR data. The overall pipeline is threefold: (1) We proposes a robot navigation framework that utilizes deep reinforcement learning (DRL), conceptualizing the observation as the convex obstacle-free region, a departure from general reliance on raw sensor inputs. (2) We design the action space, derived from the intersection of the robot's kinematic limits and the convex region, to enable efficient sampling of inherently collision-free reference points. These actions assists in guiding the robot to move towards the goal and interact with other obstacles during navigation. (3) We employ model predictive control (MPC) to track the trajectory formed by the reference points while satisfying constraints imposed by the convex obstacle-free region and the robot's kinodynamic limits. The effectiveness of proposed improvements has been validated through two sets of ablation studies and a comparative experiment against the Timed Elastic Band (TEB), demonstrating improved navigation performance in crowded and complex environments. △ Less

Submitted 14 March, 2024; v1 submitted 3 March, 2024; originally announced March 2024.

Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models

Authors: Ziting Wen, Oscar Pizarro, Stefan Williams

Abstract: Fine-tuning the pre-trained model with active learning holds promise for reducing annotation costs. However, this combination introduces significant computational costs, particularly with the growing scale of pre-trained models. Recent research has proposed proxy-based active learning, which pre-computes features to reduce computational costs. Yet, this approach often incurs a significant loss in… ▽ More Fine-tuning the pre-trained model with active learning holds promise for reducing annotation costs. However, this combination introduces significant computational costs, particularly with the growing scale of pre-trained models. Recent research has proposed proxy-based active learning, which pre-computes features to reduce computational costs. Yet, this approach often incurs a significant loss in active learning performance, which may even outweigh the computational cost savings. In this paper, we argue the performance drop stems not only from pre-computed features' inability to distinguish between categories of labeled samples, resulting in the selection of redundant samples but also from the tendency to compromise valuable pre-trained information when fine-tuning with samples selected through the proxy model. To address this issue, we propose a novel method called aligned selection via proxy to update pre-computed features while selecting a proper training method to inherit valuable pre-training information. Extensive experiments validate that our method significantly improves the total cost of efficient active learning while maintaining computational efficiency. △ Less

Submitted 2 March, 2024; originally announced March 2024.

Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning

Authors: Ruiqian Nai, Zixin Wen, Ji Li, Yuanzhi Li, Yang Gao

Abstract: In representation learning, a disentangled representation is highly desirable as it encodes generative factors of data in a separable and compact pattern. Researchers have advocated leveraging disentangled representations to complete downstream tasks with encouraging empirical evidence. This paper further investigates the necessity of disentangled representation in downstream applications. Specifi… ▽ More In representation learning, a disentangled representation is highly desirable as it encodes generative factors of data in a separable and compact pattern. Researchers have advocated leveraging disentangled representations to complete downstream tasks with encouraging empirical evidence. This paper further investigates the necessity of disentangled representation in downstream applications. Specifically, we show that dimension-wise disentangled representations are unnecessary on a fundamental downstream task, abstract visual reasoning. We provide extensive empirical evidence against the necessity of disentanglement, covering multiple datasets, representation learning methods, and downstream network architectures. Furthermore, our findings suggest that the informativeness of representations is a better indicator of downstream performance than disentanglement. Finally, the positive correlation between informativeness and disentanglement explains the claimed usefulness of disentangled representations in previous works. The source code is available at https://github.com/Richard-coder-Nai/disentanglement-lib-necessity.git. △ Less

Submitted 1 March, 2024; originally announced March 2024.

Comments: Accepted to AAAI-2024

A Search for Radio Pulsars in Supernova Remnants Using FAST with One Pulsar Discovered

Authors: Zhen Zhang, Wen-Ming Yan, Jian-Ping Yuan, Na Wang, Jun-Tao Bai, Zhi-Gang Wen, Bao-Da Li, Jin-Tao Xie, De Zhao, Yu-Bin Wang, Nan-Nan Zhai

Abstract: We report on the results of a search for radio pulsars in five supernova remnants (SNRs) with FAST. The observations were made using the 19-beam receiver in the Snapshot mode. The integration time for each pointing is 10 min. We discovered a new pulsar PSR J1845$-$0306 which has a spin period of 983.6 ms and a dispersion measure of 444.6$\pm$2.0 cm$^{-3}$ pc in observations of SNR G29.6+0.1. To ju… ▽ More We report on the results of a search for radio pulsars in five supernova remnants (SNRs) with FAST. The observations were made using the 19-beam receiver in the Snapshot mode. The integration time for each pointing is 10 min. We discovered a new pulsar PSR J1845$-$0306 which has a spin period of 983.6 ms and a dispersion measure of 444.6$\pm$2.0 cm$^{-3}$ pc in observations of SNR G29.6+0.1. To judge the association between the pulsar and the SNR, further verification is needed. We also re-detected some known pulsars in the data from SNRs G29.6+0.1 and G29.7$-$0.3. No pulsars were detected in observations of other three SNRs. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Comments: 6 pages, 2 figures, 2 tables published in CPL

Journal ref: Chin. Phys. Lett. 2024, 41 (2): 029701 February 2024

Two-stage Generative Question Answering on Temporal Knowledge Graph Using Large Language Models

Authors: Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, Dongsheng Li

Abstract: Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper fir… ▽ More Temporal knowledge graph question answering (TKGQA) poses a significant challenge task, due to the temporal constraints hidden in questions and the answers sought from dynamic structured knowledge. Although large language models (LLMs) have made considerable progress in their reasoning ability over structured data, their application to the TKGQA task is a relatively unexplored area. This paper first proposes a novel generative temporal knowledge graph question answering framework, GenTKGQA, which guides LLMs to answer temporal questions through two phases: Subgraph Retrieval and Answer Generation. First, we exploit LLM's intrinsic knowledge to mine temporal constraints and structural links in the questions without extra training, thus narrowing down the subgraph search space in both temporal and structural dimensions. Next, we design virtual knowledge indicators to fuse the graph neural network signals of the subgraph and the text representations of the LLM in a non-shallow way, which helps the open-source LLM deeply understand the temporal order and structural dependencies among the retrieved facts through instruction tuning. Experimental results on two widely used datasets demonstrate the superiority of our model. △ Less

Submitted 23 July, 2024; v1 submitted 26 February, 2024; originally announced February 2024.

Comments: Accepted by ACL(Findings) 2024

Investigation of profile shifting and subpulse movement in PSR J0344-0901 with FAST

Authors: H. M. Tedila, R. Yuen, N. Wang, D. Li, Z. G. Wen, W. M. Yan, J. P. Yuan, X. H. Han, P. Wang, W. W. Zhu, S. J. Dang, S. Q. Wang, J. T. Xie, Q. D. Wu, Sh. Khasanov, FAST Collaboration

Abstract: We report two phenomena detected in PSR J0344$-$0901 from two observations conducted at frequency centered at 1.25 GHz using the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The first phenomenon manifests as shifting in the pulse emission to later longitudinal phases and then gradually returns to its original location. The event lasts for about 216 pulse periods, with an average s… ▽ More We report two phenomena detected in PSR J0344$-$0901 from two observations conducted at frequency centered at 1.25 GHz using the Five-hundred-meter Aperture Spherical radio Telescope (FAST). The first phenomenon manifests as shifting in the pulse emission to later longitudinal phases and then gradually returns to its original location. The event lasts for about 216 pulse periods, with an average shift of about $0.7^\circ$ measured at the peak of the integrated profile. Changes in the polarization position angle (PPA) are detected around the trailing edge of the profile, together with an increase in the profile width. The second phenomenon is characterized by the apparent movement of subpulses, which results in different subpulse track patterns across the profile window. For the first time in this pulsar, we identify four emission modes, each with unique subpulse movement, and determine the pattern periods for three of the emission modes. Pulse nulling was not detected. Modeling of the changes in the PPA using the rotating vector model gives an inclination angle of $75.12^\circ \pm 3.80^\circ$ and an impact parameter of $-3.17^\circ \pm 5.32^\circ$ for this pulsar. We speculate that the subpulse movement may be related to the shifting of the pulse emission. △ Less

Submitted 22 February, 2024; originally announced February 2024.

SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning

Authors: Zhihao Wen, Jie Zhang, Yuan Fang

Abstract: Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiven… ▽ More Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiveness of these Transformer-based LLMs, resulting in suboptimal performances in downstream tasks. In this paper, we present SIBO, which is a SImple BOoster to enhance PEFT, by injecting an initial residual. SIBO is straightforward and readily extensible to a range of state-of-the-art PEFT techniques to alleviate over-smoothing and enhance performance. Extensive experiments on 22 benchmark datasets demonstrate that SIBO significantly enhances the performance of various strong baselines, achieving up to 15.7% and 23.5% improvement over existing PEFT methods on the arithmetic and commonsense reasoning tasks, respectively. △ Less

Submitted 1 June, 2024; v1 submitted 19 February, 2024; originally announced February 2024.

Comments: Accepted by ACL 2024, 17 pages

Rethinking Large Language Model Architectures for Sequential Recommendations

Authors: Hanbing Wang, Xiaorui Liu, Wenqi Fan, Xiangyu Zhao, Venkataramana Kini, Devendra Yadav, Fei Wang, Zhen Wen, Jiliang Tang, Hui Liu

Abstract: Recently, sequential recommendation has been adapted to the LLM paradigm to enjoy the power of LLMs. LLM-based methods usually formulate recommendation information into natural language and the model is trained to predict the next item in an auto-regressive manner. Despite their notable success, the substantial computational overhead of inference poses a significant obstacle to their real-world ap… ▽ More Recently, sequential recommendation has been adapted to the LLM paradigm to enjoy the power of LLMs. LLM-based methods usually formulate recommendation information into natural language and the model is trained to predict the next item in an auto-regressive manner. Despite their notable success, the substantial computational overhead of inference poses a significant obstacle to their real-world applicability. In this work, we endeavor to streamline existing LLM-based recommendation models and propose a simple yet highly effective model Lite-LLM4Rec. The primary goal of Lite-LLM4Rec is to achieve efficient inference for the sequential recommendation task. Lite-LLM4Rec circumvents the beam search decoding by using a straight item projection head for ranking scores generation. This design stems from our empirical observation that beam search decoding is ultimately unnecessary for sequential recommendations. Additionally, Lite-LLM4Rec introduces a hierarchical LLM structure tailored to efficiently handle the extensive contextual information associated with items, thereby reducing computational overhead while enjoying the capabilities of LLMs. Experiments on three publicly available datasets corroborate the effectiveness of Lite-LLM4Rec in both performance and inference efficiency (notably 46.8% performance improvement and 97.28% efficiency improvement on ML-1m) over existing LLM-based methods. Our implementations will be open sourced. △ Less

Submitted 14 February, 2024; originally announced February 2024.

Comments: 8 pages, 5 figures, conference

A single-snapshot inverse solver for two-species graph model of tau pathology spreading in human Alzheimer disease

Authors: Zheyu Wen, Ali Ghafouri, George Biros

Abstract: We propose a method that uses a two-species ordinary differential equation (ODE) biophysical model to characterize misfolded tau (or simply tau) protein spreading in Alzheimer disease (AD) and calibrates it from clinical data. The unknown model parameters are the initial condition (IC) for tau and three scalar parameters representing the migration, proliferation, and clearance of tau proteins. Dri… ▽ More We propose a method that uses a two-species ordinary differential equation (ODE) biophysical model to characterize misfolded tau (or simply tau) protein spreading in Alzheimer disease (AD) and calibrates it from clinical data. The unknown model parameters are the initial condition (IC) for tau and three scalar parameters representing the migration, proliferation, and clearance of tau proteins. Driven by imaging data, these parameters are estimated by formulating a constrained optimization problem with a sparsity regularization for the IC. This optimization problem is solved with a projection-based quasi-Newton algorithm. We investigate the sensitivity of our method to different algorithm parameters. We evaluate the performance of our method on both synthetic and clinical data. The latter comprises cases from the AD Neuroimaging Initiative (ADNI) and Harvard Aging Brain Study (HABS) datasets: 455 cognitively normal (CN), 212 mild cognitive impairment (MCI), and 45 AD subjects. We compare the performance of our approach to the commonly used Fisher-Kolmogorov (FK) model with a fixed IC at the entorhinal cortex (EC). Our method demonstrates an average improvement of 25.7% relative error compared to the FK model on the AD dataset. HFK also achieves an R-squared score of 0.664 for fitting AD data compared with 0.55 from FK model results. Furthermore, for cases that have longitudinal data, we estimate a subject-specific AD onset time. △ Less

Submitted 9 February, 2024; originally announced February 2024.

Comments: 10 pages, 6 figures

AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback

Authors: Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, Minlie Huang

Abstract: The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over a finite state machine (FSM) that solve… ▽ More The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over a finite state machine (FSM) that solves problems through autonomous executions and transitions over disentangled modules. This allows humans to provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop AMOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically constructed from various public datasets and enables AMOR to generalize across different knowledge environments, while the latter tailors AMOR to specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of AMOR to strong baselines, thanks to its FSM-based reasoning and process feedback mechanism. △ Less

Submitted 2 February, 2024; originally announced February 2024.

Comments: Work in progress

Few-Shot Learning on Graphs: from Meta-learning to Pre-training and Prompting

Authors: Xingtong Yu, Yuan Fang, Zemin Liu, Yuxia Wu, Zhihao Wen, Jianyuan Bo, Xinming Zhang, Steven C. H. Hoi

Abstract: Graph representation learning, a critical step in graph-centric tasks, has seen significant advancements. Earlier techniques often operate in an end-to-end setting, where performance heavily relies on the availability of ample labeled data. This constraint has spurred the emergence of few-shot learning on graphs, where only a few task-specific labels are available for each task. Given the extensiv… ▽ More Graph representation learning, a critical step in graph-centric tasks, has seen significant advancements. Earlier techniques often operate in an end-to-end setting, where performance heavily relies on the availability of ample labeled data. This constraint has spurred the emergence of few-shot learning on graphs, where only a few task-specific labels are available for each task. Given the extensive literature in this field, this survey endeavors to synthesize recent developments, provide comparative insights, and identify future directions. We systematically categorize existing studies into three major families: meta-learning approaches, pre-training approaches, and hybrid approaches, with a finer-grained classification in each family to aid readers in their method selection process. Within each category, we analyze the relationships among these methods and compare their strengths and limitations. Finally, we outline prospective future directions for few-shot learning on graphs to catalyze continued innovation in this field. △ Less

Submitted 2 March, 2024; v1 submitted 2 February, 2024; originally announced February 2024.

Multi-granularity Correspondence Learning from Long-term Noisy Videos

Authors: Yijie Lin, Jie Zhang, Zhenyu Huang, Jia Liu, Zujie Wen, Xi Peng

Abstract: Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To… ▽ More Existing video-language studies mainly focus on learning short video clips, leaving long-term temporal dependencies rarely explored due to over-high computational cost of modeling long videos. To address this issue, one feasible solution is learning the correspondence between video clips and captions, which however inevitably encounters the multi-granularity noisy correspondence (MNC) problem. To be specific, MNC refers to the clip-caption misalignment (coarse-grained) and frame-word misalignment (fine-grained), hindering temporal learning and video understanding. In this paper, we propose NOise Robust Temporal Optimal traNsport (Norton) that addresses MNC in a unified optimal transport (OT) framework. In brief, Norton employs video-paragraph and clip-caption contrastive losses to capture long-term dependencies based on OT. To address coarse-grained misalignment in video-paragraph contrast, Norton filters out the irrelevant clips and captions through an alignable prompt bucket and realigns asynchronous clip-caption pairs based on transport distance. To address the fine-grained misalignment, Norton incorporates a soft-maximum operator to identify crucial words and key frames. Additionally, Norton exploits the potential faulty negative samples in clip-caption contrast by rectifying the alignment target with OT assignment to ensure precise temporal modeling. Extensive experiments on video retrieval, videoQA, and action segmentation verify the effectiveness of our method. Code is available at https://lin-yijie.github.io/projects/Norton. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: Accepted by ICLR 2024 (oral)

Learning Representations for Clustering via Partial Information Discrimination and Cross-Level Interaction

Authors: Hai-Xin Zhang, Dong Huang, Hua-Bao Ling, Guang-Yu Zhang, Wei-jun Sun, Zi-hao Wen

Abstract: In this paper, we present a novel deep image clustering approach termed PICI, which enforces the partial information discrimination and the cross-level interaction in a joint learning framework. In particular, we leverage a Transformer encoder as the backbone, through which the masked image modeling with two paralleled augmented views is formulated. After deriving the class tokens from the masked… ▽ More In this paper, we present a novel deep image clustering approach termed PICI, which enforces the partial information discrimination and the cross-level interaction in a joint learning framework. In particular, we leverage a Transformer encoder as the backbone, through which the masked image modeling with two paralleled augmented views is formulated. After deriving the class tokens from the masked images by the Transformer encoder, three partial information learning modules are further incorporated, including the PISD module for training the auto-encoder via masked image reconstruction, the PICD module for employing two levels of contrastive learning, and the CLI module for mutual interaction between the instance-level and cluster-level subspaces. Extensive experiments have been conducted on six real-world image datasets, which demononstrate the superior clustering performance of the proposed PICI approach over the state-of-the-art deep clustering approaches. The source code is available at https://github.com/Regan-Zhang/PICI. △ Less

Submitted 24 January, 2024; originally announced January 2024.

The Study of Mode Switching behavior of PSR J0614+2229 Using the Parkes Ultra-wideband Receiver Observations

Authors: Yanqing Cai, Shijun Dang, Rai Yuen, Lunhua Shang, Feifei Kou, Jianping Yuan, Lei Zhang, Zurong Zhou, Na Wang, Qingying Li, Zhigang Wen, Wenming Yan, Shuangqiang Wang, Shengnan Sun, Habtamu Menberu Tedila, Shuo Xiao, Xin Xu, Rushuang Zhao, Qijun Zhi, Aijun Dong, Bing Zhang, Wei Li, Yingying Ren, Yujia Liu

Abstract: In this paper, we presented a detailed single pulse and polarization study of PSR J0614+2229 based on the archived data observed on 2019 August 15 (MJD 58710) and September 12 (MJD 58738) using the Ultra-wideband Low-frequency Receiver on the Parkes radio telescope. The single-pulse sequences show that this pulsar switches between two emission states, in which the emission of state A occurs earlie… ▽ More In this paper, we presented a detailed single pulse and polarization study of PSR J0614+2229 based on the archived data observed on 2019 August 15 (MJD 58710) and September 12 (MJD 58738) using the Ultra-wideband Low-frequency Receiver on the Parkes radio telescope. The single-pulse sequences show that this pulsar switches between two emission states, in which the emission of state A occurs earlier than that of state B in pulse longitude. We found that the variation in relative brightness between the two states is related to time and both states follow a simple power law very well. Based on the phase-aligned multi-frequency profiles, we found that there is a significant difference in the distributions of spectral index across the emission regions of the two states. Furthermore, we obtained the emission height evolution for the two emission states and found that, at a fixed frequency, the emission height of state A is higher than that of state B. What is even more interesting is that the emission heights of both states A and B have not changed with frequency. Our results suggest that the mode switching of this pulsar is possibly caused by changes in the emission heights that alter the distributions of spectral index across the emission regions of states A and B resulting in the frequency-dependent behaviors, i.e., intensity and pulse width. △ Less

Submitted 17 January, 2024; originally announced January 2024.

3D orientation super-resolution spatial-frequency-shift microscopy

Authors: Xiaowei Liu, Mingwei Tang, Ning Zhou, Chenlei Pang, Zhong Wen, Xu Liu, Qing Yang

Abstract: Super-resolution mapping of the 3D orientation of fluorophores reveals the alignment of biological structures where the fluorophores are tightly attached, and thus plays a vital role in studying the organization and dynamics of bio-complexes. However, current super-resolution imaging techniques are either limited to 2D orientation mapping or suffer from slow speed and the requirement of special la… ▽ More Super-resolution mapping of the 3D orientation of fluorophores reveals the alignment of biological structures where the fluorophores are tightly attached, and thus plays a vital role in studying the organization and dynamics of bio-complexes. However, current super-resolution imaging techniques are either limited to 2D orientation mapping or suffer from slow speed and the requirement of special labels in 3D orientation mapping. Here, we propose a novel polarized virtual spatial-frequency-shift effect to overcome these restrictions to achieve a universal 3D orientation super-resolution mapping capability. To demonstrate the mechanism, we simulate the imaging process and reconstruct the spatial-angular information for sparsely distributed dipoles with random 3D orientations and microfilament-like structures decorated with fluorophores oriented parallel to them. The 3D orientation distribution can be recovered with a doubled spatial resolution and an average angular precision of up to 2.39 degrees. The performance of the approach with noise has also been analyzed considering real implementation. △ Less

Submitted 22 January, 2024; v1 submitted 17 January, 2024; originally announced January 2024.

Comments: 22 pages, 5 figures

Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems

Authors: Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, Zhixing Tan, Junwu Xiong, Xinyu Kong, Zujie Wen, Ke Xu, Qi Li

Abstract: Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta,… ▽ More Large language models (LLMs) have strong capabilities in solving diverse natural language processing tasks. However, the safety and security issues of LLM systems have become the major obstacle to their widespread application. Many studies have extensively investigated risks in LLM systems and developed the corresponding mitigation strategies. Leading-edge enterprises such as OpenAI, Google, Meta, and Anthropic have also made lots of efforts on responsible LLMs. Therefore, there is a growing need to organize the existing studies and establish comprehensive taxonomies for the community. In this paper, we delve into four essential modules of an LLM system, including an input module for receiving prompts, a language model trained on extensive corpora, a toolchain module for development and deployment, and an output module for exporting LLM-generated content. Based on this, we propose a comprehensive taxonomy, which systematically analyzes potential risks associated with each module of an LLM system and discusses the corresponding mitigation strategies. Furthermore, we review prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. We hope that this paper can help LLM participants embrace a systematic perspective to build their responsible LLM systems. △ Less

Submitted 11 January, 2024; originally announced January 2024.

POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation

Authors: Shilong Pan, Zhiliang Tian, Liang Ding, Zhen Huang, Zhihua Wen, Dongsheng Li

Abstract: Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data, prompting research into unsupervised methods. Unsupervised neural machine translation (UNMT) methods, including back-translation, transfer learning, and pivot-based translation, offer practical solutions for LRL translation, but they are hindered by issues like synthetic data noise,… ▽ More Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data, prompting research into unsupervised methods. Unsupervised neural machine translation (UNMT) methods, including back-translation, transfer learning, and pivot-based translation, offer practical solutions for LRL translation, but they are hindered by issues like synthetic data noise, language bias, and error propagation, which can potentially be mitigated by Large Language Models (LLMs). LLMs have advanced NMT with in-context learning (ICL) and supervised fine-tuning methods, but insufficient training data results in poor performance in LRLs. We argue that LLMs can mitigate the linguistic noise with auxiliary languages to improve translations in LRLs. In this paper, we propose Probability-driven Meta-graph Prompter (POMP), a novel approach employing a dynamic, sampling-based graph of multiple auxiliary languages to enhance LLMs' translation capabilities for LRLs. POMP involves constructing a directed acyclic meta-graph for each source language, from which we dynamically sample multiple paths to prompt LLMs to mitigate the linguistic noise and improve translations during training. We use the BLEURT metric to evaluate the translations and back-propagate rewards, estimated by scores, to update the probabilities of auxiliary languages in the paths. Our experiments show significant improvements in the translation quality of three LRLs, demonstrating the effectiveness of our approach. △ Less

Submitted 16 January, 2024; v1 submitted 10 January, 2024; originally announced January 2024.

Homophily-Related: Adaptive Hybrid Graph Filter for Multi-View Graph Clustering

Authors: Zichen Wen, Yawen Ling, Yazhou Ren, Tianyi Wu, Jianpeng Chen, Xiaorong Pu, Zhifeng Hao, Lifang He

Abstract: Recently there is a growing focus on graph data, and multi-view graph clustering has become a popular area of research interest. Most of the existing methods are only applicable to homophilous graphs, yet the extensive real-world graph data can hardly fulfill the homophily assumption, where the connected nodes tend to belong to the same class. Several studies have pointed out that the poor perform… ▽ More Recently there is a growing focus on graph data, and multi-view graph clustering has become a popular area of research interest. Most of the existing methods are only applicable to homophilous graphs, yet the extensive real-world graph data can hardly fulfill the homophily assumption, where the connected nodes tend to belong to the same class. Several studies have pointed out that the poor performance on heterophilous graphs is actually due to the fact that conventional graph neural networks (GNNs), which are essentially low-pass filters, discard information other than the low-frequency information on the graph. Nevertheless, on certain graphs, particularly heterophilous ones, neglecting high-frequency information and focusing solely on low-frequency information impedes the learning of node representations. To break this limitation, our motivation is to perform graph filtering that is closely related to the homophily degree of the given graph, with the aim of fully leveraging both low-frequency and high-frequency signals to learn distinguishable node embedding. In this work, we propose Adaptive Hybrid Graph Filter for Multi-View Graph Clustering (AHGFC). Specifically, a graph joint process and graph joint aggregation matrix are first designed by using the intrinsic node features and adjacency relationship, which makes the low and high-frequency signals on the graph more distinguishable. Then we design an adaptive hybrid graph filter that is related to the homophily degree, which learns the node embedding based on the graph joint aggregation matrix. After that, the node embedding of each view is weighted and fused into a consensus embedding for the downstream task. Experimental results show that our proposed model performs well on six datasets containing homophilous and heterophilous graphs. △ Less

Submitted 5 January, 2024; originally announced January 2024.

Comments: Accepted by AAAI2024

Deep Unfolding Network with Spatial Alignment for multi-modal MRI reconstruction

Authors: Hao Zhang, Qi Wang, Jun Shi, Shihui Ying, Zhijie Wen

Abstract: Multi-modal Magnetic Resonance Imaging (MRI) offers complementary diagnostic information, but some modalities are limited by the long scanning time. To accelerate the whole acquisition process, MRI reconstruction of one modality from highly undersampled k-space data with another fully-sampled reference modality is an efficient solution. However, the misalignment between modalities, which is common… ▽ More Multi-modal Magnetic Resonance Imaging (MRI) offers complementary diagnostic information, but some modalities are limited by the long scanning time. To accelerate the whole acquisition process, MRI reconstruction of one modality from highly undersampled k-space data with another fully-sampled reference modality is an efficient solution. However, the misalignment between modalities, which is common in clinic practice, can negatively affect reconstruction quality. Existing deep learning-based methods that account for inter-modality misalignment perform better, but still share two main common limitations: (1) The spatial alignment task is not adaptively integrated with the reconstruction process, resulting in insufficient complementarity between the two tasks; (2) the entire framework has weak interpretability. In this paper, we construct a novel Deep Unfolding Network with Spatial Alignment, termed DUN-SA, to appropriately embed the spatial alignment task into the reconstruction process. Concretely, we derive a novel joint alignment-reconstruction model with a specially designed cross-modal spatial alignment term. By relaxing the model into cross-modal spatial alignment and multi-modal reconstruction tasks, we propose an effective algorithm to solve this model alternatively. Then, we unfold the iterative steps of the proposed algorithm and design corresponding network modules to build DUN-SA with interpretability. Through end-to-end training, we effectively compensate for spatial misalignment using only reconstruction loss, and utilize the progressively aligned reference modality to provide inter-modality prior to improve the reconstruction of the target modality. Comprehensive experiments on three real datasets demonstrate that our method exhibits superior reconstruction performance compared to state-of-the-art methods. △ Less

Submitted 28 December, 2023; originally announced December 2023.

Anderson Accelerated Gauss-Newton-guided deep learning for nonlinear inverse problems with Application to Electrical Impedance Tomography

Authors: Qingping Zhou, Guixian Xu, Zhexin Wen, Hongqiao Wang

Abstract: Physics-guided deep learning is an important prevalent research topic in scientific machine learning, which has tremendous potential in various complex applications including science and engineering. In these applications, data is expensive to acquire and high accuracy is required for making decisions. In this work, we introduce an efficient physics-guided deep learning framework for the variation… ▽ More Physics-guided deep learning is an important prevalent research topic in scientific machine learning, which has tremendous potential in various complex applications including science and engineering. In these applications, data is expensive to acquire and high accuracy is required for making decisions. In this work, we introduce an efficient physics-guided deep learning framework for the variational modeling of nonlinear inverse problems, which is then applied to solve an electrical impedance tomography (EIT) inverse problem. The framework is achieved by unrolling the proposed Anderson accelerated Gauss-Newton (GNAA) algorithm into an end-to-end deep learning method. Firstly, we show the convergence of the GNAA algorithm in both cases: Anderson depth is equal to one and Anderson depth is greater than one. Then, we propose three types of strategies by combining the complementary strengths of GNAA and deep learning: GNAA of learned regularization (GNAA-LRNet), where the singular values of the regularization matrix are learned by a deep neural network; GNAA of learned proximity (GNAA-LPNet), where the regularization proximal operator is learned by using a deep neural network; GNAA of plug-and-play method (GNAA-PnPNet) where the regularization proximal operator is replaced by a pre-trained deep denoisers. Lastly, we present some numerical experiments to illustrate that the proposed approaches greatly improve the convergence rate and the quality of inverse solutions. △ Less

Submitted 19 December, 2023; originally announced December 2023.

MSC Class: 78A46; 68U10; 68T07

Adaptive Isogeometric Topology Optimization of Shell Structures based on PHT-splines

Authors: Zepeng Wen, Qiong Pan, Xiaoya Zhai, Hongmei Kang, Falai Chen

Abstract: This paper proposes an Adaptive Isogeometric Topology Optimization framework for shell structures based on PHT-splines (PHT-AITO). In this framework, the design domain, displacement, and density are represented by PHT-splines. Leveraging the local refinement capability of PHT-splines, mesh elements defining the density function are adaptively refined to achieve a suitable resolution at the interfa… ▽ More This paper proposes an Adaptive Isogeometric Topology Optimization framework for shell structures based on PHT-splines (PHT-AITO). In this framework, the design domain, displacement, and density are represented by PHT-splines. Leveraging the local refinement capability of PHT-splines, mesh elements defining the density function are adaptively refined to achieve a suitable resolution at the interface between solid and void regions. This addresses the issue of excessive degrees of freedom resulting from global refinement. The refinement of the mesh elements is driven by their density. During the optimization of the density on a refined mesh, the initial value of the density is inherited from the optimization results on the previous mesh to accelerate the iteration process and maintain the stability of the optimized structure. Numerical experiments on various shell structures have verified the effectiveness of PHT-AITO. Compared with isogeometric topology optimization based on tensor-product splines, PHT-AITO can significantly reduce the degrees of freedom in the optimization problem, thereby improving computational efficiency. △ Less

Submitted 12 December, 2023; originally announced December 2023.

Dynamically configured physics-informed neural network in topology optimization applications

Authors: Jichao Yin, Ziming Wen, Shuhao Li, Yaya Zhanga, Hu Wang

Abstract: Integration of machine learning (ML) into the topology optimization (TO) framework is attracting increasing attention, but data acquisition in data-driven models is prohibitive. Compared with popular ML methods, the physics-informed neural network (PINN) can avoid generating enormous amounts of data when solving forward problems and additionally provide better inference. To this end, a dynamically… ▽ More Integration of machine learning (ML) into the topology optimization (TO) framework is attracting increasing attention, but data acquisition in data-driven models is prohibitive. Compared with popular ML methods, the physics-informed neural network (PINN) can avoid generating enormous amounts of data when solving forward problems and additionally provide better inference. To this end, a dynamically configured PINN-based topology optimization (DCPINN-TO) method is proposed. The DCPINN is composed of two subnetworks, namely the backbone neural network (NN) and the coefficient NN, where the coefficient NN has fewer trainable parameters. The designed architecture aims to dynamically configure trainable parameters; that is, an inexpensive NN is used to replace an expensive one at certain optimization cycles. Furthermore, an active sampling strategy is proposed to selectively sample collocations depending on the pseudo-densities at each optimization cycle. In this manner, the number of collocations will decrease with the optimization process but will hardly affect it. The Gaussian integral is used to calculate the strain energy of elements, which yields a byproduct of decoupling the mapping of the material at the collocations. Several examples with different resolutions validate the feasibility of the DCPINN-TO method, and multiload and multiconstraint problems are employed to illustrate its generalization. In addition, compared to finite element analysis-based TO (FEA-TO), the accuracy of the displacement prediction and optimization results indicate that the DCPINN-TO method is effective and efficient. △ Less

Submitted 12 December, 2023; originally announced December 2023.

Comments: 31 pages, 22 figures

AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes

Authors: Rao Fu, Zehao Wen, Zichen Liu, Srinath Sridhar

Abstract: Inspired by cognitive theories, we introduce AnyHome, a framework that translates any text into well-structured and textured indoor scenes at a house-scale. By prompting Large Language Models (LLMs) with designed templates, our approach converts provided textual narratives into amodal structured representations. These representations guarantee consistent and realistic spatial layouts by directing… ▽ More Inspired by cognitive theories, we introduce AnyHome, a framework that translates any text into well-structured and textured indoor scenes at a house-scale. By prompting Large Language Models (LLMs) with designed templates, our approach converts provided textual narratives into amodal structured representations. These representations guarantee consistent and realistic spatial layouts by directing the synthesis of a geometry mesh within defined constraints. A Score Distillation Sampling process is then employed to refine the geometry, followed by an egocentric inpainting process that adds lifelike textures to it. AnyHome stands out with its editability, customizability, diversity, and realism. The structured representations for scenes allow for extensive editing at varying levels of granularity. Capable of interpreting texts ranging from simple labels to detailed narratives, AnyHome generates detailed geometries and textures that outperform existing methods in both quantitative and qualitative measures. △ Less

Submitted 28 July, 2024; v1 submitted 11 December, 2023; originally announced December 2023.

Comments: accepted by ECCV 2024

GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognition

Authors: Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, Jianhua Tao

Abstract: Recently, GPT-4 with Vision (GPT-4V) has demonstrated remarkable visual capabilities across various tasks, but its performance in emotion recognition has not been fully evaluated. To bridge this gap, we present the quantitative evaluation results of GPT-4V on 21 benchmark datasets covering 6 tasks: visual sentiment analysis, tweet sentiment analysis, micro-expression recognition, facial emotion re… ▽ More Recently, GPT-4 with Vision (GPT-4V) has demonstrated remarkable visual capabilities across various tasks, but its performance in emotion recognition has not been fully evaluated. To bridge this gap, we present the quantitative evaluation results of GPT-4V on 21 benchmark datasets covering 6 tasks: visual sentiment analysis, tweet sentiment analysis, micro-expression recognition, facial emotion recognition, dynamic facial emotion recognition, and multimodal emotion recognition. This paper collectively refers to these tasks as ``Generalized Emotion Recognition (GER)''. Through experimental analysis, we observe that GPT-4V exhibits strong visual understanding capabilities in GER tasks. Meanwhile, GPT-4V shows the ability to integrate multimodal clues and exploit temporal information, which is also critical for emotion recognition. However, it's worth noting that GPT-4V is primarily designed for general domains and cannot recognize micro-expressions that require specialized knowledge. To the best of our knowledge, this paper provides the first quantitative assessment of GPT-4V for GER tasks. We have open-sourced the code and encourage subsequent researchers to broaden the evaluation scope by including more tasks and datasets. Our code and evaluation results are available at: https://github.com/zeroQiaoba/gpt4v-emotion. △ Less

Submitted 17 March, 2024; v1 submitted 7 December, 2023; originally announced December 2023.

SPROUT: Authoring Programming Tutorials with Interactive Visualization of Large Language Model Generation Process

Authors: Yihan Liu, Zhen Wen, Luoxuan Weng, Ollie Woodman, Yi Yang, Wei Chen

Abstract: The rapid development of large language models (LLMs), such as ChatGPT, has revolutionized the efficiency of creating programming tutorials. LLMs can be instructed with text prompts to generate comprehensive text descriptions of code snippets. However, the lack of transparency in the end-to-end generation process has hindered the understanding of model behavior and limited user control over the ge… ▽ More The rapid development of large language models (LLMs), such as ChatGPT, has revolutionized the efficiency of creating programming tutorials. LLMs can be instructed with text prompts to generate comprehensive text descriptions of code snippets. However, the lack of transparency in the end-to-end generation process has hindered the understanding of model behavior and limited user control over the generated results. To tackle this challenge, we introduce a novel approach that breaks down the programming tutorial creation task into actionable steps. By employing the tree-of-thought method, LLMs engage in an exploratory process to generate diverse and faithful programming tutorials. We then present SPROUT, an authoring tool equipped with a series of interactive visualizations that empower users to have greater control and understanding of the programming tutorial creation process. A formal user study demonstrated the effectiveness of SPROUT, showing that our tool assists users to actively participate in the programming tutorial creation process, leading to more reliable and customizable results. By providing users with greater control and understanding, SPROUT enhances the user experience and improves the overall quality of programming tutorial. A free copy of this paper and all supplemental materials are available at https://osf.io/uez2t/?view_only=5102e958802341daa414707646428f86. △ Less

Submitted 4 December, 2023; originally announced December 2023.

An Augmented Lagrangian Primal-Dual Semismooth Newton Method for Multi-Block Composite Optimization

Authors: Zhanwang Deng, Kangkang Deng, Jiang Hu, Zaiwen Wen

Abstract: In this paper, we develop a novel primal-dual semismooth Newton method for solving linearly constrained multi-block convex composite optimization problems. First, a differentiable augmented Lagrangian (AL) function is constructed by utilizing the Moreau envelopes of the nonsmooth functions. It enables us to derive an equivalent saddle point problem and establish the strong AL duality under the Sla… ▽ More In this paper, we develop a novel primal-dual semismooth Newton method for solving linearly constrained multi-block convex composite optimization problems. First, a differentiable augmented Lagrangian (AL) function is constructed by utilizing the Moreau envelopes of the nonsmooth functions. It enables us to derive an equivalent saddle point problem and establish the strong AL duality under the Slater's condition. Consequently, a semismooth system of nonlinear equations is formulated to characterize the optimality of the original problem instead of the inclusion-form KKT conditions. We then develop a semismooth Newton method, called ALPDSN, which uses purely second-order steps and a nonmonotone line search based globalization strategy. Through a connection to the inexact first-order steps when the regularization parameter is sufficiently large, the global convergence of ALPDSN is established. Under the regularity conditions, partial smoothness, the local error bound, and the strict complementarity, we show that both the primal and the dual iteration sequences possess a superlinear convergence rate and provide concrete examples where these regularity conditions are met. Numerical results on the image restoration with two regularization terms and the corrected tensor nuclear norm problem are presented to demonstrate the high efficiency and robustness of our ALPDSN. △ Less

Submitted 15 May, 2024; v1 submitted 2 December, 2023; originally announced December 2023.

Comments: 27 pages

RLHF and IIA: Perverse Incentives

Authors: Wanqiao Xu, Shi Dong, Xiuyuan Lu, Grace Lam, Zheng Wen, Benjamin Van Roy

Abstract: Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms. Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and learning algorithms. △ Less

Submitted 1 February, 2024; v1 submitted 2 December, 2023; originally announced December 2023.

Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity

Authors: Tianqin Li, Ziqi Wen, Yangfan Li, Tai Sing Lee

Abstract: Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. What could be the design principles in human visual systems that led to this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous… ▽ More Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. What could be the design principles in human visual systems that led to this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network. We found that enforcing the sparse coding constraint using a non-differential Top-K operation can lead to the emergence of structural encoding in neurons in convolutional neural networks, resulting in a smooth decomposition of objects into parts and subparts and endowing the networks with shape bias. We demonstrated this emergence of shape bias and its functional benefits for different network structures with various datasets. For object recognition convolutional neural networks, the shape bias leads to greater robustness against style and pattern change distraction. For the image synthesis generative adversary networks, the emerged shape bias leads to more coherent and decomposable structures in the synthesized images. Ablation studies suggest that sparse codes tend to encode structures, whereas the more distributed codes tend to favor texture. Our code is host at the github repository: \url{https://github.com/Crazy-Jack/nips2023_shape_vs_texture} △ Less

Submitted 29 October, 2023; originally announced October 2023.

Comments: Published as NeurIPS 2023 (Oral)

Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approach

Authors: Dengwang Tang, Rahul Jain, Botao Hao, Zheng Wen

Abstract: In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (paramet… ▽ More In this paper, we study the problem of efficient online reinforcement learning in the infinite horizon setting when there is an offline dataset to start with. We assume that the offline dataset is generated by an expert but with unknown level of competence, i.e., it is not perfect and not necessarily using the optimal policy. We show that if the learning agent models the behavioral policy (parameterized by a competence parameter) used by the expert, it can do substantially better in terms of minimizing cumulative regret, than if it doesn't do that. We establish an upper bound on regret of the exact informed PSRL algorithm that scales as $\tilde{O}(\sqrt{T})$. This requires a novel prior-dependent regret analysis of Bayesian online learning algorithms for the infinite horizon setting. We then propose the Informed RLSVI algorithm to efficiently approximate the iPSRL algorithm. △ Less

Submitted 1 February, 2024; v1 submitted 17 October, 2023; originally announced October 2023.

Comments: 22 pages

MSC Class: 93E35

Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection

Authors: Cunhang Fan, Mingming Ding, Jianhua Tao, Ruibo Fu, Jiangyan Yi, Zhengqi Wen, Zhao Lv

Abstract: Most research in synthetic speech detection (SSD) focuses on improving performance on standard noise-free datasets. However, in actual situations, noise interference is usually present, causing significant performance degradation in SSD systems. To improve noise robustness, this paper proposes a dual-branch knowledge distillation synthetic speech detection (DKDSSD) method. Specifically, a parallel… ▽ More Most research in synthetic speech detection (SSD) focuses on improving performance on standard noise-free datasets. However, in actual situations, noise interference is usually present, causing significant performance degradation in SSD systems. To improve noise robustness, this paper proposes a dual-branch knowledge distillation synthetic speech detection (DKDSSD) method. Specifically, a parallel data flow of the clean teacher branch and the noisy student branch is designed, and interactive fusion module and response-based teacher-student paradigms are proposed to guide the training of noisy data from both the data distribution and decision-making perspectives. In the noisy student branch, speech enhancement is introduced initially for denoising, aiming to reduce the interference of strong noise. The proposed interactive fusion combines denoised features and noisy features to mitigate the impact of speech distortion and ensure consistency with the data distribution of the clean branch. The teacher-student paradigm maps the student's decision space to the teacher's decision space, enabling noisy speech to behave similarly to clean speech. Additionally, a joint training method is employed to optimize both branches for achieving global optimality. Experimental results based on multiple datasets demonstrate that the proposed method performs effectively in noisy environments and maintains its performance in cross-dataset experiments. Source code is available at https://github.com/fchest/DKDSSD. △ Less

Submitted 16 April, 2024; v1 submitted 13 October, 2023; originally announced October 2023.

Does resistance to style-transfer equal Global Shape Bias? Measuring network sensitivity to global shape configuration

Authors: Ziqi Wen, Tianqin Li, Zhi Jing, Tai Sing Lee

Abstract: Deep learning models are known to exhibit a strong texture bias, while human tends to rely heavily on global shape structure for object recognition. The current benchmark for evaluating a model's global shape bias is a set of style-transferred images with the assumption that resistance to the attack of style transfer is related to the development of global structure sensitivity in the model. In th… ▽ More Deep learning models are known to exhibit a strong texture bias, while human tends to rely heavily on global shape structure for object recognition. The current benchmark for evaluating a model's global shape bias is a set of style-transferred images with the assumption that resistance to the attack of style transfer is related to the development of global structure sensitivity in the model. In this work, we show that networks trained with style-transfer images indeed learn to ignore style, but its shape bias arises primarily from local detail. We provide a \textbf{Disrupted Structure Testbench (DiST)} as a direct measurement of global structure sensitivity. Our test includes 2400 original images from ImageNet-1K, each of which is accompanied by two images with the global shapes of the original image disrupted while preserving its texture via the texture synthesis program. We found that \textcolor{black}{(1) models that performed well on the previous cue-conflict dataset do not fare well in the proposed DiST; (2) the supervised trained Vision Transformer (ViT) lose its global spatial information from positional embedding, leading to no significant advantages over Convolutional Neural Networks (CNNs) on DiST. While self-supervised learning methods, especially mask autoencoder significantly improves the global structure sensitivity of ViT. (3) Improving the global structure sensitivity is orthogonal to resistance to style-transfer, indicating that the relationship between global shape structure and local texture detail is not an either/or relationship. Training with DiST images and style-transferred images are complementary, and can be combined to train network together to enhance the global shape sensitivity and robustness of local features.} Our code will be hosted in github: https://github.com/leelabcnbc/DiST △ Less

Submitted 29 February, 2024; v1 submitted 11 October, 2023; originally announced October 2023.

Interpretable Traffic Event Analysis with Bayesian Networks

Authors: Tong Yuan, Jian Yang, Zeyi Wen

Abstract: Although existing machine learning-based methods for traffic accident analysis can provide good quality results to downstream tasks, they lack interpretability which is crucial for this critical problem. This paper proposes an interpretable framework based on Bayesian Networks for traffic accident prediction. To enable the ease of interpretability, we design a dataset construction pipeline to feed… ▽ More Although existing machine learning-based methods for traffic accident analysis can provide good quality results to downstream tasks, they lack interpretability which is crucial for this critical problem. This paper proposes an interpretable framework based on Bayesian Networks for traffic accident prediction. To enable the ease of interpretability, we design a dataset construction pipeline to feed the traffic data into the framework while retaining the essential traffic data information. With a concrete case study, our framework can derive a Bayesian Network from a dataset based on the causal relationships between weather and traffic events across the United States. Consequently, our framework enables the prediction of traffic accidents with competitive accuracy while examining how the probability of these events changes under different conditions, thus illustrating transparent relationships between traffic and weather events. Additionally, the visualization of the network simplifies the analysis of relationships between different variables, revealing the primary causes of traffic accidents and ultimately providing a valuable reference for reducing traffic accidents. △ Less

Submitted 10 October, 2023; originally announced October 2023.

Comments: 11 pages, 7 figures

MSC Class: 62F15 ACM Class: G.3

GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidence

Authors: Zhihua Wen, Zhiliang Tian, Wei Wu, Yuxin Yang, Yanqi Shi, Zhen Huang, Dongsheng Li

Abstract: Conditional story generation is significant in human-machine interaction, particularly in producing stories with complex plots. While Large language models (LLMs) perform well on multiple NLP tasks, including story generation, it is challenging to generate stories with both complex and creative plots. Existing methods often rely on detailed prompts to guide LLMs to meet target conditions, which in… ▽ More Conditional story generation is significant in human-machine interaction, particularly in producing stories with complex plots. While Large language models (LLMs) perform well on multiple NLP tasks, including story generation, it is challenging to generate stories with both complex and creative plots. Existing methods often rely on detailed prompts to guide LLMs to meet target conditions, which inadvertently restrict the creative potential of the generated stories. We argue that leveraging information from exemplary human-written stories facilitates generating more diverse plotlines. Delving deeper into story details helps build complex and credible plots. In this paper, we propose a retrieval-au\textbf{G}mented sto\textbf{R}y generation framework with a f\textbf{O}rest of e\textbf{V}id\textbf{E}nce (GROVE) to enhance stories' complexity. We build a retrieval repository for target conditions to produce few-shot examples to prompt LLMs. Additionally, we design an ``asking-why'' prompting scheme that extracts a forest of evidence, providing compensation for the ambiguities that may occur in the generated story. This iterative process uncovers underlying story backgrounds. Finally, we select the most fitting chains of evidence from the evidence forest and integrate them into the generated story, thereby enhancing the narrative's complexity and credibility. Experimental results and numerous examples verify the effectiveness of our method. △ Less

Submitted 23 October, 2023; v1 submitted 8 October, 2023; originally announced October 2023.

Comments: Findings of EMNLP 2023

Design Principles of Robust Multi-Armed Bandit Framework in Video Recommendations

Authors: Belhassen Bayar, Phanideep Gampa, Ainur Yessenalina, Zhen Wen

Abstract: Current multi-armed bandit approaches in recommender systems (RS) have focused more on devising effective exploration techniques, while not adequately addressing common exploitation challenges related to distributional changes and item cannibalization. Little work exists to guide the design of robust bandit frameworks that can address these frequent challenges in RS. In this paper, we propose a ne… ▽ More Current multi-armed bandit approaches in recommender systems (RS) have focused more on devising effective exploration techniques, while not adequately addressing common exploitation challenges related to distributional changes and item cannibalization. Little work exists to guide the design of robust bandit frameworks that can address these frequent challenges in RS. In this paper, we propose a new design principles to (i) make bandit models robust to time-variant metadata signals, (ii) less prone to item cannibalization, and (iii) prevent their weights fluctuating due to data sparsity. Through a series of experiments, we systematically examine the influence of several important bandit design choices. We demonstrate the advantage of our proposed design principles at making bandit models robust to dynamic behavioral changes through in-depth analyses. Noticeably, we show improved relative gain compared to a baseline bandit model not incorporating our design choices of up to $11.88\%$ and $44.85\%$, respectively in ROC-AUC and PR-AUC. Case studies about fairness in recommending specific popular and unpopular titles are presented, to demonstrate the robustness of our proposed design at addressing popularity biases. △ Less

Submitted 24 September, 2023; originally announced October 2023.

Comments: RecSys CARS 2023 Workshop paper

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

Authors: Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, Jiantao Jiao

Abstract: Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as… ▽ More Large Language Models (LLMs) can acquire extensive world knowledge through pre-training on large corpora. However, due to exposure to low-quality data, LLMs may exhibit harmful behavior without aligning with human values. The dominant approach for steering LLMs towards beneficial behavior involves Reinforcement Learning with Human Feedback (RLHF), with Proximal Policy Optimization (PPO) serving as the default RL optimizer. Despite its effectiveness, PPO has limitations when optimizing rewards trained from comparison-based loss. Primarily, PPO is not invariant to equivalent reward functions containing identical preference information due to the need to calibrate the reward scale. Additionally, PPO's necessity for token-wise updates introduces complexity in both function approximation and algorithm design compared to trajectory-wise optimization. This paper proposes a new framework, reinforcement learning with relative feedback, and a novel trajectory-wise policy gradient algorithm, Pairwise Proximal Policy Optimization (P3O) that operates directly on comparative rewards. We show theoretically that P3O is invariant to equivalent rewards and avoids the complexity of PPO. Empirical evaluations demonstrate that P3O outperforms PPO in the KL-Reward trade-off and can align with human preferences as well as or better than prior methods. In summary, this work introduces a simpler yet effective approach for aligning LLMs to human preferences through relative feedback. △ Less

Submitted 9 October, 2023; v1 submitted 29 September, 2023; originally announced October 2023.

Comments: 19 pages, 5 figures

Sharper Convergence Guarantees for Federated Learning with Partial Model Personalization

Authors: Yiming Chen, Liyuan Cao, Kun Yuan, Zaiwen Wen

Abstract: Partial model personalization, which encompasses both shared and personal variables in its formulation, is a critical optimization problem in federated learning. It balances individual client needs with collective knowledge utilization, and serves as a general formulation covering various key scenarios, ranging from fully shared to fully personalized federated learning. This paper introduces two e… ▽ More Partial model personalization, which encompasses both shared and personal variables in its formulation, is a critical optimization problem in federated learning. It balances individual client needs with collective knowledge utilization, and serves as a general formulation covering various key scenarios, ranging from fully shared to fully personalized federated learning. This paper introduces two effective algorithms, FedAvg-P and Scaffold-P, to solve this problem and provides sharp convergence analyses, quantifying the influence of gradient variance, local steps, and partial client sampling on their performance. Our established rates surpass existing results and, meanwhile, are based on more relaxed assumptions. Additionally, our analyses are also applicable to fully shared or fully personalized federated learning, matching or even outperforming their best known convergence rates. Numerical experiments corroborate our theoretical findings. △ Less

Submitted 29 September, 2023; originally announced September 2023.

Residual Speaker Representation for One-Shot Voice Conversion

Authors: Le Xu, Jiangyan Yi, Tao Wang, Yong Ren, Rongxiu Zhong, Zhengqi Wen, Jianhua Tao

Abstract: Recently, there have been significant advancements in voice conversion, resulting in high-quality performance. However, there are still two critical challenges in this field. Firstly, current voice conversion methods have limited robustness when encountering unseen speakers. Secondly, they also have limited ability to control timbre representation. To address these challenges, this paper presents… ▽ More Recently, there have been significant advancements in voice conversion, resulting in high-quality performance. However, there are still two critical challenges in this field. Firstly, current voice conversion methods have limited robustness when encountering unseen speakers. Secondly, they also have limited ability to control timbre representation. To address these challenges, this paper presents a novel approach that leverages tokens of multi-layer residual approximations to enhance robustness when dealing with unseen speakers, called the residual speaker module. Introducing multi-layer approximations facilitates the separation of information from the timbre, enabling effective control over timbre in voice conversion. The proposed method outperforms baselines in subjective and objective evaluations, demonstrating superior performance and increased robustness. Our demo page is publicly available. △ Less

Submitted 11 August, 2024; v1 submitted 15 September, 2023; originally announced September 2023.

Comments: Accepted by INTERSPEECH2024

Resolution-independent generative models based on operator learning for physics-constrained Bayesian inverse problems

Authors: Xinchao Jiang, Xin Wang, Ziming Wen, Hu Wang

Abstract: The Bayesian inference approach is widely used to tackle inverse problems due to its versatile and natural ability to handle ill-posedness. However, it often faces challenges when dealing with situations involving continuous fields or large-resolution discrete representations (high-dimensional). Moreover, the prior distribution of unknown parameters is commonly difficult to be determined. In this… ▽ More The Bayesian inference approach is widely used to tackle inverse problems due to its versatile and natural ability to handle ill-posedness. However, it often faces challenges when dealing with situations involving continuous fields or large-resolution discrete representations (high-dimensional). Moreover, the prior distribution of unknown parameters is commonly difficult to be determined. In this study, an Operator Learning-based Generative Adversarial Network (OL-GAN) is proposed and integrated into the Bayesian inference framework to handle these issues. Unlike most Bayesian approaches, the distinctive characteristic of the proposed method is to learn the joint distribution of parameters and responses. By leveraging the trained generative model, the posteriors of the unknown parameters can theoretically be approximated by any sampling algorithm (e.g., Markov Chain Monte Carlo, MCMC) in a low-dimensional latent space shared by the components of the joint distribution. The latent space is typically a simple and easy-to-sample distribution (e.g., Gaussian, uniform), which significantly reduces the computational cost associated with the Bayesian inference while avoiding prior selection concerns. Furthermore, incorporating operator learning enables resolution-independent in the generator. Predictions can be obtained at desired coordinates, and inversions can be performed even if the observation data are misaligned with the training data. Finally, the effectiveness of the proposed method is validated through several numerical experiments. △ Less

Submitted 25 August, 2023; originally announced August 2023.

Voucher Abuse Detection with Prompt-based Fine-tuning on Graph Neural Networks

Authors: Zhihao Wen, Yuan Fang, Yihan Liu, Yang Guo, Shuji Hao

Abstract: Voucher abuse detection is an important anomaly detection problem in E-commerce. While many GNN-based solutions have emerged, the supervised paradigm depends on a large quantity of labeled data. A popular alternative is to adopt self-supervised pre-training using label-free data, and further fine-tune on a downstream task with limited labels. Nevertheless, the "pre-train, fine-tune" paradigm is of… ▽ More Voucher abuse detection is an important anomaly detection problem in E-commerce. While many GNN-based solutions have emerged, the supervised paradigm depends on a large quantity of labeled data. A popular alternative is to adopt self-supervised pre-training using label-free data, and further fine-tune on a downstream task with limited labels. Nevertheless, the "pre-train, fine-tune" paradigm is often plagued by the objective gap between pre-training and downstream tasks. Hence, we propose VPGNN, a prompt-based fine-tuning framework on GNNs for voucher abuse detection. We design a novel graph prompting function to reformulate the downstream task into a similar template as the pretext task in pre-training, thereby narrowing the objective gap. Extensive experiments on both proprietary and public datasets demonstrate the strength of VPGNN in both few-shot and semi-supervised scenarios. Moreover, an online deployment of VPGNN in a production environment shows a 23.4% improvement over two existing deployed models. △ Less

Submitted 30 August, 2023; v1 submitted 19 August, 2023; originally announced August 2023.

Comments: 7 pages, Accepted by CIKM23 Applied Research Track

On the Optimal Lower and Upper Complexity Bounds for a Class of Composite Optimization Problems

Authors: Zhenyuan Zhu, Fan Chen, Junyu Zhang, Zaiwen Wen

Abstract: We study the optimal lower and upper complexity bounds for finding approximate solutions to the composite problem $\min_x\ f(x)+h(Ax-b)$, where $f$ is smooth and $h$ is convex. Given access to the proximal operator of $h$, for strongly convex, convex, and nonconvex $f$, we design efficient first order algorithms with complexities $\tilde{O}\left(κ_A\sqrt{κ_f}\log\left(1/ε\right)\right)$,… ▽ More We study the optimal lower and upper complexity bounds for finding approximate solutions to the composite problem $\min_x\ f(x)+h(Ax-b)$, where $f$ is smooth and $h$ is convex. Given access to the proximal operator of $h$, for strongly convex, convex, and nonconvex $f$, we design efficient first order algorithms with complexities $\tilde{O}\left(κ_A\sqrt{κ_f}\log\left(1/ε\right)\right)$, $\tilde{O}\left(κ_A\sqrt{L_f}D/\sqrtε\right)$, and $\tilde{O}\left(κ_A L_fΔ/ε^2\right)$, respectively. Here, $κ_A$ is the condition number of the matrix $A$ in the composition, $L_f$ is the smoothness constant of $f$, and $κ_f$ is the condition number of $f$ in the strongly convex case. $D$ is the initial point distance and $Δ$ is the initial function value gap. Tight lower complexity bounds for the three cases are also derived and they match the upper bounds up to logarithmic factors, thereby demonstrating the optimality of both the upper and lower bounds proposed in this paper. △ Less

Submitted 12 August, 2023; originally announced August 2023.

MSC Class: 90C25; 90C26; 90C46; 90C60