Showing 1–50 of 3,599 results for author: Huang, Z

Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

Authors: Zihao Sheng, Zilin Huang, Sikai Chen

Abstract: Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency compared to model-free RL by utilizing a virtual environment model. However, it is challenging to obtain sufficiently accurate representations of the environmental dynamics due to uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performa… ▽ More Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency compared to model-free RL by utilizing a virtual environment model. However, it is challenging to obtain sufficiently accurate representations of the environmental dynamics due to uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performance of model-based RL. Furthermore, while model-based RL can improve sample efficiency, it often still requires substantial training time to learn from scratch, potentially limiting its advantages over model-free approaches. To address these challenges, this paper introduces a knowledge-informed model-based residual reinforcement learning framework aimed at enhancing learning efficiency by infusing established expert knowledge into the learning process and avoiding the issue of beginning from zero. Our approach integrates traffic expert knowledge into a virtual environment model, employing the Intelligent Driver Model (IDM) for basic dynamics and neural networks for residual dynamics, thus ensuring adaptability to complex scenarios. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch. The proposed approach is applied to CAV trajectory control tasks for the dissipation of stop-and-go waves in mixed traffic flow. Experimental results demonstrate that our proposed approach enables the CAV agent to achieve superior performance in trajectory control compared to the baseline agents in terms of sample efficiency, traffic flow smoothness and traffic mobility. The source code and supplementary materials are available at https://github.com/zihaosheng/traffic-expertise-RL/. △ Less

Submitted 30 August, 2024; originally announced August 2024.

Stochastic Layer-Wise Shuffle: A Good Practice to Improve Vision Mamba Training

Authors: Zizheng Huang, Haoxing Chen, Jiaqi Li, Jun Lan, Huijia Zhu, Weiqiang Wang, Limin Wang

Abstract: Recent Vision Mamba models not only have much lower complexity for processing higher resolution images and longer videos but also the competitive performance with Vision Transformers (ViTs). However, they are stuck into overfitting and thus only present up to base size (about 80M). It is still unclear how vanilla Vision Mamba (Vim) can be efficiently scaled up to larger sizes, which is essentially… ▽ More Recent Vision Mamba models not only have much lower complexity for processing higher resolution images and longer videos but also the competitive performance with Vision Transformers (ViTs). However, they are stuck into overfitting and thus only present up to base size (about 80M). It is still unclear how vanilla Vision Mamba (Vim) can be efficiently scaled up to larger sizes, which is essentially for further exploitation. In this paper, we propose a stochastic layer-wise shuffle regularization, which empowers successfully scaling non-hierarchical Vision Mamba to a large size (about 300M) in a supervised setting. Specifically, our base and large-scale ShuffleMamba models can outperform the supervised ViTs of similar size by 0.8\% and 1.0\% classification accuracy on ImageNet1k, respectively, without auxiliary data. When evaluated on the ADE20K semantic segmentation and COCO detection tasks, our ShuffleMamba models also show significant improvements. Without bells and whistles, the stochastic layer-wise shuffle has the following highlights: (1) \textit{Plug and play:} it does not change model architectures and will be omitted in inference. (2) \textit{Simple but effective:} it can improve the overfitting in Vim training and only introduce random token permutation operations. (3) \textit{Intuitive:} the token sequences in deeper layers are more likely to be shuffled as they are expected to be more semantic and less sensitive to patch positions. Code and models will be available at https://github.com/huangzizheng01/ShuffleMamba. △ Less

Submitted 30 August, 2024; originally announced August 2024.

Pair Counting without Binning -- A New Approach to Correlation Functions in Clustering Statistics

Authors: Shiyu Yue, Longlong Feng, Wenjie Ju, Jun Pan, Zhiqi Huang, Feng Fang, Zhuoyang Li, Yan-Chuan Cai, Weishan Zhu

Abstract: This paper presents a novel perspective on correlation functions in the clustering analysis of the large-scale structure of the universe. We first recognise that pair counting in bins of radial separation is equivalent to evaluating counts-in-cells (CIC), which can be modelled using a filtered density field with a binning-window function. This insight leads to an in situ expression for the two-poi… ▽ More This paper presents a novel perspective on correlation functions in the clustering analysis of the large-scale structure of the universe. We first recognise that pair counting in bins of radial separation is equivalent to evaluating counts-in-cells (CIC), which can be modelled using a filtered density field with a binning-window function. This insight leads to an in situ expression for the two-point correlation function (2PCF). Essentially, the core idea underlying our method is to introduce a window function to define the binning scheme, enabling pair-counting without binning. This approach develops a concept of generalised 2PCF, which extends beyond conventional discrete pair counting by accommodating non-sharp-edged window functions. To extend this framework to N-point correlation functions (NPCF) using current optimal edge-corrected estimators, we developed a binning scheme independent of the specific parameterisation of polyhedral configurations. In particular, we demonstrate a fast algorithm for the three-point correlation function (3PCF), where triplet counting is accomplished by assigning either a spherical tophat or a Gaussian filter to each vertex of triangles. Additionally, we derive analytical expressions for the 3PCF using a multipole expansion in Legendre polynomials, accounting for filtered field (binning) corrections. Numerical tests using several suites of N-body simulation samples show that our approach aligns remarkably well with the theoretical predictions. Our method provides an exact solution for quantifying binning effects in practical measurements and offers a high-speed algorithm, enabling high-order clustering analysis in extremely large datasets from ongoing and upcoming surveys such as Euclid, LSST, and DESI. △ Less

Submitted 29 August, 2024; originally announced August 2024.

Comments: 17 pages, 12 figures, submitted to MNRAS

Quasi-Lindblad pseudomode theory for open quantum systems

Authors: Gunhee Park, Zhen Huang, Yuanran Zhu, Chao Yang, Garnet Kin-Lic Chan, Lin Lin

Abstract: We introduce a new framework to study the dynamics of open quantum systems with linearly coupled Gaussian baths. Our approach replaces the continuous bath with an auxiliary discrete set of pseudomodes with dissipative dynamics, but we further relax the complete positivity requirement in the Lindblad master equation and formulate a quasi-Lindblad pseudomode theory. We show that this quasi-Lindblad… ▽ More We introduce a new framework to study the dynamics of open quantum systems with linearly coupled Gaussian baths. Our approach replaces the continuous bath with an auxiliary discrete set of pseudomodes with dissipative dynamics, but we further relax the complete positivity requirement in the Lindblad master equation and formulate a quasi-Lindblad pseudomode theory. We show that this quasi-Lindblad pseudomode formulation directly leads to a representation of the bath correlation function in terms of a complex weighted sum of complex exponentials, an expansion that is known to be rapidly convergent in practice and thus leads to a compact set of pseudomodes. The pseudomode representation is not unique and can differ by a gauge choice. When the global dynamics can be simulated exactly, the system dynamics is unique and independent of the specific pseudomode representation. However, the gauge choice may affect the stability of the global dynamics, and we provide an analysis of why and when the global dynamics can retain stability despite losing positivity. We showcase the performance of this formulation across various spectral densities in both bosonic and fermionic problems, finding significant improvements over conventional pseudomode formulations. △ Less

Submitted 28 August, 2024; originally announced August 2024.

Comments: 13 pages, 6 figures (main text); 8 pages, 1 figure (Supplementary Material)

MEET-U project II: Curvature perturbations from kinetic preheating after $α$-attractor inflation

Authors: Zhiqi Huang, Xichang Ouyang, Yu Cui, Jianqi Liu, Yanhong Yao, Zehong Qiu, Guangyao Yu, Lu Huang, Zhuoyang Li, Chi-Fong Wong

Abstract: Preheating at the end of inflation is a violent nonlinear process that efficiently transfers the energy of the inflaton to a second field, the preheat field. When the preheat field is light during inflation and its background value modulates the preheating process, the superhorizon isocurvature perturbations of the preheat field may be converted to curvature perturbations that leave an imprint on… ▽ More Preheating at the end of inflation is a violent nonlinear process that efficiently transfers the energy of the inflaton to a second field, the preheat field. When the preheat field is light during inflation and its background value modulates the preheating process, the superhorizon isocurvature perturbations of the preheat field may be converted to curvature perturbations that leave an imprint on the cosmic microwave background and the large-scale structure of the universe. We use high-precision lattice simulations to study kinetic preheating after $α$-attractor inflation, a case where the effective mass of the preheat field is naturally suppressed during inflation. By comparing the expansion e-folds between different Hubble patches, we find that the conversion from isocurvature perturbations to curvature perturbations is very inefficient and can hardly be detected by cosmological observations. △ Less

Submitted 27 August, 2024; originally announced August 2024.

Report number: SYSU-SPA-2024 MSC Class: 83F05 ACM Class: J.2

CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis

Authors: Weijia Li, Jun He, Junyan Ye, Huaping Zhong, Zhimeng Zheng, Zilong Huang, Dahua Lin, Conghui He

Abstract: Satellite-to-street view synthesis aims at generating a realistic street-view image from its corresponding satellite-view image. Although stable diffusion models have exhibit remarkable performance in a variety of image generation applications, their reliance on similar-view inputs to control the generated structure or texture restricts their application to the challenging cross-view synthesis tas… ▽ More Satellite-to-street view synthesis aims at generating a realistic street-view image from its corresponding satellite-view image. Although stable diffusion models have exhibit remarkable performance in a variety of image generation applications, their reliance on similar-view inputs to control the generated structure or texture restricts their application to the challenging cross-view synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion model for satellite-to-street view synthesis. To address the challenges posed by the large discrepancy across views, we design the satellite scene structure estimation and cross-view texture mapping modules to construct the structural and textural controls for street-view image synthesis. We further design a cross-view control guided denoising process that incorporates the above controls via an enhanced cross-view attention module. To achieve a more comprehensive evaluation of the synthesis results, we additionally design a GPT-based scoring method as a supplement to standard evaluation metrics. We also explore the effect of different data sources (e.g., text, maps, building heights, and multi-temporal satellite imagery) on this task. Results on three public cross-view datasets show that CrossViewDiff outperforms current state-of-the-art on both standard and GPT-based evaluation metrics, generating high-quality street-view panoramas with more realistic structures and textures across rural, suburban, and urban scenes. The code and models of this work will be released at https://opendatalab.github.io/CrossViewDiff/. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: 21 pages, 11 figures

SWE-bench-java: A GitHub Issue Resolving Benchmark for Java

Authors: Daoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, Yifan Shi, Wei Liu, Dong Chen, Zongshuai Qi, Hao Yu, Lei Yu, Dezhi Ran, Muhan Zeng, Bo Shen, Pan Bian, Guangtai Liang, Bei Guan, Pengjie Huang, Tao Xie, Yongji Wang, Qianxiang Wang

Abstract: GitHub issue resolving is a critical task in software engineering, recently gaining significant attention in both industry and academia. Within this task, SWE-bench has been released to evaluate issue resolving capabilities of large language models (LLMs), but has so far only focused on Python version. However, supporting more programming languages is also important, as there is a strong demand in… ▽ More GitHub issue resolving is a critical task in software engineering, recently gaining significant attention in both industry and academia. Within this task, SWE-bench has been released to evaluate issue resolving capabilities of large language models (LLMs), but has so far only focused on Python version. However, supporting more programming languages is also important, as there is a strong demand in industry. As a first step toward multilingual support, we have developed a Java version of SWE-bench, called SWE-bench-java. We have publicly released the dataset, along with the corresponding Docker-based evaluation environment and leaderboard, which will be continuously maintained and updated in the coming months. To verify the reliability of SWE-bench-java, we implement a classic method SWE-agent and test several powerful LLMs on it. As is well known, developing a high-quality multi-lingual benchmark is time-consuming and labor-intensive, so we welcome contributions through pull requests or collaboration to accelerate its iteration and refinement, paving the way for fully automated programming. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: This work is in progress

All-microwave readout, spectroscopy, and dynamic polarization of individual nuclear spins in a crystal

Authors: J. Travesedo, J. O'Sullivan, L. Pallegoix, Z. W. Huang, P. Hogan, P. Goldner, T. Chaneliere, S. Bertaina, D. Esteve, P. Abgrall, D. Vion, E. Flurin, P. Bertet

Abstract: Pushing the sensitivity of nuclear magnetic resonance spectroscopy to the single spin level would have a major impact in chemistry and biology and is the goal of intense research efforts. Individual nuclear spins have been detected via their hyperfine coupling to an individual electronic paramagnetic system, itself measured by optical or electrical means. These methods are however only applicable… ▽ More Pushing the sensitivity of nuclear magnetic resonance spectroscopy to the single spin level would have a major impact in chemistry and biology and is the goal of intense research efforts. Individual nuclear spins have been detected via their hyperfine coupling to an individual electronic paramagnetic system, itself measured by optical or electrical means. These methods are however only applicable when suitable optical transitions or electron-spin-to-charge conversion mechanisms exist, and a more universal method is currently lacking. Here, we report spectroscopic measurements of individual $^{183}\mathrm{W}$ nuclear spins in a CaWO$_4$ crystal via their hyperfine interaction with a neighboring $\mathrm{Er}^{3+}$ ion detected by microwave photon counting at millikelvin temperatures. We observe real-time quantum jumps of the nuclear spin state, a proof of their individual nature. We perform single-spin ELDOR-detected NMR spectroscopy by microwave driving the zero- and double-quantum transitions of the $^{183}$W--Er$^{3+}$ coupled system. By repeated driving of these transitions, we also achieve single-spin solid-effect dynamical nuclear polarization. Relying exclusively on microwave driving and microwave detection, the methods reported here apply in principle to any nuclear spin coupled to a paramagnetic impurity, and therefore open the way to single-nuclear-spin spectroscopy in a large class of samples. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Making Large Language Models Better Planners with Reasoning-Decision Alignment

Authors: Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, Xiaodan Liang

Abstract: Data-driven approaches for autonomous driving (AD) have been widely adopted in the past decade but are confronted with dataset bias and uninterpretability. Inspired by the knowledge-driven nature of human driving, recent approaches explore the potential of large language models (LLMs) to improve understanding and decision-making in traffic scenarios. They find that the pretrain-finetune paradigm o… ▽ More Data-driven approaches for autonomous driving (AD) have been widely adopted in the past decade but are confronted with dataset bias and uninterpretability. Inspired by the knowledge-driven nature of human driving, recent approaches explore the potential of large language models (LLMs) to improve understanding and decision-making in traffic scenarios. They find that the pretrain-finetune paradigm of LLMs on downstream data with the Chain-of-Thought (CoT) reasoning process can enhance explainability and scene understanding. However, such a popular strategy proves to suffer from the notorious problems of misalignment between the crafted CoTs against the consequent decision-making, which remains untouched by previous LLM-based AD methods. To address this problem, we motivate an end-to-end decision-making model based on multimodality-augmented LLM, which simultaneously executes CoT reasoning and carries out planning results. Furthermore, we propose a reasoning-decision alignment constraint between the paired CoTs and planning results, imposing the correspondence between reasoning and decision-making. Moreover, we redesign the CoTs to enable the model to comprehend complex scenarios and enhance decision-making performance. We dub our proposed large language planners with reasoning-decision alignment as RDA-Driver. Experimental evaluations on the nuScenes and DriveLM-nuScenes benchmarks demonstrate the effectiveness of our RDA-Driver in enhancing the performance of end-to-end AD systems. Specifically, our RDA-Driver achieves state-of-the-art planning performance on the nuScenes dataset with 0.80 L2 error and 0.32 collision rate, and also achieves leading results on challenging DriveLM-nuScenes benchmarks with 0.82 L2 error and 0.38 collision rate. △ Less

Submitted 25 August, 2024; originally announced August 2024.

Ultrafast Charge Transfer Dynamics at the MoS$_2$/Au Interface Observed via Optical Spectroscopy under Ambient Conditions

Authors: Tao Yang, Zhipeng Huang, Stephan Sleziona, Eckart Hasselbrink, Peter Kratzer, Marika Schleberger, R. Kramer Campen, Yujin Tong

Abstract: To take advantage of the exceptional properties of atomically thin transition metal dichalcogenides (TMDC) for advanced devices and catalysts, integration with metallic surfaces is an efficacious approach for facilitating charge carrier injection and extraction from TMDC monolayers. Light-matter interactions predominantly occur at the K point in TMDC monolayers, making the charge carrier dynamics… ▽ More To take advantage of the exceptional properties of atomically thin transition metal dichalcogenides (TMDC) for advanced devices and catalysts, integration with metallic surfaces is an efficacious approach for facilitating charge carrier injection and extraction from TMDC monolayers. Light-matter interactions predominantly occur at the K point in TMDC monolayers, making the charge carrier dynamics at this point essential for their optimal performance. However, direct access to and comprehensive understanding of the charge carrier dynamics at the K point of TMDC monolayer on a metal substrate remains challenging. In this study, we employed azimuth- and polarization-dependent final-state sum frequency generation (FS-SFG) spectroscopy to investigate the ultrafast dynamics of charge transfer at the K point of a MoS$_2$ monolayer interfaced with an Au substrate. We observed an ultrafast injection (sub-20 fs) of photoexcited hot electrons from the Au substrate to the conduction band minimum (CBM) of the MoS$_2$ monolayer. Subsequently, driven by an internal electric field induced by charge redistribution, injected hot electrons in MoS$_2$ experience a relaxation and fast return ($\sim2$ ps) from the CBM and a trap state mediated slow return ($\sim60$ ps) process. The direct optical observation of the full electron dynamics at the K point of MoS$_2$ monolayer in ambient conditions provides valuable insights into the mechanisms of charge carrier transfer across the TMDC-metal interface, informing the design of advanced TMDC-based devices with enhanced charge transfer rates. △ Less

Submitted 24 August, 2024; originally announced August 2024.

Comments: 16 pages, 3 figures and supplemental material

MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning

Authors: Zhenyu Zhang, Guangyao Chen, Yixiong Zou, Zhimeng Huang, Yuhua Li, Ruixuan Li

Abstract: Humans exhibit a remarkable ability to learn quickly from a limited number of labeled samples, a capability that starkly contrasts with that of current machine learning systems. Unsupervised Few-Shot Learning (U-FSL) seeks to bridge this divide by reducing reliance on annotated datasets during initial training phases. In this work, we first quantitatively assess the impacts of Masked Image Modelin… ▽ More Humans exhibit a remarkable ability to learn quickly from a limited number of labeled samples, a capability that starkly contrasts with that of current machine learning systems. Unsupervised Few-Shot Learning (U-FSL) seeks to bridge this divide by reducing reliance on annotated datasets during initial training phases. In this work, we first quantitatively assess the impacts of Masked Image Modeling (MIM) and Contrastive Learning (CL) on few-shot learning tasks. Our findings highlight the respective limitations of MIM and CL in terms of discriminative and generalization abilities, which contribute to their underperformance in U-FSL contexts. To address these trade-offs between generalization and discriminability in unsupervised pretraining, we introduce a novel paradigm named Masked Image Contrastive Modeling (MICM). MICM creatively combines the targeted object learning strength of CL with the generalized visual feature learning capability of MIM, significantly enhancing its efficacy in downstream few-shot learning inference. Extensive experimental analyses confirm the advantages of MICM, demonstrating significant improvements in both generalization and discrimination capabilities for few-shot learning. Our comprehensive quantitative evaluations further substantiate the superiority of MICM, showing that our two-stage U-FSL framework based on MICM markedly outperforms existing leading baselines. △ Less

Submitted 23 August, 2024; originally announced August 2024.

Comments: ACMMM 2024 (Oral)

Focused Discriminative Training For Streaming CTC-Trained Automatic Speech Recognition Models

Authors: Adnan Haider, Xingyu Na, Erik McDermott, Tim Ng, Zhen Huang, Xiaodan Zhuang

Abstract: This paper introduces a novel training framework called Focused Discriminative Training (FDT) to further improve streaming word-piece end-to-end (E2E) automatic speech recognition (ASR) models trained using either CTC or an interpolation of CTC and attention-based encoder-decoder (AED) loss. The proposed approach presents a novel framework to identify and improve a model's recognition on challengi… ▽ More This paper introduces a novel training framework called Focused Discriminative Training (FDT) to further improve streaming word-piece end-to-end (E2E) automatic speech recognition (ASR) models trained using either CTC or an interpolation of CTC and attention-based encoder-decoder (AED) loss. The proposed approach presents a novel framework to identify and improve a model's recognition on challenging segments of an audio. Notably, this training framework is independent of hidden Markov models (HMMs) and lattices, eliminating the need for substantial decision-making regarding HMM topology, lexicon, and graph generation, as typically required in standard discriminative training approaches. Compared to additional fine-tuning with MMI or MWER loss on the encoder, FDT is shown to be more effective in achieving greater reductions in Word Error Rate (WER) on streaming models trained on LibriSpeech. Additionally, this method is shown to be effective in further improving a converged word-piece streaming E2E model trained on 600k hours of assistant and dictation dataset. △ Less

Submitted 23 August, 2024; originally announced August 2024.

Comments: UK Speech 2024, Submitted to SLT 2024

Examining the Commitments and Difficulties Inherent in Multimodal Foundation Models for Street View Imagery

Authors: Zhenyuan Yang, Xuhui Lin, Qinyi He, Ziye Huang, Zhengliang Liu, Hanqi Jiang, Peng Shu, Zihao Wu, Yiwei Li, Stephen Law, Gengchen Mai, Tianming Liu, Tao Yang

Abstract: The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture i… ▽ More The emergence of Large Language Models (LLMs) and multimodal foundation models (FMs) has generated heightened interest in their applications that integrate vision and language. This paper investigates the capabilities of ChatGPT-4V and Gemini Pro for Street View Imagery, Built Environment, and Interior by evaluating their performance across various tasks. The assessments include street furniture identification, pedestrian and car counts, and road width measurement in Street View Imagery; building function classification, building age analysis, building height analysis, and building structure classification in the Built Environment; and interior room classification, interior design style analysis, interior furniture counts, and interior length measurement in Interior. The results reveal proficiency in length measurement, style analysis, question answering, and basic image understanding, but highlight limitations in detailed recognition and counting tasks. While zero-shot learning shows potential, performance varies depending on the problem domains and image complexities. This study provides new insights into the strengths and weaknesses of multimodal foundation models for practical challenges in Street View Imagery, Built Environment, and Interior. Overall, the findings demonstrate foundational multimodal intelligence, emphasizing the potential of FMs to drive forward interdisciplinary applications at the intersection of computer vision and language. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Towards measuring fairness in speech recognition: Fair-Speech dataset

Authors: Irina-Elena Veliche, Zhuangqun Huang, Vineeth Ayyat Kochaniyan, Fuchun Peng, Ozlem Kalinli, Michael L. Seltzer

Abstract: The current public datasets for speech recognition (ASR) tend not to focus specifically on the fairness aspect, such as performance across different demographic groups. This paper introduces a novel dataset, Fair-Speech, a publicly released corpus to help researchers evaluate their ASR models for accuracy across a diverse set of self-reported demographic information, such as age, gender, ethnicity… ▽ More The current public datasets for speech recognition (ASR) tend not to focus specifically on the fairness aspect, such as performance across different demographic groups. This paper introduces a novel dataset, Fair-Speech, a publicly released corpus to help researchers evaluate their ASR models for accuracy across a diverse set of self-reported demographic information, such as age, gender, ethnicity, geographic variation and whether the participants consider themselves native English speakers. Our dataset includes approximately 26.5K utterances in recorded speech by 593 people in the United States, who were paid to record and submit audios of themselves saying voice commands. We also provide ASR baselines, including on models trained on transcribed and untranscribed social media videos and open source models. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Automatic Organ and Pan-cancer Segmentation in Abdomen CT: the FLARE 2023 Challenge

Authors: Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Ershuai Wang, Qin Zhou, Ziyan Huang, Pengju Lyu, Jian He, Bo Wang

Abstract: Organ and cancer segmentation in abdomen Computed Tomography (CT) scans is the prerequisite for precise cancer diagnosis and treatment. Most existing benchmarks and algorithms are tailored to specific cancer types, limiting their ability to provide comprehensive cancer analysis. This work presents the first international competition on abdominal organ and pan-cancer segmentation by providing a lar… ▽ More Organ and cancer segmentation in abdomen Computed Tomography (CT) scans is the prerequisite for precise cancer diagnosis and treatment. Most existing benchmarks and algorithms are tailored to specific cancer types, limiting their ability to provide comprehensive cancer analysis. This work presents the first international competition on abdominal organ and pan-cancer segmentation by providing a large-scale and diverse dataset, including 4650 CT scans with various cancer types from over 40 medical centers. The winning team established a new state-of-the-art with a deep learning-based cascaded framework, achieving average Dice Similarity Coefficient scores of 92.3% for organs and 64.9% for lesions on the hidden multi-national testing set. The dataset and code of top teams are publicly available, offering a benchmark platform to drive further innovations https://codalab.lisn.upsaclay.fr/competitions/12239. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: MICCAI 2024 FLARE Challenge Summary

Stochastic Online Correlated Selection

Authors: Ziyun Chen, Zhiyi Huang, Enze Sun

Abstract: We study Stochastic Online Correlated Selection (SOCS), a family of online rounding algorithms for Non-IID Stochastic Online Submodular Welfare Maximization and special cases such as Online Stochastic Matching, Stochastic AdWords, and Stochastic Display Ads. At each step, the algorithm sees an online item's type and fractional allocation, then immediately allocates it to an agent. We propose a met… ▽ More We study Stochastic Online Correlated Selection (SOCS), a family of online rounding algorithms for Non-IID Stochastic Online Submodular Welfare Maximization and special cases such as Online Stochastic Matching, Stochastic AdWords, and Stochastic Display Ads. At each step, the algorithm sees an online item's type and fractional allocation, then immediately allocates it to an agent. We propose a metric called the convergence rate for the quality of SOCS. This is cleaner than most metrics in the OCS literature. We propose a Type Decomposition that reduces SOCS to the two-way special case. First, we sample a surrogate type with half-integer allocation. The rounding is trivial for a one-way type fully allocated to an agent. For a two-way type split equally between two agents, we round it using two-way SOCS. We design the distribution of surrogate types to get two-way types as often as possible while respecting the original fractional allocation in expectation. Following this framework, we make progress on numerous problems: 1) Online Stochastic Matching: We improve the state-of-the-art $0.666$ competitive ratio for unweighted/vertex-weighted matching to $0.69$. 2) Query-Commit Matching: We enhance the ratio to $0.705$ in the Query-Commit model, improving the best previous $0.696$ and $0.662$ for unweighted and vertex-weighted matching. 3) Stochastic AdWords: We give a $0.6338$ competitive algorithm, breaking the $1-\frac{1}{e}$ barrier and answering a decade-old open question. 4) AdWords: The framework applies to the adversarial model if the rounding is oblivious to future items' distributions. We get the first multi-way OCS for AdWords, addressing an open question about OCS. This gives a $0.504$ competitive ratio for AdWords, improving the previous $0.501$. 5) Stochastic Display Ads: We design a $0.644$ competitive algorithm, breaking the $1-\frac{1}{e}$ barrier. △ Less

Submitted 22 August, 2024; originally announced August 2024.

All Five-point Kaluza-Klein Correlators and Hidden 8d Symmetry in $\rm AdS_5\times S^3$

Authors: Zhongjie Huang, Bo Wang, Ellis Ye Yuan, Jiarong Zhang

Abstract: We systematically compute five-point correlators of chiral primary operators with arbitrary Kaluza-Klein charges at tree-level in $\mathrm{AdS}_5\times\mathrm{S}^3$, and obtain a unified formula. This result serves as the first concrete confirmation for the existence of the hidden eight-dimensional symmetries at the level of five points. We systematically compute five-point correlators of chiral primary operators with arbitrary Kaluza-Klein charges at tree-level in $\mathrm{AdS}_5\times\mathrm{S}^3$, and obtain a unified formula. This result serves as the first concrete confirmation for the existence of the hidden eight-dimensional symmetries at the level of five points. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: 5 pages, 2 figures and 3 appendices + ancillary files for the results

Recent Advances on Machine Learning for Computational Fluid Dynamics: A Survey

Authors: Haixin Wang, Yadi Cao, Zijie Huang, Yuxuan Liu, Peiyan Hu, Xiao Luo, Zezheng Song, Wanjia Zhao, Jilin Liu, Jinan Sun, Shikun Zhang, Long Wei, Yue Wang, Tailin Wu, Zhi-Ming Ma, Yizhou Sun

Abstract: This paper explores the recent advancements in enhancing Computational Fluid Dynamics (CFD) tasks through Machine Learning (ML) techniques. We begin by introducing fundamental concepts, traditional methods, and benchmark datasets, then examine the various roles ML plays in improving CFD. The literature systematically reviews papers in recent five years and introduces a novel classification for for… ▽ More This paper explores the recent advancements in enhancing Computational Fluid Dynamics (CFD) tasks through Machine Learning (ML) techniques. We begin by introducing fundamental concepts, traditional methods, and benchmark datasets, then examine the various roles ML plays in improving CFD. The literature systematically reviews papers in recent five years and introduces a novel classification for forward modeling: Data-driven Surrogates, Physics-Informed Surrogates, and ML-assisted Numerical Solutions. Furthermore, we also review the latest ML methods in inverse design and control, offering a novel classification and providing an in-depth discussion. Then we highlight real-world applications of ML for CFD in critical scientific and engineering disciplines, including aerodynamics, combustion, atmosphere & ocean science, biology fluid, plasma, symbolic regression, and reduced order modeling. Besides, we identify key challenges and advocate for future research directions to address these challenges, such as multi-scale representation, physical knowledge encoding, scientific foundation model and automatic scientific discovery. This review serves as a guide for the rapidly expanding ML for CFD community, aiming to inspire insights for future advancements. We draw the conclusion that ML is poised to significantly transform CFD research by enhancing simulation accuracy, reducing computational time, and enabling more complex analyses of fluid dynamics. The paper resources can be viewed at https://github.com/WillDreamer/Awesome-AI4CFD. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: 22 pages, 6 figures

RePair: Automated Program Repair with Process-based Feedback

Authors: Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, Yu Su

Abstract: The gap between the trepidation of program reliability and the expense of repairs underscores the indispensability of Automated Program Repair (APR). APR is instrumental in transforming vulnerable programs into more robust ones, bolstering program reliability while simultaneously diminishing the financial burden of manual repairs. Commercial-scale language models (LM) have taken APR to unprecedent… ▽ More The gap between the trepidation of program reliability and the expense of repairs underscores the indispensability of Automated Program Repair (APR). APR is instrumental in transforming vulnerable programs into more robust ones, bolstering program reliability while simultaneously diminishing the financial burden of manual repairs. Commercial-scale language models (LM) have taken APR to unprecedented levels. However, the emergence reveals that for models fewer than 100B parameters, making single-step modifications may be difficult to achieve the desired effect. Moreover, humans interact with the LM through explicit prompts, which hinders the LM from receiving feedback from compiler and test cases to automatically optimize its repair policies. In this literature, we explore how small-scale LM (less than 20B) achieve excellent performance through process supervision and feedback. We start by constructing a dataset named CodeNet4Repair, replete with multiple repair records, which supervises the fine-tuning of a foundational model. Building upon the encouraging outcomes of reinforcement learning, we develop a reward model that serves as a critic, providing feedback for the fine-tuned LM's action, progressively optimizing its policy. During inference, we require the LM to generate solutions iteratively until the repair effect no longer improves or hits the maximum step limit. The results show that process-based not only outperforms larger outcome-based generation methods, but also nearly matches the performance of closed-source commercial large-scale LMs. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: 15 pages, 13 figures

Journal ref: ACL 2024 Findings

Revisiting Evolutionary Program Repair via Code Language Model

Authors: Yunan Wang, Tingyu Guo, Zilong Huang, Yuan Yuan

Abstract: Software defects are an inherent part of software development and maintenance. To address these defects, Automated Program Repair (APR) has been developed to fix bugs automatically. With the advent of Large Language Models, Code Language Models (CLMs) trained on code corpora excels in code generation, making them suitable for APR applications. Despite this progress, a significant limitation remain… ▽ More Software defects are an inherent part of software development and maintenance. To address these defects, Automated Program Repair (APR) has been developed to fix bugs automatically. With the advent of Large Language Models, Code Language Models (CLMs) trained on code corpora excels in code generation, making them suitable for APR applications. Despite this progress, a significant limitation remains: many bugs necessitate multi-point edits for repair, yet current CLM-based APRs are restricted to single-point bug fixes, which severely narrows the scope of repairable bugs. Moreover, these tools typically only consider the direct context of the buggy line when building prompts for the CLM, leading to suboptimal repair outcomes due to the limited information provided. This paper introduces a novel approach, ARJA-CLM, which integrates the multiobjective evolutionary algorithm with CLM to fix multilocation bugs in Java projects. We also propose a context-aware prompt construction stratege, which enriches the prompt with additional information about accessible fields and methods for the CLM generating candidate statements. Our experiments on the Defects4J and APR-2024 competition benchmark demonstrate that ARJA-CLM surpasses many state-of-the-art repair systems, and performs well on multi-point bugs. The results also reveal that CLMs effectively utilize the provided field and method information within context-aware prompts to produce candidate statements. △ Less

Submitted 26 August, 2024; v1 submitted 19 August, 2024; originally announced August 2024.

In-Flight Performance of Spider's 280 GHz Receivers

Authors: Elle C. Shaw, P. A. R. Ade, S. Akers, M. Amiri, J. Austermann, J. Beall, D. T. Becker, S. J. Benton, A. S. Bergman, J. J. Bock, J. R. Bond, S. A. Bryan, H. C. Chiang, C. R. Contaldi, R. S. Domagalski, O. Doré, S. M. Duff, A. J. Duivenvoorden, H. K. Eriksen, M. Farhang, J. P. Filippini, L. M. Fissel, A. A. Fraisse, K. Freese, M. Galloway , et al. (62 additional authors not shown)

Abstract: SPIDER is a balloon-borne instrument designed to map the cosmic microwave background at degree-angular scales in the presence of Galactic foregrounds. SPIDER has mapped a large sky area in the Southern Hemisphere using more than 2000 transition-edge sensors (TESs) during two NASA Long Duration Balloon flights above the Antarctic continent. During its first flight in January 2015, SPIDER observed i… ▽ More SPIDER is a balloon-borne instrument designed to map the cosmic microwave background at degree-angular scales in the presence of Galactic foregrounds. SPIDER has mapped a large sky area in the Southern Hemisphere using more than 2000 transition-edge sensors (TESs) during two NASA Long Duration Balloon flights above the Antarctic continent. During its first flight in January 2015, SPIDER observed in the 95 GHz and 150 GHz frequency bands, setting constraints on the B-mode signature of primordial gravitational waves. Its second flight in the 2022-23 season added new receivers at 280 GHz, each using an array of TESs coupled to the sky through feedhorns formed from stacks of silicon wafers. These receivers are optimized to produce deep maps of polarized Galactic dust emission over a large sky area, providing a unique data set with lasting value to the field. In this work, we describe the instrument's performance during SPIDER's second flight. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Comments: Submitted to SPIE Astronomical Telescopes + Instrumentation 2024, JATIS

Stream-Based Ground Segmentation for Real-Time LiDAR Point Cloud Processing on FPGA

Authors: Xiao Zhang, Zhanhong Huang, Garcia Gonzalez Antony, Witek Jachimczyk, Xinming Huang

Abstract: This paper presents a novel and fast approach for ground plane segmentation in a LiDAR point cloud, specifically optimized for processing speed and hardware efficiency on FPGA hardware platforms. Our approach leverages a channel-based segmentation method with an advanced angular data repair technique and a cross-eight-way flood-fill algorithm. This innovative approach significantly reduces the num… ▽ More This paper presents a novel and fast approach for ground plane segmentation in a LiDAR point cloud, specifically optimized for processing speed and hardware efficiency on FPGA hardware platforms. Our approach leverages a channel-based segmentation method with an advanced angular data repair technique and a cross-eight-way flood-fill algorithm. This innovative approach significantly reduces the number of iterations while ensuring the high accuracy of the segmented ground plane, which makes the stream-based hardware implementation possible. To validate the proposed approach, we conducted extensive experiments on the SemanticKITTI dataset. We introduced a bird's-eye view (BEV) evaluation metric tailored for the area representation of LiDAR segmentation tasks. Our method demonstrated superior performance in terms of BEV areas when compared to the existing approaches. Moreover, we presented an optimized hardware architecture targeted on a Zynq-7000 FPGA, compatible with LiDARs of various channel densities, i.e., 32, 64, and 128 channels. Our FPGA implementation operating at 160 MHz significantly outperforms the traditional computing platforms, which is 12 to 25 times faster than the CPU-based solutions and up to 6 times faster than the GPU-based solution, in addition to the benefit of low power consumption. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Parallel Processing of Point Cloud Ground Segmentation for Mechanical and Solid-State LiDARs

Authors: Xiao Zhang, Zhanhong Huang, Garcia Gonzalez Antony, Witek Jachimczyk, Xinming Huang

Abstract: In this study, we introduce a novel parallel processing framework for real-time point cloud ground segmentation on FPGA platforms, aimed at adapting LiDAR algorithms to the evolving landscape from mechanical to solid-state LiDAR (SSL) technologies. Focusing on the ground segmentation task, we explore parallel processing techniques on existing approaches and adapt them to real-world SSL data handli… ▽ More In this study, we introduce a novel parallel processing framework for real-time point cloud ground segmentation on FPGA platforms, aimed at adapting LiDAR algorithms to the evolving landscape from mechanical to solid-state LiDAR (SSL) technologies. Focusing on the ground segmentation task, we explore parallel processing techniques on existing approaches and adapt them to real-world SSL data handling. We validated frame-segmentation based parallel processing methods using point-based, voxel-based, and range-image-based ground segmentation approaches on the SemanticKITTI dataset based on mechanical LiDAR. The results revealed the superior performance and robustness of the range-image method, especially in its resilience to slicing. Further, utilizing a custom dataset from our self-built Camera-SSLSS equipment, we examined regular SSL data frames and validated the effectiveness of our parallel approach for SSL sensor. Additionally, our pioneering implementation of range-image ground segmentation on FPGA for SSL sensors demonstrated significant processing speed improvements and resource efficiency, achieving processing rates up to 50.3 times faster than conventional CPU setups. These findings underscore the potential of parallel processing strategies to significantly enhance LiDAR technologies for advanced perception tasks in autonomous systems. Post-publication, both the data and the code will be made available on GitHub. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Comments: 5 pages

FFAA: Multimodal Large Language Model based Explainable Open-World Face Forgery Analysis Assistant

Authors: Zhengchao Huang, Bin Xia, Zicheng Lin, Zhun Mou, Wenming Yang

Abstract: The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptions of these aspects, making it d… ▽ More The rapid advancement of deepfake technologies has sparked widespread public concern, particularly as face forgery poses a serious threat to public information security. However, the unknown and diverse forgery techniques, varied facial features and complex environmental factors pose significant challenges for face forgery analysis. Existing datasets lack descriptions of these aspects, making it difficult for models to distinguish between real and forged faces using only visual information amid various confounding factors. In addition, existing methods do not yield user-friendly and explainable results, complicating the understanding of the model's decision-making process. To address these challenges, we introduce a novel Open-World Face Forgery Analysis VQA (OW-FFA-VQA) task and the corresponding benchmark. To tackle this task, we first establish a dataset featuring a diverse collection of real and forged face images with essential descriptions and reliable forgery reasoning. Base on this dataset, we introduce FFAA: Face Forgery Analysis Assistant, consisting of a fine-tuned Multimodal Large Language Model (MLLM) and Multi-answer Intelligent Decision System (MIDS). By integrating hypothetical prompts with MIDS, the impact of fuzzy classification boundaries is effectively mitigated, enhancing the model's robustness. Extensive experiments demonstrate that our method not only provides user-friendly explainable results but also significantly boosts accuracy and robustness compared to previous methods. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Comments: 17 pages, 18 figures; project page: https://ffaa-vl.github.io

Data-driven Conditional Instrumental Variables for Debiasing Recommender Systems

Authors: Zhirong Huang, Shichao Zhang, Debo Cheng, Jiuyong Li, Lin Liu, Guangquan Lu

Abstract: In recommender systems, latent variables can cause user-item interaction data to deviate from true user preferences. This biased data is then used to train recommendation models, further amplifying the bias and ultimately compromising both recommendation accuracy and user satisfaction. Instrumental Variable (IV) methods are effective tools for addressing the confounding bias introduced by latent v… ▽ More In recommender systems, latent variables can cause user-item interaction data to deviate from true user preferences. This biased data is then used to train recommendation models, further amplifying the bias and ultimately compromising both recommendation accuracy and user satisfaction. Instrumental Variable (IV) methods are effective tools for addressing the confounding bias introduced by latent variables; however, identifying a valid IV is often challenging. To overcome this issue, we propose a novel data-driven conditional IV (CIV) debiasing method for recommender systems, called CIV4Rec. CIV4Rec automatically generates valid CIVs and their corresponding conditioning sets directly from interaction data, significantly reducing the complexity of IV selection while effectively mitigating the confounding bias caused by latent variables in recommender systems. Specifically, CIV4Rec leverages a variational autoencoder (VAE) to generate the representations of the CIV and its conditional set from interaction data, followed by the application of least squares to derive causal representations for click prediction. Extensive experiments on two real-world datasets, Movielens-10M and Douban-Movie, demonstrate that our CIV4Rec successfully identifies valid CIVs, effectively reduces bias, and consequently improves recommendation accuracy. △ Less

Submitted 18 August, 2024; originally announced August 2024.

Debiased Contrastive Representation Learning for Mitigating Dual Biases in Recommender Systems

Authors: Zhirong Huang, Shichao Zhang, Debo Cheng, Jiuyong Li, Lin Liu, Guixian Zhang

Abstract: In recommender systems, popularity and conformity biases undermine recommender effectiveness by disproportionately favouring popular items, leading to their over-representation in recommendation lists and causing an unbalanced distribution of user-item historical data. We construct a causal graph to address both biases and describe the abstract data generation mechanism. Then, we use it as a guide… ▽ More In recommender systems, popularity and conformity biases undermine recommender effectiveness by disproportionately favouring popular items, leading to their over-representation in recommendation lists and causing an unbalanced distribution of user-item historical data. We construct a causal graph to address both biases and describe the abstract data generation mechanism. Then, we use it as a guide to develop a novel Debiased Contrastive Learning framework for Mitigating Dual Biases, called DCLMDB. In DCLMDB, both popularity bias and conformity bias are handled in the model training process by contrastive learning to ensure that user choices and recommended items are not unduly influenced by conformity and popularity. Extensive experiments on two real-world datasets, Movielens-10M and Netflix, show that DCLMDB can effectively reduce the dual biases, as well as significantly enhance the accuracy and diversity of recommendations. △ Less

Submitted 18 August, 2024; originally announced August 2024.

FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model

Authors: Ziyu Yao, Xuxin Cheng, Zhiqi Huang

Abstract: Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they… ▽ More Talking head generation is a significant research topic that still faces numerous challenges. Previous works often adopt generative adversarial networks or regression models, which are plagued by generation quality and average facial shape problem. Although diffusion models show impressive generative ability, their exploration in talking head generation remains unsatisfactory. This is because they either solely use the diffusion model to obtain an intermediate representation and then employ another pre-trained renderer, or they overlook the feature decoupling of complex facial details, such as expressions, head poses and appearance textures. Therefore, we propose a Facial Decoupled Diffusion model for Talking head generation called FD2Talk, which fully leverages the advantages of diffusion models and decouples the complex facial details through multi-stages. Specifically, we separate facial details into motion and appearance. In the initial phase, we design the Diffusion Transformer to accurately predict motion coefficients from raw audio. These motions are highly decoupled from appearance, making them easier for the network to learn compared to high-dimensional RGB images. Subsequently, in the second phase, we encode the reference image to capture appearance textures. The predicted facial and head motions and encoded appearance then serve as the conditions for the Diffusion UNet, guiding the frame generation. Benefiting from decoupling facial details and fully leveraging diffusion models, extensive experiments substantiate that our approach excels in enhancing image quality and generating more accurate and diverse results compared to previous state-of-the-art methods. △ Less

Submitted 18 August, 2024; originally announced August 2024.

Comments: Accepted by ACM Multimedia 2024

V2X-VLM: End-to-End V2X Cooperative Autonomous Driving Through Large Vision-Language Models

Authors: Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, Bin Ran

Abstract: Advancements in autonomous driving have increasingly focused on end-to-end (E2E) systems that manage the full spectrum of driving tasks, from environmental perception to vehicle navigation and control. This paper introduces V2X-VLM, an innovative E2E vehicle-infrastructure cooperative autonomous driving (VICAD) framework with large vision-language models (VLMs). V2X-VLM is designed to enhance situ… ▽ More Advancements in autonomous driving have increasingly focused on end-to-end (E2E) systems that manage the full spectrum of driving tasks, from environmental perception to vehicle navigation and control. This paper introduces V2X-VLM, an innovative E2E vehicle-infrastructure cooperative autonomous driving (VICAD) framework with large vision-language models (VLMs). V2X-VLM is designed to enhance situational awareness, decision-making, and ultimate trajectory planning by integrating data from vehicle-mounted cameras, infrastructure sensors, and textual information. The strength of the comprehensive multimodel data fusion of the VLM enables precise and safe E2E trajectory planning in complex and dynamic driving scenarios. Validation on the DAIR-V2X dataset demonstrates that V2X-VLM outperforms existing state-of-the-art methods in cooperative autonomous driving. △ Less

Submitted 17 August, 2024; originally announced August 2024.

GSLAMOT: A Tracklet and Query Graph-based Simultaneous Locating, Mapping, and Multiple Object Tracking System

Authors: Shuo Wang, Yongcai Wang, Zhimin Xu, Yongyu Guo, Wanting Li, Zhe Huang, Xuewei Bai, Deying Li

Abstract: For interacting with mobile objects in unfamiliar environments, simultaneously locating, mapping, and tracking the 3D poses of multiple objects are crucially required. This paper proposes a Tracklet Graph and Query Graph-based framework, i.e., GSLAMOT, to address this challenge. GSLAMOT utilizes camera and LiDAR multimodal information as inputs and divides the representation of the dynamic scene i… ▽ More For interacting with mobile objects in unfamiliar environments, simultaneously locating, mapping, and tracking the 3D poses of multiple objects are crucially required. This paper proposes a Tracklet Graph and Query Graph-based framework, i.e., GSLAMOT, to address this challenge. GSLAMOT utilizes camera and LiDAR multimodal information as inputs and divides the representation of the dynamic scene into a semantic map for representing the static environment, a trajectory of the ego-agent, and an online maintained Tracklet Graph (TG) for tracking and predicting the 3D poses of the detected mobile objects. A Query Graph (QG) is constructed in each frame by object detection to query and update TG. For accurate object association, a Multi-criteria Star Graph Association (MSGA) method is proposed to find matched objects between the detections in QG and the predicted tracklets in TG. Then, an Object-centric Graph Optimization (OGO) method is proposed to simultaneously optimize the TG, the semantic map, and the agent trajectory. It triangulates the detected objects into the map to enrich the map's semantic information. We address the efficiency issues to handle the three tightly coupled tasks in parallel. Experiments are conducted on KITTI, Waymo, and an emulated Traffic Congestion dataset that highlights challenging scenarios. Experiments show that GSLAMOT enables accurate crowded object tracking while conducting SLAM accurately in challenging scenarios, demonstrating more excellent performances than the state-of-the-art methods. The code and dataset are at https://gslamot.github.io. △ Less

Submitted 17 August, 2024; originally announced August 2024.

Comments: 11 pages, 9 figures, ACM MM 2024

SSNeRF: Sparse View Semi-supervised Neural Radiance Fields with Augmentation

Authors: Xiao Cao, Beibei Lin, Bo Wang, Zhiyong Huang, Robby T. Tan

Abstract: Sparse view NeRF is challenging because limited input images lead to an under constrained optimization problem for volume rendering. Existing methods address this issue by relying on supplementary information, such as depth maps. However, generating this supplementary information accurately remains problematic and often leads to NeRF producing images with undesired artifacts. To address these arti… ▽ More Sparse view NeRF is challenging because limited input images lead to an under constrained optimization problem for volume rendering. Existing methods address this issue by relying on supplementary information, such as depth maps. However, generating this supplementary information accurately remains problematic and often leads to NeRF producing images with undesired artifacts. To address these artifacts and enhance robustness, we propose SSNeRF, a sparse view semi supervised NeRF method based on a teacher student framework. Our key idea is to challenge the NeRF module with progressively severe sparse view degradation while providing high confidence pseudo labels. This approach helps the NeRF model become aware of noise and incomplete information associated with sparse views, thus improving its robustness. The novelty of SSNeRF lies in its sparse view specific augmentations and semi supervised learning mechanism. In this approach, the teacher NeRF generates novel views along with confidence scores, while the student NeRF, perturbed by the augmented input, learns from the high confidence pseudo labels. Our sparse view degradation augmentation progressively injects noise into volume rendering weights, perturbs feature maps in vulnerable layers, and simulates sparse view blurriness. These augmentation strategies force the student NeRF to recognize degradation and produce clearer rendered views. By transferring the student's parameters to the teacher, the teacher gains increased robustness in subsequent training iterations. Extensive experiments demonstrate the effectiveness of our SSNeRF in generating novel views with less sparse view degradation. We will release code upon acceptance. △ Less

Submitted 17 August, 2024; originally announced August 2024.

Multi-Antenna Broadband Backscatter Communications

Authors: Hao Chen, Zhizhi Huang, Ying-Chang Liang, Robert Schober

Abstract: Backscatter communication offers a promising solution to connect massive Internet-of-Things (IoT) devices with low cost and high energy efficiency. Nevertheless, its inherently passive nature limits transmission reliability, thereby hindering improvements in communication range and data rate. To overcome these challenges, we introduce a bistatic broadband backscatter communication (BBBC) system, w… ▽ More Backscatter communication offers a promising solution to connect massive Internet-of-Things (IoT) devices with low cost and high energy efficiency. Nevertheless, its inherently passive nature limits transmission reliability, thereby hindering improvements in communication range and data rate. To overcome these challenges, we introduce a bistatic broadband backscatter communication (BBBC) system, which equips the backscatter device (BD) with multiple antennas. In the proposed BBBC system, a radio frequency (RF) source directs a sinusoidal signal to the BD, facilitating single-carrier block transmission at the BD. Meanwhile, without requiring channel state information (CSI), cyclic delay diversity (CDD) is employed at the multi-antenna BD to enhance transmission reliability through additional cyclically delayed backscattered signals. We also propose a receiver design that includes preprocessing of the time-domain received signal, pilot-based parameter estimation, and frequency-domain equalization, enabling low-complexity detection of the backscattered signal. Leveraging the matched filter bound (MFB), we analyze the achievable diversity gains in terms of outage probability. Our analysis reveals that spatial diversity is achievable under general Rayleigh fading conditions, and both frequency and spatial diversity are attainable in scenarios where the forward link experiences a line-of-sight (LoS) channel. Simulation results validate the effectiveness of the proposed BBBC system. As the number of BD antennas increases, our results show that the proposed scheme not only enhances array gain but also improves diversity order, significantly reducing both outage probability and bit error rate (BER). Consequently, it outperforms conventional schemes that yield only minor gains. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Penny-Wise and Pound-Foolish in Deepfake Detection

Authors: Yabin Wang, Zhiwu Huang, Su Zhou, Adam Prugel-Bennett, Xiaopeng Hong

Abstract: The diffusion of deepfake technologies has sparked serious concerns about its potential misuse across various domains, prompting the urgent need for robust detection methods. Despite advancement, many current approaches prioritize short-term gains at expense of long-term effectiveness. This paper critiques the overly specialized approach of fine-tuning pre-trained models solely with a penny-wise o… ▽ More The diffusion of deepfake technologies has sparked serious concerns about its potential misuse across various domains, prompting the urgent need for robust detection methods. Despite advancement, many current approaches prioritize short-term gains at expense of long-term effectiveness. This paper critiques the overly specialized approach of fine-tuning pre-trained models solely with a penny-wise objective on a single deepfake dataset, while disregarding the pound-wise balance for generalization and knowledge retention. To address this "Penny-Wise and Pound-Foolish" issue, we propose a novel learning framework (PoundNet) for generalization of deepfake detection on a pre-trained vision-language model. PoundNet incorporates a learnable prompt design and a balanced objective to preserve broad knowledge from upstream tasks (object classification) while enhancing generalization for downstream tasks (deepfake detection). We train PoundNet on a standard single deepfake dataset, following common practice in the literature. We then evaluate its performance across 10 public large-scale deepfake datasets with 5 main evaluation metrics-forming the largest benchmark test set for assessing the generalization ability of deepfake detection models, to our knowledge. The comprehensive benchmark evaluation demonstrates the proposed PoundNet is significantly less "Penny-Wise and Pound-Foolish", achieving a remarkable improvement of 19% in deepfake detection performance compared to state-of-the-art methods, while maintaining a strong performance of 63% on object classification tasks, where other deepfake detection models tend to be ineffective. Code and data are open-sourced at https://github.com/iamwangyabin/PoundNet. △ Less

Submitted 15 August, 2024; originally announced August 2024.

Layerwise Recurrent Router for Mixture-of-Experts

Authors: Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, Jie Fu

Abstract: The scaling of large language models (LLMs) has revolutionized their capabilities in various tasks, yet this growth must be matched with efficient computational strategies. The Mixture-of-Experts (MoE) architecture stands out for its ability to scale model size without significantly increasing training costs. Despite their advantages, current MoE models often display parameter inefficiency. For in… ▽ More The scaling of large language models (LLMs) has revolutionized their capabilities in various tasks, yet this growth must be matched with efficient computational strategies. The Mixture-of-Experts (MoE) architecture stands out for its ability to scale model size without significantly increasing training costs. Despite their advantages, current MoE models often display parameter inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion parameters might perform comparably to a standard model with 6.7 billion parameters. Being a crucial part of MoE, current routers in different layers independently assign tokens without leveraging historical routing information, potentially leading to suboptimal token-expert combinations and the parameter inefficiency problem. To alleviate this issue, we introduce the Layerwise Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated Recurrent Unit (GRU) to establish dependencies between routing decisions across consecutive layers. Such layerwise recurrence can be efficiently parallelly computed for input tokens and introduces negotiable costs. Our extensive empirical evaluations demonstrate that RMoE-based language models consistently outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel computation stage orthogonal to existing methods, allowing seamless compatibility with other MoE architectures. Our analyses attribute RMoE's gains to its effective cross-layer information sharing, which also improves expert selection and diversity. Our code is at https://github.com/qiuzh20/RMoE △ Less

Submitted 13 August, 2024; originally announced August 2024.

Emergent superconductivity and pair density wave at antiphase boundaries of charge density wave order in kagome metals

Authors: Xianghe Han, Hui Chen, Hengxin Tan, Zhongyi Cao, Zihao Huang, Yuhan Ye, Zhen Zhao, Chengmin Shen, Haitao Yang, Binghai Yan, Ziqiang Wang, Hong-Jun Gao

Abstract: Central to the layered kagome lattice superconductors AV3Sb5 (A = K, Cs, Rb) is a cascade of novel quantum states triggered by an unconventional charge density wave (CDW) order. The three-dimensional (3D) order involves a 2x2x2 phase coherent stacking of 2x2 charge density modulations in the kagome plane at low temperatures, exhibiting a CDW energy gap and evidence for time-reversal symmetry break… ▽ More Central to the layered kagome lattice superconductors AV3Sb5 (A = K, Cs, Rb) is a cascade of novel quantum states triggered by an unconventional charge density wave (CDW) order. The three-dimensional (3D) order involves a 2x2x2 phase coherent stacking of 2x2 charge density modulations in the kagome plane at low temperatures, exhibiting a CDW energy gap and evidence for time-reversal symmetry breaking. Here we report the discovery of emergent superconductivity and primary pair density wave (PDW) at the antiphase boundaries and stacking faults of bulk CDW order. We find that the π-phase shift dislocations can naturally appear on the surface as the Cs atoms form 2x2 superstructures that are out of phase with the bulk CDW. An incipient narrow band of surface states inside bulk CDW gap emerge close to the Fermi level where a particle-hole symmetric energy gap develops. We demonstrate that the energy gap originates from a novel quasi-2D kagome superconducting state (Tc ~ 5.4 K) intertwined with bulk CDW order, exhibiting an unprecedented vortex core spectrum and spatial modulations of the superconducting gap consistent with a 4x4 PDW. Intriguingly, the 2D kagome superconductivity is shown to be tunable on and off by atomically manipulating the Cs atoms on the surface. Our findings provide fresh new insights for understanding the interplay between the unconventional CDW and superconductivity in kagome metals and a pathway for atomic manipulation and topological defects engineering of quantum many-body states in correlated materials. △ Less

Submitted 12 August, 2024; originally announced August 2024.

MV2DFusion: Leveraging Modality-Specific Object Semantics for Multi-Modal 3D Detection

Authors: Zitian Wang, Zehao Huang, Yulu Gao, Naiyan Wang, Si Liu

Abstract: The rise of autonomous vehicles has significantly increased the demand for robust 3D object detection systems. While cameras and LiDAR sensors each offer unique advantages--cameras provide rich texture information and LiDAR offers precise 3D spatial data--relying on a single modality often leads to performance limitations. This paper introduces MV2DFusion, a multi-modal detection framework that in… ▽ More The rise of autonomous vehicles has significantly increased the demand for robust 3D object detection systems. While cameras and LiDAR sensors each offer unique advantages--cameras provide rich texture information and LiDAR offers precise 3D spatial data--relying on a single modality often leads to performance limitations. This paper introduces MV2DFusion, a multi-modal detection framework that integrates the strengths of both worlds through an advanced query-based fusion mechanism. By introducing an image query generator to align with image-specific attributes and a point cloud query generator, MV2DFusion effectively combines modality-specific object semantics without biasing toward one single modality. Then the sparse fusion process can be accomplished based on the valuable object semantics, ensuring efficient and accurate object detection across various scenarios. Our framework's flexibility allows it to integrate with any image and point cloud-based detectors, showcasing its adaptability and potential for future advancements. Extensive evaluations on the nuScenes and Argoverse2 datasets demonstrate that MV2DFusion achieves state-of-the-art performance, particularly excelling in long-range detection scenarios. △ Less

Submitted 12 August, 2024; originally announced August 2024.

PS-TTL: Prototype-based Soft-labels and Test-Time Learning for Few-shot Object Detection

Authors: Yingjie Gao, Yanan Zhang, Ziyue Huang, Nanqing Liu, Di Huang

Abstract: In recent years, Few-Shot Object Detection (FSOD) has gained widespread attention and made significant progress due to its ability to build models with a good generalization power using extremely limited annotated data. The fine-tuning based paradigm is currently dominating this field, where detectors are initially pre-trained on base classes with sufficient samples and then fine-tuned on novel on… ▽ More In recent years, Few-Shot Object Detection (FSOD) has gained widespread attention and made significant progress due to its ability to build models with a good generalization power using extremely limited annotated data. The fine-tuning based paradigm is currently dominating this field, where detectors are initially pre-trained on base classes with sufficient samples and then fine-tuned on novel ones with few samples, but the scarcity of labeled samples of novel classes greatly interferes precisely fitting their data distribution, thus hampering the performance. To address this issue, we propose a new framework for FSOD, namely Prototype-based Soft-labels and Test-Time Learning (PS-TTL). Specifically, we design a Test-Time Learning (TTL) module that employs a mean-teacher network for self-training to discover novel instances from test data, allowing detectors to learn better representations and classifiers for novel classes. Furthermore, we notice that even though relatively low-confidence pseudo-labels exhibit classification confusion, they still tend to recall foreground. We thus develop a Prototype-based Soft-labels (PS) strategy through assessing similarities between low-confidence pseudo-labels and category prototypes as soft-labels to unleash their potential, which substantially mitigates the constraints posed by few-shot samples. Extensive experiments on both the VOC and COCO benchmarks show that PS-TTL achieves the state-of-the-art, highlighting its effectiveness. The code and model are available at https://github.com/gaoyingjay/PS-TTL. △ Less

Submitted 10 August, 2024; originally announced August 2024.

Comments: Accepted to ACM MM 2024

Auslander-type conditions and weakly Gorenstein algebras

Authors: Zhaoyong Huang

Abstract: Let $R$ be an Artin algebra. Under certain Auslander-type conditions, we give some equivalent characterizations of (weakly) Gorenstein algebras in terms of the properties of Gorenstein projective modules and modules satisfying Auslander-type conditions. As applications, we provide some support for several homological conjectures. In particular, we prove that if $R$ is left quasi Auslander, then… ▽ More Let $R$ be an Artin algebra. Under certain Auslander-type conditions, we give some equivalent characterizations of (weakly) Gorenstein algebras in terms of the properties of Gorenstein projective modules and modules satisfying Auslander-type conditions. As applications, we provide some support for several homological conjectures. In particular, we prove that if $R$ is left quasi Auslander, then $R$ is Gorenstein if and only if it is (left and) right weakly Gorenstein; and that if $R$ satisfies the Auslander condition, then $R$ is Gorenstein if and only if it is left or right weakly Gorenstein. This is a reduction of an Auslander--Reiten's conjecture, which states that $R$ is Gorenstein if $R$ satisfies the Auslander condition. △ Less

Submitted 10 August, 2024; originally announced August 2024.

Comments: 16 pages; accepted for publication in Bulletin of the London Mathematical Society

MSC Class: 16E65; 16E10; 18G25

One-dimensional spin-flipping topological edge state laser

Authors: Jhih-Sheng Wu, Zhen-Ting Huang, Meng-Ting Han, Yen-Hsun Chen, Tien-Chang Lu

Abstract: Topological edge states manifest spin-momentum-locking propagation as a primary consequence of topological crystals. However, experimental studies on spin manipulation and the resulting propagation of these states are lacking. Here, we demonstrate experimentally spin manipulation of topological edge states by the boundary conditions of the one-dimensional path. Armchair boundaries at the endpoints… ▽ More Topological edge states manifest spin-momentum-locking propagation as a primary consequence of topological crystals. However, experimental studies on spin manipulation and the resulting propagation of these states are lacking. Here, we demonstrate experimentally spin manipulation of topological edge states by the boundary conditions of the one-dimensional path. Armchair boundaries at the endpoints of the path induce spin-flipping back-scattering, resulting in a novel one-dimensional resonance -- traveling resonance. Remarkably, we demonstrate lasing of this one-dimensional traveling resonance. Our findings hold significant potential for practical applications in spin manipulation of light. △ Less

Submitted 9 August, 2024; originally announced August 2024.

Comments: 9 pages, 6 figures

Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?

Authors: Yu Zhao, Lina Gong, Zhiqiu Huang, Yongwei Wang, Mingqiang Wei, Fei Wu

Abstract: Vulnerability detection is garnering increasing attention in software engineering, since code vulnerabilities possibly pose significant security. Recently, reusing various code pre-trained models has become common for code embedding without providing reasonable justifications in vulnerability detection. The premise for casually utilizing pre-trained models (PTMs) is that the code embeddings genera… ▽ More Vulnerability detection is garnering increasing attention in software engineering, since code vulnerabilities possibly pose significant security. Recently, reusing various code pre-trained models has become common for code embedding without providing reasonable justifications in vulnerability detection. The premise for casually utilizing pre-trained models (PTMs) is that the code embeddings generated by different PTMs would generate a similar impact on the performance. Is that TRUE? To answer this important question, we systematically investigate the effects of code embedding generated by ten different code PTMs on the performance of vulnerability detection, and get the answer, i.e., that is NOT true. We observe that code embedding generated by various code PTMs can indeed influence the performance and selecting an embedding technique based on parameter scales and embedding dimension is not reliable. Our findings highlight the necessity of quantifying and evaluating the characteristics of code embedding generated by various code PTMs to understand the effects. To achieve this goal, we analyze the numerical representation and data distribution of code embedding generated by different PTMs to evaluate differences and characteristics. Based on these insights, we propose Coding-PTMs, a recommendation framework to assist engineers in selecting optimal code PTMs for their specific vulnerability detection tasks. Specifically, we define thirteen code embedding metrics across three dimensions (i.e., statistics, norm, and distribution) for constructing a specialized code PTM recommendation dataset. We then employ a Random Forest classifier to train a recommendation model and identify the optimal code PTMs from the candidate model zoo. △ Less

Submitted 9 August, 2024; originally announced August 2024.

Comments: Accepted by ASE 2024

DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mamba

Authors: Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, Marcelo H. Ang Jr

Abstract: Motion planning is a challenging task to generate safe and feasible trajectories in highly dynamic and complex environments, forming a core capability for autonomous vehicles. In this paper, we propose DRAMA, the first Mamba-based end-to-end motion planner for autonomous vehicles. DRAMA fuses camera, LiDAR Bird's Eye View images in the feature space, as well as ego status information, to generate… ▽ More Motion planning is a challenging task to generate safe and feasible trajectories in highly dynamic and complex environments, forming a core capability for autonomous vehicles. In this paper, we propose DRAMA, the first Mamba-based end-to-end motion planner for autonomous vehicles. DRAMA fuses camera, LiDAR Bird's Eye View images in the feature space, as well as ego status information, to generate a series of future ego trajectories. Unlike traditional transformer-based methods with quadratic attention complexity for sequence length, DRAMA is able to achieve a less computationally intensive attention complexity, demonstrating potential to deal with increasingly complex scenarios. Leveraging our Mamba fusion module, DRAMA efficiently and effectively fuses the features of the camera and LiDAR modalities. In addition, we introduce a Mamba-Transformer decoder that enhances the overall planning performance. This module is universally adaptable to any Transformer-based model, especially for tasks with long sequence inputs. We further introduce a novel feature state dropout which improves the planner's robustness without increasing training and inference times. Extensive experimental results show that DRAMA achieves higher accuracy on the NAVSIM dataset compared to the baseline Transfuser, with fewer parameters and lower computational costs. △ Less

Submitted 14 August, 2024; v1 submitted 7 August, 2024; originally announced August 2024.

MoExtend: Tuning New Experts for Modality and Task Extension

Authors: Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou

Abstract: Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fi… ▽ More Large language models (LLMs) excel in various tasks but are primarily trained on text data, limiting their application scope. Expanding LLM capabilities to include vision-language understanding is vital, yet training them on multimodal data from scratch is challenging and costly. Existing instruction tuning methods, e.g., LLAVA, often connects a pretrained CLIP vision encoder and LLMs via fully fine-tuning LLMs to bridge the modality gap. However, full fine-tuning is plagued by catastrophic forgetting, i.e., forgetting previous knowledge, and high training costs particularly in the era of increasing tasks and modalities. To solve this issue, we introduce MoExtend, an effective framework designed to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. MoExtend seamlessly integrates new experts into pre-trained MoE models, endowing them with novel knowledge without the need to tune pretrained models such as MoE and vision encoders. This approach enables rapid adaptation and extension to new modal data or tasks, effectively addressing the challenge of accommodating new modalities within LLMs. Furthermore, MoExtend avoids tuning pretrained models, thus mitigating the risk of catastrophic forgetting. Experimental results demonstrate the efficacy and efficiency of MoExtend in enhancing the multimodal capabilities of LLMs, contributing to advancements in multimodal AI research. Code: https://github.com/zhongshsh/MoExtend. △ Less

Submitted 6 August, 2024; originally announced August 2024.

Comments: ACL 2024 - SRW

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Authors: Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

Abstract: Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Curren… ▽ More Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 285 datasets across 39 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 52%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI. △ Less

Submitted 9 August, 2024; v1 submitted 6 August, 2024; originally announced August 2024.

FastEdit: Fast Text-Guided Single-Image Editing via Semantic-Aware Diffusion Fine-Tuning

Authors: Zhi Chen, Zecheng Zhao, Yadan Luo, Zi Huang

Abstract: Conventional Text-guided single-image editing approaches require a two-step process, including fine-tuning the target text embedding for over 1K iterations and the generative model for another 1.5K iterations. Although it ensures that the resulting image closely aligns with both the input image and the target text, this process often requires 7 minutes per image, posing a challenge for practical a… ▽ More Conventional Text-guided single-image editing approaches require a two-step process, including fine-tuning the target text embedding for over 1K iterations and the generative model for another 1.5K iterations. Although it ensures that the resulting image closely aligns with both the input image and the target text, this process often requires 7 minutes per image, posing a challenge for practical application due to its time-intensive nature. To address this bottleneck, we introduce FastEdit, a fast text-guided single-image editing method with semantic-aware diffusion fine-tuning, dramatically accelerating the editing process to only 17 seconds. FastEdit streamlines the generative model's fine-tuning phase, reducing it from 1.5K to a mere 50 iterations. For diffusion fine-tuning, we adopt certain time step values based on the semantic discrepancy between the input image and target text. Furthermore, FastEdit circumvents the initial fine-tuning step by utilizing an image-to-image model that conditions on the feature space, rather than the text embedding space. It can effectively align the target text prompt and input image within the same feature space and save substantial processing time. Additionally, we apply the parameter-efficient fine-tuning technique LoRA to U-net. With LoRA, FastEdit minimizes the model's trainable parameters to only 0.37\% of the original size. At the same time, we can achieve comparable editing outcomes with significantly reduced computational overhead. We conduct extensive experiments to validate the editing performance of our approach and show promising editing capabilities, including content addition, style transfer, background replacement, and posture manipulation, etc. △ Less

Submitted 6 August, 2024; originally announced August 2024.

Comments: Technical Report

Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline

Authors: Tianqi Wei, Zhi Chen, Zi Huang, Xin Yu

Abstract: Existing plant disease classification models have achieved remarkable performance in recognizing in-laboratory diseased images. However, their performance often significantly degrades in classifying in-the-wild images. Furthermore, we observed that in-the-wild plant images may exhibit similar appearances across various diseases (i.e., small inter-class discrepancy) while the same diseases may look… ▽ More Existing plant disease classification models have achieved remarkable performance in recognizing in-laboratory diseased images. However, their performance often significantly degrades in classifying in-the-wild images. Furthermore, we observed that in-the-wild plant images may exhibit similar appearances across various diseases (i.e., small inter-class discrepancy) while the same diseases may look quite different (i.e., large intra-class variance). Motivated by this observation, we propose an in-the-wild multimodal plant disease recognition dataset that contains the largest number of disease classes but also text-based descriptions for each disease. Particularly, the newly provided text descriptions are introduced to provide rich information in textual modality and facilitate in-the-wild disease classification with small inter-class discrepancy and large intra-class variance issues. Therefore, our proposed dataset can be regarded as an ideal testbed for evaluating disease recognition methods in the real world. In addition, we further present a strong yet versatile baseline that models text descriptions and visual data through multiple prototypes for a given class. By fusing the contributions of multimodal prototypes in classification, our baseline can effectively address the small inter-class discrepancy and large intra-class variance issues. Remarkably, our baseline model can not only classify diseases but also recognize diseases in few-shot or training-free scenarios. Extensive benchmarking results demonstrate that our proposed in-the-wild multimodal dataset sets many new challenges to the plant disease recognition task and there is a large space to improve for future works. △ Less

Submitted 6 August, 2024; originally announced August 2024.

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

Authors: Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, Tao Wang

Abstract: Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental q… ▽ More Significant advancements has recently been achieved in the field of multi-modal large language models (MLLMs), demonstrating their remarkable capabilities in understanding and reasoning across diverse tasks. However, these models are often trained for specific tasks and rely on task-specific input-output formats, limiting their applicability to a broader range of tasks. This raises a fundamental question: Can we develop a unified approach to represent and handle different multi-modal tasks to maximize the generalizability of MLLMs? In this paper, we propose UnifiedMLLM, a comprehensive model designed to represent various tasks using a unified representation. Our model exhibits strong capabilities in comprehending the implicit intent of user instructions and preforming reasoning. In addition to generating textual responses, our model also outputs task tokens and grounding tokens, serving as indicators of task types and task granularity. These outputs are subsequently routed through the task router and directed to specific expert models for task completion. To train our model, we construct a task-specific dataset and an 100k multi-task dataset encompassing complex scenarios. Employing a three-stage training strategy, we equip our model with robust reasoning and task processing capabilities while preserving its generalization capacity and knowledge reservoir. Extensive experiments showcase the impressive performance of our unified representation approach across various tasks, surpassing existing methodologies. Furthermore, our approach exhibits exceptional scalability and generality. Our code, model, and dataset will be available at \url{https://github.com/lzw-lzw/UnifiedMLLM}. △ Less

Submitted 5 August, 2024; originally announced August 2024.

On the two problems in Ramsey achievement games

Authors: Zhong Huang, Yusuke Kobayashi, Yaping Mao, Bo Ning, Xiumin Wang

Abstract: Let $p,q$ be two integers with $p\geq q$. Given a finite graph $F$ with no isolated vertices, the generalized Ramsey achievement game of $F$ on the complete graph $K_n$, denoted by $(p,q;K_n,F,+)$, is played by two players called Alice and Bob. In each round, Alice firstly chooses $p$ uncolored edges $e_1,e_2,...,e_p$ and colors it blue, then Bob chooses $q$ uncolored edge $f_1,f_2,...,f_q$ and co… ▽ More Let $p,q$ be two integers with $p\geq q$. Given a finite graph $F$ with no isolated vertices, the generalized Ramsey achievement game of $F$ on the complete graph $K_n$, denoted by $(p,q;K_n,F,+)$, is played by two players called Alice and Bob. In each round, Alice firstly chooses $p$ uncolored edges $e_1,e_2,...,e_p$ and colors it blue, then Bob chooses $q$ uncolored edge $f_1,f_2,...,f_q$ and colors it red; the player who can first complete the formation of $F$ in his (or her) color is the winner. The generalized achievement number of $F$, denoted by ${a}(p,q;F)$ is defined to be the smallest $n$ for which Alice has a winning strategy. If $p=q=1$, then it is denoted by ${a}(F)$, which is the classical achievement number of $F$ introduced by Harary in 1982. If Alice aims to form a blue $F$, and the goal of Bob is to try to stop him, this kind of game is called the first player game by Bollobás. Let ${a}^*(F)$ be the smallest positive integer $n$ for which Alice has a winning strategy in the first player game. A conjecture due to Harary states that the minimum value of ${a}(T)$ is realized when $T$ is a path and the maximum value of ${a}(T)$ is realized when $T$ is a star among all trees $T$ of order $n$. He also asked which graphs $F$ satisfy $a^*(F)=a(F)$? In this paper, we proved that $n\leq {a}(p,q;T)\leq n+q\left\lfloor (n-2)/p \right\rfloor$ for all trees $T$ of order $n$, and obtained a lower bound of ${a}(p,q;K_{1,n-1})$, where $K_{1,n-1}$ is a star. We proved that the minimum value of ${a}(T)$ is realized when $T$ is a path which gives a positive solution to the first part of Harary's conjecture, and ${a}(T)\leq 2n-2$ for all trees of order $n$. We also proved that for $n\geq 3$, we have $2n-2-\sqrt{(4n-8)\ln (4n-4)}\leq a(K_{1,n-1})\leq 2n-2$ with the help of a theorem of Alon, Krivelevich, Spencer and Szabó. We proved that $a^*(P_n)=a(P_n)$ for a path $P_n$. △ Less

Submitted 2 August, 2024; originally announced August 2024.

Comments: 13 pages

3DPX: Progressive 2D-to-3D Oral Image Reconstruction with Hybrid MLP-CNN Networks

Authors: Xiaoshuang Li, Mingyuan Meng, Zimo Huang, Lei Bi, Eduardo Delamare, Dagan Feng, Bin Sheng, Jinman Kim

Abstract: Panoramic X-ray (PX) is a prevalent modality in dental practice for its wide availability and low cost. However, as a 2D projection image, PX does not contain 3D anatomical information, and therefore has limited use in dental applications that can benefit from 3D information, e.g., tooth angular misa-lignment detection and classification. Reconstructing 3D structures directly from 2D PX has recent… ▽ More Panoramic X-ray (PX) is a prevalent modality in dental practice for its wide availability and low cost. However, as a 2D projection image, PX does not contain 3D anatomical information, and therefore has limited use in dental applications that can benefit from 3D information, e.g., tooth angular misa-lignment detection and classification. Reconstructing 3D structures directly from 2D PX has recently been explored to address limitations with existing methods primarily reliant on Convolutional Neural Networks (CNNs) for direct 2D-to-3D mapping. These methods, however, are unable to correctly infer depth-axis spatial information. In addition, they are limited by the in-trinsic locality of convolution operations, as the convolution kernels only capture the information of immediate neighborhood pixels. In this study, we propose a progressive hybrid Multilayer Perceptron (MLP)-CNN pyra-mid network (3DPX) for 2D-to-3D oral PX reconstruction. We introduce a progressive reconstruction strategy, where 3D images are progressively re-constructed in the 3DPX with guidance imposed on the intermediate recon-struction result at each pyramid level. Further, motivated by the recent ad-vancement of MLPs that show promise in capturing fine-grained long-range dependency, our 3DPX integrates MLPs and CNNs to improve the semantic understanding during reconstruction. Extensive experiments on two large datasets involving 464 studies demonstrate that our 3DPX outperforms state-of-the-art 2D-to-3D oral reconstruction methods, including standalone MLP and transformers, in reconstruction quality, and also im-proves the performance of downstream angular misalignment classification tasks. △ Less

Submitted 2 August, 2024; originally announced August 2024.

Comments: accepted by MICCAI 2024

Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach

Authors: Yabin Zhu, Qianwu Wang, Chenglong Li, Jin Tang, Zhixiang Huang

Abstract: The complementary benefits from visible and thermal infrared data are widely utilized in various computer vision task, such as visual tracking, semantic segmentation and object detection, but rarely explored in Multiple Object Tracking (MOT). In this work, we contribute a large-scale Visible-Thermal video benchmark for MOT, called VT-MOT. VT-MOT has the following main advantages. 1) The data is la… ▽ More The complementary benefits from visible and thermal infrared data are widely utilized in various computer vision task, such as visual tracking, semantic segmentation and object detection, but rarely explored in Multiple Object Tracking (MOT). In this work, we contribute a large-scale Visible-Thermal video benchmark for MOT, called VT-MOT. VT-MOT has the following main advantages. 1) The data is large scale and high diversity. VT-MOT includes 582 video sequence pairs, 401k frame pairs from surveillance, drone, and handheld platforms. 2) The cross-modal alignment is highly accurate. We invite several professionals to perform both spatial and temporal alignment frame by frame. 3) The annotation is dense and high-quality. VT-MOT has 3.99 million annotation boxes annotated and double-checked by professionals, including heavy occlusion and object re-acquisition (object disappear and reappear) challenges. To provide a strong baseline, we design a simple yet effective tracking framework, which effectively fuses temporal information and complementary information of two modalities in a progressive manner, for robust visible-thermal MOT. A comprehensive experiment are conducted on VT-MOT and the results prove the superiority and effectiveness of the proposed method compared with state-of-the-art methods. From the evaluation results and analysis, we specify several potential future directions for visible-thermal MOT. The project is released in https://github.com/wqw123wqw/PFTrack. △ Less

Submitted 1 August, 2024; originally announced August 2024.

From 2015 to 2023: How Machine Learning Aids Natural Product Analysis

Authors: Suwen Shi, Ziwei Huang, Xingxin Gu, Xu Lin, Chaoying Zhong, Junjie Hang, Jianli Lin, Claire Chenwen Zhong, Lin Zhang, Yu Li, Junjie Huang

Abstract: In recent years, conventional chemistry techniques have faced significant challenges due to their inherent limitations, struggling to cope with the increasing complexity and volume of data generated in contemporary research endeavors. Computational methodologies represent robust tools in the field of chemistry, offering the capacity to harness potent machine-learning models to yield insightful ana… ▽ More In recent years, conventional chemistry techniques have faced significant challenges due to their inherent limitations, struggling to cope with the increasing complexity and volume of data generated in contemporary research endeavors. Computational methodologies represent robust tools in the field of chemistry, offering the capacity to harness potent machine-learning models to yield insightful analytical outcomes. This review delves into the spectrum of computational strategies available for natural product analysis and constructs a research framework for investigating both qualitative and quantitative chemistry problems. Our objective is to present a novel perspective on the symbiosis of machine learning and chemistry, with the potential to catalyze a transformation in the field of natural product analysis. △ Less

Submitted 17 July, 2024; originally announced August 2024.

Comments: 19 pages, 4 figures

CERT-ED: Certifiably Robust Text Classification for Edit Distance

Authors: Zhuoqun Huang, Neil G Marchant, Olga Ohrimenko, Benjamin I. P. Rubinstein

Abstract: With the growing integration of AI in daily life, ensuring the robustness of systems to inference-time attacks is crucial. Among the approaches for certifying robustness to such adversarial examples, randomized smoothing has emerged as highly promising due to its nature as a wrapper around arbitrary black-box models. Previous work on randomized smoothing in natural language processing has primaril… ▽ More With the growing integration of AI in daily life, ensuring the robustness of systems to inference-time attacks is crucial. Among the approaches for certifying robustness to such adversarial examples, randomized smoothing has emerged as highly promising due to its nature as a wrapper around arbitrary black-box models. Previous work on randomized smoothing in natural language processing has primarily focused on specific subsets of edit distance operations, such as synonym substitution or word insertion, without exploring the certification of all edit operations. In this paper, we adapt Randomized Deletion (Huang et al., 2023) and propose, CERTified Edit Distance defense (CERT-ED) for natural language classification. Through comprehensive experiments, we demonstrate that CERT-ED outperforms the existing Hamming distance method RanMASK (Zeng et al., 2023) in 4 out of 5 datasets in terms of both accuracy and the cardinality of the certificate. By covering various threat models, including 5 direct and 5 transfer attacks, our method improves empirical robustness in 38 out of 50 settings. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Comments: 22 pages, 3 figures, 12 tables. Include 11 pages of appendices