Showing 51–100 of 891 results for author: Dai, Y

Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?

Authors: Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, Li Shen

Abstract: Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL), yet it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a pr… ▽ More Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL), yet it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba in offline RL (dubbed DeMa) from the aspect of data structures and network architectures with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements due to the fact that DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify that the hidden attention mechanism is key to its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations from eight Atari games demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous state-of-the-art methods, outdoing Decision Transformer (DT) by 80\% with 30\% fewer parameters, and exceeds DT in MuJoCo with only a quarter of the parameters. △ Less

Submitted 20 May, 2024; originally announced May 2024.

Comments: 20 pages, 8 figures

Stability for Nash Equilibrium Problems

Authors: Ruoyu Diao, Yu-Hong Dai, Liwei Zhang

Abstract: This paper is devoted to studying the stability properties of the Karush-Kuhn-Tucker (KKT) solution mapping $S_{\rm KKT}$ for Nash equilibrium problems (NEPs) with canonical perturbations. Firstly, we obtain an exact characterization of the strong regularity of $S_{\rm KKT}$ and a sufficient condition that is easy to verify. Secondly, we propose equivalent conditions for the continuously different… ▽ More This paper is devoted to studying the stability properties of the Karush-Kuhn-Tucker (KKT) solution mapping $S_{\rm KKT}$ for Nash equilibrium problems (NEPs) with canonical perturbations. Firstly, we obtain an exact characterization of the strong regularity of $S_{\rm KKT}$ and a sufficient condition that is easy to verify. Secondly, we propose equivalent conditions for the continuously differentiable single-valued localization of $S_{\rm KKT}$. Thirdly, the isolated calmness of $S_{\rm KKT}$ is studied based on two conditions: Property A and Property B, and Property B proves to be sufficient for the robustness of both $E(p)$ and $S_{\rm KKT}$ under the convex assumptions, where $E(p)$ denotes the Nash equilibria at perturbation $p$. Furthermore, we establish that studying the stability properties of the NEP with canonical perturbations is equivalent to studying those of the NEP with only tilt perturbations based on the prior discussions. Finally, we provide detailed characterizations of stability for NEPs whose each individual player solves a quadratic programming (QP) problem. △ Less

Submitted 18 May, 2024; originally announced May 2024.

Comments: 34 pages, 1 figure

MSC Class: 90C30; 90C31; 49J53

A Complete 16 $μ$m selected Galaxy Sample at $z \sim 1$. II: Morphological Analysis

Authors: Piaoran Liang, Y. Sophia Dai, Jia-Sheng Huang, Cheng Cheng, Shi Yaru

Abstract: We present morphological analysis of the 16$μ$m flux-density-limited galaxy sample at 0.8$<z<$1.3 from arXiv:2103.04585. At the targeted redshift, the 16$μ$m emission corresponds to the Polycyclic aromatic hydrocarbon (PAH) feature from intense star formation, or dust heated by AGN (Active galactic nuclei). Our sample of 479 galaxies are dominated by Luminous Infrared Galaxies (LIRGs, 67\%) in thr… ▽ More We present morphological analysis of the 16$μ$m flux-density-limited galaxy sample at 0.8$<z<$1.3 from arXiv:2103.04585. At the targeted redshift, the 16$μ$m emission corresponds to the Polycyclic aromatic hydrocarbon (PAH) feature from intense star formation, or dust heated by AGN (Active galactic nuclei). Our sample of 479 galaxies are dominated by Luminous Infrared Galaxies (LIRGs, 67\%) in three CANDLES fields (EGS, GOODS-N, and GOODS-S), and are further divided into AGN dominated, star-forming dominated, composite, and blue compact galaxies by their spectral energy distribution (SED) types. The majority of our sample (71\%) have disky morphologies, with the few AGN dominated galaxies being more bulge-dominanted than the star-forming dominated and composite galaxies. The distribution of our sample on the Gini vs. M$_{\text{20}}$ plane is consistent with previous studies, where the Sérsic index $n$ shows an increasing trend towards the smaller M$_{\text{20}}$ and higher Gini region below the dividing line for mergers. The subsample of ULIRGs follow a steep size-mass relation that is closer to the early-type galaxies. In addition, as the 4.5 $μ$m luminosity excess ($L_{4.5}^{Exc}$, proxy for AGN strength) increases, our sample appear to be more bulge-dominated (i.e. higher $n$). Based on the sSFR and compactness ($log_{10}Σ_{1.5}, Σ_{1.5}=M_*/R_e^{1.5}$) diagram, the majority of our LIRG-dominated galaxy sample follow a secular evolution track, and their distribution can be explained without involving any merging activities. Out of the 16 ULIRGs in our sample, six are compact with strong AGN contributions, likely evolving along the fast-track from more violent activities. △ Less

Submitted 11 May, 2024; originally announced May 2024.

Comments: 21 pages, 8 figures, 3 tables

A Survey on Visualization Approaches in Political Science for Social and Political Factors: Progress to Date and Future Opportunities

Authors: Dongyun Han, Abdullah-Al-Raihan Nayeem, Jason Windett, Yaoyao Dai, Benjamin Radford, Isaac Cho

Abstract: Politics is the set of activities related to strategic decision-making in groups. Political scientists study the strategic interactions between states, institutions, politicians, and citizens; they seek to understand the causes and consequences of those decisions and interactions. While some decisions might alleviate social problems, others might lead to disasters such as war and conflict. Data vi… ▽ More Politics is the set of activities related to strategic decision-making in groups. Political scientists study the strategic interactions between states, institutions, politicians, and citizens; they seek to understand the causes and consequences of those decisions and interactions. While some decisions might alleviate social problems, others might lead to disasters such as war and conflict. Data visualization approaches have the potential to assist political scientists in their studies by providing visual contexts. However, political researchers' perspectives on data visualization are unclear. This paper examines political scientists' perspectives on visualization and how they apply data visualization in their research. We discovered a growing trend in the use of graphs in political science journals. However, we also found a knowledge gap between the political science and visualization domains, such as effective visualization techniques for tasks and the use of color studied by visualization researchers. To reduce this gap, we survey visualization techniques applicable to the political scientists' research and report the visual analytics systems implemented for and evaluated by political scientists. At the end of this paper, we present an outline of future opportunities, including research topics and methodologies, for multidisciplinary research in political science and data analytics. Through this paper, we expect visualization researchers to get a better grasp of the political science domain, as well as broaden the possibility of future visualization approaches from a multidisciplinary perspective. △ Less

Submitted 9 May, 2024; originally announced May 2024.

MIPI 2024 Challenge on Demosaic for HybridEVS Camera: Methods and Results

Authors: Yaqi Wu, Zhihao Fan, Xiaofeng Chu, Jimmy S. Ren, Xiaoming Li, Zongsheng Yue, Chongyi Li, Shangcheng Zhou, Ruicheng Feng, Yuekun Dai, Peiqing Yang, Chen Change Loy, Senyan Xu, Zhijing Sun, Jiaying Zhu, Yurui Zhu, Xueyang Fu, Zheng-Jun Zha, Jun Cao, Cheng Li, Shu Chen, Liang Ma, Shiyang Zhou, Haijin Zeng, Kai Feng , et al. (24 additional authors not shown)

Abstract: The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photogra… ▽ More The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). Building on the achievements of the previous MIPI Workshops held at ECCV 2022 and CVPR 2023, we introduce our third MIPI challenge including three tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Nighttime Flare Removal track on MIPI 2024. In total, 170 participants were successfully registered, and 14 teams submitted results in the final testing phase. The developed solutions in this challenge achieved state-of-the-art performance on Nighttime Flare Removal. More details of this challenge and the link to the dataset can be found at https://mipi-challenge.org/MIPI2024/. △ Less

Submitted 8 May, 2024; originally announced May 2024.

Comments: MIPI@CVPR2024. Website: https://mipi-challenge.org/MIPI2024/

Non-rigid Structure-from-Motion: Temporally-smooth Procrustean Alignment and Spatially-variant Deformation Modeling

Authors: Jiawei Shi, Hui Deng, Yuchao Dai

Abstract: Even though Non-rigid Structure-from-Motion (NRSfM) has been extensively studied and great progress has been made, there are still key challenges that hinder their broad real-world applications: 1) the inherent motion/rotation ambiguity requires either explicit camera motion recovery with extra constraint or complex Procrustean Alignment; 2) existing low-rank modeling of the global shape can over-… ▽ More Even though Non-rigid Structure-from-Motion (NRSfM) has been extensively studied and great progress has been made, there are still key challenges that hinder their broad real-world applications: 1) the inherent motion/rotation ambiguity requires either explicit camera motion recovery with extra constraint or complex Procrustean Alignment; 2) existing low-rank modeling of the global shape can over-penalize drastic deformations in the 3D shape sequence. This paper proposes to resolve the above issues from a spatial-temporal modeling perspective. First, we propose a novel Temporally-smooth Procrustean Alignment module that estimates 3D deforming shapes and adjusts the camera motion by aligning the 3D shape sequence consecutively. Our new alignment module remedies the requirement of complex reference 3D shape during alignment, which is more conductive to non-isotropic deformation modeling. Second, we propose a spatial-weighted approach to enforce the low-rank constraint adaptively at different locations to accommodate drastic spatially-variant deformation reconstruction better. Our modeling outperform existing low-rank based methods, and extensive experiments across different datasets validate the effectiveness of our method. △ Less

Submitted 23 June, 2024; v1 submitted 7 May, 2024; originally announced May 2024.

Comments: Accepted by CVPR 2024; V2 adds new experiments

Memristive switching in the surface of a charge-density-wave topological semimetal

Authors: Jianwen Ma, Xianghao Meng, Binhua Zhang, Yuxiang Wang, Yicheng Mou, Wenting Lin, Yannan Dai, Luqiu Chen, Haonan Wang, Haoqi Wu, Jiaming Gu, Jiayu Wang, Yuhan Du, Chunsen Liu, Wu Shi, Zhenzhong Yang, Bobo Tian, Lin Miao, Peng Zhou, Chun-Gang Duan, Changsong Xu, Xiang Yuan, Cheng Zhang

Abstract: Owing to the outstanding properties provided by nontrivial band topology, topological phases of matter are considered as a promising platform towards low-dissipation electronics, efficient spin-charge conversion, and topological quantum computation. Achieving ferroelectricity in topological materials enables the non-volatile control of the quantum states, which could greatly facilitate topological… ▽ More Owing to the outstanding properties provided by nontrivial band topology, topological phases of matter are considered as a promising platform towards low-dissipation electronics, efficient spin-charge conversion, and topological quantum computation. Achieving ferroelectricity in topological materials enables the non-volatile control of the quantum states, which could greatly facilitate topological electronic research. However, ferroelectricity is generally incompatible with systems featuring metallicity due to the screening effect of free carriers. In this study, we report the observation of memristive switching based on the ferroelectric surface state of a topological semimetal (TaSe4)2I. We find that the surface state of (TaSe4)2I presents out-of-plane ferroelectric polarization due to surface reconstruction. With the combination of ferroelectric surface and charge-density-wave-gapped bulk states, an electric switchable barrier height can be achieved in (TaSe4)2I-metal contact. By employing a multi-terminal grounding design, we manage to construct a prototype ferroelectric memristor based on (TaSe4)2I with on/off ratio up to 10^3, endurance over 10^3 cycles, and good retention characteristics. The origin of the ferroelectric surface state is further investigated by first-principles calculations, which reveals an interplay between ferroelectricity and band topology. The emergence of ferroelectricity in (TaSe4)2I not only demonstrates it as a rare but essential case of ferroelectric topological materials, but also opens new routes towards the implementation of topological materials in functional electronic devices. △ Less

Submitted 6 May, 2024; originally announced May 2024.

Comments: 14 pages, 5 figures

Journal ref: Science Bulletin 69(13), 2042-2049 (2024)

Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets

Authors: Xuelong Geng, Tianyi Xu, Kun Wei, Bingshen Mu, Hongfei Xue, He Wang, Yangze Li, Pengcheng Guo, Yuhang Dai, Longhao Li, Mingchen Shao, Lei Xie

Abstract: Large Language Models (LLMs) have demonstrated unparalleled effectiveness in various NLP tasks, and integrating LLMs with automatic speech recognition (ASR) is becoming a mainstream paradigm. Building upon this momentum, our research delves into an in-depth examination of this paradigm on a large open-source Chinese dataset. Specifically, our research aims to evaluate the impact of various configu… ▽ More Large Language Models (LLMs) have demonstrated unparalleled effectiveness in various NLP tasks, and integrating LLMs with automatic speech recognition (ASR) is becoming a mainstream paradigm. Building upon this momentum, our research delves into an in-depth examination of this paradigm on a large open-source Chinese dataset. Specifically, our research aims to evaluate the impact of various configurations of speech encoders, LLMs, and projector modules in the context of the speech foundation encoder-LLM ASR paradigm. Furthermore, we introduce a three-stage training approach, expressly developed to enhance the model's ability to align auditory and textual information. The implementation of this approach, alongside the strategic integration of ASR components, enabled us to achieve the SOTA performance on the AISHELL-1, Test_Net, and Test_Meeting test sets. Our analysis presents an empirical foundation for future research in LLM-based ASR systems and offers insights into optimizing performance using Chinese datasets. We will publicly release all scripts used for data preparation, training, inference, and scoring, as well as pre-trained models and training logs to promote reproducible research. △ Less

Submitted 6 May, 2024; v1 submitted 3 May, 2024; originally announced May 2024.

SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning

Authors: Yexiao He, Ziyao Wang, Zheyu Shen, Guoheng Sun, Yucong Dai, Yongkai Wu, Hongyi Wang, Ang Li

Abstract: The pre-trained Large Language Models (LLMs) can be adapted for many downstream tasks and tailored to align with human preferences through fine-tuning. Recent studies have discovered that LLMs can achieve desirable performance with only a small amount of high-quality data, suggesting that a large amount of the data in these extensive datasets is redundant or even harmful. Identifying high-quality… ▽ More The pre-trained Large Language Models (LLMs) can be adapted for many downstream tasks and tailored to align with human preferences through fine-tuning. Recent studies have discovered that LLMs can achieve desirable performance with only a small amount of high-quality data, suggesting that a large amount of the data in these extensive datasets is redundant or even harmful. Identifying high-quality data from vast datasets to curate small yet effective datasets has emerged as a critical challenge. In this paper, we introduce SHED, an automated dataset refinement framework based on Shapley value for instruction fine-tuning. SHED eliminates the need for human intervention or the use of commercial LLMs. Moreover, the datasets curated through SHED exhibit transferability, indicating they can be reused across different LLMs with consistently high performance. We conduct extensive experiments to evaluate the datasets curated by SHED. The results demonstrate SHED's superiority over state-of-the-art methods across various tasks and LLMs; notably, datasets comprising only 10% of the original data selected by SHED achieve performance comparable to or surpassing that of the full datasets. △ Less

Submitted 23 April, 2024; originally announced May 2024.

MIPI 2024 Challenge on Nighttime Flare Removal: Methods and Results

Authors: Yuekun Dai, Dafeng Zhang, Xiaoming Li, Zongsheng Yue, Chongyi Li, Shangchen Zhou, Ruicheng Feng, Peiqing Yang, Zhezhu Jin, Guanqun Liu, Chen Change Loy, Lize Zhang, Shuai Liu, Chaoyu Feng, Luyang Wang, Shuan Chen, Guangqi Shao, Xiaotao Wang, Lei Lei, Qirui Yang, Qihua Cheng, Zhiqiang Xu, Yihao Liu, Huanjing Yue, Jingyu Yang , et al. (38 additional authors not shown)

Abstract: The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photogra… ▽ More The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). Building on the achievements of the previous MIPI Workshops held at ECCV 2022 and CVPR 2023, we introduce our third MIPI challenge including three tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Nighttime Flare Removal track on MIPI 2024. In total, 170 participants were successfully registered, and 14 teams submitted results in the final testing phase. The developed solutions in this challenge achieved state-of-the-art performance on Nighttime Flare Removal. More details of this challenge and the link to the dataset can be found at https://mipi-challenge.org/MIPI2024/. △ Less

Submitted 27 May, 2024; v1 submitted 30 April, 2024; originally announced April 2024.

Comments: CVPR 2024 Mobile Intelligent Photography and Imaging (MIPI) Workshop--Nighttime Flare Removal Challenge Report. Website: https://mipi-challenge.org/MIPI2024/

Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras

Authors: Jun Yu, Yutong Dai, Xiaokang Liu, Jin Huang, Yishan Shen, Ke Zhang, Rong Zhou, Eashan Adhikarla, Wenxuan Ye, Yixin Liu, Zhaoming Kong, Kai Zhang, Yilong Yin, Vinod Namboodiri, Brian D. Davison, Jason H. Moore, Yong Chen

Abstract: MTL is a learning paradigm that effectively leverages both task-specific and shared information to address multiple related tasks simultaneously. In contrast to STL, MTL offers a suite of benefits that enhance both the training process and the inference efficiency. MTL's key advantages encompass streamlined model architecture, performance enhancement, and cross-domain generalizability. Over the pa… ▽ More MTL is a learning paradigm that effectively leverages both task-specific and shared information to address multiple related tasks simultaneously. In contrast to STL, MTL offers a suite of benefits that enhance both the training process and the inference efficiency. MTL's key advantages encompass streamlined model architecture, performance enhancement, and cross-domain generalizability. Over the past twenty years, MTL has become widely recognized as a flexible and effective approach in various fields, including CV, NLP, recommendation systems, disease prognosis and diagnosis, and robotics. This survey provides a comprehensive overview of the evolution of MTL, encompassing the technical aspects of cutting-edge methods from traditional approaches to deep learning and the latest trend of pretrained foundation models. Our survey methodically categorizes MTL techniques into five key areas: regularization, relationship learning, feature propagation, optimization, and pre-training. This categorization not only chronologically outlines the development of MTL but also dives into various specialized strategies within each category. Furthermore, the survey reveals how the MTL evolves from handling a fixed set of tasks to embracing a more flexible approach free from task or modality constraints. It explores the concepts of task-promptable and -agnostic training, along with the capacity for ZSL, which unleashes the untapped potential of this historically coveted learning paradigm. Overall, we hope this survey provides the research community with a comprehensive overview of the advancements in MTL from its inception in 1997 to the present in 2023. We address present challenges and look ahead to future possibilities, shedding light on the opportunities and potential avenues for MTL research in a broad manner. This project is publicly available at https://github.com/junfish/Awesome-Multitask-Learning. △ Less

Submitted 29 April, 2024; originally announced April 2024.

Comments: 60 figures, 116 pages, 500+ references

Prompt Customization for Continual Learning

Authors: Yong Dai, Xiaopeng Hong, Yabin Wang, Zhiheng Ma, Dongmei Jiang, Yaowei Wang

Abstract: Contemporary continual learning approaches typically select prompts from a pool, which function as supplementary inputs to a pre-trained model. However, this strategy is hindered by the inherent noise of its selection approach when handling increasing tasks. In response to these challenges, we reformulate the prompting approach for continual learning and propose the prompt customization (PC) metho… ▽ More Contemporary continual learning approaches typically select prompts from a pool, which function as supplementary inputs to a pre-trained model. However, this strategy is hindered by the inherent noise of its selection approach when handling increasing tasks. In response to these challenges, we reformulate the prompting approach for continual learning and propose the prompt customization (PC) method. PC mainly comprises a prompt generation module (PGM) and a prompt modulation module (PMM). In contrast to conventional methods that employ hard prompt selection, PGM assigns different coefficients to prompts from a fixed-sized pool of prompts and generates tailored prompts. Moreover, PMM further modulates the prompts by adaptively assigning weights according to the correlations between input data and corresponding prompts. We evaluate our method on four benchmark datasets for three diverse settings, including the class, domain, and task-agnostic incremental learning tasks. Experimental results demonstrate consistent improvement (by up to 16.2\%), yielded by the proposed method, over the state-of-the-art (SOTA) techniques. △ Less

Submitted 27 April, 2024; originally announced April 2024.

Comments: ACM MM

Soft X-ray prompt emission from a high-redshift gamma-ray burst EP240315a

Authors: Y. Liu, H. Sun, D. Xu, D. S. Svinkin, J. Delaunay, N. R. Tanvir, H. Gao, C. Zhang, Y. Chen, X. -F. Wu, B. Zhang, W. Yuan, J. An, G. Bruni, D. D. Frederiks, G. Ghirlanda, J. -W. Hu, A. Li, C. -K. Li, J. -D. Li, D. B. Malesani, L. Piro, G. Raman, R. Ricci, E. Troja , et al. (170 additional authors not shown)

Abstract: Long gamma-ray bursts (GRBs) are believed to originate from core collapse of massive stars. High-redshift GRBs can probe the star formation and reionization history of the early universe, but their detection remains rare. Here we report the detection of a GRB triggered in the 0.5--4 keV band by the Wide-field X-ray Telescope (WXT) on board the Einstein Probe (EP) mission, designated as EP240315a,… ▽ More Long gamma-ray bursts (GRBs) are believed to originate from core collapse of massive stars. High-redshift GRBs can probe the star formation and reionization history of the early universe, but their detection remains rare. Here we report the detection of a GRB triggered in the 0.5--4 keV band by the Wide-field X-ray Telescope (WXT) on board the Einstein Probe (EP) mission, designated as EP240315a, whose bright peak was also detected by the Swift Burst Alert Telescope and Konus-Wind through off-line analyses. At a redshift of $z=4.859$, EP240315a showed a much longer and more complicated light curve in the soft X-ray band than in gamma-rays. Benefiting from a large field-of-view ($\sim$3600 deg$^2$) and a high sensitivity, EP-WXT captured the earlier engine activation and extended late engine activity through a continuous detection. With a peak X-ray flux at the faint end of previously known high-$z$ GRBs, the detection of EP240315a demonstrates the great potential for EP to study the early universe via GRBs. △ Less

Submitted 25 April, 2024; originally announced April 2024.

Comments: 41 pages, 8 figures, 7 tables

TOP-Nav: Legged Navigation Integrating Terrain, Obstacle and Proprioception Estimation

Authors: Junli Ren, Yikai Liu, Yingru Dai, Junfeng Long, Guijin Wang

Abstract: Legged navigation is typically examined within open-world, off-road, and challenging environments. In these scenarios, estimating external disturbances requires a complex synthesis of multi-modal information. This underlines a major limitation in existing works that primarily focus on avoiding obstacles. In this work, we propose TOP-Nav, a novel legged navigation framework that integrates a compre… ▽ More Legged navigation is typically examined within open-world, off-road, and challenging environments. In these scenarios, estimating external disturbances requires a complex synthesis of multi-modal information. This underlines a major limitation in existing works that primarily focus on avoiding obstacles. In this work, we propose TOP-Nav, a novel legged navigation framework that integrates a comprehensive path planner with Terrain awareness, Obstacle avoidance and close-loop Proprioception. TOP-Nav underscores the synergies between vision and proprioception in both path and motion planning. Within the path planner, we present and integrate a terrain estimator that enables the robot to select waypoints on terrains with higher traversability while effectively avoiding obstacles. In the motion planning level, we not only implement a locomotion controller to track the navigation commands, but also construct a proprioception advisor to provide motion evaluations for the path planner. Based on the close-loop motion feedback, we make online corrections for the vision-based terrain and obstacle estimations. Consequently, TOP-Nav achieves open-world navigation that the robot can handle terrains or disturbances beyond the distribution of prior knowledge and overcomes constraints imposed by visual conditions. Building upon extensive experiments conducted in both simulation and real-world environments, TOP-Nav demonstrates superior performance in open-world navigation compared to existing methods. △ Less

Submitted 12 July, 2024; v1 submitted 23 April, 2024; originally announced April 2024.

An inexact augmented Lagrangian algorithm for unsymmetric saddle-point systems

Authors: N. Huang, Y. -H. Dai, D. Orban, M. A. Saunders

Abstract: Augmented Lagrangian (AL) methods are a well known class of algorithms for solving constrained optimization problems. They have been extended to the solution of saddle-point systems of linear equations. We study an AL (SPAL) algorithm for unsymmetric saddle-point systems and derive convergence and semi-convergence properties, even when the system is singular. At each step, our SPAL requires the ex… ▽ More Augmented Lagrangian (AL) methods are a well known class of algorithms for solving constrained optimization problems. They have been extended to the solution of saddle-point systems of linear equations. We study an AL (SPAL) algorithm for unsymmetric saddle-point systems and derive convergence and semi-convergence properties, even when the system is singular. At each step, our SPAL requires the exact solution of a linear system of the same size but with an SPD (2,2) block. To improve efficiency, we introduce an inexact SPAL algorithm. We establish its convergence properties under reasonable assumptions. Specifically, we use a gradient method, known as the Barzilai-Borwein (BB) method, to solve the linear system at each iteration. We call the result the augmented Lagrangian BB (SPALBB) algorithm and study its convergence. Numerical experiments on test problems from Navier-Stokes equations and coupled Stokes-Darcy flow show that SPALBB is more robust and efficient than BICGSTAB and GMRES. SPALBB often requires the least CPU time, especially on large systems. △ Less

Submitted 22 April, 2024; originally announced April 2024.

Report number: G-2024-30 MSC Class: 65F10; 65F50

TAVGBench: Benchmarking Text to Audible-Video Generation

Authors: Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, Yuchao Dai

Abstract: The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 th… ▽ More The Text to Audible-Video Generation (TAVG) task involves generating videos with accompanying audio based on text descriptions. Achieving this requires skillful alignment of both audio and video elements. To support research in this field, we have developed a comprehensive Text to Audible-Video Generation Benchmark (TAVGBench), which contains over 1.7 million clips with a total duration of 11.8 thousand hours. We propose an automatic annotation pipeline to ensure each audible video has detailed descriptions for both its audio and video contents. We also introduce the Audio-Visual Harmoni score (AVHScore) to provide a quantitative measure of the alignment between the generated audio and video modalities. Additionally, we present a baseline model for TAVG called TAVDiffusion, which uses a two-stream latent diffusion model to provide a fundamental starting point for further research in this area. We achieve the alignment of audio and video by employing cross-attention and contrastive learning. Through extensive experiments and evaluations on TAVGBench, we demonstrate the effectiveness of our proposed model under both conventional metrics and our proposed metrics. △ Less

Submitted 22 April, 2024; originally announced April 2024.

Comments: Technical Report. Project page:https://github.com/OpenNLPLab/TAVGBench

BERT: Accelerating Vital Signs Measurement for Bioradar with An Efficient Recursive Technique

Authors: Chengyao Tang, Yongpeng Dai, Zhi Li, Yongping Song, Fulai Liang, Tian Jin

Abstract: Recent years have witnessed the great advance of bioradar system in smart sensing of vital signs (VS) for human healthcare monitoring. As an important part of VS sensing process, VS measurement aims to capture the chest wall micromotion induced by the human respiratory and cardiac activities. Unfortunately, the existing VS measurement methods using bioradar have encountered bottlenecks in making a… ▽ More Recent years have witnessed the great advance of bioradar system in smart sensing of vital signs (VS) for human healthcare monitoring. As an important part of VS sensing process, VS measurement aims to capture the chest wall micromotion induced by the human respiratory and cardiac activities. Unfortunately, the existing VS measurement methods using bioradar have encountered bottlenecks in making a trade-off between time cost and measurement accuracy. To break this bottleneck, this letter proposes an efficient recursive technique (BERT) heuristically, based on the observation that the features of bioradar VS meet the conditions of Markov model. Extensive experimental results validate that BERT measurement yields lower time costs, competitive estimates of heart rate, breathing rate, and heart rate variability. Our BERT method is promising us a new and superior option to measure VS for bioradar. This work seeks not only to solve the current issue of how to accelerate VS measurement with an acceptable accuracy, but also to inspire creative new ideas that spur further advances in this promising field in the future. △ Less

Submitted 20 April, 2024; originally announced April 2024.

Comments: 4pages, 8 figures, submitted to the IEEE for possible publication

Existence of Hebel-Slichter peak in unconventional kagome superconductors

Authors: Yi Dai, Andreas Kreisel, Brian M. Andersen

Abstract: We perform a theoretical investigation of the spin susceptibility of unconventional superconductivity on the kagome lattice. Despite the existence of a sign-changing gap structure, which sums to zero over the Fermi surface, we show that such unconventional pairing states may exhibit a Hebel-Slichter peak in the temperature-dependent spin-lattice relaxation rate. It originates from destructive subl… ▽ More We perform a theoretical investigation of the spin susceptibility of unconventional superconductivity on the kagome lattice. Despite the existence of a sign-changing gap structure, which sums to zero over the Fermi surface, we show that such unconventional pairing states may exhibit a Hebel-Slichter peak in the temperature-dependent spin-lattice relaxation rate. It originates from destructive sublattice interference effects. For the same reason, unconventional pairing states on the kagome lattice tend not to exhibit a neutron resonance peak. These results supplement previous theoretical studies of the surprising robustness of sign-changing gap structures to disorder on the kagome lattice. Taken together these findings imply that unconventional superconductivity on the kagome lattice is deceptive in the sense that its properties may appear similar to conventional non-sign-changing superconductivity. These results may be of relevance to the superconducting state of the kagome superconductors $A$V$_3$Sb$_5$ ($A$: K, Rb, Cs) and CsTi$_3$Bi$_5$. △ Less

Submitted 16 April, 2024; originally announced April 2024.

Comments: 10 pages, 8 figures

Report number: NBI CMT 2024

Self-playing Adversarial Language Game Enhances LLM Reasoning

Authors: Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, Nan Du

Abstract: We explore the self-play training procedure of large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's u… ▽ More We explore the self-play training procedure of large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate around a target word only visible to the attacker. The attacker aims to induce the defender to speak the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players should have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by self-play in this adversarial language game (SPAG). With this goal, we select several open-source LLMs and let each act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performances uniformly improve on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLMs' reasoning abilities. The code is at https://github.com/Linear95/SPAG. △ Less

Submitted 23 May, 2024; v1 submitted 16 April, 2024; originally announced April 2024.

Comments: Preprint

A cut-and-project perspective for linearized Bregman iterations

Authors: Yu-Hong Dai, Kangkang Deng, Hui Zhang

Abstract: The linearized Bregman iterations (LBreI) and its variants are powerful tools for finding sparse or low-rank solutions to underdetermined linear systems. In this study, we propose a cut-and-project perspective for the linearized Bregman method via a bilevel optimization formulation, along with a new unified algorithmic framework. The new perspective not only encompasses various existing linearized… ▽ More The linearized Bregman iterations (LBreI) and its variants are powerful tools for finding sparse or low-rank solutions to underdetermined linear systems. In this study, we propose a cut-and-project perspective for the linearized Bregman method via a bilevel optimization formulation, along with a new unified algorithmic framework. The new perspective not only encompasses various existing linearized Bregman iteration variants as specific instances, but also allows us to extend the linearized Bregman method to solve more general inverse problems. We provide a completed convergence result of the proposed algorithmic framework, including convergence guarantees to feasible points and optimal solutions, and the sublinear convergence rate. Moreover, we introduce the Bregman distance growth condition to ensure linear convergence. At last, our findings are illustrated via numerical tests. △ Less

Submitted 15 April, 2024; originally announced April 2024.

MSC Class: 90C25; 90C30; 65K05; 49M37

Prior-agnostic Multi-scale Contrastive Text-Audio Pre-training for Parallelized TTS Frontend Modeling

Authors: Quanxiu Wang, Hui Huang, Mingjie Wang, Yong Dai, Jinzuomu Zhong, Benlai Tang

Abstract: Over the past decade, a series of unflagging efforts have been dedicated to developing highly expressive and controllable text-to-speech (TTS) systems. In general, the holistic TTS comprises two interconnected components: the frontend module and the backend module. The frontend excels in capturing linguistic representations from the raw text input, while the backend module converts linguistic cues… ▽ More Over the past decade, a series of unflagging efforts have been dedicated to developing highly expressive and controllable text-to-speech (TTS) systems. In general, the holistic TTS comprises two interconnected components: the frontend module and the backend module. The frontend excels in capturing linguistic representations from the raw text input, while the backend module converts linguistic cues to speech. The research community has shown growing interest in the study of the frontend component, recognizing its pivotal role in text-to-speech systems, including Text Normalization (TN), Prosody Boundary Prediction (PBP), and Polyphone Disambiguation (PD). Nonetheless, the limitations posed by insufficient annotated textual data and the reliance on homogeneous text signals significantly undermine the effectiveness of its supervised learning. To evade this obstacle, a novel two-stage TTS frontend prediction pipeline, named TAP-FM, is proposed in this paper. Specifically, during the first learning phase, we present a Multi-scale Contrastive Text-audio Pre-training protocol (MC-TAP), which hammers at acquiring richer insights via multi-granularity contrastive pre-training in an unsupervised manner. Instead of mining homogeneous features in prior pre-training approaches, our framework demonstrates the ability to delve deep into both global and local text-audio semantic and acoustic representations. Furthermore, a parallelized TTS frontend model is delicately devised to execute TN, PD, and PBP prediction tasks, respectively in the second stage. Finally, extensive experiments illustrate the superiority of our proposed method, achieving state-of-the-art performance. △ Less

Submitted 14 April, 2024; originally announced April 2024.

Ferroelectrovalley in Two-Dimensional Multiferroic Lattices

Authors: Jiangyu Zhao, Yangyang Feng, Ying Dai, Baibiao Huang, Yandong Ma

Abstract: Engineering valley index is essential and highly sought for valley physics, but currently it is exclusively based on the paradigm of the challenging ferrovalley with spin-orientation reversal under magnetic field. Here, an alternative strategy, i.e., the so-called ferroelectrovalley, is proposed to tackle the insurmountable spin-orientation reversal, which reveres valley index with the feasible fe… ▽ More Engineering valley index is essential and highly sought for valley physics, but currently it is exclusively based on the paradigm of the challenging ferrovalley with spin-orientation reversal under magnetic field. Here, an alternative strategy, i.e., the so-called ferroelectrovalley, is proposed to tackle the insurmountable spin-orientation reversal, which reveres valley index with the feasible ferroelectricity. Using symmetry arguments and tight-binding model, the C_2 rotation is unveiled to be able to take the place of time reversal for operating valley index in two-dimensional multiferroic kagome lattices, which enables the ferroelectricity-engineered valley index, thereby generating the concept of ferroelectrovalley. Based on first-principles calculations, this concept is further demonstrated in the breathing kagome lattice of single-layer Ti3Br8, wherein ferroelectricity couples the breathing process. These findings open a new direction for valleytronics and two-dimensional materials research. △ Less

Submitted 13 April, 2024; originally announced April 2024.

Topology-engineered orbital Hall effect in two-dimensional ferromagnets

Authors: Zhiqi Chen, Runhan Li, Yingxi Bai, Ning Mao, Mahmoud Zeer, Dongwook Go, Ying Dai, Baibiao Huang, Yuriy Mokrousov, Chengwang Niu

Abstract: Recent advances in manipulation of orbital angular momentum (OAM) within the paradigm of orbitronics present a promising avenue for the design of future electronic devices. In this context, the recently observed orbital Hall effect (OHE) occupies a special place. Here, focusing on both the second-order topological and quantum anomalous Hall insulators in two-dimensional ferromagnets, we demonstrat… ▽ More Recent advances in manipulation of orbital angular momentum (OAM) within the paradigm of orbitronics present a promising avenue for the design of future electronic devices. In this context, the recently observed orbital Hall effect (OHE) occupies a special place. Here, focusing on both the second-order topological and quantum anomalous Hall insulators in two-dimensional ferromagnets, we demonstrate that topological phase transitions present an efficient and straightforward way to engineer the OHE, where the OAM distribution can be controlled by the nature of the band inversion. Using first-principles calculations, we identify Janus RuBrCl and three septuple layers of MnBi$_2$Te$_4$ as experimentally feasible examples of the proposed mechanism of OHE engineering by topology. With our work we open up new possibilities for innovative applications in topological spintronics and orbitronics. △ Less

Submitted 11 April, 2024; originally announced April 2024.

Comments: 4 figures

Research on fine co-focus adjustment method for segmented solar telescope

Authors: Kunyan Wang, Yichun Dai, Bin Wang, Xu Tan, Dehua Yang, Zhenyu Jin

Abstract: For segmented telescopes, achieving fine co-focus adjustment is essential for realizing co-phase adjustment and maintenance, which involves adjusting the millimeter-scale piston between segments to fall within the capture range of the co-phase detection system. CGST proposes using a SHWFS for piston detection during the co-focus adjustment stage. However, the residual piston after adjustment excee… ▽ More For segmented telescopes, achieving fine co-focus adjustment is essential for realizing co-phase adjustment and maintenance, which involves adjusting the millimeter-scale piston between segments to fall within the capture range of the co-phase detection system. CGST proposes using a SHWFS for piston detection during the co-focus adjustment stage. However, the residual piston after adjustment exceeds the capture range of the broadband PSF phasing algorithm$(\pm 30 μm) $, and the multi-wavelength PSF algorithm requires even higher precision in co-focus adjustment. To improve the co-focus adjustment accuracy of CGST, a fine co-focus adjustment based on cross-calibration is proposed. This method utilizes a high-precision detector to calibrate and fit the measurements from the SHWFS, thereby reducing the impact of atmospheric turbulence and systematic errors on piston measurement accuracy during co-focus adjustment. Simulation results using CGST demonstrate that the proposed method significantly enhances adjustment accuracy compared to the SHWFS detection method. Additionally, the residual piston after fine co-focus adjustment using this method falls within the capture range of the multi-wavelength PSF algorithm. To verify the feasibility of this method, experiments were conducted on an 800mm ring segmented mirror system, successfully achieving fine co-focus adjustment where the remaining piston of all segments fell within $\pm 15 μm$. △ Less

Submitted 11 April, 2024; originally announced April 2024.

Emission-line galaxies at $z\sim1$ from near-IR HST Slitless Spectroscopy: metallicities, star formation rates and redshift confirmations from VLT/FORS2 spectroscopy

Authors: K. Boyett, A. J. Bunker, J Chevallard, A. J. Battisti, A. L. Henry, S. Wilkins, M. A. Malkan, J. Caruana, H. Atek, I. Baronchelli, J. Colbert, Y. S. Dai, Jonathan. P. Gardner, M. Rafelski, C. Scarlata, H. I. Teplitz, X. Wang

Abstract: We follow up emission line galaxies identified through the near-infrared slitless HST/WFC3 WISP survey with VLT/FORS2 optical spectroscopy. Over 4 WISP fields, we targetted 85 of 138 line emission objects at $0.4<z<2$ identified in WFC3 spectroscopy. Half the galaxies are fainter than $H_{AB}=24$mag, and would not have been included in many well-known surveys based on broad-band magnitude selectio… ▽ More We follow up emission line galaxies identified through the near-infrared slitless HST/WFC3 WISP survey with VLT/FORS2 optical spectroscopy. Over 4 WISP fields, we targetted 85 of 138 line emission objects at $0.4<z<2$ identified in WFC3 spectroscopy. Half the galaxies are fainter than $H_{AB}=24$mag, and would not have been included in many well-known surveys based on broad-band magnitude selection. We confirm 95% of the initial WFC3 grism redshifts in the 38 cases where we detect lines in FORS2 spectroscopy. However, for targets which exhibited a single emission line in WFC3, up to 65% at $z<1.28$ did not have expected emission lines detected in FORS2 and hence may be spurious (although this false-detection rate improves to 33% using the latest public WISP emission line catalogue). From the Balmer decrement the extinction of the WISP galaxies is consistent with $A($H$α)=1$mag. From SED fits to multi-band photometry including Spitzer $3.6μ$m, we find a median stellar mass of $\log_{10}(M/M_{\odot})=8.94$. Our emission-line-selected galaxies tend to lie above the star-forming main sequence (i.e. higher specific star formation rates). Using [OIII], [OII] and H$β$ lines to derive gas-phase metallicities, we find typically sub-solar metallicities, decreasing with redshift. Our WISP galaxies lie below the $z=0$ mass-metallicity relation, and galaxies with higher star formation rates tend to have lower metallicity. Finally, we find a strong increase with redshift of the H$α$ rest-frame equivalent width in this emission-line selected sample, with higher $EW_0$ galaxies having larger [OIII]/H$β$ and O32 ratios on average, suggesting lower metallicity or higher ionisation parameter in these extreme emission line galaxies. △ Less

Submitted 30 August, 2024; v1 submitted 11 April, 2024; originally announced April 2024.

Comments: Accepted to MNRAS

A Proximal-Gradient Method for Constrained Optimization

Authors: Yutong Dai, Xiaoyi Qu, Daniel P. Robinson

Abstract: We present a new algorithm for solving optimization problems with objective functions that are the sum of a smooth function and a (potentially) nonsmooth regularization function, and nonlinear equality constraints. The algorithm may be viewed as an extension of the well-known proximal-gradient method that is applicable when constraints are not present. To account for nonlinear equality constraints… ▽ More We present a new algorithm for solving optimization problems with objective functions that are the sum of a smooth function and a (potentially) nonsmooth regularization function, and nonlinear equality constraints. The algorithm may be viewed as an extension of the well-known proximal-gradient method that is applicable when constraints are not present. To account for nonlinear equality constraints, we combine a decomposition procedure for computing trial steps with an exact merit function for determining trial step acceptance. Under common assumptions, we show that both the proximal parameter and merit function parameter eventually remain fixed, and then prove a worst-case complexity result for the maximum number of iterations before an iterate satisfying approximate first-order optimality conditions for a given tolerance is computed. Our preliminary numerical results indicate that our approach has great promise, especially in terms of returning approximate solutions that are structured (e.g., sparse solutions when a one-norm regularizer is used). △ Less

Submitted 10 April, 2024; originally announced April 2024.

MSC Class: 49M37; 65K05; 65K10; 65Y20; 68Q25; 90C30; 90C60

3D Geometry-aware Deformable Gaussian Splatting for Dynamic View Synthesis

Authors: Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang, Feng Zhu, Yuchao Dai

Abstract: In this paper, we propose a 3D geometry-aware deformable Gaussian Splatting method for dynamic view synthesis. Existing neural radiance fields (NeRF) based solutions learn the deformation in an implicit manner, which cannot incorporate 3D scene geometry. Therefore, the learned deformation is not necessarily geometrically coherent, which results in unsatisfactory dynamic view synthesis and 3D dynam… ▽ More In this paper, we propose a 3D geometry-aware deformable Gaussian Splatting method for dynamic view synthesis. Existing neural radiance fields (NeRF) based solutions learn the deformation in an implicit manner, which cannot incorporate 3D scene geometry. Therefore, the learned deformation is not necessarily geometrically coherent, which results in unsatisfactory dynamic view synthesis and 3D dynamic reconstruction. Recently, 3D Gaussian Splatting provides a new representation of the 3D scene, building upon which the 3D geometry could be exploited in learning the complex 3D deformation. Specifically, the scenes are represented as a collection of 3D Gaussian, where each 3D Gaussian is optimized to move and rotate over time to model the deformation. To enforce the 3D scene geometry constraint during deformation, we explicitly extract 3D geometry features and integrate them in learning the 3D deformation. In this way, our solution achieves 3D geometry-aware deformation modeling, which enables improved dynamic view synthesis and 3D dynamic reconstruction. Extensive experimental results on both synthetic and real datasets prove the superiority of our solution, which achieves new state-of-the-art performance. The project is available at https://npucvr.github.io/GaGS/ △ Less

Submitted 14 April, 2024; v1 submitted 9 April, 2024; originally announced April 2024.

Comments: Accepted by CVPR 2024. Project page: https://npucvr.github.io/GaGS/

A model for heating the super-hot corona in solar active regions

Authors: Zekun Lu, Feng Chen, M. D. Ding, Can Wang, Yu Dai, Xin Cheng

Abstract: What physical mechanisms heat the outer solar or stellar atmosphere to million-Kelvin temperatures is a fundamental but long-standing open question. In particular, the solar corona in active region cores contains an even hotter component reaching ten million Kelvin, manifesting as persistent coronal loops in extreme ultraviolet and soft X-ray images, which imposes a more stringent energy budget. H… ▽ More What physical mechanisms heat the outer solar or stellar atmosphere to million-Kelvin temperatures is a fundamental but long-standing open question. In particular, the solar corona in active region cores contains an even hotter component reaching ten million Kelvin, manifesting as persistent coronal loops in extreme ultraviolet and soft X-ray images, which imposes a more stringent energy budget. Here, we present a self-consistent coronal heating model using a state-of-the-art three-dimensional radiative magnetohydrodynamics simulation. We find that the continuous magnetic flux emergence in active regions keeps driving magnetic reconnections that release energy impulsively but, on time average, persistently. As a result, numerous sub-structures are heated to ten million Kelvin and then evolve independently, which collectively form long-lived and stable coronal loops as in observations. This provides a heating model explaining the origin of the super-hot coronal plasma and the persistence of hot coronal loops in emerging active regions. △ Less

Submitted 8 April, 2024; originally announced April 2024.

Comments: 34 pages, 14 figures

Journal ref: 8 April 2024, Nature Astronomy

WFC3 Infrared Spectroscopic Parallel (WISP) Survey: Photometric and Emission Line Data Release

Authors: A. J. Battisti, M. B. Bagley, M. Rafelski, I. Baronchelli, Y. S. Dai, A. L. Henry, H. Atek, J. Colbert, M. A. Malkan, P. J. McCarthy, C. Scarlata, B. Siana, H. I. Teplitz, A. Alavi, K. Boyett, A. J. Bunker, J. P. Gardner, N. P. Hathi, D. Masters, V. Mehta, M. Rutkowski, K. Shahinyan, B. Sunnquist, X. Wang

Abstract: We present reduced images and catalogues of photometric and emission line data ($\sim$230,000 and $\sim$8,000 sources, respectively) for the WFC3 Infrared Spectroscopic Parallel (WISP) Survey. These data are made publicly available on the Mikulski Archive for Space Telescopes (MAST) and include reduced images from various facilities: ground-based $ugri$, HST WFC3, and Spitzer IRAC (Infrared Array… ▽ More We present reduced images and catalogues of photometric and emission line data ($\sim$230,000 and $\sim$8,000 sources, respectively) for the WFC3 Infrared Spectroscopic Parallel (WISP) Survey. These data are made publicly available on the Mikulski Archive for Space Telescopes (MAST) and include reduced images from various facilities: ground-based $ugri$, HST WFC3, and Spitzer IRAC (Infrared Array Camera). Coverage in at least one additional filter beyond the WFC3/IR data are available for roughly half of the fields (227 out of 483), with $\sim$20% (86) having coverage in six or more filters from $u$-band to IRAC 3.6$μ$m (0.35-3.6$μ$m). For the lower spatial resolution (and shallower) ground-based and IRAC data, we perform PSF-matched, prior-based, deconfusion photometry (i.e., forced-photometry) using the TPHOT software to optimally extract measurements or upper limits. We present the methodology and software used for the WISP emission line detection and visual inspection. The former adopts a continuous wavelet transformation that significantly reduces the number of spurious sources as candidates before the visual inspection stage. We combine both WISP catalogues and perform SED fitting on galaxies with reliable spectroscopic redshifts and multi-band photometry to measure their stellar masses. We stack WISP spectra as functions of stellar mass and redshift and measure average emission line fluxes and ratios. We find that WISP emission line sources are typically `normal' star-forming galaxies based on the Mass-Excitation diagram ([OIII]/H$β$ vs. $M_\star$; $0.74<z_\mathrm{grism}<2.31$), the galaxy main sequence (SFR vs. $M_\star$; $0.30<z_\mathrm{grism}<1.45$), $S_{32}$ ratio vs. $M_\star$ ($0.30<z_\mathrm{grism}<0.73$), and $O_{32}$ and $R_{23}$ ratios vs. $M_\star$ ($1.27<z_\mathrm{grism}<1.45$). △ Less

Submitted 6 April, 2024; originally announced April 2024.

Comments: 36 pages, 21 figures, 17 tables. Accepted for publication in MNRAS. The WISP Photometric and Emission Line catalogues and reduced images are in the process of being added as HLSPs to the WISP MAST website (https://archive.stsci.edu/prepds/wisp/). Please email the first-author (provided in paper) to request access to files prior to the MAST release

GENEVIC: GENetic data Exploration and Visualization via Intelligent interactive Console

Authors: Anindita Nath, Savannah Mwesigwa, Yulin Dai, Xiaoqian Jiang, Zhongming Zhao

Abstract: Summary: The vast generation of genetic data poses a significant challenge in efficiently uncovering valuable knowledge. Introducing GENEVIC, an AI-driven chat framework that tackles this challenge by bridging the gap between genetic data generation and biomedical knowledge discovery. Leveraging generative AI, notably ChatGPT, it serves as a biologist's 'copilot'. It automates the analysis, retrie… ▽ More Summary: The vast generation of genetic data poses a significant challenge in efficiently uncovering valuable knowledge. Introducing GENEVIC, an AI-driven chat framework that tackles this challenge by bridging the gap between genetic data generation and biomedical knowledge discovery. Leveraging generative AI, notably ChatGPT, it serves as a biologist's 'copilot'. It automates the analysis, retrieval, and visualization of customized domain-specific genetic information, and integrates functionalities to generate protein interaction networks, enrich gene sets, and search scientific literature from PubMed, Google Scholar, and arXiv, making it a comprehensive tool for biomedical research. In its pilot phase, GENEVIC is assessed using a curated database that ranks genetic variants associated with Alzheimer's disease, schizophrenia, and cognition, based on their effect weights from the Polygenic Score Catalog, thus enabling researchers to prioritize genetic variants in complex diseases. GENEVIC's operation is user-friendly, accessible without any specialized training, secured by Azure OpenAI's HIPAA-compliant infrastructure, and evaluated for its efficacy through real-time query testing. As a prototype, GENEVIC is set to advance genetic research, enabling informed biomedical decisions. Availability and implementation: GENEVIC is publicly accessible at https://genevic-anath2024.streamlit.app. The underlying code is open-source and available via GitHub at https://github.com/anath2110/GENEVIC.git. △ Less

Submitted 4 April, 2024; originally announced April 2024.

The impact of geopolitical risk on the international agricultural market: Empirical analysis based on the GJR-GARCH-MIDAS model

Authors: Yun-Shi Dai, Peng-Fei Dai, Wei-Xing Zhou

Abstract: The current international landscape is turbulent and unstable, with frequent outbreaks of geopolitical conflicts worldwide. Geopolitical risk has emerged as a significant threat to regional and global peace, stability, and economic prosperity, causing serious disruptions to the global food system and food security. Focusing on the international food market, this paper builds different dimensions o… ▽ More The current international landscape is turbulent and unstable, with frequent outbreaks of geopolitical conflicts worldwide. Geopolitical risk has emerged as a significant threat to regional and global peace, stability, and economic prosperity, causing serious disruptions to the global food system and food security. Focusing on the international food market, this paper builds different dimensions of geopolitical risk measures based on the random matrix theory and constructs single- and two-factor GJR-GARCH-MIDAS models with fixed time span and rolling window, respectively, to investigate the impact of geopolitical risk on food market volatility. The findings indicate that modeling based on rolling window performs better in describing the overall volatility of the wheat, maize, soybean, and rice markets, and the two-factor models generally exhibit stronger explanatory power in most cases. In terms of short-term fluctuations, all four staple food markets demonstrate obvious volatility clustering and high volatility persistence, without significant asymmetry. Regarding long-term volatility, the realized volatility of wheat, maize, and soybean significantly exacerbates their long-run market volatility. Additionally, geopolitical risks of different dimensions show varying directions and degrees of effects in explaining the long-term market volatility of the four staple food commodities. This study contributes to the understanding of the macro-drivers of food market fluctuations, provides useful information for investment using agricultural futures, and offers valuable insights into maintaining the stable operation of food markets and safeguarding global food security. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Comments: 38 pages, 3 figures, 11 tables

Parallel Proportional Fusion of Spiking Quantum Neural Network for Optimizing Image Classification

Authors: Zuyu Xu, Kang Shen, Pengnian Cai, Tao Yang, Yuanming Hu, Shixian Chen, Yunlai Zhu, Zuheng Wu, Yuehua Dai, Jun Wang, Fei Yang

Abstract: The recent emergence of the hybrid quantum-classical neural network (HQCNN) architecture has garnered considerable attention due to the potential advantages associated with integrating quantum principles to enhance various facets of machine learning algorithms and computations. However, the current investigated serial structure of HQCNN, wherein information sequentially passes from one network to… ▽ More The recent emergence of the hybrid quantum-classical neural network (HQCNN) architecture has garnered considerable attention due to the potential advantages associated with integrating quantum principles to enhance various facets of machine learning algorithms and computations. However, the current investigated serial structure of HQCNN, wherein information sequentially passes from one network to another, often imposes limitations on the trainability and expressivity of the network. In this study, we introduce a novel architecture termed Parallel Proportional Fusion of Quantum and Spiking Neural Networks (PPF-QSNN). The dataset information is simultaneously fed into both the spiking neural network and the variational quantum circuits, with the outputs amalgamated in proportion to their individual contributions. We systematically assess the impact of diverse PPF-QSNN parameters on network performance for image classification, aiming to identify the optimal configuration. Numerical results on the MNIST dataset unequivocally illustrate that our proposed PPF-QSNN outperforms both the existing spiking neural network and the serial quantum neural network across metrics such as accuracy, loss, and robustness. This study introduces a novel and effective amalgamation approach for HQCNN, thereby laying the groundwork for the advancement and application of quantum advantage in artificial intelligent computations. △ Less

Submitted 1 April, 2024; originally announced April 2024.

Slightly Shift New Classes to Remember Old Classes for Video Class-Incremental Learning

Authors: Jian Jiao, Yu Dai, Hefei Mei, Heqian Qiu, Chuanyang Gong, Shiyuan Tang, Xinpeng Hao, Hongliang Li

Abstract: Recent video class-incremental learning usually excessively pursues the accuracy of the newly seen classes and relies on memory sets to mitigate catastrophic forgetting of the old classes. However, limited storage only allows storing a few representative videos. So we propose SNRO, which slightly shifts the features of new classes to remember old classes. Specifically, SNRO contains Examples Spars… ▽ More Recent video class-incremental learning usually excessively pursues the accuracy of the newly seen classes and relies on memory sets to mitigate catastrophic forgetting of the old classes. However, limited storage only allows storing a few representative videos. So we propose SNRO, which slightly shifts the features of new classes to remember old classes. Specifically, SNRO contains Examples Sparse(ES) and Early Break(EB). ES decimates at a lower sample rate to build memory sets and uses interpolation to align those sparse frames in the future. By this, SNRO stores more examples under the same memory consumption and forces the model to focus on low-semantic features which are harder to be forgotten. EB terminates the training at a small epoch, preventing the model from overstretching into the high-semantic space of the current task. Experiments on UCF101, HMDB51, and UESTC-MMEA-CL datasets show that SNRO performs better than other approaches while consuming the same memory consumption. △ Less

Submitted 31 March, 2024; originally announced April 2024.

Stochastic Approximation Proximal Subgradient Method for Stochastic Convex-Concave Minimax Optimization

Authors: Yu-Hong Dai, Jiani Wang, Liwei Zhang

Abstract: This paper presents a stochastic approximation proximal subgradient (SAPS) method for stochastic convex-concave minimax optimization. By accessing unbiased and variance bounded approximate subgradients, we show that this algorithm exhibits ${\rm O}(N^{-1/2})$ expected convergence rate of the minimax optimality measure if the parameters in the algorithm are properly chosen, where $N$ denotes the nu… ▽ More This paper presents a stochastic approximation proximal subgradient (SAPS) method for stochastic convex-concave minimax optimization. By accessing unbiased and variance bounded approximate subgradients, we show that this algorithm exhibits ${\rm O}(N^{-1/2})$ expected convergence rate of the minimax optimality measure if the parameters in the algorithm are properly chosen, where $N$ denotes the number of iterations. Moreover, we show that the algorithm has ${\rm O}(\log(N)N^{-1/2})$ minimax optimality measure bound with high probability. Further we study a specific stochastic convex-concave minimax optimization problems arising from stochastic convex conic optimization problems, which the the bounded subgradient condition is fail. To overcome the lack of the bounded subgradient conditions in convex-concave minimax problems, we propose a linearized stochastic approximation augmented Lagrange (LSAAL) method and prove that this algorithm exhibits ${\rm O}(N^{-1/2})$ expected convergence rate for the minimax optimality measure and ${\rm O}(\log^2(N)N^{-1/2})$ minimax optimality measure bound with high probability as well. Preliminary numerical results demonstrate the effect of the SAPS and LSAAL methods. △ Less

Submitted 29 March, 2024; originally announced March 2024.

MSC Class: 90C30

RELI11D: A Comprehensive Multimodal Human Motion Dataset and Method

Authors: Ming Yan, Yan Zhang, Shuqiang Cai, Shuqi Fan, Xincheng Lin, Yudi Dai, Siqi Shen, Chenglu Wen, Lan Xu, Yuexin Ma, Cheng Wang

Abstract: Comprehensive capturing of human motions requires both accurate captures of complex poses and precise localization of the human within scenes. Most of the HPE datasets and methods primarily rely on RGB, LiDAR, or IMU data. However, solely using these modalities or a combination of them may not be adequate for HPE, particularly for complex and fast movements. For holistic human motion understanding… ▽ More Comprehensive capturing of human motions requires both accurate captures of complex poses and precise localization of the human within scenes. Most of the HPE datasets and methods primarily rely on RGB, LiDAR, or IMU data. However, solely using these modalities or a combination of them may not be adequate for HPE, particularly for complex and fast movements. For holistic human motion understanding, we present RELI11D, a high-quality multimodal human motion dataset involves LiDAR, IMU system, RGB camera, and Event camera. It records the motions of 10 actors performing 5 sports in 7 scenes, including 3.32 hours of synchronized LiDAR point clouds, IMU measurement data, RGB videos and Event steams. Through extensive experiments, we demonstrate that the RELI11D presents considerable challenges and opportunities as it contains many rapid and complex motions that require precise location. To address the challenge of integrating different modalities, we propose LEIR, a multimodal baseline that effectively utilizes LiDAR Point Cloud, Event stream, and RGB through our cross-attention fusion strategy. We show that LEIR exhibits promising results for rapid motions and daily motions and that utilizing the characteristics of multiple modalities can indeed improve HPE performance. Both the dataset and source code will be released publicly to the research community, fostering collaboration and enabling further exploration in this field. △ Less

Submitted 28 March, 2024; originally announced March 2024.

Comments: CVPR2024, Project website: http://www.lidarhumanmotion.net/reli11d/

Close Major-merger Pairs at $z=0$: Star-forming Galaxies with Pseudobulges

Authors: Chuan He, Cong Kevin Xu, Ute Lisenfeld, Y Sophia Dai, Taotao Fang, Jia-Sheng Huang, Wei Wang, Qingzheng Yu

Abstract: We present a study of star-forming galaxies (SFGs) with pseudobulges (bulges with Sérsic index $\rm n < 2$) in a local close major-merger galaxy pair sample (H-KPAIR). With data from new aperture photometries in the optical and near-infrared bands (aperture size of 7\;kpc) and from the literature, we find that the mean Age of central stellar populations in Spirals with pseudobulges is consistent w… ▽ More We present a study of star-forming galaxies (SFGs) with pseudobulges (bulges with Sérsic index $\rm n < 2$) in a local close major-merger galaxy pair sample (H-KPAIR). With data from new aperture photometries in the optical and near-infrared bands (aperture size of 7\;kpc) and from the literature, we find that the mean Age of central stellar populations in Spirals with pseudobulges is consistent with that of disky galaxies and is nearly constant against the bulge-to-total ratio (B/T). Paired Spirals have a slightly lower fraction of pure disk galaxies ($\rm B/T \leq 0.1$) than their counterparts in the control sample. Compared to SFGs with classical bulges, those with pseudobulges have a higher ($>2\;σ$) mean of specific star formation rate (sSFR) enhancement ($\rm sSFR_{enh} = 0.33\pm0.07$ vs $\rm sSFR_{enh} = 0.12\pm0.06$) and broader scatter (by $\sim 1$\;dex). The eight SFGs that have the highest $\rm sSFR_{enh}$ in the sample all have pseudobulges. A majority (69\%) of paired SFGs with strong enhancement (having sSFR more than 5 times the median of the control galaxies) have pseudobulges. The Spitzer data show that the pseudobulges in these galaxies are tightly linked to nuclear/circum-nuclear starbursts. Pseudobulge SFGs in S+S and in S+E pairs have significantly ($>3\;σ$) different sSFR enhancement, with the means of $\rm sSFR_{enh} = 0.45\pm0.08$ and $-0.04\pm0.11$, respectively. We find a decrease in the sSFR enhancements with the density of the environment for SFGs with pseudobulges. Since a high fraction (5/11) of pseudobulge SFGs in S+E pairs are in rich groups/clusters (local density $\rm N_{1Mpc} \geq 7$), the dense environment might be the cause for their low $\rm sSFR_{enh}$. △ Less

Submitted 28 March, 2024; originally announced March 2024.

Comments: Accepted for publication in RAA, ?? pages, 10 figures, 4 tables

Learning Inclusion Matching for Animation Paint Bucket Colorization

Authors: Yuekun Dai, Shangchen Zhou, Qinyue Li, Chongyi Li, Chen Change Loy

Abstract: Colorizing line art is a pivotal task in the production of hand-drawn cel animation. This typically involves digital painters using a paint bucket tool to manually color each segment enclosed by lines, based on RGB values predetermined by a color designer. This frame-by-frame process is both arduous and time-intensive. Current automated methods mainly focus on segment matching. This technique migr… ▽ More Colorizing line art is a pivotal task in the production of hand-drawn cel animation. This typically involves digital painters using a paint bucket tool to manually color each segment enclosed by lines, based on RGB values predetermined by a color designer. This frame-by-frame process is both arduous and time-intensive. Current automated methods mainly focus on segment matching. This technique migrates colors from a reference to the target frame by aligning features within line-enclosed segments across frames. However, issues like occlusion and wrinkles in animations often disrupt these direct correspondences, leading to mismatches. In this work, we introduce a new learning-based inclusion matching pipeline, which directs the network to comprehend the inclusion relationships between segments rather than relying solely on direct visual correspondences. Our method features a two-stage pipeline that integrates a coarse color warping module with an inclusion matching module, enabling more nuanced and accurate colorization. To facilitate the training of our network, we also develope a unique dataset, referred to as PaintBucket-Character. This dataset includes rendered line arts alongside their colorized counterparts, featuring various 3D characters. Extensive experiments demonstrate the effectiveness and superiority of our method over existing techniques. △ Less

Submitted 27 March, 2024; originally announced March 2024.

Comments: accepted to CVPR 2024. Project Page: https://ykdai.github.io/projects/InclusionMatching

Interfacial magnetic spin Hall effect in van der Waals Fe3GeTe2/MoTe2 heterostructure

Authors: Yudi Dai, Junlin Xiong, Yanfeng Ge, Bin Cheng, Lizheng Wang, Pengfei Wang, Zenglin Liu, Shengnan Yan, Cuiwei Zhang, Xianghan Xu, Youguo Shi, Sang-Wook Cheong, Cong Xiao, Shengyuan A. Yang, Shi-Jun Liang, Feng Miao

Abstract: The spin Hall effect (SHE) allows efficient generation of spin polarization or spin current through charge current and plays a crucial role in the development of spintronics. While SHE typically occurs in non-magnetic materials and is time-reversal even, exploring time-reversal-odd (T-odd) SHE, which couples SHE to magnetization in ferromagnetic materials, offers a new charge-spin conversion mecha… ▽ More The spin Hall effect (SHE) allows efficient generation of spin polarization or spin current through charge current and plays a crucial role in the development of spintronics. While SHE typically occurs in non-magnetic materials and is time-reversal even, exploring time-reversal-odd (T-odd) SHE, which couples SHE to magnetization in ferromagnetic materials, offers a new charge-spin conversion mechanism with new functionalities. Here, we report the observation of giant T-odd SHE in Fe3GeTe2/MoTe2 van der Waals heterostructure, representing a previously unidentified interfacial magnetic spin Hall effect (interfacial-MSHE). Through rigorous symmetry analysis and theoretical calculations, we attribute the interfacial-MSHE to a symmetry-breaking induced spin current dipole at the vdW interface. Furthermore, we show that this linear effect can be used for implementing multiply-accumulate operations and binary convolutional neural networks with cascaded multi-terminal devices. Our findings uncover an interfacial T-odd charge-spin conversion mechanism with promising potential for energy-efficient in-memory computing. △ Less

Submitted 26 March, 2024; originally announced March 2024.

Journal ref: Nature Communications 15, 1129 (2024)

AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behavior

Authors: Zhouhong Gu, Xiaoxuan Zhu, Haoran Guo, Lin Zhang, Yin Cai, Hao Shen, Jiangjie Chen, Zheyu Ye, Yifei Dai, Yan Gao, Yao Hu, Hongwei Feng, Yanghua Xiao

Abstract: Language significantly influences the formation and evolution of Human emergent behavior, which is crucial in understanding collective intelligence within human societies. Considering that the study of how language affects human behavior needs to put it into the dynamic scenarios in which it is used, we introduce AgentGroupChat in this paper, a simulation that delves into the complex role of langu… ▽ More Language significantly influences the formation and evolution of Human emergent behavior, which is crucial in understanding collective intelligence within human societies. Considering that the study of how language affects human behavior needs to put it into the dynamic scenarios in which it is used, we introduce AgentGroupChat in this paper, a simulation that delves into the complex role of language in shaping collective behavior through interactive debate scenarios. Central to this simulation are characters engaging in dynamic conversation interactions. To enable simulation, we introduce the Verbal Strategist Agent, utilizing large language models to enhance interaction strategies by incorporating elements of persona and action. We set four narrative scenarios based on AgentGroupChat to demonstrate the simulation's capacity to mimic complex language use in group dynamics. Evaluations focus on aligning agent behaviors with human expectations and the emergence of collective behaviors within the simulation. Results reveal that emergent behaviors materialize from a confluence of factors: a conducive environment for extensive information exchange, characters with diverse traits, high linguistic comprehension, and strategic adaptability. During discussions on ``the impact of AI on humanity'' in AgentGroupChat simulation, philosophers commonly agreed that ``AI could enhance societal welfare with judicious limitations'' and even come to a conclusion that ``the essence of true intelligence encompasses understanding the necessity to constrain self abilities''. Additionally, in the competitive domain of casting for primary roles in films in AgentGroupChat, certain actors were ready to reduce their remuneration or accept lesser roles, motivated by their deep-seated desire to contribute to the project. △ Less

Submitted 4 April, 2024; v1 submitted 20 March, 2024; originally announced March 2024.

LSKNet: A Foundation Lightweight Backbone for Remote Sensing

Authors: Yuxuan Li, Xiang Li, Yimian Dai, Qibin Hou, Li Liu, Yongxiang Liu, Ming-Ming Cheng, Jian Yang

Abstract: Remote sensing images pose distinct challenges for downstream tasks due to their inherent complexity. While a considerable amount of research has been dedicated to remote sensing classification, object detection and semantic segmentation, most of these studies have overlooked the valuable prior knowledge embedded within remote sensing scenarios. Such prior knowledge can be useful because remote se… ▽ More Remote sensing images pose distinct challenges for downstream tasks due to their inherent complexity. While a considerable amount of research has been dedicated to remote sensing classification, object detection and semantic segmentation, most of these studies have overlooked the valuable prior knowledge embedded within remote sensing scenarios. Such prior knowledge can be useful because remote sensing objects may be mistakenly recognized without referencing a sufficiently long-range context, which can vary for different objects. This paper considers these priors and proposes a lightweight Large Selective Kernel Network (LSKNet) backbone. LSKNet can dynamically adjust its large spatial receptive field to better model the ranging context of various objects in remote sensing scenarios. To our knowledge, large and selective kernel mechanisms have not been previously explored in remote sensing images. Without bells and whistles, our lightweight LSKNet sets new state-of-the-art scores on standard remote sensing classification, object detection and semantic segmentation benchmarks. Our comprehensive analysis further validated the significance of the identified priors and the effectiveness of LSKNet. The code is available at https://github.com/zcablii/LSKNet. △ Less

Submitted 23 June, 2024; v1 submitted 18 March, 2024; originally announced March 2024.

Comments: arXiv admin note: substantial text overlap with arXiv:2303.09030

GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time

Authors: Hao Li, Yuanyuan Gao, Chenming Wu, Dingwen Zhang, Yalun Dai, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, Junwei Han

Abstract: This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative… ▽ More This paper presents GGRt, a novel approach to generalizable novel view synthesis that alleviates the need for real camera poses, complexity in processing high-resolution images, and lengthy optimization processes, thus facilitating stronger applicability of 3D Gaussian Splatting (3D-GS) in real-world scenarios. Specifically, we design a novel joint learning framework that consists of an Iterative Pose Optimization Network (IPO-Net) and a Generalizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism, the proposed framework can inherently estimate robust relative pose information from the image observations and thus primarily alleviate the requirement of real camera poses. Moreover, we implement a deferred back-propagation mechanism that enables high-resolution training and inference, overcoming the resolution constraints of previous methods. To enhance the speed and efficiency, we further introduce a progressive Gaussian cache module that dynamically adjusts during training and inference. As the first pose-free generalizable 3D-GS framework, GGRt achieves inference at $\ge$ 5 FPS and real-time rendering at $\ge$ 100 FPS. Through extensive experimentation, we demonstrate that our method outperforms existing NeRF-based pose-free techniques in terms of inference speed and effectiveness. It can also approach the real pose-based 3D-GS methods. Our contributions provide a significant leap forward for the integration of computer vision and computer graphics into practical applications, offering state-of-the-art results on LLFF, KITTI, and Waymo Open datasets and enabling real-time rendering for immersive experiences. △ Less

Submitted 18 March, 2024; v1 submitted 15 March, 2024; originally announced March 2024.

Comments: Project page: https://3d-aigc.github.io/GGRt

RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception

Authors: Ruiyang Hao, Siqi Fan, Yingru Dai, Zhenlin Zhang, Chenxi Li, Yuntian Wang, Haibao Yu, Wenxian Yang, Jirui Yuan, Zaiqing Nie

Abstract: The value of roadside perception, which could extend the boundaries of autonomous driving and traffic management, has gradually become more prominent and acknowledged in recent years. However, existing roadside perception approaches only focus on the single-infrastructure sensor system, which cannot realize a comprehensive understanding of a traffic area because of the limited sensing range and bl… ▽ More The value of roadside perception, which could extend the boundaries of autonomous driving and traffic management, has gradually become more prominent and acknowledged in recent years. However, existing roadside perception approaches only focus on the single-infrastructure sensor system, which cannot realize a comprehensive understanding of a traffic area because of the limited sensing range and blind spots. Orienting high-quality roadside perception, we need Roadside Cooperative Perception (RCooper) to achieve practical area-coverage roadside perception for restricted traffic areas. Rcooper has its own domain-specific challenges, but further exploration is hindered due to the lack of datasets. We hence release the first real-world, large-scale RCooper dataset to bloom the research on practical roadside cooperative perception, including detection and tracking. The manually annotated dataset comprises 50k images and 30k point clouds, including two representative traffic scenes (i.e., intersection and corridor). The constructed benchmarks prove the effectiveness of roadside cooperation perception and demonstrate the direction of further research. Codes and dataset can be accessed at: https://github.com/AIR-THU/DAIR-RCooper. △ Less

Submitted 31 March, 2024; v1 submitted 15 March, 2024; originally announced March 2024.

Comments: Accepted by CVPR2024. 10 pages with 6 figures

ACM Class: I.4.8; I.5.4

Polarized Charge Dynamics of a Novel Charge Density Wave in Kagome FeGe

Authors: Shaohui Yi, Zhiyu Liao, Qi Wang, Haiyang Ma, Jianpeng Liu, Xiaokun Teng, Pengcheng Dai, Yaomin Dai, Jianzhou Zhao, Yanpeng Qi, Bing Xu, Xianggang Qiu

Abstract: We report on the charge dynamics of kagome FeGe, an antiferromagnet with a charge density wave (CDW) transition at $T_{\mathrm{CDW}} \simeq 105$ K, using polarized infrared spectroscopy and band structure calculations. We reveal a pronounced optical anisotropy, various excitations associated with flat bands and van Hove singularities (VHSs), and a moderate level of electronic correlations. Notably… ▽ More We report on the charge dynamics of kagome FeGe, an antiferromagnet with a charge density wave (CDW) transition at $T_{\mathrm{CDW}} \simeq 105$ K, using polarized infrared spectroscopy and band structure calculations. We reveal a pronounced optical anisotropy, various excitations associated with flat bands and van Hove singularities (VHSs), and a moderate level of electronic correlations. Notably, there are two types of remarkable spectral weight (SW) redistributions for above and below $T_{\mathrm{CDW}}$. The former involves a transfer between incoherent and coherent excitations driven by the magnetic splitting-induced elevation of flat bands. The latter manifests itself as a sudden change of SW from low to high energies for both $a$ and $c$ directions, suggesting a first-order transition and the three-dimensional nature of CDW. These anomalies in SW significantly differ from those observed in other kagome metals like CsV$_3$Sb$_5$, where the nesting of VHSs results in a pronounced CDW gap feature. Instead, our findings can be accounted for by the jump of VHSs relative to the Fermi energy via a first-order structural transition involving large partial Ge1-dimerization. Our study thus unveils a complex interplay among structure, magnetism, electronic correlations, and charge order in FeGe, offering valuable insights for a comprehensive understanding of CDW order in kagome systems. △ Less

Submitted 14 March, 2024; originally announced March 2024.

Comments: 7 pages, 3 figures

OPEN TEACH: A Versatile Teleoperation System for Robotic Manipulation

Authors: Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, Lerrel Pinto

Abstract: Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphol… ▽ More Open-sourced, user-friendly tools form the bedrock of scientific advancement across disciplines. The widespread adoption of data-driven learning has led to remarkable progress in multi-fingered dexterity, bimanual manipulation, and applications ranging from logistics to home robotics. However, existing data collection platforms are often proprietary, costly, or tailored to specific robotic morphologies. We present OPEN TEACH, a new teleoperation system leveraging VR headsets to immerse users in mixed reality for intuitive robot control. Built on the affordable Meta Quest 3, which costs $500, OPEN TEACH enables real-time control of various robots, including multi-fingered hands and bimanual arms, through an easy-to-use app. Using natural hand gestures and movements, users can manipulate robots at up to 90Hz with smooth visual feedback and interface widgets offering closeup environment views. We demonstrate the versatility of OPEN TEACH across 38 tasks on different robots. A comprehensive user study indicates significant improvement in teleoperation capability over the AnyTeleop framework. Further experiments exhibit that the collected data is compatible with policy learning on 10 dexterous and contact-rich manipulation tasks. Currently supporting Franka, xArm, Jaco, and Allegro platforms, OPEN TEACH is fully open-sourced to promote broader adoption. Videos are available at https://open-teach.github.io/. △ Less

Submitted 12 March, 2024; originally announced March 2024.

Harmonious Group Choreography with Trajectory-Controllable Diffusion

Authors: Yuqin Dai, Wanlu Zhu, Ronghui Li, Zeping Ren, Xiangzheng Zhou, Xiu Li, Jun Li, Jian Yang

Abstract: Creating group choreography from music has gained attention in cultural entertainment and virtual reality, aiming to coordinate visually cohesive and diverse group movements. Despite increasing interest, recent works face challenges in achieving aesthetically appealing choreography, primarily for two key issues: multi-dancer collision and single-dancer foot slide. To address these issues, we propo… ▽ More Creating group choreography from music has gained attention in cultural entertainment and virtual reality, aiming to coordinate visually cohesive and diverse group movements. Despite increasing interest, recent works face challenges in achieving aesthetically appealing choreography, primarily for two key issues: multi-dancer collision and single-dancer foot slide. To address these issues, we propose a Trajectory-Controllable Diffusion (TCDiff), a novel approach that harnesses non-overlapping trajectories to facilitate coherent dance movements. Specifically, to tackle dancer collisions, we introduce a Dance-Beat Navigator capable of generating trajectories for multiple dancers based on the music, complemented by a Distance-Consistency loss to maintain appropriate spacing among trajectories within a reasonable threshold. To mitigate foot sliding, we present a Footwork Adaptor that utilizes trajectory displacement from adjacent frames to enable flexible footwork, coupled with a Relative Forward-Kinematic loss to adjust the positioning of individual dancers' root nodes and joints. Extensive experiments demonstrate that our method achieves state-of-the-art results. △ Less

Submitted 13 August, 2024; v1 submitted 10 March, 2024; originally announced March 2024.

LTGC: Long-tail Recognition via Leveraging LLMs-driven Generated Content

Authors: Qihao Zhao, Yalun Dai, Hao Li, Wei Hu, Fan Zhang, Jun Liu

Abstract: Long-tail recognition is challenging because it requires the model to learn good representations from tail categories and address imbalances across all categories. In this paper, we propose a novel generative and fine-tuning framework, LTGC, to handle long-tail recognition via leveraging generated content. Firstly, inspired by the rich implicit knowledge in large-scale models (e.g., large language… ▽ More Long-tail recognition is challenging because it requires the model to learn good representations from tail categories and address imbalances across all categories. In this paper, we propose a novel generative and fine-tuning framework, LTGC, to handle long-tail recognition via leveraging generated content. Firstly, inspired by the rich implicit knowledge in large-scale models (e.g., large language models, LLMs), LTGC leverages the power of these models to parse and reason over the original tail data to produce diverse tail-class content. We then propose several novel designs for LTGC to ensure the quality of the generated data and to efficiently fine-tune the model using both the generated and original data. The visualization demonstrates the effectiveness of the generation module in LTGC, which produces accurate and diverse tail data. Additionally, the experimental results demonstrate that our LTGC outperforms existing state-of-the-art methods on popular long-tailed benchmarks. △ Less

Submitted 26 May, 2024; v1 submitted 9 March, 2024; originally announced March 2024.

Comments: CVPR 2024, Oral

CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning

Authors: Yanqi Dai, Dong Jing, Nanyi Fei, Zhiwu Lu

Abstract: Visual instruction tuning is a key training stage of large multimodal models (LMMs). Nevertheless, the common practice of indiscriminately mixing instruction-following data from various tasks may result in suboptimal overall performance due to different instruction formats and knowledge domains across tasks. To mitigate this issue, we propose a novel Comprehensive Task Balancing (CoTBal) algorithm… ▽ More Visual instruction tuning is a key training stage of large multimodal models (LMMs). Nevertheless, the common practice of indiscriminately mixing instruction-following data from various tasks may result in suboptimal overall performance due to different instruction formats and knowledge domains across tasks. To mitigate this issue, we propose a novel Comprehensive Task Balancing (CoTBal) algorithm for multi-task visual instruction tuning of LMMs. To our knowledge, this is the first work that explores multi-task optimization in visual instruction tuning. Specifically, we consider two key dimensions for task balancing: (1) Inter-Task Contribution, the phenomenon where learning one task potentially enhances the performance in other tasks, attributable to the overlapping knowledge domains, and (2) Intra-Task Difficulty, which refers to the learning difficulty within a single task. By quantifying these two dimensions with performance-based metrics, task balancing is thus enabled by assigning more weights to tasks that offer substantial contributions to others, receive minimal contributions from others, and also have great intra-task difficulties. Experiments show that our CoTBal leads to superior overall performance in multi-task visual instruction tuning. △ Less

Submitted 7 March, 2024; originally announced March 2024.

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Authors: Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Jiefeng Ma, Haotian Wang, Chin-Hui Lee

Abstract: Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting p… ▽ More Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH) to systematically describe the relationship between modality bias and robustness against missing modality in multimodal systems. Building on these findings, we propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality and to maintain performance and robustness simultaneously. Finally, to address an entirely missing modality, we adopt adapters to dynamically switch decision strategies. The effectiveness of our proposed approach is evaluated and validated through a series of comprehensive experiments using the MISP2021 and MISP2022 datasets. Our code is available at https://github.com/dalision/ModalBiasAVSR △ Less

Submitted 7 March, 2024; originally announced March 2024.

Comments: the paper is accepted by CVPR2024

The true number density of massive galaxies in the early Universe revealed by JWST/MIRI

Authors: Tao Wang, Hanwen Sun, Luwenjia Zhou, Ke Xu, Cheng Cheng, Zhaozhou Li, Yangyao Chen, H. J. Mo, Avishai Dekel, Xianzhong Zheng, Zheng Cai, Tiacheng Yang, Y. -S. Dai, David Elbaz, J. -S. Huang

Abstract: One of the main challenges in galaxy formation that has emerged recently is the early assembly of massive galaxies. The observed number density and the maximum stellar mass ($M_{\star}$) of massive galaxies in the early Universe appear to be higher than model predictions, which may pose a serious problem to the LCDM cosmology. A major limitation in many previous studies is the large uncertainty in… ▽ More One of the main challenges in galaxy formation that has emerged recently is the early assembly of massive galaxies. The observed number density and the maximum stellar mass ($M_{\star}$) of massive galaxies in the early Universe appear to be higher than model predictions, which may pose a serious problem to the LCDM cosmology. A major limitation in many previous studies is the large uncertainty in estimating $M_{\star}$ due to the lack of constraints in the rest-frame near-infrared part of the spectral energy distribution, which is critical to determining $M_{\star}$ accurately. Here we use data from a large JWST/MIRI survey in the PRIMER program to carry out a systematic analysis of massive galaxies at $z \sim 3-8$, leveraging photometric constraints at rest-frame $\gtrsim 1 μ$m. We find a significant reduction in the number and mass densities of massive galaxies at $z > 5$ compared to earlier results that did not use the MIRI photometry. Within the standard $Λ$CDM cosmology, our results require a moderate increase in the baryon-to-star conversion efficiency ($ε$) towards higher redshifts and higher $M_{\star}$. For the most massive galaxies at $z\sim 8$, the required $ε$ is $\sim 0.3$, in comparison to $ε\sim 0.14$ for typical low-redshift galaxies. Our findings are consistent with models assuming suppressed stellar feedback due to the high gas density and the associated short free-fall time expected for massive halos at high redshift. △ Less

Submitted 4 March, 2024; originally announced March 2024.

Comments: 23 pages, 10 figures, submitted

Generalized Coronal Loop Scaling Laws and Their Implication for Turbulence in Solar Active Region Loops

Authors: Y. Dai, J. J. Xiang, M. D. Ding

Abstract: Recent coronal loop modeling has emphasized the importance of combining both Coulomb collisions and turbulent scattering to characterize field-aligned thermal conduction, which invokes a hybrid loop model. In this work we generalize the hybrid model by incorporating nonuniform heating and cross section that are both formulated by a power-law function of temperature. Based on the hybrid model solut… ▽ More Recent coronal loop modeling has emphasized the importance of combining both Coulomb collisions and turbulent scattering to characterize field-aligned thermal conduction, which invokes a hybrid loop model. In this work we generalize the hybrid model by incorporating nonuniform heating and cross section that are both formulated by a power-law function of temperature. Based on the hybrid model solutions, we construct scaling laws that relate loop-top temperature ($T_a$) and heating rate ($H_a$) to other loop parameters. It is found that the loop-top properties for turbulent loops are additionally power-law functions of turbulent mean free path ($λ_T$), with the functional forms varying from situation to situation that depends on the specification of the heating and/or areal parameters. More importantly, both a sufficiently footpoint-concentrated heating and a cross-sectional expansion with height can effectively weaken (strengthen) the negative (positive) power-law dependence of $T_a$ ($H_a$) on $λ_T$. The reason lies in a notable reduction of heat flux by footpoint heating and/or cross-sectional expansion in the turbulence-dominated coronal part, where turbulent scattering introduces a much weaker dependence of the conduction coefficient on temperature. In this region, therefore, the reduction of the heat flux predominately relies on a backward flattening of the temperature gradient. Through numerical modeling that incorporates more realistic conditions, this scenario is further consolidated. Our results have important implication for solar active region (AR) loops. With the factors of nonuniform heating and cross section taken into account, AR loops can bear relatively stronger turbulence while still keeping a physically reasonable temperature for nonflaring loops. △ Less

Submitted 4 March, 2024; originally announced March 2024.

Comments: 18 pages, 8 figures and 2 tables, accepted for publication in The Astrophysical Journal