Showing 1–50 of 891 results for author: Dai, Y

Serial and Parallel Two-Column Probing for Mixed-Integer Programming

Authors: Yongzheng Dai, Chen Chen

Abstract: Probing in mixed-integer programming (MIP) is a technique of temporarily fixing variables to discover implications that are useful to branch-and-cut solvers. Such fixing is typically performed one variable at a time -- this paper develops instead a two-column probing scheme that instead fixes a pair of variables per iteration. Although the scheme involves more work per iteration compared to the on… ▽ More Probing in mixed-integer programming (MIP) is a technique of temporarily fixing variables to discover implications that are useful to branch-and-cut solvers. Such fixing is typically performed one variable at a time -- this paper develops instead a two-column probing scheme that instead fixes a pair of variables per iteration. Although the scheme involves more work per iteration compared to the one-column approach, stronger implied bounds as well as more conflicts identified may compensate. Indeed, our prototype implementation was awarded first prize at the MIP Workshop 2024 Computational Competition on novel presolving approaches. This paper presents the aforementioned (serial) prototype and additionally develops an efficient parallelization, leveraging hardware acceleration to further improve overall solve times. Compared to serial two-column probing, our parallel version sacrifices some strength per-pair probed in exchange for greatly increasing the total number of such probings; computational experiments demonstrate its promise. △ Less

Submitted 29 August, 2024; originally announced August 2024.

Comments: 22 pages, 2 figures, 3 charts

MSC Class: 90C10

Adversarial Network Optimization under Bandit Feedback: Maximizing Utility in Non-Stationary Multi-Hop Networks

Authors: Yan Dai, Longbo Huang

Abstract: Stochastic Network Optimization (SNO) concerns scheduling in stochastic queueing systems. It has been widely studied in network theory. Classical SNO algorithms require network conditions to be stationary with time, which fails to capture the non-stationary components in many real-world scenarios. Many existing algorithms also assume knowledge of network conditions before decision, which rules out… ▽ More Stochastic Network Optimization (SNO) concerns scheduling in stochastic queueing systems. It has been widely studied in network theory. Classical SNO algorithms require network conditions to be stationary with time, which fails to capture the non-stationary components in many real-world scenarios. Many existing algorithms also assume knowledge of network conditions before decision, which rules out applications where unpredictability presents. Motivated by these issues, we consider Adversarial Network Optimization (ANO) under bandit feedback. Specifically, we consider the task of *i)* maximizing some unknown and time-varying utility function associated to scheduler's actions, where *ii)* the underlying network is a non-stationary multi-hop one whose conditions change arbitrarily with time, and *iii)* only bandit feedback (effect of actually deployed actions) is revealed after decisions. Our proposed `UMO2` algorithm ensures network stability and also matches the utility maximization performance of any "mildly varying" reference policy up to a polynomially decaying gap. To our knowledge, no previous ANO algorithm handled multi-hop networks or achieved utility guarantees under bandit feedback, whereas ours can do both. Technically, our method builds upon a novel integration of online learning into Lyapunov analyses: To handle complex inter-dependencies among queues in multi-hop networks, we propose meticulous techniques to balance online learning and Lyapunov arguments. To tackle the learning obstacles due to potentially unbounded queue sizes, we design a new online linear optimization algorithm that automatically adapts to loss magnitudes. To maximize utility, we propose a bandit convex optimization algorithm with novel queue-dependent learning rate scheduling that suites drastically varying queue lengths. Our new insights in online learning can be of independent interest. △ Less

Submitted 28 August, 2024; originally announced August 2024.

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Authors: Qianqian Xie, Dong Li, Mengxi Xiao, Zihao Jiang, Ruoyu Xiang, Xiao Zhang, Zhengyu Chen, Yueru He, Weiguang Han, Yuzhe Yang, Shunian Chen, Yifei Zhang, Lihang Shen, Daniel Kim, Zhiwei Liu, Zheheng Luo, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Zhiyuan Yao, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu , et al. (14 additional authors not shown)

Abstract: Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs}, a series of Financial LLMs. We begin with FinLLaMA, pre-trained on a 52 billion token financial corpus, incorporating text, table… ▽ More Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs}, a series of Financial LLMs. We begin with FinLLaMA, pre-trained on a 52 billion token financial corpus, incorporating text, tables, and time-series data to embed comprehensive financial knowledge. FinLLaMA is then instruction fine-tuned with 573K financial instructions, resulting in FinLLaMA-instruct, which enhances task performance. Finally, we present FinLLaVA, a multimodal LLM trained with 1.43M image-text instructions to handle complex financial data types. Extensive evaluations demonstrate FinLLaMA's superior performance over LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT in both zero-shot and few-shot settings across 19 and 4 datasets, respectively. FinLLaMA-instruct outperforms GPT-4 and other Financial LLMs on 15 datasets. FinLLaVA excels in understanding tables and charts across 4 multimodal tasks. Additionally, FinLLaMA achieves impressive Sharpe Ratios in trading simulations, highlighting its robust financial application capabilities. We will continually maintain and improve our models and benchmarks to support ongoing innovation in academia and industry. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: 33 pages, 13 figures

Target-Oriented Object Grasping via Multimodal Human Guidance

Authors: Pengwei Xie, Siang Chen, Dingchang Hu, Yixiang Dai, Kaiqin Yang, Guijin Wang

Abstract: In the context of human-robot interaction and collaboration scenarios, robotic grasping still encounters numerous challenges. Traditional grasp detection methods generally analyze the entire scene to predict grasps, leading to redundancy and inefficiency. In this work, we reconsider 6-DoF grasp detection from a target-referenced perspective and propose a Target-Oriented Grasp Network (TOGNet). TOG… ▽ More In the context of human-robot interaction and collaboration scenarios, robotic grasping still encounters numerous challenges. Traditional grasp detection methods generally analyze the entire scene to predict grasps, leading to redundancy and inefficiency. In this work, we reconsider 6-DoF grasp detection from a target-referenced perspective and propose a Target-Oriented Grasp Network (TOGNet). TOGNet specifically targets local, object-agnostic region patches to predict grasps more efficiently. It integrates seamlessly with multimodal human guidance, including language instructions, pointing gestures, and interactive clicks. Thus our system comprises two primary functional modules: a guidance module that identifies the target object in 3D space and TOGNet, which detects region-focal 6-DoF grasps around the target, facilitating subsequent motion planning. Through 50 target-grasping simulation experiments in cluttered scenes, our system achieves a success rate improvement of about 13.7%. In real-world experiments, we demonstrate that our method excels in various target-oriented grasping scenarios. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: Accepted by ECCV 2024 Workshop on Assistive Computer Vision and Robotics (ACVR 2024)

TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition

Authors: Tianwei Lin, Jiang Liu, Wenqiao Zhang, Zhaocheng Li, Yang Dai, Haoyuan Li, Zhelun Yu, Wanggui He, Juncheng Li, Hao Jiang, Siliang Tang, Yueting Zhuang

Abstract: While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multidimensional task scenarios. To address this issue, one straightforward solution is to introduce task-specific LoRA modules as domain experts, leveraging the modeling of multiple experts' capabilities and thus en… ▽ More While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multidimensional task scenarios. To address this issue, one straightforward solution is to introduce task-specific LoRA modules as domain experts, leveraging the modeling of multiple experts' capabilities and thus enhancing the general capability of multi-task learning. Despite promising, these additional components often add complexity to the training and inference process, contravening the efficient characterization of PEFT designed for. Considering this, we introduce an innovative PEFT method, TeamLoRA, consisting of a collaboration and competition module for experts, and thus achieving the right balance of effectiveness and efficiency: (i) For collaboration, a novel knowledge-sharing and -organizing mechanism is devised to appropriately reduce the scale of matrix operations, thereby boosting the training and inference speed. (ii) For competition, we propose leveraging a game-theoretic interaction mechanism for experts, encouraging experts to transfer their domain-specific knowledge while facing diverse downstream tasks, and thus enhancing the performance. By doing so, TeamLoRA elegantly connects the experts as a "Team" with internal collaboration and competition, enabling a faster and more accurate PEFT paradigm for multi-task learning. To validate the superiority of TeamLoRA, we curate a comprehensive multi-task evaluation(CME) benchmark to thoroughly assess the capability of multi-task learning. Experiments conducted on our CME and other benchmarks indicate the effectiveness and efficiency of TeamLoRA. Our project is available at https://github.com/Lin-Tianwei/TeamLoRA. △ Less

Submitted 19 August, 2024; originally announced August 2024.

Enhanced Barrier-Smoothing Technique for Bilevel Optimization with Nonsmooth Mappings

Authors: Mengwei Xu, Yu-Hong Dai, Xin-Wei Liu, Bo Wang

Abstract: Bilevel optimization problems, encountered in fields such as economics, engineering, and machine learning, pose significant computational challenges due to their hierarchical structure and constraints at both upper and lower levels. Traditional gradient-based methods are effective for unconstrained bilevel programs with unique lower level solutions, but struggle with constrained bilevel problems d… ▽ More Bilevel optimization problems, encountered in fields such as economics, engineering, and machine learning, pose significant computational challenges due to their hierarchical structure and constraints at both upper and lower levels. Traditional gradient-based methods are effective for unconstrained bilevel programs with unique lower level solutions, but struggle with constrained bilevel problems due to the nonsmoothness of lower level solution mappings. To overcome these challenges, this paper introduces the Enhanced Barrier-Smoothing Algorithm (EBSA), a novel approach that integrates gradient-based techniques with an augmented Lagrangian framework. EBSA utilizes innovative smoothing functions to approximate the primal-dual solution mapping of the lower level problem, and then transforms the bilevel problem into a sequence of smooth single-level problems. This approach not only addresses the nonsmoothness but also enhances convergence properties. Theoretical analysis demonstrates its superiority in achieving Clarke and, under certain conditions, Bouligand stationary points for bilevel problems. Both theoretical analysis and preliminary numerical experiments confirm the robustness and efficiency of EBSA. △ Less

Submitted 20 August, 2024; v1 submitted 18 August, 2024; originally announced August 2024.

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Authors: Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles , et al. (2 additional authors not shown)

Abstract: This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tas… ▽ More This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our pre-trained base model exhibits strong in-context learning capabilities and the instruction-tuned model demonstrates competitive performance among open-source LMMs with similar model sizes. In addition, we introduce a safety-tuned model with DPO, aiming to mitigate harmful behaviors such as hallucinations and improve safety. We open-source our models, curated large-scale datasets, and our fine-tuning codebase to facilitate further advancements in LMM research. Associated resources will be available on our project page above. △ Less

Submitted 28 August, 2024; v1 submitted 16 August, 2024; originally announced August 2024.

MSG-Chart: Multimodal Scene Graph for ChartQA

Authors: Yue Dai, Soyeon Caren Han, Wei Liu

Abstract: Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual… ▽ More Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart. This graph module can be easily integrated with different vision transformers as inductive bias. Our experiments demonstrate that incorporating the proposed graph module enhances the understanding of charts' elements' structure and semantics, thereby improving performance on publicly available benchmarks, ChartQA and OpenCQA. △ Less

Submitted 9 August, 2024; originally announced August 2024.

Comments: Accpeted by CIKM Short 2024

MMRole: A Comprehensive Framework for Developing and Evaluating Multimodal Role-Playing Agents

Authors: Yanqi Dai, Huanran Hu, Lei Wang, Shengjie Jin, Xu Chen, Zhiwu Lu

Abstract: Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research. However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities. To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comp… ▽ More Recently, Role-Playing Agents (RPAs) have garnered increasing attention for their potential to deliver emotional value and facilitate sociological research. However, existing studies are primarily confined to the textual modality, unable to simulate humans' multimodal perceptual capabilities. To bridge this gap, we introduce the concept of Multimodal Role-Playing Agents (MRPAs), and propose a comprehensive framework, MMRole, for their development and evaluation, which comprises a personalized multimodal dataset and a robust evaluation method. Specifically, we construct a large-scale, high-quality dataset, MMRole-Data, consisting of 85 characters, 11K images, and 14K single or multi-turn dialogues. Additionally, we present a robust evaluation method, MMRole-Eval, encompassing eight metrics across three dimensions, where a reward model is trained to score MRPAs with the constructed ground-truth data for comparison. Moreover, we develop the first specialized MRPA, MMRole-Agent. Extensive evaluation results demonstrate the improved performance of MMRole-Agent and highlight the primary challenges in developing MRPAs, emphasizing the need for enhanced multimodal understanding and role-playing consistency. The data, code, and models will be available at https://github.com/YanqiDai/MMRole. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Pick of the Bunch: Detecting Infrared Small Targets Beyond Hit-Miss Trade-Offs via Selective Rank-Aware Attention

Authors: Yimian Dai, Peiwen Pan, Yulei Qian, Yuxuan Li, Xiang Li, Jian Yang, Huan Wan

Abstract: Infrared small target detection faces the inherent challenge of precisely localizing dim targets amidst complex background clutter. Traditional approaches struggle to balance detection precision and false alarm rates. To break this dilemma, we propose SeRankDet, a deep network that achieves high accuracy beyond the conventional hit-miss trade-off, by following the ``Pick of the Bunch'' principle.… ▽ More Infrared small target detection faces the inherent challenge of precisely localizing dim targets amidst complex background clutter. Traditional approaches struggle to balance detection precision and false alarm rates. To break this dilemma, we propose SeRankDet, a deep network that achieves high accuracy beyond the conventional hit-miss trade-off, by following the ``Pick of the Bunch'' principle. At its core lies our Selective Rank-Aware Attention (SeRank) module, employing a non-linear Top-K selection process that preserves the most salient responses, preventing target signal dilution while maintaining constant complexity. Furthermore, we replace the static concatenation typical in U-Net structures with our Large Selective Feature Fusion (LSFF) module, a dynamic fusion strategy that empowers SeRankDet with adaptive feature integration, enhancing its ability to discriminate true targets from false alarms. The network's discernment is further refined by our Dilated Difference Convolution (DDC) module, which merges differential convolution aimed at amplifying subtle target characteristics with dilated convolution to expand the receptive field, thereby substantially improving target-background separation. Despite its lightweight architecture, the proposed SeRankDet sets new benchmarks in state-of-the-art performance across multiple public datasets. The code is available at https://github.com/GrokCV/SeRankDet. △ Less

Submitted 7 August, 2024; originally announced August 2024.

HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection

Authors: Yuxin Wang, Duanyu Feng, Yongfu Dai, Zhengyu Chen, Jimin Huang, Sophia Ananiadou, Qianqian Xie, Hao Wang

Abstract: Data serves as the fundamental foundation for advancing deep learning, particularly tabular data presented in a structured format, which is highly conducive to modeling. However, even in the era of LLM, obtaining tabular data from sensitive domains remains a challenge due to privacy or copyright concerns. Hence, exploring how to effectively use models like LLMs to generate realistic and privacy-pr… ▽ More Data serves as the fundamental foundation for advancing deep learning, particularly tabular data presented in a structured format, which is highly conducive to modeling. However, even in the era of LLM, obtaining tabular data from sensitive domains remains a challenge due to privacy or copyright concerns. Hence, exploring how to effectively use models like LLMs to generate realistic and privacy-preserving synthetic tabular data is urgent. In this paper, we take a step forward to explore LLMs for tabular data synthesis and privacy protection, by introducing a new framework HARMONIC for tabular data generation and evaluation. In the tabular data generation of our framework, unlike previous small-scale LLM-based methods that rely on continued pre-training, we explore the larger-scale LLMs with fine-tuning to generate tabular data and enhance privacy. Based on idea of the k-nearest neighbors algorithm, an instruction fine-tuning dataset is constructed to inspire LLMs to discover inter-row relationships. Then, with fine-tuning, LLMs are trained to remember the format and connections of the data rather than the data itself, which reduces the risk of privacy leakage. In the evaluation part of our framework, we develop specific privacy risk metrics DLT for LLM synthetic data generation, as well as performance evaluation metrics LLE for downstream LLM tasks. Our experiments find that this tabular data generation framework achieves equivalent performance to existing methods with better privacy, which also demonstrates our evaluation framework for the effectiveness of synthetic data and privacy risks in LLM scenarios. △ Less

Submitted 5 August, 2024; originally announced August 2024.

The most distant HI galaxies discovered by the 500 m dish FAST

Authors: Hongwei Xi, Bo Peng, Lister Staveley-Smith, Bi-Qing For, Bin Liu, Ru-Rong Chen, Lei Yu, Dejian Ding, Wei-Jian Guo, Hu Zou, Suijian Xue, Jing Wang, Thomas G. Brink, WeiKang Zheng, Alexei V. Filippenko, Yi Yang, Jianyan Wei, Y. Sophia Dai, Zi-Jian Li, Zizhao He, Chengzi Jiang, Alexei Moiseev, Sergey Kotov

Abstract: Neutral hydrogen (HI) is the primary component of the cool interstellar medium (ISM) and is the reservoir of fuel for star formation. Owing to the sensitivity of existing radio telescopes, our understanding of the evolution of the ISM in galaxies remains limited, as it is based on only a few hundred galaxies detected in HI beyond the local Universe. With the high sensitivity of the Five-hundred-me… ▽ More Neutral hydrogen (HI) is the primary component of the cool interstellar medium (ISM) and is the reservoir of fuel for star formation. Owing to the sensitivity of existing radio telescopes, our understanding of the evolution of the ISM in galaxies remains limited, as it is based on only a few hundred galaxies detected in HI beyond the local Universe. With the high sensitivity of the Five-hundred-meter Aperture Spherical radio Telescope (FAST), we carried out a blind HI search, the FAST Ultra-Deep Survey (FUDS), which extends to redshifts up to 0.42 and a sensitivity of 50 $\rm μJy \cdot beam^{-1}$. Here, we report the first discovery of six galaxies in HI at $z>0.38$. For these galaxies, the FAST angular resolution of $\sim\,4'$ corresponds to a mean linear size of $\sim1.3\,h_{70}^{-1}\,$Mpc. These galaxies are among the most distant HI emission detections known, with one having the most massive HI content ($10^{10.93 \pm 0.04}~h_{70}^{-2}\, \rm M_\odot$). Using recent data from the DESI survey, and new observations with the Hale, BTA, and Keck telescopes, optical counterparts are detected for all galaxies within the 3-$σ$ positional uncertainty ($0.5\,h_{70}^{-1}\,$Mpc) and $\rm 200\,km \cdot s^{-1}$ in recession velocity. Assuming that the dominant source of HI is the identified optical counterpart, we find an evidence of evolution in the HI content of galaxies over the last 4.2 Gyr. Our new high-redshift HI galaxy sample provides the opportunity to better investigate the evolution of cool gas in galaxies. A larger sample size in the future will allow us to refine our knowledge of the formation and evolution of galaxies. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Comments: 14 pages, 6 figures, 3 tables

Journal ref: ApJL, 966(2024), L36

Deep Learning-Based Longitudinal Prediction of Childhood Myopia Progression Using Fundus Image Sequences and Baseline Refraction Data

Authors: Mengtian Kang, Yansong Hu, Shuo Gao, Yuanyuan Liu, Hongbei Meng, Xuemeng Li, Xuhang Chen, Hubin Zhao, Jing Fu, Guohua Hu, Wei Wang, Yanning Dai, Arokia Nathan, Peter Smielewski, Ningli Wang, Shiming Li

Abstract: Childhood myopia constitutes a significant global health concern. It exhibits an escalating prevalence and has the potential to evolve into severe, irreversible conditions that detrimentally impact familial well-being and create substantial economic costs. Contemporary research underscores the importance of precisely predicting myopia progression to enable timely and effective interventions, there… ▽ More Childhood myopia constitutes a significant global health concern. It exhibits an escalating prevalence and has the potential to evolve into severe, irreversible conditions that detrimentally impact familial well-being and create substantial economic costs. Contemporary research underscores the importance of precisely predicting myopia progression to enable timely and effective interventions, thereby averting severe visual impairment in children. Such predictions predominantly rely on subjective clinical assessments, which are inherently biased and resource-intensive, thus hindering their widespread application. In this study, we introduce a novel, high-accuracy method for quantitatively predicting the myopic trajectory and myopia risk in children using only fundus images and baseline refraction data. This approach was validated through a six-year longitudinal study of 3,408 children in Henan, utilizing 16,211 fundus images and corresponding refractive data. Our method based on deep learning demonstrated predictive accuracy with an error margin of 0.311D per year and AUC scores of 0.944 and 0.995 for forecasting the risks of developing myopia and high myopia, respectively. These findings confirm the utility of our model in supporting early intervention strategies and in significantly reducing healthcare costs, particularly by obviating the need for additional metadata and repeated consultations. Furthermore, our method was designed to rely only on fundus images and refractive error data, without the need for meta data or multiple inquiries from doctors, strongly reducing the associated medical costs and facilitating large-scale screening. Our model can even provide good predictions based on only a single time measurement. Consequently, the proposed method is an important means to reduce medical inequities caused by economic disparities. △ Less

Submitted 31 July, 2024; originally announced July 2024.

Fine-grained Metrics for Point Cloud Semantic Segmentation

Authors: Zhuheng Lu, Ting Wu, Yuewei Dai, Weiqing Li, Zhiyong Su

Abstract: Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others. Because of this, the majority of categories and large objects are favored in the existing evaluation metrics. This paper suggests fine-grained mIoU a… ▽ More Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others. Because of this, the majority of categories and large objects are favored in the existing evaluation metrics. This paper suggests fine-grained mIoU and mAcc for a more thorough assessment of point cloud segmentation algorithms in order to address these issues. Richer statistical information is provided for models and datasets by these fine-grained metrics, which also lessen the bias of current semantic segmentation metrics towards large objects. The proposed metrics are used to train and assess various semantic segmentation algorithms on three distinct indoor and outdoor semantic segmentation datasets. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Comments: PRCV 2024

Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming

Authors: Yate Ge, Yi Dai, Run Shan, Kechun Li, Yuanda Hu, Xiaohua Sun

Abstract: End-user development allows everyday users to tailor service robots or applications to their needs. One user-friendly approach is natural language programming. However, it encounters challenges such as an expansive user expression space and limited support for debugging and editing, which restrict its application in end-user programming. The emergence of large language models (LLMs) offers promisi… ▽ More End-user development allows everyday users to tailor service robots or applications to their needs. One user-friendly approach is natural language programming. However, it encounters challenges such as an expansive user expression space and limited support for debugging and editing, which restrict its application in end-user programming. The emergence of large language models (LLMs) offers promising avenues for the translation and interpretation between human language instructions and the code executed by robots, but their application in end-user programming systems requires further study. We introduce Cocobo, a natural language programming system with interactive diagrams powered by LLMs. Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations. Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Comments: This is the preprint version of a paper accepted for presentation at the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2024

Background Semantics Matter: Cross-Task Feature Exchange Network for Clustered Infrared Small Target Detection With Sky-Annotated Dataset

Authors: Yimian Dai, Mengxuan Xiao, Yiming Zhu, Huan Wang, Kehua Guo, Jian Yang

Abstract: Infrared small target detection poses unique challenges due to the scarcity of intrinsic target features and the abundance of similar background distractors. We argue that background semantics play a pivotal role in distinguishing visually similar objects for this task. To address this, we introduce a new task -- clustered infrared small target detection, and present DenseSIRST, a novel benchmark… ▽ More Infrared small target detection poses unique challenges due to the scarcity of intrinsic target features and the abundance of similar background distractors. We argue that background semantics play a pivotal role in distinguishing visually similar objects for this task. To address this, we introduce a new task -- clustered infrared small target detection, and present DenseSIRST, a novel benchmark dataset that provides per-pixel semantic annotations for background regions, enabling the transition from sparse to dense target detection. Leveraging this dataset, we propose the Background-Aware Feature Exchange Network (BAFE-Net), which transforms the detection paradigm from a single task focused on the foreground to a multi-task architecture that jointly performs target detection and background semantic segmentation. BAFE-Net introduces a cross-task feature hard-exchange mechanism to embed target and background semantics between the two tasks. Furthermore, we propose the Background-Aware Gaussian Copy-Paste (BAG-CP) method, which selectively pastes small targets into sky regions during training, avoiding the creation of false alarm targets in complex non-sky backgrounds. Extensive experiments validate the effectiveness of BAG-CP and BAFE-Net in improving target detection accuracy while reducing false alarms. The DenseSIRST dataset, code, and trained models are available at https://github.com/GrokCV/BAFE-Net. △ Less

Submitted 29 July, 2024; originally announced July 2024.

A Low-Frequency Vibration Experimental Platform for University Physics Experiment Designed by LabVIEW

Authors: Yangjie Dai, Leijian Wang, Wenbin Wu, Aiping Chen, Dawei Gu

Abstract: Virtual instrument technology has been increasingly used in university physics experiment teaching. An experimental platform is specifically constructed for studying low-frequency vibrations in university physics, which is based on a computer and its internal sound card, along with a program developed in LabVIEW programming environment to perform control and measurement on our experimental platfor… ▽ More Virtual instrument technology has been increasingly used in university physics experiment teaching. An experimental platform is specifically constructed for studying low-frequency vibrations in university physics, which is based on a computer and its internal sound card, along with a program developed in LabVIEW programming environment to perform control and measurement on our experimental platform. The proposed platform effectively replaces the conventional signal generator and oscilloscope traditionally used in such experiments by integrating virtual instruments and essential experimental equipment. The platform offers various functionalities, such as synchronous transmission and reception of low-frequency signals, frequency measurement, dynamic frequency sweep measurement, and measurement using the three-point approximation method. The proposed platform has been successfully applied in experiments involving forced vibration, resonance of tuning forks, and dynamic measurement of Young's modulus. Unlike conventional low-frequency vibration experiments, the proposed experimental platform optimizes efficiency, reduces costs, and offers opportunities for enhancing the instructional content of experiments. Furthermore, the incorporation of state-of-the-art computer technology enhances students' engagement and enthusiasm for learning. △ Less

Submitted 27 July, 2024; originally announced July 2024.

Comments: 13 pages, 8 figures, 2 supplementary files

THEA-Code: an Autoencoder-Based IDS-correcting Code for DNA Storage

Authors: Alan J. X. Guo, Mengyi Wei, Yufan Dai, Yali Wei, Pengchen Zhang

Abstract: The insertion, deletion, substitution (IDS) correcting code has garnered increased attention due to significant advancements in DNA storage that emerged recently. Despite this, the pursuit of optimal solutions in IDS-correcting codes remains an open challenge, drawing interest from both theoretical and engineering perspectives. This work introduces a pioneering approach named THEA-code. The propos… ▽ More The insertion, deletion, substitution (IDS) correcting code has garnered increased attention due to significant advancements in DNA storage that emerged recently. Despite this, the pursuit of optimal solutions in IDS-correcting codes remains an open challenge, drawing interest from both theoretical and engineering perspectives. This work introduces a pioneering approach named THEA-code. The proposed method follows a heuristic idea of employing an end-to-end autoencoder for the integrated encoding and decoding processes. To address the challenges associated with deploying an autoencoder as an IDS-correcting code, we propose innovative techniques, including the differentiable IDS channel, the entropy constraint on the codeword, and the auxiliary reconstruction of the source sequence. These strategies contribute to the successful convergence of the autoencoder, resulting in a deep learning-based IDS-correcting code with commendable performance. Notably, THEA-Code represents the first instance of a deep learning-based code that is independent of conventional coding frameworks in the IDS-correcting domain. Comprehensive experiments, including an ablation study, provide a detailed analysis and affirm the effectiveness of THEA-Code. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Sparse Prior Is Not All You Need: When Differential Directionality Meets Saliency Coherence for Infrared Small Target Detection

Authors: Fei Zhou, Maixia Fu, Yulei Qian, Jian Yang, Yimian Dai

Abstract: Infrared small target detection is crucial for the efficacy of infrared search and tracking systems. Current tensor decomposition methods emphasize representing small targets with sparsity but struggle to separate targets from complex backgrounds due to insufficient use of intrinsic directional information and reduced target visibility during decomposition. To address these challenges, this study… ▽ More Infrared small target detection is crucial for the efficacy of infrared search and tracking systems. Current tensor decomposition methods emphasize representing small targets with sparsity but struggle to separate targets from complex backgrounds due to insufficient use of intrinsic directional information and reduced target visibility during decomposition. To address these challenges, this study introduces a Sparse Differential Directionality prior (SDD) framework. SDD leverages the distinct directional characteristics of targets to differentiate them from the background, applying mixed sparse constraints on the differential directional images and continuity difference matrix of the temporal component, both derived from Tucker decomposition. We further enhance target detectability with a saliency coherence strategy that intensifies target contrast against the background during hierarchical decomposition. A Proximal Alternating Minimization-based (PAM) algorithm efficiently solves our proposed model. Experimental results on several real-world datasets validate our method's effectiveness, outperforming ten state-of-the-art methods in target detection and clutter suppression. Our code is available at https://github.com/GrokCV/SDD. △ Less

Submitted 22 July, 2024; originally announced July 2024.

Comments: Submitted to IEEE TIM, Minor Revision

OE-BevSeg: An Object Informed and Environment Aware Multimodal Framework for Bird's-eye-view Vehicle Semantic Segmentation

Authors: Jian Sun, Yuqi Dai, Chi-Man Vong, Qing Xu, Shengbo Eben Li, Jianqiang Wang, Lei He, Keqiang Li

Abstract: Bird's-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. It realizes ego-vehicle surrounding environment perception by projecting 2D multi-view images into 3D world space. Recently, BEV segmentation has made notable progress, attributed to better view transformation modules, larger image encoders, or more temporal information. However, there are still two issu… ▽ More Bird's-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. It realizes ego-vehicle surrounding environment perception by projecting 2D multi-view images into 3D world space. Recently, BEV segmentation has made notable progress, attributed to better view transformation modules, larger image encoders, or more temporal information. However, there are still two issues: 1) a lack of effective understanding and enhancement of BEV space features, particularly in accurately capturing long-distance environmental features and 2) recognizing fine details of target objects. To address these issues, we propose OE-BevSeg, an end-to-end multimodal framework that enhances BEV segmentation performance through global environment-aware perception and local target object enhancement. OE-BevSeg employs an environment-aware BEV compressor. Based on prior knowledge about the main composition of the BEV surrounding environment varying with the increase of distance intervals, long-sequence global modeling is utilized to improve the model's understanding and perception of the environment. From the perspective of enriching target object information in segmentation results, we introduce the center-informed object enhancement module, using centerness information to supervise and guide the segmentation head, thereby enhancing segmentation performance from a local enhancement perspective. Additionally, we designed a multimodal fusion branch that integrates multi-view RGB image features with radar/LiDAR features, achieving significant performance improvements. Extensive experiments show that, whether in camera-only or multimodal fusion BEV segmentation tasks, our approach achieves state-of-the-art results by a large margin on the nuScenes dataset for vehicle segmentation, demonstrating superior applicability in the field of autonomous driving. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration

Authors: Tianyu Wang, Sheng Li, Bingyao Li, Yue Dai, Ao Li, Geng Yuan, Yufei Ding, Youtao Zhang, Xulong Tang

Abstract: Continuous learning (CL) has emerged as one of the most popular deep learning paradigms deployed in modern cloud GPUs. Specifically, CL has the capability to continuously update the model parameters (through model retraining) and use the updated model (if available) to serve overtime arriving inference requests. It is generally beneficial to co-locate the retraining and inference together to enabl… ▽ More Continuous learning (CL) has emerged as one of the most popular deep learning paradigms deployed in modern cloud GPUs. Specifically, CL has the capability to continuously update the model parameters (through model retraining) and use the updated model (if available) to serve overtime arriving inference requests. It is generally beneficial to co-locate the retraining and inference together to enable timely model updates and avoid model transfer overheads. This brings the need for GPU sharing among retraining and inferences. Meanwhile, multiple CL workloads can share the modern GPUs in the cloud, leading to multi-tenancy execution. In this paper, we observe that prior GPU-sharing techniques are not optimized for multi-tenancy CL workloads. Specifically, they do not coherently consider the accuracy of the retraining model and the inference service level objective (SLO) attainment. Moreover, they cannot accommodate the overtime dynamics (e.g., inference arrival intensity) in CL execution. In this paper, we propose MIGRator, a novel GPU reconfiguration runtime that dynamically performs GPU reconfiguration for multi-tenancy CL workloads. MIGRator is based on the recent NVIDIA multi-instance GPU (MIG) to mitigate resource contention and formulates the reconfiguration optimization into Integer Linear Programming (ILP) to dynamically identify, reconfigure, and allocate the GPU instances. MIGRator leverages the "Goodput" metric in the ILP objective function to consider both inference SLO attainment and model accuracy in the reconfiguration exploration. We evaluate MIGRator using representative multi-tenancy CL workloads. The results show our approach outperforms the state-of-the-art GPU sharing techniques (i.e., Ekya, Astraea, and PARIS) by 17\%, 21\%, and 20\%, respectively. △ Less

Submitted 17 July, 2024; originally announced July 2024.

LTRL: Boosting Long-tail Recognition via Reflective Learning

Authors: Qihao Zhao, Yalun Dai, Shen Lin, Wei Hu, Fan Zhang, Jun Liu

Abstract: In real-world scenarios, where knowledge distributions exhibit long-tail. Humans manage to master knowledge uniformly across imbalanced distributions, a feat attributed to their diligent practices of reviewing, summarizing, and correcting errors. Motivated by this learning process, we propose a novel learning paradigm, called reflecting learning, in handling long-tail recognition. Our method integ… ▽ More In real-world scenarios, where knowledge distributions exhibit long-tail. Humans manage to master knowledge uniformly across imbalanced distributions, a feat attributed to their diligent practices of reviewing, summarizing, and correcting errors. Motivated by this learning process, we propose a novel learning paradigm, called reflecting learning, in handling long-tail recognition. Our method integrates three processes for reviewing past predictions during training, summarizing and leveraging the feature relation across classes, and correcting gradient conflict for loss functions. These designs are lightweight enough to plug and play with existing long-tail learning methods, achieving state-of-the-art performance in popular long-tail visual benchmarks. The experimental results highlight the great potential of reflecting learning in dealing with long-tail recognition. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Comments: ECCV2024

Hierarchical and Decoupled BEV Perception Learning Framework for Autonomous Driving

Authors: Yuqi Dai, Jian Sun, Shengbo Eben Li, Qing Xu, Jianqiang Wang, Lei He, Keqiang Li

Abstract: Perception is essential for autonomous driving system. Recent approaches based on Bird's-eye-view (BEV) and deep learning have made significant progress. However, there exists challenging issues including lengthy development cycles, poor reusability, and complex sensor setups in perception algorithm development process. To tackle the above challenges, this paper proposes a novel hierarchical BEV p… ▽ More Perception is essential for autonomous driving system. Recent approaches based on Bird's-eye-view (BEV) and deep learning have made significant progress. However, there exists challenging issues including lengthy development cycles, poor reusability, and complex sensor setups in perception algorithm development process. To tackle the above challenges, this paper proposes a novel hierarchical BEV perception paradigm, aiming to provide a library of fundamental perception modules and user-friendly graphical interface, enabling swift construction of customized models. We conduct the Pretrain-Finetune strategy to effectively utilize large scale public datasets and streamline development processes. Moreover, we present a Multi-Module Learning (MML) approach, enhancing performance through synergistic and iterative training of multiple models. Extensive experimental results on the Nuscenes dataset demonstrate that our approach renders significant improvement over the traditional training scheme. △ Less

Submitted 25 July, 2024; v1 submitted 17 July, 2024; originally announced July 2024.

Small exciton effective mass in QL Bi2Se2Te: A material platform towards high-temperature excitonic condensate

Authors: Yuanyuan Wang, Ying Dai, Baibiao Huang, Yee Sin Ang, Wei Wei

Abstract: Using first-principles simulations combined with many-body calculations, we show that two-dimensional free-standing quintuple-layer Bi2Se2Te is an inversion symmetric monolayer expected to achieve spatially indirect exciton with large exciton radius, small exciton effective mass and long exciton lifetime. Such system is theoretically predicted to be a promising platform for realizing excitonic Bos… ▽ More Using first-principles simulations combined with many-body calculations, we show that two-dimensional free-standing quintuple-layer Bi2Se2Te is an inversion symmetric monolayer expected to achieve spatially indirect exciton with large exciton radius, small exciton effective mass and long exciton lifetime. Such system is theoretically predicted to be a promising platform for realizing excitonic Bose-Einstein condensation and superfluid due to its high phase transition temperatures of ~257 K and ~64.25 K for the BEC and excitonic superfluid, respectively. The importance of spin-orbit coupling is revealed, and the angular momentum selection rules for photon absorption are discussed. This finding suggests the potential of QL Bi2Se2Te monolayer with exotic bosonic bound states provides as a tantalizing high-temperature platform to probe excitonic physics. △ Less

Submitted 16 July, 2024; originally announced July 2024.

Reshaping the Online Data Buffering and Organizing Mechanism for Continual Test-Time Adaptation

Authors: Zhilin Zhu, Xiaopeng Hong, Zhiheng Ma, Weijun Zhuang, Yaohui Ma, Yong Dai, Yaowei Wang

Abstract: Continual Test-Time Adaptation (CTTA) involves adapting a pre-trained source model to continually changing unsupervised target domains. In this paper, we systematically analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting under continual domain shifts. To address these challenges, we reshape the online data bu… ▽ More Continual Test-Time Adaptation (CTTA) involves adapting a pre-trained source model to continually changing unsupervised target domains. In this paper, we systematically analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting under continual domain shifts. To address these challenges, we reshape the online data buffering and organizing mechanism for CTTA. We propose an uncertainty-aware buffering approach to identify and aggregate significant samples with high certainty from the unsupervised, single-pass data stream. Based on this, we propose a graph-based class relation preservation constraint to overcome catastrophic forgetting. Furthermore, a pseudo-target replay objective is used to mitigate error accumulation. Extensive experiments demonstrate the superiority of our method in both segmentation and classification CTTA tasks. Code is available at https://github.com/z1358/OBAO. △ Less

Submitted 18 July, 2024; v1 submitted 12 July, 2024; originally announced July 2024.

Comments: This is the preprint version of our paper and supplemental material to appear in ECCV 2024

Coupling multi-space topologies in 2D ferromagnetic lattice

Authors: Zhonglin He, Wenhui Du, Kaiying Dou, Ying Dai, Baibiao Huang, Yandong Ma

Abstract: Topology can manifest topological magnetism (e.g., skyrmion and bimeron) in real space and quantum anomalous Hall (QAH) state in momentum space, which have changed the modern conceptions of matter phase. While the topologies in different spaces are widely studied separately, their coexistence and coupling in single phase is seldomly explored. Here, we report a novel phenomenon that arises from the… ▽ More Topology can manifest topological magnetism (e.g., skyrmion and bimeron) in real space and quantum anomalous Hall (QAH) state in momentum space, which have changed the modern conceptions of matter phase. While the topologies in different spaces are widely studied separately, their coexistence and coupling in single phase is seldomly explored. Here, we report a novel phenomenon that arises from the interaction of topological magnetism and band topology, the multi-space topology, in 2D ferromagnetic lattice. Based on continuum theory and tight-binding model, we reveal that the interconnection between skyrmion/bimeron and QAH state generates distinctive localized chiral bound states (CBSs). With moderating topological magnetism through magnetic field, the multi-space topologies accompanied with different CBSs can be reversed, facilitating the coupling of multi-space topologies. By performing firstprinciples and atomic spin model simulations, we further demonstrate such multi-space topologies and their coupling in monolayer Cr2NSb. These results represent an important step towards the development of multispace topological phenomena in 2D lattice. △ Less

Submitted 12 July, 2024; originally announced July 2024.

Invisible Optical Adversarial Stripes on Traffic Sign against Autonomous Vehicles

Authors: Dongfang Guo, Yuting Wu, Yimin Dai, Pengfei Zhou, Xin Lou, Rui Tan

Abstract: Camera-based computer vision is essential to autonomous vehicle's perception. This paper presents an attack that uses light-emitting diodes and exploits the camera's rolling shutter effect to create adversarial stripes in the captured images to mislead traffic sign recognition. The attack is stealthy because the stripes on the traffic sign are invisible to human. For the attack to be threatening,… ▽ More Camera-based computer vision is essential to autonomous vehicle's perception. This paper presents an attack that uses light-emitting diodes and exploits the camera's rolling shutter effect to create adversarial stripes in the captured images to mislead traffic sign recognition. The attack is stealthy because the stripes on the traffic sign are invisible to human. For the attack to be threatening, the recognition results need to be stable over consecutive image frames. To achieve this, we design and implement GhostStripe, an attack system that controls the timing of the modulated light emission to adapt to camera operations and victim vehicle movements. Evaluated on real testbeds, GhostStripe can stably spoof the traffic sign recognition results for up to 94\% of frames to a wrong class when the victim vehicle passes the road section. In reality, such attack effect may fool victim vehicles into life-threatening incidents. We discuss the countermeasures at the levels of camera sensor, perception model, and autonomous driving system. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Journal ref: In Proceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services (MobiSys 2024), 534-546

Plasmonic Vortices Host Magnetoelectric Interactions

Authors: Atreyie Ghosh, Sena Yang, Yanan Dai, W. Vincent Liu, Hrvoje Petek

Abstract: The vector cross product and pseudoscalar dot products of electric (E) and magnetic (H) fields are separately finite in vacuum transverse electric and magnetic (TEM) plane waves, and angular momentum structured light. Current theories of interactions beyond the standard model of particle physics invoke non-zero dot(E,H) as the source term in the axion law that describes interactions with the cosmo… ▽ More The vector cross product and pseudoscalar dot products of electric (E) and magnetic (H) fields are separately finite in vacuum transverse electric and magnetic (TEM) plane waves, and angular momentum structured light. Current theories of interactions beyond the standard model of particle physics invoke non-zero dot(E,H) as the source term in the axion law that describes interactions with the cosmological dark matter axion particles outside of the quartet of Maxwells equations. The non-zero dot(E,H) also drives relativistic spin-charge magnetoelectric excitations of axion quasiparticles at a distinctively higher condensed matter scale in magnetic and topological materials. Yet, how to drive coherent dot(E,H) responses is unknown, and provides motivation to examine the field polarizations in structured light on a deep sub-diffraction limited spatial scale and sub-optical cycle temporal scale by ultrafast nonlinear photoemission electron microscopy. By analytical theory and ultrafast coherent photoemission electron microscopy, we image dot(E,H) fields in surface plasmon polariton vortex cores at subwavelength scales, where we find that the magnetoelectric relative to the dipole density is intensified on a ~10 nm diameter scale as a universal property of plasmonic vortex fields. The generation and nanoscale localization of dot(E,H) fields introduces the magnetoelectric symmetry class, having the parity and time reversal broken, but the joint parity-time reversal symmetry preserved. The ability to image the optical fields of plasmonic vortex cores opens the research of ultrafast microscopy of magnetoelectric responses and interactions with axion quasiparticles in solid state materials. △ Less

Submitted 9 July, 2024; originally announced July 2024.

Comments: 5 Figures

Journal ref: Physical Review RESEARCH 6, 013163 (2024)

A Broadband Algorithm for Adiabatic Mode Evolution and its Application on Polarization Splitter-Rotator on LNOI Platform

Authors: Geng Chen, Chijun Li, Xuanhao Wang, An Pan, Junjie Wei, Yuankang Huang, Siyu Lu, Yiqi Dai, Xiangyu Meng, Cheng Zeng, Jinsong Xia

Abstract: Adiabatic mode evolution waveguides (AMEWs) are widely utilized in integrated photonics, including tapered waveguides, edge couplers, mode converters, splitters, etc. An analytical theory and a novel AMEW design algorithm are developed to create shortcuts to adiabaticity (STA). This new algorithm is effective in shortening the total length of the AMEW while maintaining the desired wavelength range… ▽ More Adiabatic mode evolution waveguides (AMEWs) are widely utilized in integrated photonics, including tapered waveguides, edge couplers, mode converters, splitters, etc. An analytical theory and a novel AMEW design algorithm are developed to create shortcuts to adiabaticity (STA). This new algorithm is effective in shortening the total length of the AMEW while maintaining the desired wavelength range. Moreover, this analytical algorithm requires much fewer computing resources than traditional numerical algorithms. With the new algorithm, we demonstrate a broadband and highly efficient polarization splitter-rotator (PSR) on a lithium-niobate-on-insulator (LNOI) platform with an LN thickness of 500 nm. According to our simulation, the length of the PSR is shortened by 3.5 times compared to the linear design. The fabricated PSR, with a total length of 2 mm, exhibits an insertion loss (IL) of 0.8 dB and a polarization extinction ratio (ER) of 12.2 dB over a wavelength range exceeding 76 nm. △ Less

Submitted 22 July, 2024; v1 submitted 6 July, 2024; originally announced July 2024.

Comments: 16 pages, 6 figures, 2 tables

NOEMA formIng Cluster survEy (NICE): Characterizing eight massive galaxy groups at $1.5 < z < 4$ in the COSMOS field

Authors: Nikolaj B. Sillassen, Shuowen Jin, Georgios E. Magdis, Emanuele Daddi, Tao Wang, Shiying Lu, Hanwen Sun, Vinod Arumugam, Daizhong Liu, Malte Brinch, Chiara D'Eugenio, Raphael Gobat, Carlos Gómez-Guijarro, Michael Rich, Eva Schinnerer, Veronica Strazzullo, Qinghua Tan, Francesco Valentino, Yijun Wang, Mengyuan Xiao, Luwenjia Zhou, David Blánquez-Sesé, Zheng Cai, Yanmei Chen, Laure Ciesla , et al. (19 additional authors not shown)

Abstract: The NOEMA formIng Cluster survEy (NICE) is a large program targeting 69 massive galaxy group candidates at $z>2$ in six deep fields. We report spectroscopic confirmation of eight groups at $1.65\leq z\leq3.61$ in COSMOS. Homogeneously selected as significant overdensities of red IRAC sources with red Herschel colors, four groups are confirmed by CO and [CI] with NOEMA 3mm observations, three are c… ▽ More The NOEMA formIng Cluster survEy (NICE) is a large program targeting 69 massive galaxy group candidates at $z>2$ in six deep fields. We report spectroscopic confirmation of eight groups at $1.65\leq z\leq3.61$ in COSMOS. Homogeneously selected as significant overdensities of red IRAC sources with red Herschel colors, four groups are confirmed by CO and [CI] with NOEMA 3mm observations, three are confirmed with ALMA, and one is confirmed by H$α$ from Subaru/FMOS. We constructed the integrated FIR SEDs for the eight groups, obtaining total IR SFR $=260-1300~{\rm M_\odot}$~yr$^{-1}$. We adopted six methods to estimate the dark matter masses, including stellar mass to halo mass relations, overdensity with galaxy bias, and NFW profile fitting to radial stellar mass density. We found the radial stellar mass density are consistent with a NFW profile, supporting that they are collapsed structures hosted by a single dark matter halo. The best halo mass estimates are $\log(M_{\rm h}/{\rm M_\odot})=12.8-13.7$ with uncertainty of 0.3 dex. From halo mass estimates, we derive baryonic accretion rate ${\rm BAR}=(1-8)\times10^{3}\,{\rm M_{\odot}/yr}$ for this sample. We find a quasi-linear correlation between the integrated SFR/BAR and the theoretical halo mass limit for cold streams, $M_{\rm stream}/M_{\rm h}$, with ${\rm SFR/BAR}=10^{-0.46\pm0.22}\left({M_{\rm stream}/M_{\rm h}}\right)^{0.71\pm0.16}$ with a scatter of $0.40\,{\rm dex}$. Further, we compare halo masses and stellar masses with simulations, and find all structures are consistent with being progenitors of $M_{\rm h}(z=0)>10^{14}\,{\rm M_{\odot}}$ galaxy clusters, and the most massive central galaxies have stellar masses consistent with brightest cluster galaxies (BCGs) progenitors in the TNG300 simulation. The results strongly suggest these structures are forming massive galaxy clusters via baryonic and dark matter accretion. △ Less

Submitted 5 July, 2024; v1 submitted 3 July, 2024; originally announced July 2024.

Comments: 44 pages (27pp appendix), 32 figures, 18 tables, accepted for publication in A&A

Enhancing Stability for Large Models Training in Constrained Bandwidth Networks

Authors: Yun Dai, Tejas Dharamsi, Byron Hsu, Tao Song, Hamed Firooz

Abstract: Training extremely large language models with billions of parameters is a computationally intensive task that pushes the limits of current data parallel training systems. While techniques like ZeRO++ have enabled efficient distributed training of such giant models on inexpensive low-bandwidth clusters, they can suffer from convergence issues due to potential race conditions in the hierarchical par… ▽ More Training extremely large language models with billions of parameters is a computationally intensive task that pushes the limits of current data parallel training systems. While techniques like ZeRO++ have enabled efficient distributed training of such giant models on inexpensive low-bandwidth clusters, they can suffer from convergence issues due to potential race conditions in the hierarchical partitioning (hpZ) scheme employed to reduce cross-machine communication. In this work, we first show how these race conditions cause instability when training models with billions of parameters. We then propose a modification to the partitioning algorithm that addresses these convergence challenges while maintaining competitive training efficiency. Empirical evaluation on training the multi-billion parameters Falcon Models and Llama-2 models demonstrates the updated algorithm's ability to achieve reliable convergence on these massive models, where stock ZeRO++ hpZ fails to converge. The updated algorithm enables robust training of larger models with 98\% throughput and model training speed improvement without sacrificing the quality of convergence. △ Less

Submitted 31 July, 2024; v1 submitted 27 June, 2024; originally announced July 2024.

Timing and Scintillation Studies of Pulsars in Globular Cluster M3 (NGC 5272) with FAST

Authors: Baoda Li, Li-yun Zhang, Jumei Yao, Dejiang Yin, Ralph P. Eatough, Minghui Li, Yifeng Li, Yujie Lian, Yu Pan, Yinfeng Dai, Yaowei Li, Xingnan Zhang, Tianhao Su, Yuxiao Wu, Tong Liu, Kuo Liu, Lin Wang, Lei Qian, Zhichen Pan

Abstract: We present the phase-connected timing solutions of all the five pulsars in globular cluster (GC) M3 (NGC 5272), namely PSRs M3A to F (PSRs J1342+2822A to F), with the exception of PSR M3C, from FAST archival data. In these timing solutions, those of PSRs M3E, and F are obtained for the first time. We find that PSRs M3E and F have low mass companions, and are in circular orbits with periods of 7.1… ▽ More We present the phase-connected timing solutions of all the five pulsars in globular cluster (GC) M3 (NGC 5272), namely PSRs M3A to F (PSRs J1342+2822A to F), with the exception of PSR M3C, from FAST archival data. In these timing solutions, those of PSRs M3E, and F are obtained for the first time. We find that PSRs M3E and F have low mass companions, and are in circular orbits with periods of 7.1 and 3.0 days, respectively. For PSR M3C, we have not detected it in all the 41 observations. We found no X-ray counterparts for these pulsars in archival Chandra images in the band of 0.2-20 keV. We noticed that the pulsars in M3 seem to be native. From the Auto-Correlation Function (ACF) analysis of the M3A's and M3B's dynamic spectra, the scintillation timescale ranges from $7.0\pm0.3$ min to $60.0\pm0.6$ min, and the scintillation bandwidth ranges from $4.6\pm0.2$ MHz to $57.1\pm1.1$ MHz. The measured scintillation bandwidths from the dynamic spectra indicate strong scintillation, and the scattering medium is anisotropic. From the secondary spectra, we captured a scintillation arc only for PSR M3B with a curvature of $649\pm23 {\rm m}^{-1} {\rm mHz}^{-2}$. △ Less

Submitted 26 June, 2024; originally announced June 2024.

Comments: 14 pages, 4 figures, accepted for publication in The Astrophysical Journal

MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling

Authors: Yuxin Dai, Qi Wang, Jingsen Zhu, Dianbing Xi, Yuchi Huo, Chen Qian, Ying He

Abstract: We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based… ▽ More We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model that utilizes multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method effectively estimates indirect illumination, including self-shadowing and internal reflections, which improves the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through qualitative and quantitative evaluation of several scenarios, especially in challenging scenarios with complex shadows, we demonstrate that our method achieves state-of-the-art performance on decomposition results. Additionally, our optimized explicit geometry enables applications such as scene editing, relighting, and material editing with modern graphics engines or CAD software. The source code is available at https://brabbitdousha.github.io/MIRReS/ △ Less

Submitted 24 June, 2024; v1 submitted 24 June, 2024; originally announced June 2024.

Comments: 16 pages, 14 figures

Electrical switching of Ising-superconducting nonreciprocity for quantum neuronal transistor

Authors: Junlin Xiong, Jiao Xie, Bin Cheng, Yudi Dai, Xinyu Cui, Lizheng Wang, Zenglin Liu, Ji Zhou, Naizhou Wang, Xianghan Xu, Xianhui Chen, Sang-Wook Cheong, Shi-Jun Liang, Feng Miao

Abstract: Nonreciprocal quantum transport effect is mainly governed by the symmetry breaking of the material systems and is gaining extensive attention in condensed matter physics. Realizing electrical switching of the polarity of the nonreciprocal transport without external magnetic field is essential to the development of nonreciprocal quantum devices. However, electrical switching of superconducting nonr… ▽ More Nonreciprocal quantum transport effect is mainly governed by the symmetry breaking of the material systems and is gaining extensive attention in condensed matter physics. Realizing electrical switching of the polarity of the nonreciprocal transport without external magnetic field is essential to the development of nonreciprocal quantum devices. However, electrical switching of superconducting nonreciprocity remains yet to be achieved. Here, we report the observation of field-free electrical switching of nonreciprocal Ising superconductivity in Fe3GeTe2/NbSe2 van der Waals (vdW) heterostructure. By taking advantage of this electrically switchable superconducting nonreciprocity, we demonstrate a proof-of-concept nonreciprocal quantum neuronal transistor, which allows for implementing the XOR logic gate and faithfully emulating biological functionality of a cortical neuron in the brain. Our work provides a promising pathway to realize field-free and electrically switchable nonreciprocity of quantum transport and demonstrate its potential in exploring neuromorphic quantum devices with both functionality and performance beyond the traditional devices. △ Less

Submitted 20 June, 2024; originally announced June 2024.

Journal ref: Nature Communications 15, 4953 (2024)

VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

Authors: Ziyang Meng, Yu Dai, Zezheng Gong, Shaoxiong Guo, Minglong Tang, Tongquan Wei

Abstract: Recent advances in Large Vision-Language Models (LVLMs) have significantly improve performance in image comprehension tasks, such as formatted charts and rich-content images. Yet, Graphical User Interface (GUI) pose a greater challenge due to their structured format and detailed textual information. Existing LVLMs often overly depend on internal knowledge and neglect image content, resulting in ha… ▽ More Recent advances in Large Vision-Language Models (LVLMs) have significantly improve performance in image comprehension tasks, such as formatted charts and rich-content images. Yet, Graphical User Interface (GUI) pose a greater challenge due to their structured format and detailed textual information. Existing LVLMs often overly depend on internal knowledge and neglect image content, resulting in hallucinations and incorrect responses in GUI comprehension. To address these issues, we introduce VGA, a fine-tuned model designed for comprehensive GUI understanding. Our model aims to enhance the interpretation of visual data of GUI and reduce hallucinations. We first construct a Vision Question Answering (VQA) dataset of 63.8k high-quality examples with our propose Referent Method, which ensures the model's responses are highly depend on visual content within the image. We then design a two-stage fine-tuning method called Foundation and Advanced Comprehension (FAC) to enhance both the model's ability to extract information from image content and alignment with human intent. Experiments show that our approach enhances the model's ability to extract information from images and achieves state-of-the-art results in GUI understanding tasks. Our dataset and fine-tuning script will be released soon. △ Less

Submitted 21 June, 2024; v1 submitted 20 June, 2024; originally announced June 2024.

Comments: 18 pages

MSC Class: 68-04 68-04 ACM Class: I.2.7; I.2.10

Using Geometrical information to Measure the Vibration of A Swaying Millimeter-wave Radar

Authors: Chengyao Tang, Yongpeng Dai, Zhi Li, Tian Jin

Abstract: This paper presents two new, simple yet effective approaches to measure the vibration of a swaying millimeter-wave radar (mmRadar) utilizing geometrical information. Specifically, for the planar vibrations, we firstly establish an equation based on the area difference between the swaying mmRadar and the reference objects at different moments, which enables the quantification of planar displacement… ▽ More This paper presents two new, simple yet effective approaches to measure the vibration of a swaying millimeter-wave radar (mmRadar) utilizing geometrical information. Specifically, for the planar vibrations, we firstly establish an equation based on the area difference between the swaying mmRadar and the reference objects at different moments, which enables the quantification of planar displacement. Secondly, volume differences are also utilized with the same idea, achieving the self-vibration measurement of a swaying mmRadar for spatial vibrations. Experimental results confirm the effectiveness of our methods, demonstrating its capability to estimate both the amplitude and a crude direction of the mmRadar's self-vibration. △ Less

Submitted 19 June, 2024; originally announced June 2024.

Comments: 5 pages, 4 figures,submitted to the IEEE for publication

Applications of Explainable artificial intelligence in Earth system science

Authors: Feini Huang, Shijie Jiang, Lu Li, Yongkun Zhang, Ye Zhang, Ruqing Zhang, Qingliang Li, Danxi Li, Wei Shangguan, Yongjiu Dai

Abstract: In recent years, artificial intelligence (AI) rapidly accelerated its influence and is expected to promote the development of Earth system science (ESS) if properly harnessed. In application of AI to ESS, a significant hurdle lies in the interpretability conundrum, an inherent problem of black-box nature arising from the complexity of AI algorithms. To address this, explainable AI (XAI) offers a s… ▽ More In recent years, artificial intelligence (AI) rapidly accelerated its influence and is expected to promote the development of Earth system science (ESS) if properly harnessed. In application of AI to ESS, a significant hurdle lies in the interpretability conundrum, an inherent problem of black-box nature arising from the complexity of AI algorithms. To address this, explainable AI (XAI) offers a set of powerful tools that make the models more transparent. The purpose of this review is twofold: First, to provide ESS scholars, especially newcomers, with a foundational understanding of XAI, serving as a primer to inspire future research advances; second, to encourage ESS professionals to embrace the benefits of AI, free from preconceived biases due to its lack of interpretability. We begin with elucidating the concept of XAI, along with typical methods. We then delve into a review of XAI applications in the ESS literature, highlighting the important role that XAI has played in facilitating communication with AI model decisions, improving model diagnosis, and uncovering scientific insights. We identify four significant challenges that XAI faces within the ESS, and propose solutions. Furthermore, we provide a comprehensive illustration of multifaceted perspectives. Given the unique challenges in ESS, an interpretable hybrid approach that seamlessly integrates AI with domain-specific knowledge appears to be a promising way to enhance the utility of AI in ESS. A visionary outlook for ESS envisions a harmonious blend where process-based models govern the known, AI models explore the unknown, and XAI bridges the gap by providing explanations. △ Less

Submitted 12 June, 2024; originally announced June 2024.

Exploiting Overlap Information in Chance-constrained Program with Random Right-hand Side

Authors: Wei Lv, Wei-Kun Chen, Yu-Hong Dai, Xiao-Jiao Tong

Abstract: We consider the chance-constrained program (CCP) with random right-hand side under a finite discrete distribution. It is known that the standard mixed integer linear programming (MILP) reformulation of the CCP is generally difficult to solve by general-purpose solvers as the branch-and-cut search trees are enormously large, partly due to the weak linear programming relaxation. In this paper, we id… ▽ More We consider the chance-constrained program (CCP) with random right-hand side under a finite discrete distribution. It is known that the standard mixed integer linear programming (MILP) reformulation of the CCP is generally difficult to solve by general-purpose solvers as the branch-and-cut search trees are enormously large, partly due to the weak linear programming relaxation. In this paper, we identify another reason for this phenomenon: the intersection of the feasible regions of the subproblems in the search tree could be nonempty, leading to a wasteful duplication of effort in exploring the uninteresting overlap in the search tree. To address the newly identified challenge and enhance the capability of the MILP-based approach in solving CCPs, we first show that the overlap in the search tree can be completely removed by a family of valid nonlinear if-then constraints, and then propose two practical approaches to tackle the highly nonlinear if-then constraints. In particular, we use the concept of dominance relations between different scenarios of the random variables, and propose a novel branching, called dominance-based branching, which is able to create a valid partition of the problem with a much smaller overlap than the classic variable branching. Moreover, we develop overlap-oriented node pruning and variable fixing techniques, applied at each node of the search tree, to remove more overlaps in the search tree. Computational results demonstrate the effectiveness of the proposed dominance-based branching and overlap-oriented node pruning and variable fixing techniques in reducing the search tree size and improving the overall solution efficiency. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: 34 pages, 3 figures, submitted for possible publication

MSC Class: 90C11; 90C15

MIPI 2024 Challenge on Few-shot RAW Image Denoising: Methods and Results

Authors: Xin Jin, Chunle Guo, Xiaoming Li, Zongsheng Yue, Chongyi Li, Shangchen Zhou, Ruicheng Feng, Yuekun Dai, Peiqing Yang, Chen Change Loy, Ruoqi Li, Chang Liu, Ziyi Wang, Yao Du, Jingjing Yang, Long Bao, Heng Sun, Xiangyu Kong, Xiaoxia Xing, Jinlong Wu, Yuanyang Xue, Hyunhee Park, Sejun Song, Changho Kim, Jingfan Tan , et al. (17 additional authors not shown)

Abstract: The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photogra… ▽ More The increasing demand for computational photography and imaging on mobile platforms has led to the widespread development and integration of advanced image sensors with novel algorithms in camera systems. However, the scarcity of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). Building on the achievements of the previous MIPI Workshops held at ECCV 2022 and CVPR 2023, we introduce our third MIPI challenge including three tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Few-shot RAW Image Denoising track on MIPI 2024. In total, 165 participants were successfully registered, and 7 teams submitted results in the final testing phase. The developed solutions in this challenge achieved state-of-the-art erformance on Few-shot RAW Image Denoising. More details of this challenge and the link to the dataset can be found at https://mipichallenge.org/MIPI2024. △ Less

Submitted 11 June, 2024; originally announced June 2024.

Comments: CVPR 2024 Mobile Intelligent Photography and Imaging (MIPI) Workshop--Few-shot RAWImage Denoising Challenge Report. Website: https://mipi-challenge.org/MIPI2024/

The Prevalence of Resonance Among Young, Close-in Planets

Authors: Fei Dai, Max Goldberg, Konstantin Batygin, Jennifer van Saders, Eugene Chiang, Nick Choksi, Rixin Li, Erik A. Petigura, Gregory J. Gilbert, Sarah C. Millholland, Yuan-Zhe Dai, Luke Bouma, Lauren M. Weiss, Joshua N. Winn

Abstract: Multiple planets undergoing disk migration may be captured into a chain of mean-motion resonances with the innermost planet parked near the disk's inner edge. Subsequent dynamical evolution may disrupt these resonances, leading to the non-resonant configurations typically observed among {\it Kepler} planets that are Gyrs old. In this scenario, resonant configurations are expected to be more common… ▽ More Multiple planets undergoing disk migration may be captured into a chain of mean-motion resonances with the innermost planet parked near the disk's inner edge. Subsequent dynamical evolution may disrupt these resonances, leading to the non-resonant configurations typically observed among {\it Kepler} planets that are Gyrs old. In this scenario, resonant configurations are expected to be more common in younger systems. This prediction can now be tested, thanks to recent discoveries of young planets, particularly those in stellar clusters, by NASA's {\it TESS} mission. We divided the known planetary systems into three age groups: young ($<$100-Myr-old), adolescent (0.1-1-Gyr-old), and mature ($>1$-Gyr-old). The fraction of neighboring planet pairs having period ratios within a few percent of a first-order commensurability (e.g.~4:3, 3:2, or 2:1) is 70$\pm$15\% for young pairs, 24$\pm$8\% for adolescent pairs, and 15$\pm$2\% for mature pairs. The fraction of systems with at least one nearly commensurable pair (either first or second-order) is 86$\pm13$\% among young systems, 38$\pm12$\% for adolescent systems, and 23$\pm3$\% for mature systems. First-order commensurabilities prevail across all age groups, with an admixture of second-order commensurabilities. Commensurabilities are more common in systems with high planet multiplicity and low mutual inclinations. Observed period ratios often deviate from perfect commensurability by $\sim$1\% even among young planets, too large to be explained by resonant repulsion with equilibrium eccentricity tides. We also find that super-Earths in the radius gap ($1.5-1.9R_\oplus$) are less likely to be near-resonant (11.9$\pm2.0\%$) compared to Earth-sized planets ($R_p<1R_\oplus$; 25.3$\pm4.4\%$) or mini-Neptunes ($1.9R_\oplus \leq R_p<2.5R_\oplus$; 14.4$\pm1.8\%$). △ Less

Submitted 21 August, 2024; v1 submitted 10 June, 2024; originally announced June 2024.

Comments: 17 pages, 9 figures, accepted to AAS

An efficient branch-and-cut approach for large-scale competitive facility location problems with limited choice rule

Authors: Wei-Kun Chen, Wei-Yang Zhang, Yan-Ru Wang, Shahin Gelareh, Yu-Hong Dai

Abstract: In the paper, we consider the competitive facility location problem with limited choice rule (CFLPLCR), which attempts to open a subset of facilities to maximize the net profit of a newcomer company, requiring customers to patronize only a limited number of opening facilities and an outside option. We propose an efficient branch-and-cut (B&C) approach for the CFLPLCR based on newly proposed mixed… ▽ More In the paper, we consider the competitive facility location problem with limited choice rule (CFLPLCR), which attempts to open a subset of facilities to maximize the net profit of a newcomer company, requiring customers to patronize only a limited number of opening facilities and an outside option. We propose an efficient branch-and-cut (B&C) approach for the CFLPLCR based on newly proposed mixed integer linear programming (MILP) formulations. Specifically, by establishing the submodularity of the probability function, we develop an MILP formulation for the CFLPLCR using the submodular inequalities. For the special case where each customer patronizes at most one open facility and the outside option, we show that the submodular inequalities can characterize the convex hull of the considered set and provide a compact MILP formulation. Moreover, for the general case, we strengthen the submodular inequalities by sequential lifting, resulting in a class of facet-defining inequalities. The proposed lifted submodular inequalities are shown to be stronger than the classic submodular inequalities, enabling to obtain another MILP formulation with a tighter linear programming (LP) relaxation. By extensive numerical experiments, we show that the proposed B&C approach outperforms the state-of-the-art generalized Benders decomposition approach by at least one order of magnitude. Furthermore, it enables to solve CFLPLCR instances with 10000 customers and 2000 facilities. △ Less

Submitted 9 June, 2024; originally announced June 2024.

Comments: 29 pages, 2 figures, submitted for possible publication

MSC Class: 90C11

Observation of floating surface state in obstructed atomic insulator candidate NiP$_2$

Authors: Xiang-Rui Liu, Ming-Yuan Zhu, Yuanwen Feng, Meng Zeng, Xiao-Ming Ma, Yu-Jie Hao, Yue Dai, Rong-Hao Luo, Kohei Yamagami, Yi Liu, Shengtao Cui, Zhe Sun, Jia-Yu Liu, Zhengtai Liu, Mao Ye, Dawei Shen, Bing Li, Chang Liu

Abstract: Obstructed atomic insulator is recently proposed as an unconventional material, in which electric charge centers localized at sites away from the atoms. A half-filling surface state would emerge at specific interfaces cutting through these charge centers and avoid intersecting any atoms. In this article, we utilized angle-resolved photoemission spectroscopy and density functional theory calculatio… ▽ More Obstructed atomic insulator is recently proposed as an unconventional material, in which electric charge centers localized at sites away from the atoms. A half-filling surface state would emerge at specific interfaces cutting through these charge centers and avoid intersecting any atoms. In this article, we utilized angle-resolved photoemission spectroscopy and density functional theory calculations to study one of the obstructed atomic insulator candidates, NiP$_2$. A floating surface state with large effective mass that is isolated from all bulk states is resolved on the (100) cleavage plane, distinct from previously reported surface states in obstructed atomic insulators that are merged into bulk bands. Density functional theory calculation results elucidate that this floating surface state is originated from the obstructed Wannier charge centers, albeit underwent surface reconstruction that splits the half-filled obstructed surface state. Our findings not only shed lights on the spectroscopy study of obstructed atomic insulators and obstructed surface states, but also provide possible route for development of new catalysts. △ Less

Submitted 16 June, 2024; v1 submitted 8 June, 2024; originally announced June 2024.

Comments: 21 pages, 5 figures

DenoDet: Attention as Deformable Multi-Subspace Feature Denoising for Target Detection in SAR Images

Authors: Yimian Dai, Minrui Zou, Yuxuan Li, Xiang Li, Kang Ni, Jian Yang

Abstract: Synthetic Aperture Radar (SAR) target detection has long been impeded by inherent speckle noise and the prevalence of diminutive, ambiguous targets. While deep neural networks have advanced SAR target detection, their intrinsic low-frequency bias and static post-training weights falter with coherent noise and preserving subtle details across heterogeneous terrains. Motivated by traditional SAR ima… ▽ More Synthetic Aperture Radar (SAR) target detection has long been impeded by inherent speckle noise and the prevalence of diminutive, ambiguous targets. While deep neural networks have advanced SAR target detection, their intrinsic low-frequency bias and static post-training weights falter with coherent noise and preserving subtle details across heterogeneous terrains. Motivated by traditional SAR image denoising, we propose DenoDet, a network aided by explicit frequency domain transform to calibrate convolutional biases and pay more attention to high-frequencies, forming a natural multi-scale subspace representation to detect targets from the perspective of multi-subspace denoising. We design TransDeno, a dynamic frequency domain attention module that performs as a transform domain soft thresholding operation, dynamically denoising across subspaces by preserving salient target signals and attenuating noise. To adaptively adjust the granularity of subspace processing, we also propose a deformable group fully-connected layer (DeGroFC) that dynamically varies the group conditioned on the input features. Without bells and whistles, our plug-and-play TransDeno sets state-of-the-art scores on multiple SAR target detection datasets. The code is available at https://github.com/GrokCV/GrokSAR. △ Less

Submitted 10 August, 2024; v1 submitted 4 June, 2024; originally announced June 2024.

Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

Authors: Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, Xing Sun

Abstract: In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality… ▽ More In the quest for artificial general intelligence, Multi-modal Large Language Models (MLLMs) have emerged as a focal point in recent advancements. However, the predominant focus remains on developing their capabilities in static image understanding. The potential of MLLMs in processing sequential visual data is still insufficiently explored, highlighting the absence of a comprehensive, high-quality assessment of their performance. In this paper, we introduce Video-MME, the first-ever full-spectrum, Multi-Modal Evaluation benchmark of MLLMs in Video analysis. Our work distinguishes from existing benchmarks through four key features: 1) Diversity in video types, spanning 6 primary visual domains with 30 subfields to ensure broad scenario generalizability; 2) Duration in temporal dimension, encompassing both short-, medium-, and long-term videos, ranging from 11 seconds to 1 hour, for robust contextual dynamics; 3) Breadth in data modalities, integrating multi-modal inputs besides video frames, including subtitles and audios, to unveil the all-round capabilities of MLLMs; 4) Quality in annotations, utilizing rigorous manual labeling by expert annotators to facilitate precise and reliable model assessment. 900 videos with a total of 254 hours are manually selected and annotated by repeatedly viewing all the video content, resulting in 2,700 question-answer pairs. With Video-MME, we extensively evaluate various state-of-the-art MLLMs, including GPT-4 series and Gemini 1.5 Pro, as well as open-source image models like InternVL-Chat-V1.5 and video models like LLaVA-NeXT-Video. Our experiments reveal that Gemini 1.5 Pro is the best-performing commercial model, significantly outperforming the open-source models. Our dataset along with these findings underscores the need for further improvements in handling longer sequences and multi-modal data. Project Page: https://video-mme.github.io △ Less

Submitted 16 June, 2024; v1 submitted 31 May, 2024; originally announced May 2024.

Comments: Project Page: https://video-mme.github.io

You Only Scan Once: Efficient Multi-dimension Sequential Modeling with LightNet

Authors: Zhen Qin, Yuxin Mao, Xuyang Shen, Dong Li, Jing Zhang, Yuchao Dai, Yiran Zhong

Abstract: Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism in linear attention presents challenges when applied to multi-dimensional sequence modeling tasks, such as image processing and multi-modal learning. In these scenarios, the utilization of sequential scanning to establis… ▽ More Linear attention mechanisms have gained prominence in causal language models due to their linear computational complexity and enhanced speed. However, the inherent decay mechanism in linear attention presents challenges when applied to multi-dimensional sequence modeling tasks, such as image processing and multi-modal learning. In these scenarios, the utilization of sequential scanning to establish a global receptive field necessitates multiple scans for multi-dimensional data, thereby leading to inefficiencies. This paper identifies the inefficiency caused by a multiplicative linear recurrence and proposes an efficient alternative additive linear recurrence to avoid the issue, as it can handle multi-dimensional data within a single scan. We further develop an efficient multi-dimensional sequential modeling framework called LightNet based on the new recurrence. Moreover, we present two new multi-dimensional linear relative positional encoding methods, MD-TPE and MD-LRPE to enhance the model's ability to discern positional information in multi-dimensional scenarios. Our empirical evaluations across various tasks, including image classification, image generation, bidirectional language modeling, and autoregressive language modeling, demonstrate the efficacy of LightNet, showcasing its potential as a versatile and efficient solution for multi-dimensional sequential modeling. △ Less

Submitted 31 May, 2024; originally announced May 2024.

Comments: Technical report. Yiran Zhong is the corresponding author. The code is available at https://github.com/OpenNLPLab/LightNet

BackdoorIndicator: Leveraging OOD Data for Proactive Backdoor Detection in Federated Learning

Authors: Songze Li, Yanbo Dai

Abstract: In a federated learning (FL) system, decentralized data owners (clients) could upload their locally trained models to a central server, to jointly train a global model. Malicious clients may plant backdoors into the global model through uploading poisoned local models, causing misclassification to a target class when encountering attacker-defined triggers. Existing backdoor defenses show inconsist… ▽ More In a federated learning (FL) system, decentralized data owners (clients) could upload their locally trained models to a central server, to jointly train a global model. Malicious clients may plant backdoors into the global model through uploading poisoned local models, causing misclassification to a target class when encountering attacker-defined triggers. Existing backdoor defenses show inconsistent performance under different system and adversarial settings, especially when the malicious updates are made statistically close to the benign ones. In this paper, we first reveal the fact that planting subsequent backdoors with the same target label could significantly help to maintain the accuracy of previously planted backdoors, and then propose a novel proactive backdoor detection mechanism for FL named BackdoorIndicator, which has the server inject indicator tasks into the global model leveraging out-of-distribution (OOD) data, and then utilizing the fact that any backdoor samples are OOD samples with respect to benign samples, the server, who is completely agnostic of the potential backdoor types and target labels, can accurately detect the presence of backdoors in uploaded models, via evaluating the indicator tasks. We perform systematic and extensive empirical studies to demonstrate the consistently superior performance and practicality of BackdoorIndicator over baseline defenses, across a wide range of system and adversarial settings. △ Less

Submitted 31 May, 2024; originally announced May 2024.

FAST Discovery of Eight Isolated Millisecond Pulsars in NGC 6517

Authors: Dejiang Yin, Li-yun Zhang, Lei Qian, Ralph P. Eatough, Baoda Li, Duncan R. Lorimer, Yinfeng Dai, Yaowei Li, Xingnan Zhang, Minghui Li, Tianhao Su, Yuxiao Wu, Yu Pan, Yujie Lian, Tong Liu, Zhen Yan, Zhichen Pan

Abstract: We present the discovery of 8 isolated millisecond pulsars in Globular Cluster (GC) NGC 6517 using the Five-Hundred-meter Aperture Spherical radio Telescope (FAST). The spin periods of those pulsars (namely PSR J1801-0857K to R, or, NGC 6517K to R) are all shorter than 10 ms. With these discoveries, NGC 6517 is currently the GC with the most known pulsars in the FAST sky. The largest difference in… ▽ More We present the discovery of 8 isolated millisecond pulsars in Globular Cluster (GC) NGC 6517 using the Five-Hundred-meter Aperture Spherical radio Telescope (FAST). The spin periods of those pulsars (namely PSR J1801-0857K to R, or, NGC 6517K to R) are all shorter than 10 ms. With these discoveries, NGC 6517 is currently the GC with the most known pulsars in the FAST sky. The largest difference in dispersion measure of the pulsars in NGC 6517 is 11.2 cm$^{-3}$ pc, the second among all GCs. The fraction of isolated pulsars in this GC (16 of 17, 94$\%$) is consistent with previous studies indicating an overabundance of isolated pulsars in the densest GCs, especially in those undergoing cluster core collapse. Considering the FAST GC pulsar discoveries, we modeled the GC pulsar population using the empirical Bayesian method described by Turk and Lorimer with the recent counts. Using this approach, we find that the expected number of potential pulsars in GCs seems to be correlated with the central escape velocity, hence, the GCs Liller 1, NGC 6441, M54 (NGC 6715), and $ω$-Cen (NGC 5139) are expected to host the largest numbers of pulsars. △ Less

Submitted 28 May, 2024; originally announced May 2024.

Comments: 21 pages, 2 figures, accepted for publication in The Astrophysical Journal Letters

Variational Bayes for Federated Continual Learning

Authors: Dezhong Yao, Sanmu Li, Yutong Dai, Zhiqiang Xu, Shengshan Hu, Peilin Zhao, Lichao Sun

Abstract: Federated continual learning (FCL) has received increasing attention due to its potential in handling real-world streaming data, characterized by evolving data distributions and varying client classes over time. The constraints of storage limitations and privacy concerns confine local models to exclusively access the present data within each learning cycle. Consequently, this restriction induces p… ▽ More Federated continual learning (FCL) has received increasing attention due to its potential in handling real-world streaming data, characterized by evolving data distributions and varying client classes over time. The constraints of storage limitations and privacy concerns confine local models to exclusively access the present data within each learning cycle. Consequently, this restriction induces performance degradation in model training on previous data, termed "catastrophic forgetting". However, existing FCL approaches need to identify or know changes in data distribution, which is difficult in the real world. To release these limitations, this paper directs attention to a broader continuous framework. Within this framework, we introduce Federated Bayesian Neural Network (FedBNN), a versatile and efficacious framework employing a variational Bayesian neural network across all clients. Our method continually integrates knowledge from local and historical data distributions into a single model, adeptly learning from new data distributions while retaining performance on historical distributions. We rigorously evaluate FedBNN's performance against prevalent methods in federated learning and continual learning using various metrics. Experimental analyses across diverse datasets demonstrate that FedBNN achieves state-of-the-art results in mitigating forgetting. △ Less

Submitted 23 May, 2024; originally announced May 2024.

Observation of Spin Splitting in Room-Temperature Metallic Antiferromagnet CrSb

Authors: Meng Zeng, Ming-Yuan Zhu, Yu-Peng Zhu, Xiang-Rui Liu, Xiao-Ming Ma, Yu-Jie Hao, Pengfei Liu, Gexing Qu, Yichen Yang, Zhicheng Jiang, Kohei Yamagami, Masashi Arita, Xiaoqian Zhang, Tian-Hao Shao, Yue Dai, Kenya Shimada, Zhengtai Liu, Mao Ye, Yaobo Huang, Qihang Liu, Chang Liu

Abstract: Recently, unconventional antiferromagnets that enable the splitting of electronic spins have been theoretically proposed and experimentally realized, where the magnetic sublattices containing moments pointing at different directions are connected by a novel set of symmetries. Such spin splitting (SS) is substantial, $k$-dependent, and independent of the spin-orbit coupling strength, making these m… ▽ More Recently, unconventional antiferromagnets that enable the splitting of electronic spins have been theoretically proposed and experimentally realized, where the magnetic sublattices containing moments pointing at different directions are connected by a novel set of symmetries. Such spin splitting (SS) is substantial, $k$-dependent, and independent of the spin-orbit coupling strength, making these magnets promising materials for antiferromagnetic spintronics. Here, combined with angle-resolved photoemission spectroscopy (ARPES) and density functional theory (DFT) calculations, we perform a systematic study on CrSb, a metallic spin-split antiferromagnet candidate with $T_N$ = 703 K. Our data reveals the electronic structure of CrSb along both out-of-plane and in-plane momentum directions, which renders anisotropic $k$-dependent SS and agrees well with the calculational results. The magnitude of such SS reaches up to at least 0.8 eV at non-high-symmetry momentum points, which is significantly higher than the largest known SOC-induced SS. This compound expands the choice of materials in the field of antiferromagnetic spintronics and is likely to stimulate subsequent investigations of high-efficiency spintronic devices that are functional at room temperature. △ Less

Submitted 21 May, 2024; originally announced May 2024.

Comments: 14 pages, 4 figures

Molecule-induced surface second-order nonlinearity in an inversion symmetric microcavity

Authors: Ru Wang, Yue Dai, Jinsong Cheng, Ruoyu Wang, Xiaoqin Shen

Abstract: Inversion symmetry eliminates the second-order nonlinear responses in materials commonly used in silicon photonics with electric-dipole approximation. The lack of effective methods to induce the second-order nonlinearity in silicon photonic materials prevents their applications in second-order nonlinear integrated photonics. Here, we experimentally demonstrate a surface second-order nonlinear opti… ▽ More Inversion symmetry eliminates the second-order nonlinear responses in materials commonly used in silicon photonics with electric-dipole approximation. The lack of effective methods to induce the second-order nonlinearity in silicon photonic materials prevents their applications in second-order nonlinear integrated photonics. Here, we experimentally demonstrate a surface second-order nonlinear optics approach for boosting the second harmonic (SH) generation process in a silica microcavity. By leveraging the molecule-induced surface second-order nonlinearity, a record high SH efficiency of about 6.7% W-1 is achieved in a silica microcavity functionalized with a surface asymmetrically-aligned molecular monolayer, which is enhanced of two to four orders of magnitude compared to that before molecule-functionalization. Furthermore, we derive the equations that govern the surface second-order nonlinear process in inversion symmetric microcavities. Our method not only enables high efficiency second-order nonlinear frequency conversions in silica photonics, but also can apply to other inversion symmetric material platforms for integrated photonics. △ Less

Submitted 21 May, 2024; originally announced May 2024.

Comments: 22