Showing 151–200 of 622 results for author: Wen, Y

Assessing and Understanding Creativity in Large Language Models

Authors: Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li, Yunji Chen

Abstract: In the field of natural language processing, the rapid development of large language model (LLM) has attracted more and more attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. The assessment of LLM creativity needs to consider differences from humans, requiring multi-dimensional measurement while balancing accuracy… ▽ More In the field of natural language processing, the rapid development of large language model (LLM) has attracted more and more attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. The assessment of LLM creativity needs to consider differences from humans, requiring multi-dimensional measurement while balancing accuracy and efficiency. This paper aims to establish an efficient framework for assessing the level of creativity in LLMs. By adapting the modified Torrance Tests of Creative Thinking, the research evaluates the creative performance of various LLMs across 7 tasks, emphasizing 4 criteria including Fluency, Flexibility, Originality, and Elaboration. In this context, we develop a comprehensive dataset of 700 questions for testing and an LLM-based evaluation method. In addition, this study presents a novel analysis of LLMs' responses to diverse prompts and role-play situations. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration. Besides, the use of prompts and the role-play settings of the model significantly influence creativity. Additionally, the experimental results also indicate that collaboration among multiple LLMs can enhance originality. Notably, our findings reveal a consensus between human evaluations and LLMs regarding the personality traits that influence creativity. The findings underscore the significant impact of LLM design on creativity and bridges artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications. △ Less

Submitted 23 January, 2024; originally announced January 2024.

Citizen Science for IceCube: Name that Neutrino

Authors: R. Abbasi, M. Ackermann, J. Adams, S. K. Agarwalla, J. A. Aguilar, M. Ahlers, J. M. Alameddine, N. M. Amin, K. Andeen, G. Anton, C. Argüelles, Y. Ashida, S. Athanasiadou, L. Ausborm, S. N. Axani, X. Bai, A. Balagopal V., M. Baricevic, S. W. Barwick, V. Basu, R. Bay, J. J. Beatty, J. Becker Tjus, J. Beise, C. Bellenghi , et al. (391 additional authors not shown)

Abstract: Name that Neutrino is a citizen science project where volunteers aid in classification of events for the IceCube Neutrino Observatory, an immense particle detector at the geographic South Pole. From March 2023 to September 2023, volunteers did classifications of videos produced from simulated data of both neutrino signal and background interactions. Name that Neutrino obtained more than 128,000 cl… ▽ More Name that Neutrino is a citizen science project where volunteers aid in classification of events for the IceCube Neutrino Observatory, an immense particle detector at the geographic South Pole. From March 2023 to September 2023, volunteers did classifications of videos produced from simulated data of both neutrino signal and background interactions. Name that Neutrino obtained more than 128,000 classifications by over 1,800 registered volunteers that were compared to results obtained by a deep neural network machine-learning algorithm. Possible improvements for both Name that Neutrino and the deep neural network are discussed. △ Less

Submitted 22 January, 2024; originally announced January 2024.

Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

Authors: Kohei Uehara, Nabarun Goswami, Hanqin Wang, Toshiaki Baba, Kohtaro Tanaka, Tomohiro Hashimoto, Kai Wang, Rei Ito, Takagi Naoya, Ryo Umagami, Yingyi Wen, Tanachai Anakewat, Tatsuya Harada

Abstract: The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of large Vision-and-Language Models (VLMs) that are not only accurate but also have explicit reasoning capabilities. This paper presents a novel approach to develop a VLM with the ability to conduct explicit reasoning based on visual content and textual instructions. We… ▽ More The increasing demand for intelligent systems capable of interpreting and reasoning about visual content requires the development of large Vision-and-Language Models (VLMs) that are not only accurate but also have explicit reasoning capabilities. This paper presents a novel approach to develop a VLM with the ability to conduct explicit reasoning based on visual content and textual instructions. We introduce a system that can ask a question to acquire necessary knowledge, thereby enhancing the robustness and explicability of the reasoning process. To this end, we developed a novel dataset generated by a Large Language Model (LLM), designed to promote chain-of-thought reasoning combined with a question-asking mechanism. The dataset covers a range of tasks, from common ones like caption generation to specialized VQA tasks that require expert knowledge. Furthermore, using the dataset we created, we fine-tuned an existing VLM. This training enabled the models to generate questions and perform iterative reasoning during inference. The results demonstrated a stride toward a more robust, accurate, and interpretable VLM, capable of reasoning explicitly and seeking information proactively when confronted with ambiguous visual input. △ Less

Submitted 17 July, 2024; v1 submitted 18 January, 2024; originally announced January 2024.

Measurement of Born cross section of $e^{+}e^{-}\rightarrowΣ^{+}\barΣ^{-}$ at center-of-mass energies between 3.510 and 4.951 GeV

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (632 additional authors not shown)

Abstract: Using 24.1 fb$^{-1}$ of $e^{+}e^{-}$ collision data collected with the BESIII detector at the BEPCII collider, the Born cross sections and effective form factors of the $e^{+}e^{-}\rightarrowΣ^{+}\barΣ^{-}$ reaction are measured. The measurements are performed at center-of-mass energies ranging from 3.510 to 4.951 GeV. No significant evidence for the decay of the charmonium(-like) states,… ▽ More Using 24.1 fb$^{-1}$ of $e^{+}e^{-}$ collision data collected with the BESIII detector at the BEPCII collider, the Born cross sections and effective form factors of the $e^{+}e^{-}\rightarrowΣ^{+}\barΣ^{-}$ reaction are measured. The measurements are performed at center-of-mass energies ranging from 3.510 to 4.951 GeV. No significant evidence for the decay of the charmonium(-like) states, $ψ(3770)$, $ψ(4040)$, $ψ(4160)$, $Y(4230)$, $Y(4360)$, $ψ(4415)$, and $Y(4660)$, into a $Σ^{+}\barΣ^{-}$ final state is observed. Consequently, upper limits for the products of the branching fractions and the electronic partial widths at the 90% confidence level are reported for these decays. △ Less

Submitted 6 May, 2024; v1 submitted 10 January, 2024; originally announced January 2024.

Comments: 22 pages, 3 figures, 3 tables, consistent with the publication in JHEP05(2024)022

Journal ref: JHEP05(2024)022

InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding

Authors: Qiaoling Chen, Diandian Gu, Guoteng Wang, Xun Chen, YingTong Xiong, Ting Huang, Qinghao Hu, Xin Jin, Yonggang Wen, Tianwei Zhang, Peng Sun

Abstract: Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space… ▽ More Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization. △ Less

Submitted 22 January, 2024; v1 submitted 17 January, 2024; originally announced January 2024.

Improved measurements of the Dalitz decays $η/η'\rightarrowγe^{+}e^{-}$

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (618 additional authors not shown)

Abstract: Based on a data sample of 10 billion $J/ψ$ events collected with the BESIII detector, improved measurements of the Dalitz decays $η/η'\rightarrowγe^+e^-$ are performed, where the $η$ and $η'$ are produced through the radiative decays $J/ψ\rightarrowγη/η'$. The branching fractions of $η\rightarrowγe^+e^-$ and $η'\rightarrowγe^+e^-$ are measured to be $(7.07 \pm 0.05 \pm 0.23)\times10^{-3}$ and… ▽ More Based on a data sample of 10 billion $J/ψ$ events collected with the BESIII detector, improved measurements of the Dalitz decays $η/η'\rightarrowγe^+e^-$ are performed, where the $η$ and $η'$ are produced through the radiative decays $J/ψ\rightarrowγη/η'$. The branching fractions of $η\rightarrowγe^+e^-$ and $η'\rightarrowγe^+e^-$ are measured to be $(7.07 \pm 0.05 \pm 0.23)\times10^{-3}$ and $(4.83\pm0.07\pm0.14)\times10^{-4}$, respectively. Within the single pole model, the parameter of electromagnetic transition form factor for $η\rightarrowγe^+e^-$ is determined to be $Λ_η=(0.749 \pm 0.027 \pm 0.007)~ {\rm GeV}/c^{2}$. Within the multi-pole model, we extract the electromagnetic transition form factors for $η'\rightarrowγe^+e^-$ to be $Λ_{η'} = (0.802 \pm 0.007\pm 0.008)~ {\rm GeV}/c^{2}$ and $γ_{η'} = (0.113\pm0.010\pm0.002)~ {\rm GeV}/c^{2}$. The results are consistent with both theoretical predictions and previous measurements. The characteristic sizes of the interaction regions for the $η$ and $η'$ are calculated to be $(0.645 \pm 0.023 \pm 0.007 )~ {\rm fm}$ and $(0.596 \pm 0.005 \pm 0.006)~ {\rm fm}$, respectively. In addition, we search for the dark photon in $η/η^\prime\rightarrowγe^{+}e^{-}$, and the upper limits of the branching fractions as a function of the dark photon are given at 90\% confidence level. △ Less

Submitted 5 April, 2024; v1 submitted 17 January, 2024; originally announced January 2024.

Journal ref: Phys.Rev.D 109 (2024) 7, 072001

First study of antihyperon-nucleon scattering $\barΛp\rightarrow\barΛp$ and measurement of $Λp\rightarrowΛp$ cross section

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (634 additional authors not shown)

Abstract: Using $(10.087\pm0.044)\times10^{9}$ $J/ψ$ events collected with the BESIII detector at the BEPCII storage ring, the processes $Λp\rightarrowΛp$ and $\barΛp\rightarrow\barΛp$ are studied, where the $Λ/\barΛ$ baryons are produced in the process $J/ψ\rightarrowΛ\barΛ$ and the protons are the hydrogen nuclei in the cooling oil of the beam pipe. Clear signals are observed for the two reactions. The cr… ▽ More Using $(10.087\pm0.044)\times10^{9}$ $J/ψ$ events collected with the BESIII detector at the BEPCII storage ring, the processes $Λp\rightarrowΛp$ and $\barΛp\rightarrow\barΛp$ are studied, where the $Λ/\barΛ$ baryons are produced in the process $J/ψ\rightarrowΛ\barΛ$ and the protons are the hydrogen nuclei in the cooling oil of the beam pipe. Clear signals are observed for the two reactions. The cross sections in $-0.9\leq\rm{cos}θ_{Λ/\barΛ}\leq0.9$ are measured to be $σ(Λp\rightarrowΛp)=(12.2\pm1.6_{\rm{stat}}\pm1.1_{\rm{sys}})$ mb and $σ(\barΛ p\rightarrow\barΛ p)=(17.5\pm2.1_{\rm{stat}}\pm1.6_{\rm{sys}})$ mb at the $Λ/\barΛ$ momentum of $1.074$ GeV/$c$ within a range of $\pm0.017$ GeV/$c$, where the $θ_{Λ/\barΛ}$ are the scattering angles of the $Λ/\barΛ$ in the $Λp/\barΛp$ rest frames. Furthermore, the differential cross sections of the two reactions are also measured, where there is a slight tendency of forward scattering for $Λp\rightarrowΛp$, and a strong forward peak for $\barΛp\rightarrow\barΛp$. We present an approach to extract the total elastic cross sections by extrapolation. The study of $\barΛp\rightarrow\barΛp$ represents the first study of antihyperon-nucleon scattering, and these new measurements will serve as important inputs for the theoretical understanding of the (anti)hyperon-nucleon interaction. △ Less

Submitted 18 May, 2024; v1 submitted 17 January, 2024; originally announced January 2024.

Comments: 9 pages, 5 figures

Enhancing Campus Mobility: Achievements and Challenges of Autonomous Shuttle "Snow Lion''

Authors: Yingbing Chen, Jie Cheng, Sheng Wang, Hongji Liu, Xiaodong Mei, Xiaoyang Yan, Mingkai Tang, Ge Sun, Ya Wen, Junwei Cai, Xupeng Xie, Lu Gan, Mandan Chao, Ren Xin, Ming Liu, Jianhao Jiao, Kangcheng Liu, Lujia Wang

Abstract: The rapid evolution of autonomous vehicles (AVs) has significantly influenced global transportation systems. In this context, we present ``Snow Lion'', an autonomous shuttle meticulously designed to revolutionize on-campus transportation, offering a safer and more efficient mobility solution for students, faculty, and visitors. The primary objective of this research is to enhance campus mobility b… ▽ More The rapid evolution of autonomous vehicles (AVs) has significantly influenced global transportation systems. In this context, we present ``Snow Lion'', an autonomous shuttle meticulously designed to revolutionize on-campus transportation, offering a safer and more efficient mobility solution for students, faculty, and visitors. The primary objective of this research is to enhance campus mobility by providing a reliable, efficient, and eco-friendly transportation solution that seamlessly integrates with existing infrastructure and meets the diverse needs of a university setting. To achieve this goal, we delve into the intricacies of the system design, encompassing sensing, perception, localization, planning, and control aspects. We evaluate the autonomous shuttle's performance in real-world scenarios, involving a 1146-kilometer road haul and the transportation of 442 passengers over a two-month period. These experiments demonstrate the effectiveness of our system and offer valuable insights into the intricate process of integrating an autonomous vehicle within campus shuttle operations. Furthermore, a thorough analysis of the lessons derived from this experience furnishes a valuable real-world case study, accompanied by recommendations for future research and development in the field of autonomous driving. △ Less

Submitted 16 January, 2024; originally announced January 2024.

Comments: 9 pages, 9 figures

WAVES: Benchmarking the Robustness of Image Watermarks

Authors: Bang An, Mucong Ding, Tahseen Rabbani, Aakriti Agrawal, Yuancheng Xu, Chenghao Deng, Sicheng Zhu, Abdirisak Mohamed, Yuxin Wen, Tom Goldstein, Furong Huang

Abstract: In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised… ▽ More In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/ △ Less

Submitted 6 June, 2024; v1 submitted 16 January, 2024; originally announced January 2024.

Comments: Accepted by ICML 2024

Observation of $ψ(3686) \to Ω^- K^+ \barΞ^0 $+c.c

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (630 additional authors not shown)

Abstract: Using $(27.12 \pm 0.14) \times 10^{8}$ $ψ(3686)$ events collected with the BESIII detector at BEPCII, the decay of $ψ(3686) \to Ω^- K^+ \barΞ^0 +c.c.$ is observed for the first time. The branching fraction of this decay is measured to be $\mathcal{B}_{ψ(3686) \to Ω^- K^+ \barΞ^0 +c.c.}=(2.78 \pm 0.40 \pm 0.18 ) \times 10^{-6}$, where the first uncertainty is statistical and the second is systemati… ▽ More Using $(27.12 \pm 0.14) \times 10^{8}$ $ψ(3686)$ events collected with the BESIII detector at BEPCII, the decay of $ψ(3686) \to Ω^- K^+ \barΞ^0 +c.c.$ is observed for the first time. The branching fraction of this decay is measured to be $\mathcal{B}_{ψ(3686) \to Ω^- K^+ \barΞ^0 +c.c.}=(2.78 \pm 0.40 \pm 0.18 ) \times 10^{-6}$, where the first uncertainty is statistical and the second is systematic. Possible baryon excited states are searched for in this decay, but no evident intermediate state is observed with the current sample size. △ Less

Submitted 15 April, 2024; v1 submitted 16 January, 2024; originally announced January 2024.

First observation of the decay $Λ^+_c\to nK^{0}_{S}π^+π^0$

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (630 additional authors not shown)

Abstract: Based on 4.5 fb$^{-1}$ of $e^{+}e^{-}$ collision data accumulated at center-of-mass energies between $4599.53$ MeV and $4698.82$ MeV with the BESIII detector, the decay $Λ_{c}^{+}\to nK_{S}^{0}π^+π^0$ is observed for the first time with a significance of $9.2σ$. The branching fraction is measured to be $(0.85\pm0.13\pm0.03)\%$, where the first uncertainty is statistical and the second systematic,… ▽ More Based on 4.5 fb$^{-1}$ of $e^{+}e^{-}$ collision data accumulated at center-of-mass energies between $4599.53$ MeV and $4698.82$ MeV with the BESIII detector, the decay $Λ_{c}^{+}\to nK_{S}^{0}π^+π^0$ is observed for the first time with a significance of $9.2σ$. The branching fraction is measured to be $(0.85\pm0.13\pm0.03)\%$, where the first uncertainty is statistical and the second systematic, which differs from the theoretical prediction based on isospin by 4.4$σ$. This indicates that there may be resonant contributions or some unknown dynamics in this decay. △ Less

Submitted 28 March, 2024; v1 submitted 11 January, 2024; originally announced January 2024.

Journal ref: Phys.Rev.D,109,053005 (2024)

PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Model

Authors: Chenghao Huang, Yanbo Cao, Yinlong Wen, Tao Zhou, Yanru Zhang

Abstract: Poker, also known as Texas Hold'em, has always been a typical research target within imperfect information games (IIGs). IIGs have long served as a measure of artificial intelligence (AI) development. Representative prior works, such as DeepStack and Libratus heavily rely on counterfactual regret minimization (CFR) to tackle heads-up no-limit Poker. However, it is challenging for subsequent resear… ▽ More Poker, also known as Texas Hold'em, has always been a typical research target within imperfect information games (IIGs). IIGs have long served as a measure of artificial intelligence (AI) development. Representative prior works, such as DeepStack and Libratus heavily rely on counterfactual regret minimization (CFR) to tackle heads-up no-limit Poker. However, it is challenging for subsequent researchers to learn CFR from previous models and apply it to other real-world applications due to the expensive computational cost of CFR iterations. Additionally, CFR is difficult to apply to multi-player games due to the exponential growth of the game tree size. In this work, we introduce PokerGPT, an end-to-end solver for playing Texas Hold'em with arbitrary number of players and gaining high win rates, established on a lightweight large language model (LLM). PokerGPT only requires simple textual information of Poker games for generating decision-making advice, thus guaranteeing the convenient interaction between AI and humans. We mainly transform a set of textual records acquired from real games into prompts, and use them to fine-tune a lightweight pre-trained LLM using reinforcement learning human feedback technique. To improve fine-tuning performance, we conduct prompt engineering on raw data, including filtering useful information, selecting behaviors of players with high win rates, and further processing them into textual instruction using multiple prompt engineering techniques. Through the experiments, we demonstrate that PokerGPT outperforms previous approaches in terms of win rate, model size, training time, and response speed, indicating the great potential of LLMs in solving IIGs. △ Less

Submitted 4 January, 2024; originally announced January 2024.

Analytical solutions for optimal photon absorption into inhomogeneous spin memories

Authors: József Zsolt Bernád, Michael Schilling, Yutian Wen, Matthias M. Müller, Tommaso Calarco, Patrice Bertet, Felix Motzoi

Abstract: We investigate for optimal photon absorption a quantum electrodynamical model of an inhomogeneously-broadened spin ensemble coupled to a single-mode cavity. We consider a one-photon input pulse and obtain a simple one-parameter form for its optimal shape for absorption in the spin ensemble. Solutions to this problem are developed without using perturbation theory concerning the spin ensemble. Furt… ▽ More We investigate for optimal photon absorption a quantum electrodynamical model of an inhomogeneously-broadened spin ensemble coupled to a single-mode cavity. We consider a one-photon input pulse and obtain a simple one-parameter form for its optimal shape for absorption in the spin ensemble. Solutions to this problem are developed without using perturbation theory concerning the spin ensemble. Furthermore, we exploit the possibility of modulating the frequency and coupling rate of the resonator. We show some optimal scenarios and demonstrate the usefulness of our approach for the design of efficient quantum memories. In particular, we find the optimal cooperativity for different parameters and identify cases where absorption with a success probability larger than $99\%$ is achieved. △ Less

Submitted 9 January, 2024; originally announced January 2024.

Comments: 18 pages, 20 figures

Explore Human Parsing Modality for Action Recognition

Authors: Jinfu Liu, Runwei Ding, Yuhang Wen, Nan Dai, Fanyang Meng, Shen Zhao, Mengyuan Liu

Abstract: Multimodal-based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, we introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts, while filtering out m… ▽ More Multimodal-based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, we introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts, while filtering out most irrelevant noise. We propose a new dual-branch framework called Ensemble Human Parsing and Pose Network (EPP-Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing festures via convolutional backbones. The two high-level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB+D and NTU RGB+D 120 benchmarks consistently verify the effectiveness of our proposed EPP-Net, which outperforms the existing action recognition methods. Our code is available at: https://github.com/liujf69/EPP-Net-Action. △ Less

Submitted 4 January, 2024; originally announced January 2024.

Comments: arXiv admin note: text overlap with arXiv:2307.07977

Adaptive Control Strategy for Quadruped Robots in Actuator Degradation Scenarios

Authors: Xinyuan Wu, Wentao Dong, Hang Lai, Yong Yu, Ying Wen

Abstract: Quadruped robots have strong adaptability to extreme environments but may also experience faults. Once these faults occur, robots must be repaired before returning to the task, reducing their practical feasibility. One prevalent concern among these faults is actuator degradation, stemming from factors like device aging or unexpected operational events. Traditionally, addressing this problem has re… ▽ More Quadruped robots have strong adaptability to extreme environments but may also experience faults. Once these faults occur, robots must be repaired before returning to the task, reducing their practical feasibility. One prevalent concern among these faults is actuator degradation, stemming from factors like device aging or unexpected operational events. Traditionally, addressing this problem has relied heavily on intricate fault-tolerant design, which demands deep domain expertise from developers and lacks generalizability. Learning-based approaches offer effective ways to mitigate these limitations, but a research gap exists in effectively deploying such methods on real-world quadruped robots. This paper introduces a pioneering teacher-student framework rooted in reinforcement learning, named Actuator Degradation Adaptation Transformer (ADAPT), aimed at addressing this research gap. This framework produces a unified control strategy, enabling the robot to sustain its locomotion and perform tasks despite sudden joint actuator faults, relying exclusively on its internal sensors. Empirical evaluations on the Unitree A1 platform validate the deployability and effectiveness of Adapt on real-world quadruped robots, and affirm the robustness and practicality of our approach. △ Less

Submitted 29 December, 2023; originally announced December 2023.

Comments: 13 pages, 14 figures, in proceeding of DAI'23

Search for a massless particle beyond the Standard Model in the $Σ^+\rightarrow p+{\rm invisible}$ decay

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (634 additional authors not shown)

Abstract: A massless particle beyond the Standard Model is searched for in the two-body decay $Σ^+\rightarrow p+{\rm invisible}$ using $(1.0087\pm0.0044)\times10^{10}$ $J/ψ$ events collected at a center-of-mass energy of $\sqrt{s}=3.097$ GeV with the BESIII detector at the BEPCII collider. No significant signal is observed, and the upper limit on the branching fraction $B(Σ^+\rightarrow p+{\rm invisible})$… ▽ More A massless particle beyond the Standard Model is searched for in the two-body decay $Σ^+\rightarrow p+{\rm invisible}$ using $(1.0087\pm0.0044)\times10^{10}$ $J/ψ$ events collected at a center-of-mass energy of $\sqrt{s}=3.097$ GeV with the BESIII detector at the BEPCII collider. No significant signal is observed, and the upper limit on the branching fraction $B(Σ^+\rightarrow p+{\rm invisible})$ is determined to be $3.2\times10^{-5}$ at the 90% confidence level. This is the first search for a flavor-changing neutral current process with missing energy in hyperon decays which plays an important role in constraining new physics models. △ Less

Submitted 5 April, 2024; v1 submitted 28 December, 2023; originally announced December 2023.

Comments: 11 pages, 5 figures

Journal ref: Phys. Lett. B 852 (2024) 138614

Observation of $χ_{cJ}\to 3(K^+K^-)$

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (632 additional authors not shown)

Abstract: By analyzing $(27.12\pm0.14)\times10^8$ $ψ(3686)$ events collected with the BESIII detector operating at the BEPCII collider, the decay processes $χ_{cJ} \to 3(K^+K^-)$ ($J=0,1,2$) are observed for the first time with statistical significances of 8.2$σ$, 8.1$σ$, and 12.4$σ$, respectively. The product branching fractions of $ψ(3686)\toγχ_{cJ}$, $χ_{cJ}\to 3(K^+K^-)$ are presented and the branching… ▽ More By analyzing $(27.12\pm0.14)\times10^8$ $ψ(3686)$ events collected with the BESIII detector operating at the BEPCII collider, the decay processes $χ_{cJ} \to 3(K^+K^-)$ ($J=0,1,2$) are observed for the first time with statistical significances of 8.2$σ$, 8.1$σ$, and 12.4$σ$, respectively. The product branching fractions of $ψ(3686)\toγχ_{cJ}$, $χ_{cJ}\to 3(K^+K^-)$ are presented and the branching fractions of $χ_{cJ}\to 3(K^+K^-)$ decays are determined to be $\mathcal{B}_{χ_{c0}\to 3(K^+K^-)}$=$(10.7\pm1.8\pm1.1)$$\times10^{-6}$, $\mathcal{B}_{χ_{c1}\to 3(K^+K^-)}$=$(4.2\pm0.9\pm0.5)$$\times10^{-6}$, and $\mathcal{B}_{χ_{c2}\to 3(K^+K^-)}$=$(7.2\pm1.1\pm0.8)$$\times10^{-6}$, where the first uncertainties are statistical and the second are systematic. △ Less

Submitted 26 December, 2023; originally announced December 2023.

Comments: 8 pages, 2 figures

AdaNAS: Adaptively Post-processing with Self-supervised Neural Architecture Search for Ensemble Rainfall Forecasts

Authors: Yingpeng Wen, Weijiang Yu, Fudan Zheng, Dan Huang, Nong Xiao

Abstract: Previous post-processing studies on rainfall forecasts using numerical weather prediction (NWP) mainly focus on statistics-based aspects, while learning-based aspects are rarely investigated. Although some manually-designed models are proposed to raise accuracy, they are customized networks, which need to be repeatedly tried and verified, at a huge cost in time and labor. Therefore, a self-supervi… ▽ More Previous post-processing studies on rainfall forecasts using numerical weather prediction (NWP) mainly focus on statistics-based aspects, while learning-based aspects are rarely investigated. Although some manually-designed models are proposed to raise accuracy, they are customized networks, which need to be repeatedly tried and verified, at a huge cost in time and labor. Therefore, a self-supervised neural architecture search (NAS) method without significant manual efforts called AdaNAS is proposed in this study to perform rainfall forecast post-processing and predict rainfall with high accuracy. In addition, we design a rainfall-aware search space to significantly improve forecasts for high-rainfall areas. Furthermore, we propose a rainfall-level regularization function to eliminate the effect of noise data during the training. Validation experiments have been performed under the cases of \emph{None}, \emph{Light}, \emph{Moderate}, \emph{Heavy} and \emph{Violent} on a large-scale precipitation benchmark named TIGGE. Finally, the average mean-absolute error (MAE) and average root-mean-square error (RMSE) of the proposed AdaNAS model are 0.98 and 2.04 mm/day, respectively. Additionally, the proposed AdaNAS model is compared with other neural architecture search methods and previous studies. Compared results reveal the satisfactory performance and superiority of the proposed AdaNAS model in terms of precipitation amount prediction and intensity classification. Concretely, the proposed AdaNAS model outperformed previous best-performing manual methods with MAE and RMSE improving by 80.5\% and 80.3\%, respectively. △ Less

Submitted 4 February, 2024; v1 submitted 26 December, 2023; originally announced December 2023.

KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph

Authors: Tiezheng Guo, Qingwen Yang, Chen Wang, Yanyi Liu, Pan Li, Jiawei Tang, Dapeng Li, Yingyou Wen

Abstract: Large language model (LLM) has achieved outstanding performance on various downstream tasks with its powerful natural language understanding and zero-shot capability, but LLM still suffers from knowledge limitation. Especially in scenarios that require long logical chains or complex reasoning, the hallucination and knowledge limitation of LLM limit its performance in question answering (QA). In th… ▽ More Large language model (LLM) has achieved outstanding performance on various downstream tasks with its powerful natural language understanding and zero-shot capability, but LLM still suffers from knowledge limitation. Especially in scenarios that require long logical chains or complex reasoning, the hallucination and knowledge limitation of LLM limit its performance in question answering (QA). In this paper, we propose a novel framework KnowledgeNavigator to address these challenges by efficiently and accurately retrieving external knowledge from knowledge graph and using it as a key factor to enhance LLM reasoning. Specifically, KnowledgeNavigator first mines and enhances the potential constraints of the given question to guide the reasoning. Then it retrieves and filters external knowledge that supports answering through iterative reasoning on knowledge graph with the guidance of LLM and the question. Finally, KnowledgeNavigator constructs the structured knowledge into effective prompts that are friendly to LLM to help its reasoning. We evaluate KnowledgeNavigator on multiple public KGQA benchmarks, the experiments show the framework has great effectiveness and generalization, outperforming previous knowledge graph enhanced LLM methods and is comparable to the fully supervised models. △ Less

Submitted 19 January, 2024; v1 submitted 25 December, 2023; originally announced December 2023.

Journal ref: Complex & Intelligent Systems (2024): 1-14

Automatic Tooth Arrangement with Joint Features of Point and Mesh Representations via Diffusion Probabilistic Models

Authors: Changsong Lei, Mengfei Xia, Shaofeng Wang, Yaqian Liang, Ran Yi, Yuhui Wen, Yongjin Liu

Abstract: Tooth arrangement is a crucial step in orthodontics treatment, in which aligning teeth could improve overall well-being, enhance facial aesthetics, and boost self-confidence. To improve the efficiency of tooth arrangement and minimize errors associated with unreasonable designs by inexperienced practitioners, some deep learning-based tooth arrangement methods have been proposed. Currently, most ex… ▽ More Tooth arrangement is a crucial step in orthodontics treatment, in which aligning teeth could improve overall well-being, enhance facial aesthetics, and boost self-confidence. To improve the efficiency of tooth arrangement and minimize errors associated with unreasonable designs by inexperienced practitioners, some deep learning-based tooth arrangement methods have been proposed. Currently, most existing approaches employ MLPs to model the nonlinear relationship between tooth features and transformation matrices to achieve tooth arrangement automatically. However, the limited datasets (which to our knowledge, have not been made public) collected from clinical practice constrain the applicability of existing methods, making them inadequate for addressing diverse malocclusion issues. To address this challenge, we propose a general tooth arrangement neural network based on the diffusion probabilistic model. Conditioned on the features extracted from the dental model, the diffusion probabilistic model can learn the distribution of teeth transformation matrices from malocclusion to normal occlusion by gradually denoising from a random variable, thus more adeptly managing real orthodontic data. To take full advantage of effective features, we exploit both mesh and point cloud representations by designing different encoding networks to extract the tooth (local) and jaw (global) features, respectively. In addition to traditional metrics ADD, PA-ADD, CSA, and ME_{rot}, we propose a new evaluation metric based on dental arch curves to judge whether the generated teeth meet the individual normal occlusion. Experimental results demonstrate that our proposed method achieves state-of-the-art tooth alignment results and satisfactory occlusal relationships between dental arches. We will publish the code and dataset. △ Less

Submitted 22 December, 2023; originally announced December 2023.

Critic-Guided Decision Transformer for Offline Reinforcement Learning

Authors: Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, Yu Qiao

Abstract: Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner. However, prevailing RCSL methods largely focus on deterministic trajectory modeling, disregarding stochastic state transitions and the diversity of… ▽ More Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner. However, prevailing RCSL methods largely focus on deterministic trajectory modeling, disregarding stochastic state transitions and the diversity of future trajectory distributions. A fundamental challenge arises from the inconsistency between the sampled returns within individual trajectories and the expected returns across multiple trajectories. Fortunately, value-based methods offer a solution by leveraging a value function to approximate the expected returns, thereby addressing the inconsistency effectively. Building upon these insights, we propose a novel approach, termed the Critic-Guided Decision Transformer (CGDT), which combines the predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer. By incorporating a learned value function, known as the critic, CGDT ensures a direct alignment between the specified target returns and the expected returns of actions. This integration bridges the gap between the deterministic nature of RCSL and the probabilistic characteristics of value-based methods. Empirical evaluations on stochastic environments and D4RL benchmark datasets demonstrate the superiority of CGDT over traditional RCSL methods. These results highlight the potential of CGDT to advance the state of the art in offline RL and extend the applicability of RCSL to a wide range of RL tasks. △ Less

Submitted 21 December, 2023; originally announced December 2023.

Comments: Accepted at AAAI 2024

Free Space Optical Integrated Sensing and Communication Based on DCO-OFDM: Performance Metrics and Resource Allocation

Authors: Yunfeng Wen, Fang Yang, Jian Song, Zhu Han

Abstract: As one of the six usage scenarios of the sixth generation (6G) mobile communication system, integrated sensing and communication (ISAC) has garnered considerable attention, and numerous studies have been conducted on radio-frequency (RF)-ISAC. Benefitting from the communication and sensing capabilities of an optical system, free space optical (FSO)-ISAC becomes a potential complement to RF-ISAC. I… ▽ More As one of the six usage scenarios of the sixth generation (6G) mobile communication system, integrated sensing and communication (ISAC) has garnered considerable attention, and numerous studies have been conducted on radio-frequency (RF)-ISAC. Benefitting from the communication and sensing capabilities of an optical system, free space optical (FSO)-ISAC becomes a potential complement to RF-ISAC. In this paper, a direct-current-biased optical orthogonal frequency division multiplexing (DCO-OFDM) scheme is proposed for FSO-ISAC. To derive the spectral efficiency for communication and the Fisher information for sensing as performance metrics, we model the clipping noise of DCO-OFDM as additive colored Gaussian noise to obtain the expression of the signal-to-noise ratio. Based on the derived performance metrics, joint power allocation problems are formulated for both communication-centric and sensing-centric scenarios. In addition, the non-convex joint optimization problems are decomposed into sub-problems for DC bias and subcarriers, which can be solved by block coordinate descent algorithms. Furthermore, numerical simulations demonstrate the proposed algorithms and reveal the trade-off between communication and sensing functionalities of the OFDM-based FSO-ISAC system. △ Less

Submitted 21 December, 2023; originally announced December 2023.

Comments: 13 pages, 8 figures

Optical Integrated Sensing and Communication: Architectures, Potentials and Challenges

Authors: Yunfeng Wen, Fang Yang, Jian Song, Zhu Han

Abstract: Integrated sensing and communication (ISAC) is viewed as a crucial component of future mobile networks and has gained much interest in both academia and industry. Similar to the emergence of radio-frequency (RF) ISAC, the integration of free space optical communication and optical sensing yields optical ISAC (O-ISAC), which is regarded as a powerful complement to its RF counterpart. In this articl… ▽ More Integrated sensing and communication (ISAC) is viewed as a crucial component of future mobile networks and has gained much interest in both academia and industry. Similar to the emergence of radio-frequency (RF) ISAC, the integration of free space optical communication and optical sensing yields optical ISAC (O-ISAC), which is regarded as a powerful complement to its RF counterpart. In this article, we first introduce the generalized system structure of O-ISAC, and then elaborate on three advantages of O-ISAC, i.e., increasing communication rate, enhancing sensing precision, and reducing interference. Next, waveform design and resource allocation of O-ISAC are discussed based on pulsed waveform, constant-modulus waveform, and multi-carrier waveform. Furthermore, we put forward future trends and challenges of O-ISAC, which are expected to provide some valuable directions for future research. △ Less

Submitted 10 March, 2024; v1 submitted 21 December, 2023; originally announced December 2023.

Comments: 7 pages, 5 figures

Measurements of $Σ$ electromagnetic form factors in the time-like region using the untagged initial-state radiation technique

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (626 additional authors not shown)

Abstract: The process $e^{+}e^{-}\toΣ^{+}\barΣ^{-}$ is studied from threshold up to 3.04 GeV/$c^2$ via the initial-state radiation technique using data with an integrated luminosity of 12.0 fb$^{-1}$, collected at center-of-mass energies between 3.773 and 4.258 GeV with the BESIII detector at the BEPCII collider. The pair production cross sections and the effective form factors of $Σ$ are measured in eleven… ▽ More The process $e^{+}e^{-}\toΣ^{+}\barΣ^{-}$ is studied from threshold up to 3.04 GeV/$c^2$ via the initial-state radiation technique using data with an integrated luminosity of 12.0 fb$^{-1}$, collected at center-of-mass energies between 3.773 and 4.258 GeV with the BESIII detector at the BEPCII collider. The pair production cross sections and the effective form factors of $Σ$ are measured in eleven $Σ^{+}\barΣ^{-}$ invariant mass intervals from threshold to 3.04 GeV/$c^2$. The results are consistent with the previous results from Belle and BESIII. Furthermore, the branching fractions of the decays $J/ψ\toΣ^{+}\barΣ^{-}$ and $ψ(3686)\toΣ^{+}\barΣ^{-}$ are determined and the obtained results are consistent with the previous results of BESIII. △ Less

Submitted 19 December, 2023; originally announced December 2023.

Comments: 13 pages, 6 figures

Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation

Authors: Yuze He, Yushi Bai, Matthieu Lin, Jenny Sheng, Yubin Hu, Qi Wang, Yu-Hui Wen, Yong-Jin Liu

Abstract: By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal… ▽ More By lifting the pre-trained 2D diffusion models into Neural Radiance Fields (NeRFs), text-to-3D generation methods have made great progress. Many state-of-the-art approaches usually apply score distillation sampling (SDS) to optimize the NeRF representations, which supervises the NeRF optimization with pre-trained text-conditioned 2D diffusion models such as Imagen. However, the supervision signal provided by such pre-trained diffusion models only depends on text prompts and does not constrain the multi-view consistency. To inject the cross-view consistency into diffusion priors, some recent works finetune the 2D diffusion model with multi-view data, but still lack fine-grained view coherence. To tackle this challenge, we incorporate multi-view image conditions into the supervision signal of NeRF optimization, which explicitly enforces fine-grained view consistency. With such stronger supervision, our proposed text-to-3D method effectively mitigates the generation of floaters (due to excessive densities) and completely empty spaces (due to insufficient densities). Our quantitative evaluations on the T$^3$Bench dataset demonstrate that our method achieves state-of-the-art performance over existing text-to-3D methods. We will make the code publicly available. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Search for 10--1000 GeV neutrinos from Gamma Ray Bursts with IceCube

Authors: IceCube Collaboration, R. Abbasi, M. Ackermann, J. Adams, S. K. Agarwalla, J. A. Aguilar, M. Ahlers, J. M. Alameddine, N. M. Amin, K. Andeen, G. Anton, C. Argüelles, Y. Ashida, S. Athanasiadou, L. Ausborm, S. N. Axani, X. Bai, A. Balagopal V., M. Baricevic, S. W. Barwick, V. Basu, R. Bay, J. J. Beatty, J. Becker Tjus, J. Beise , et al. (384 additional authors not shown)

Abstract: We present the results of a search for 10--1,000 GeV neutrinos from 2,268 gamma-ray bursts over 8 years of IceCube-DeepCore data. This work probes burst physics below the photosphere where electromagnetic radiation cannot escape. Neutrinos of tens of GeVs are predicted in sub-photospheric collision of free streaming neutrons with bulk-jet protons. In a first analysis, we searched for the most sign… ▽ More We present the results of a search for 10--1,000 GeV neutrinos from 2,268 gamma-ray bursts over 8 years of IceCube-DeepCore data. This work probes burst physics below the photosphere where electromagnetic radiation cannot escape. Neutrinos of tens of GeVs are predicted in sub-photospheric collision of free streaming neutrons with bulk-jet protons. In a first analysis, we searched for the most significant neutrino-GRB coincidence using six overlapping time windows centered on the prompt phase of each GRB. In a second analysis, we conducted a search for a group of GRBs, each individually too weak to be detectable, but potentially significant when combined. No evidence of neutrino emission is found for either analysis. The most significant neutrino coincidence is for Fermi-GBM GRB bn 140807500, with a p-value of 0.097 corrected for all trials. The binomial test used to search for a group of GRBs had a p-value of 0.65 after all trial corrections. The binomial test found a group consisting only of GRB bn 140807500 and no additional GRBs. The neutrino limits of this work complement those obtained by IceCube at TeV to PeV energies. We compare our findings for the large set of GRBs as well as GRB 221009A to the sub-photospheric neutron-proton collision model and find that GRB 221009A provides the most constraining limit on baryon loading. For a jet Lorentz factor of 300 (800), the baryon loading on GRB 221009A is lower than 3.85 (2.13) at a 90% confidence level. △ Less

Submitted 29 July, 2024; v1 submitted 12 December, 2023; originally announced December 2023.

Journal ref: ApJ 964 126 (2024)

DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Driving

Authors: Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, Hao Tian, Lewei Lu, Xizhou Zhu, Xiaogang Wang, Yu Qiao, Jifeng Dai

Abstract: Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge… ▽ More Large language models (LLMs) have opened up new possibilities for intelligent agents, endowing them with human-like thinking and cognitive abilities. In this work, we delve into the potential of large language models (LLMs) in autonomous driving (AD). We introduce DriveMLM, an LLM-based AD framework that can perform close-loop autonomous driving in realistic simulators. To this end, (1) we bridge the gap between the language decisions and the vehicle control commands by standardizing the decision states according to the off-the-shelf motion planning module. (2) We employ a multi-modal LLM (MLLM) to model the behavior planning module of a module AD system, which uses driving rules, user commands, and inputs from various sensors (e.g., camera, lidar) as input and makes driving decisions and provide explanations; This model can plug-and-play in existing AD systems such as Apollo for close-loop driving. (3) We design an effective data engine to collect a dataset that includes decision state and corresponding explanation annotation for model training and evaluation. We conduct extensive experiments and show that our model achieves 76.1 driving score on the CARLA Town05 Long, and surpasses the Apollo baseline by 4.7 points under the same settings, demonstrating the effectiveness of our model. We hope this work can serve as a baseline for autonomous driving with LLMs. Code and models shall be released at https://github.com/OpenGVLab/DriveMLM. △ Less

Submitted 25 December, 2023; v1 submitted 14 December, 2023; originally announced December 2023.

Comments: Technical Report

CartoMark: a benchmark dataset for map pattern recognition and 1 map content retrieval with machine intelligence

Authors: Xiran Zhou, Yi Wen, Honghao Li, Kaiyuan Li, Zhenfeng Shao, Zhigang Yan, Xiao Xie

Abstract: Maps are fundamental medium to visualize and represent the real word in a simple and 16 philosophical way. The emergence of the 3rd wave information has made a proportion of maps are available to be generated ubiquitously, which would significantly enrich the dimensions and perspectives to understand the characteristics of the real world. However, a majority of map dataset have never been discover… ▽ More Maps are fundamental medium to visualize and represent the real word in a simple and 16 philosophical way. The emergence of the 3rd wave information has made a proportion of maps are available to be generated ubiquitously, which would significantly enrich the dimensions and perspectives to understand the characteristics of the real world. However, a majority of map dataset have never been discovered, acquired and effectively used, and the map data used in many applications might not be completely fitted for the authentic demands of these applications. This challenge is emerged due to the lack of numerous well-labelled benchmark datasets for implementing the deep learning approaches into identifying complicated map content. Thus, we develop a large-scale benchmark dataset that includes well-labelled dataset for map text annotation recognition, map scene classification, map super-resolution reconstruction, and map style transferring. Furthermore, these well-labelled datasets would facilitate the state-of-the-art machine intelligence technologies to conduct map feature detection, map pattern recognition and map content retrieval. We hope our efforts would be useful for AI-enhanced cartographical applications. △ Less

Submitted 13 December, 2023; originally announced December 2023.

Measurements of Born Cross Sections for $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2595)^- + {\rm c.c.}$ and $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- + {\rm c.c.}$ at $\sqrt{s}=$4918.0 and 4950.9 MeV

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (620 additional authors not shown)

Abstract: Using $e^+e^-$ collision data collected with the BESIII detector operating at the BEPCII collider, the Born cross sections of $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2595)^- + \rm{c.c.}$ and $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- + \rm{c.c.}$ are measured for the first time at center-of-mass energies of $\sqrt{s}=4918.0$ and 4950.9 MeV. Non-zero cross sections are observed very close to the production threshol… ▽ More Using $e^+e^-$ collision data collected with the BESIII detector operating at the BEPCII collider, the Born cross sections of $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2595)^- + \rm{c.c.}$ and $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- + \rm{c.c.}$ are measured for the first time at center-of-mass energies of $\sqrt{s}=4918.0$ and 4950.9 MeV. Non-zero cross sections are observed very close to the production threshold. The measured Born cross sections of $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- + \rm{c.c.}$ are about $2\sim3$ times greater than those of $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2595)^- + \rm{c.c.}$, thereby indicating that the exotic structure potentially exists in the excited charmed baryons. The Born cross sections are $15.6\pm3.1\pm0.9$ pb and $29.4\pm3.7\pm2.7$ pb for $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2595)^- + \rm{c.c.}$, and are $43.4\pm4.0\pm4.1$ pb and $76.8\pm6.5\pm4.2$ pb for $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- +\rm{c.c.}$ at $\sqrt s=4918.0$ and 4950.9 MeV, respectively. Based on the polar angle distributions of the $\barΛ_{c}(2625)^-$ and $Λ_{c}(2625)^+$, the form-factor ratios $\sqrt{|G_{E}|^2 + 3|G_{M}|^2}/|G_{C}|$ are determined for $e^+e^-\to Λ_{c}^+ \barΛ_{c}(2625)^- + \rm{c.c.}$ for the first time, which are $5.95\pm4.07\pm0.15$ and $0.94\pm0.32\pm0.02$ at $\sqrt s=4918.0$ and 4950.9 MeV, respectively. All of these first uncertainties are statistical and second systematic. △ Less

Submitted 8 May, 2024; v1 submitted 13 December, 2023; originally announced December 2023.

Comments: 10 pages, 6 figures

Journal ref: Phys. Rev. D 109, L071104 (2024)

Encoder-minimal and Decoder-minimal Framework for Remote Sensing Image Dehazing

Authors: Yuanbo Wen, Tao Gao, Ziqi Li, Jing Zhang, Ting Chen

Abstract: Haze obscures remote sensing images, hindering valuable information extraction. To this end, we propose RSHazeNet, an encoder-minimal and decoder-minimal framework for efficient remote sensing image dehazing. Specifically, regarding the process of merging features within the same level, we develop an innovative module called intra-level transposed fusion module (ITFM). This module employs adaptive… ▽ More Haze obscures remote sensing images, hindering valuable information extraction. To this end, we propose RSHazeNet, an encoder-minimal and decoder-minimal framework for efficient remote sensing image dehazing. Specifically, regarding the process of merging features within the same level, we develop an innovative module called intra-level transposed fusion module (ITFM). This module employs adaptive transposed self-attention to capture comprehensive context-aware information, facilitating the robust context-aware feature fusion. Meanwhile, we present a cross-level multi-view interaction module (CMIM) to enable effective interactions between features from various levels, mitigating the loss of information due to the repeated sampling operations. In addition, we propose a multi-view progressive extraction block (MPEB) that partitions the features into four distinct components and employs convolution with varying kernel sizes, groups, and dilation factors to facilitate view-progressive feature learning. Extensive experiments demonstrate the superiority of our proposed RSHazeNet. We release the source code and all pre-trained models at \url{https://github.com/chdwyb/RSHazeNet}. △ Less

Submitted 12 December, 2023; originally announced December 2023.

Algorithms and Complexity for Congested Assignments

Authors: Jiehua Chen, Jiong Guo, Yinghui Wen

Abstract: We study the congested assignment problem as introduced by Bogomolnaia and Moulin (2023). We show that deciding whether a competitive assignment exists can be done in polynomial time, while deciding whether an envy-free assignment exists is NP-complete. We study the congested assignment problem as introduced by Bogomolnaia and Moulin (2023). We show that deciding whether a competitive assignment exists can be done in polynomial time, while deciding whether an envy-free assignment exists is NP-complete. △ Less

Submitted 12 December, 2023; originally announced December 2023.

Batched Low-Rank Adaptation of Foundation Models

Authors: Yeming Wen, Swarat Chaudhuri

Abstract: Low-Rank Adaptation (LoRA) has recently gained attention for fine-tuning foundation models by incorporating trainable low-rank matrices, thereby reducing the number of trainable parameters. While LoRA offers numerous advantages, its applicability for real-time serving to a diverse and global user base is constrained by its incapability to handle multiple task-specific adapters efficiently. This im… ▽ More Low-Rank Adaptation (LoRA) has recently gained attention for fine-tuning foundation models by incorporating trainable low-rank matrices, thereby reducing the number of trainable parameters. While LoRA offers numerous advantages, its applicability for real-time serving to a diverse and global user base is constrained by its incapability to handle multiple task-specific adapters efficiently. This imposes a performance bottleneck in scenarios requiring personalized, task-specific adaptations for each incoming request. To mitigate this constraint, we introduce Fast LoRA (FLoRA), a framework in which each input example in a minibatch can be associated with its unique low-rank adaptation weights, allowing for efficient batching of heterogeneous requests. We empirically demonstrate that FLoRA retains the performance merits of LoRA, showcasing competitive results on the MultiPL-E code generation benchmark spanning over 8 languages and a multilingual speech recognition task across 6 languages. △ Less

Submitted 25 April, 2024; v1 submitted 9 December, 2023; originally announced December 2023.

Comments: 16 pages, 3 figures

All-Sky Search for Transient Astrophysical Neutrino Emission with 10 Years of IceCube Cascade Events

Authors: R. Abbasi, M. Ackermann, J. Adams, S. K. Agarwalla, J. A. Aguilar, M. Ahlers, J. M. Alameddine, N. M. Amin, K. Andeen, G. Anton, C. Argüelles, Y. Ashida, S. Athanasiadou, L. Ausborm, S. N. Axani, X. Bai, A. Balagopal V., M. Baricevic, S. W. Barwick, V. Basu, R. Bay, J. J. Beatty, J. Becker Tjus, J. Beise, C. Bellenghi , et al. (382 additional authors not shown)

Abstract: We present the results of a time-dependent search for neutrino flares in data collected by IceCube between May 2011 and 2021. This data set contains cascade-like events originating from charged-current electron neutrino and tau neutrino interactions and all-flavor neutral-current interactions. IceCube's previous all-sky searches for neutrino flares used data sets consisting of track-like events or… ▽ More We present the results of a time-dependent search for neutrino flares in data collected by IceCube between May 2011 and 2021. This data set contains cascade-like events originating from charged-current electron neutrino and tau neutrino interactions and all-flavor neutral-current interactions. IceCube's previous all-sky searches for neutrino flares used data sets consisting of track-like events originating from charged-current muon neutrino interactions. The cascade data sets are statistically independent of the track data sets and provide a new opportunity to observe the transient all-sky landscape. This search uses the spatial, temporal, and energy information of the cascade-like events to conduct searches for the most statistically significant neutrino flares in the northern and southern skies. No statistically significant time-dependent neutrino emission was observed. For the most statistically significant location in the northern sky, $p_\mathrm{global} =$ 0.71, and in the southern sky, $p_\mathrm{global} =$ 0.51. These results are compatible with the background hypothesis. Assuming an E$^{-2.53}$ spectrum from the diffuse astrophysical neutrino flux as measured with cascades, these results are used to calculate upper limits at the 90\% confidence level on neutrino flares of varying duration and constrain the contribution of these flares to the diffuse astrophysical neutrino flux. These constraints are independent of a specified class of astrophysical objects and show that multiple unresolved transient sources may contribute to the diffuse astrophysical neutrino flux. △ Less

Submitted 11 March, 2024; v1 submitted 8 December, 2023; originally announced December 2023.

Comments: Submitted to The Astrophysical Journal

Amplitude Analysis of the Decays $D^0\toπ^+π^-π^+π^-$ and $π^+π^-π^0π^0$

Authors: BESIII Collaboration, M. Ablikim, M. N. Achasov, P. Adlarson, O. Afedulidis, X. C. Ai, R. Aliberti, A. Amoroso, Q. An, Y. Bai, O. Bakina, I. Balossino, Y. Ban, H. -R. Bao, V. Batozskaya, K. Begzsuren, N. Berger, M. Berlowski, M. Bertani, D. Bettoni, F. Bianchi, E. Bianco, A. Bortone, I. Boyko, R. A. Briere , et al. (620 additional authors not shown)

Abstract: Using $e^+e^-$ annihilation data corresponding to an integrated luminosity of 2.93 $\rm fb^{-1}$ taken at the center-of-mass energy $\sqrt{s}=3.773$~GeV with the BESIII detector, a joint amplitude analysis is performed on the decays $D^0\toπ^+π^-π^+π^-$ and $D^0\toπ^+π^-π^0π^0$(non-$η$). The fit fractions of individual components are obtained, and large interferences among the dominant components… ▽ More Using $e^+e^-$ annihilation data corresponding to an integrated luminosity of 2.93 $\rm fb^{-1}$ taken at the center-of-mass energy $\sqrt{s}=3.773$~GeV with the BESIII detector, a joint amplitude analysis is performed on the decays $D^0\toπ^+π^-π^+π^-$ and $D^0\toπ^+π^-π^0π^0$(non-$η$). The fit fractions of individual components are obtained, and large interferences among the dominant components of $D^{0}\to a_{1}(1260)π$, $D^{0}\toπ(1300)π$, $D^{0}\toρ(770)ρ(770)$ and $D^{0}\to2(ππ)_{S}$ are found in both channels. With the obtained amplitude model, the $CP$-even fractions of $D^0\to π^+π^-π^+π^-$ and $D^0\toπ^+π^-π^0π^0$(non-$η$) are determined to be $(75.2\pm1.1_{\rm stat.}\pm1.5_{\rm syst.})\%$ and $(68.9\pm1.5_{\rm stat.}\pm 2.4_{\rm syst.})\%$, respectively. The branching fractions of $D^0\to π^+π^-π^+π^-$ and $D^0\toπ^+π^-π^0π^0$(non-$η$) are measured to be $(0.688\pm0.010_{\rm stat.}\pm 0.010_{\rm syst.})\%$ and $(0.951\pm0.025_{\rm stat.}\pm 0.021_{\rm syst.})\%$, respectively. The amplitude analysis provides an important model for binning strategy in the measurements of the strong phase parameters of $D^0 \to 4π$ when used to determine the CKM angle $γ(φ_{3})$ via the $B^{-}\to D K^{-}$ decay. △ Less

Submitted 3 April, 2024; v1 submitted 5 December, 2023; originally announced December 2023.

Generative Hierarchical Temporal Transformer for Hand Action Recognition and Motion Prediction

Authors: Yilin Wen, Hao Pan, Takehiko Ohkawa, Lei Yang, Jia Pan, Yoichi Sato, Taku Komura, Wenping Wang

Abstract: We present a novel framework that concurrently tackles hand action recognition and 3D future hand motion prediction. While previous works focus on either recognition or prediction, we propose a generative Transformer VAE architecture to jointly capture both aspects, facilitating realistic motion prediction by leveraging the short-term hand motion and long-term action consistency observed across ti… ▽ More We present a novel framework that concurrently tackles hand action recognition and 3D future hand motion prediction. While previous works focus on either recognition or prediction, we propose a generative Transformer VAE architecture to jointly capture both aspects, facilitating realistic motion prediction by leveraging the short-term hand motion and long-term action consistency observed across timestamps. To ensure faithful representation of the semantic dependency and different temporal granularity of hand pose and action, our framework is decomposed into two cascaded VAE blocks. The lower pose block models short-span poses, while the upper action block models long-span action. These are connected by a mid-level feature that represents sub-second series of hand poses. Our framework is trained across multiple datasets, where pose and action blocks are trained separately to fully utilize pose-action annotations of different qualities. Evaluations show that on multiple datasets, the joint modeling of recognition and prediction improves over separate solutions, and the semantic and temporal hierarchy enables long-term pose and action modeling. △ Less

Submitted 24 December, 2023; v1 submitted 29 November, 2023; originally announced November 2023.

Panacea: Panoramic and Controllable Video Generation for Autonomous Driving

Authors: Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, Xiangyu Zhang

Abstract: The field of autonomous driving increasingly demands high-quality annotated training data. In this paper, we propose Panacea, an innovative approach to generate panoramic and controllable videos in driving scenarios, capable of yielding an unlimited numbers of diverse, annotated samples pivotal for autonomous driving advancements. Panacea addresses two critical challenges: 'Consistency' and 'Contr… ▽ More The field of autonomous driving increasingly demands high-quality annotated training data. In this paper, we propose Panacea, an innovative approach to generate panoramic and controllable videos in driving scenarios, capable of yielding an unlimited numbers of diverse, annotated samples pivotal for autonomous driving advancements. Panacea addresses two critical challenges: 'Consistency' and 'Controllability.' Consistency ensures temporal and cross-view coherence, while Controllability ensures the alignment of generated content with corresponding annotations. Our approach integrates a novel 4D attention and a two-stage generation pipeline to maintain coherence, supplemented by the ControlNet framework for meticulous control by the Bird's-Eye-View (BEV) layouts. Extensive qualitative and quantitative evaluations of Panacea on the nuScenes dataset prove its effectiveness in generating high-quality multi-view driving-scene videos. This work notably propels the field of autonomous driving by effectively augmenting the training dataset used for advanced BEV perception techniques. △ Less

Submitted 28 November, 2023; originally announced November 2023.

Comments: Project page: https://panacea-ad.github.io/

Efficient Encoding of Graphics Primitives with Simplex-based Structures

Authors: Yibo Wen, Yunfan Yang

Abstract: Grid-based structures are commonly used to encode explicit features for graphics primitives such as images, signed distance functions (SDF), and neural radiance fields (NeRF) due to their simple implementation. However, in $n$-dimensional space, calculating the value of a sampled point requires interpolating the values of its $2^n$ neighboring vertices. The exponential scaling with dimension leads… ▽ More Grid-based structures are commonly used to encode explicit features for graphics primitives such as images, signed distance functions (SDF), and neural radiance fields (NeRF) due to their simple implementation. However, in $n$-dimensional space, calculating the value of a sampled point requires interpolating the values of its $2^n$ neighboring vertices. The exponential scaling with dimension leads to significant computational overheads. To address this issue, we propose a simplex-based approach for encoding graphics primitives. The number of vertices in a simplex-based structure increases linearly with dimension, making it a more efficient and generalizable alternative to grid-based representations. Using the non-axis-aligned simplicial structure property, we derive and prove a coordinate transformation, simplicial subdivision, and barycentric interpolation scheme for efficient sampling, which resembles transformation procedures in the simplex noise algorithm. Finally, we use hash tables to store multiresolution features of all interest points in the simplicial grid, which are passed into a tiny fully connected neural network to parameterize graphics primitives. We implemented a detailed simplex-based structure encoding algorithm in C++ and CUDA using the methods outlined in our approach. In the 2D image fitting task, the proposed method is capable of fitting a giga-pixel image with 9.4% less time compared to the baseline method proposed by instant-ngp, while maintaining the same quality and compression rate. In the volumetric rendering setup, we observe a maximum 41.2% speedup when the samples are dense enough. △ Less

Submitted 26 November, 2023; originally announced November 2023.

Comments: 10 pages, 8 figures

Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach

Authors: Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, Lijuan Li, Guoliang Fan

Abstract: The remarkable progress in Large Language Models (LLMs) opens up new avenues for addressing planning and decision-making problems in Multi-Agent Systems (MAS). However, as the number of agents increases, the issues of hallucination in LLMs and coordination in MAS have become increasingly prominent. Additionally, the efficient utilization of tokens emerges as a critical consideration when employing… ▽ More The remarkable progress in Large Language Models (LLMs) opens up new avenues for addressing planning and decision-making problems in Multi-Agent Systems (MAS). However, as the number of agents increases, the issues of hallucination in LLMs and coordination in MAS have become increasingly prominent. Additionally, the efficient utilization of tokens emerges as a critical consideration when employing LLMs to facilitate the interactions among a substantial number of agents. In this paper, we develop a modular framework called LLaMAC to mitigate these challenges. LLaMAC implements a value distribution encoding similar to that found in the human brain, utilizing internal and external feedback mechanisms to facilitate collaboration and iterative reasoning among its modules. Through evaluations involving system resource allocation and robot grid transportation, we demonstrate the considerable advantages afforded by our proposed approach. △ Less

Submitted 23 January, 2024; v1 submitted 23 November, 2023; originally announced November 2023.

Comments: 13 pages, 11 figures

ADriver-I: A General World Model for Autonomous Driving

Authors: Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, Tiancai Wang

Abstract: Typically, autonomous driving adopts a modular design, which divides the full stack into perception, prediction, planning and control parts. Though interpretable, such modular design tends to introduce a substantial amount of redundancy. Recently, multimodal large language models (MLLM) and diffusion techniques have demonstrated their superior performance on comprehension and generation ability. I… ▽ More Typically, autonomous driving adopts a modular design, which divides the full stack into perception, prediction, planning and control parts. Though interpretable, such modular design tends to introduce a substantial amount of redundancy. Recently, multimodal large language models (MLLM) and diffusion techniques have demonstrated their superior performance on comprehension and generation ability. In this paper, we first introduce the concept of interleaved vision-action pair, which unifies the format of visual features and control signals. Based on the vision-action pairs, we construct a general world model based on MLLM and diffusion model for autonomous driving, termed ADriver-I. It takes the vision-action pairs as inputs and autoregressively predicts the control signal of the current frame. The generated control signals together with the historical vision-action pairs are further conditioned to predict the future frames. With the predicted next frame, ADriver-I performs further control signal prediction. Such a process can be repeated infinite times, ADriver-I achieves autonomous driving in the world created by itself. Extensive experiments are conducted on nuScenes and our large-scale private datasets. ADriver-I shows impressive performance compared to several constructed baselines. We hope our ADriver-I can provide some new insights for future autonomous driving and embodied intelligence. △ Less

Submitted 22 November, 2023; originally announced November 2023.

Comments: Tech Report

GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning

Authors: Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen, Shifeng Chen

Abstract: Recent advances in text-to-video generation have harnessed the power of diffusion models to create visually compelling content conditioned on text prompts. However, they usually encounter high computational costs and often struggle to produce videos with coherent physical motions. To tackle these issues, we propose GPT4Motion, a training-free framework that leverages the planning capability of lar… ▽ More Recent advances in text-to-video generation have harnessed the power of diffusion models to create visually compelling content conditioned on text prompts. However, they usually encounter high computational costs and often struggle to produce videos with coherent physical motions. To tackle these issues, we propose GPT4Motion, a training-free framework that leverages the planning capability of large language models such as GPT, the physical simulation strength of Blender, and the excellent image generation ability of text-to-image diffusion models to enhance the quality of video synthesis. Specifically, GPT4Motion employs GPT-4 to generate a Blender script based on a user textual prompt, which commands Blender's built-in physics engine to craft fundamental scene components that encapsulate coherent physical motions across frames. Then these components are inputted into Stable Diffusion to generate a video aligned with the textual prompt. Experimental results on three basic physical motion scenarios, including rigid object drop and collision, cloth draping and swinging, and liquid flow, demonstrate that GPT4Motion can generate high-quality videos efficiently in maintaining motion coherency and entity consistency. GPT4Motion offers new insights in text-to-video research, enhancing its quality and broadening its horizon for further explorations. △ Less

Submitted 23 April, 2024; v1 submitted 21 November, 2023; originally announced November 2023.

EDMSound: Spectrogram Based Diffusion Models for Efficient and High-Quality Audio Synthesis

Authors: Ge Zhu, Yutong Wen, Marc-André Carbonneau, Zhiyao Duan

Abstract: Audio diffusion models can synthesize a wide variety of sounds. Existing models often operate on the latent domain with cascaded phase recovery modules to reconstruct waveform. This poses challenges when generating high-fidelity audio. In this paper, we propose EDMSound, a diffusion-based generative model in spectrogram domain under the framework of elucidated diffusion models (EDM). Combining wit… ▽ More Audio diffusion models can synthesize a wide variety of sounds. Existing models often operate on the latent domain with cascaded phase recovery modules to reconstruct waveform. This poses challenges when generating high-fidelity audio. In this paper, we propose EDMSound, a diffusion-based generative model in spectrogram domain under the framework of elucidated diffusion models (EDM). Combining with efficient deterministic sampler, we achieved similar Fréchet audio distance (FAD) score as top-ranked baseline with only 10 steps and reached state-of-the-art performance with 50 steps on the DCASE2023 foley sound generation benchmark. We also revealed a potential concern regarding diffusion based audio generation models that they tend to generate samples with high perceptual similarity to the data from training data. Project page: https://agentcooper2002.github.io/EDMSound/ △ Less

Submitted 18 November, 2023; v1 submitted 14 November, 2023; originally announced November 2023.

Comments: Accepted at NeurIPS Workshop: Machine Learning for Audio (Camera Ready)

Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization

Authors: Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, Bernhard Schölkopf

Abstract: Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly larg… ▽ More Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language. △ Less

Submitted 28 April, 2024; v1 submitted 10 November, 2023; originally announced November 2023.

Comments: ICLR 2024 (v2: 34 pages, 19 figures)

Emergent Communication for Rules Reasoning

Authors: Yuxuan Guo, Yifan Hao, Rui Zhang, Enshuai Zhou, Zidong Du, Xishan Zhang, Xinkai Song, Yuanbo Wen, Yongwei Zhao, Xuehai Zhou, Jiaming Guo, Qi Yi, Shaohui Peng, Di Huang, Ruizhi Chen, Qi Guo, Yunji Chen

Abstract: Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forces agents to describe low-level perceptual features intra image or symbol contexts. In this work, in… ▽ More Research on emergent communication between deep-learning-based agents has received extensive attention due to its inspiration for linguistics and artificial intelligence. However, previous attempts have hovered around emerging communication under perception-oriented environmental settings, that forces agents to describe low-level perceptual features intra image or symbol contexts. In this work, inspired by the classic human reasoning test (namely Raven's Progressive Matrix), we propose the Reasoning Game, a cognition-oriented environment that encourages agents to reason and communicate high-level rules, rather than perceived low-level contexts. Moreover, we propose 1) an unbiased dataset (namely rule-RAVEN) as a benchmark to avoid overfitting, 2) and a two-stage curriculum agent training method as a baseline for more stable convergence in the Reasoning Game, where contexts and semantics are bilaterally drifting. Experimental results show that, in the Reasoning Game, a semantically stable and compositional language emerges to solve reasoning problems. The emerged language helps agents apply the extracted rules to the generalization of unseen context attributes, and to the transfer between different context attributes or even tasks. △ Less

Submitted 8 November, 2023; originally announced November 2023.

Hierarchical Bayesian Inference of Globular Cluster Properties

Authors: Robin Y. Wen, Joshua S. Speagle, Jeremy J. Webb, Gwendolyn M. Eadie

Abstract: We present a hierarchical Bayesian inference approach to estimating the structural properties and the phase space center of a globular cluster (GC) given the spatial and kinematic information of its stars based on lowered isothermal cluster models. As a first step towards more realistic modelling of GCs, we built a differentiable, accurate emulator of the lowered isothermal distribution function u… ▽ More We present a hierarchical Bayesian inference approach to estimating the structural properties and the phase space center of a globular cluster (GC) given the spatial and kinematic information of its stars based on lowered isothermal cluster models. As a first step towards more realistic modelling of GCs, we built a differentiable, accurate emulator of the lowered isothermal distribution function using interpolation. The reliable gradient information provided by the emulator allows the use of Hamiltonian Monte Carlo methods to sample large Bayesian models with hundreds of parameters, thereby enabling inference on hierarchical models. We explore the use of hierarchical Bayesian modelling to address several issues encountered in observations of GC including an unknown GC center, incomplete data, and measurement errors. Our approach not only avoids the common technique of radial binning but also incorporates the aforementioned uncertainties in a robust and statistically consistent way. Through demonstrating the reliability of our hierarchical Bayesian model on simulations, our work lays out the foundation for more realistic and complex modelling of real GC data. △ Less

Submitted 6 November, 2023; originally announced November 2023.

Comments: 16 pages, 12 figures, and 2 tables

A cosmic glitch in gravity

Authors: Robin Y. Wen, Lukas T. Hergt, Niayesh Afshordi, Douglas Scott

Abstract: We investigate a model that modifies general relativity on cosmological scales, specifically by having a `glitch' in the gravitational constant between the cosmological (super-horizon) and Newtonian (sub-horizon) regimes, as motivated e.g. in the Hořava-Lifshitz proposal or in the Einstein-aether framework. This gives a single-parameter extension to the standard $Λ$CDM model, which is equivalent t… ▽ More We investigate a model that modifies general relativity on cosmological scales, specifically by having a `glitch' in the gravitational constant between the cosmological (super-horizon) and Newtonian (sub-horizon) regimes, as motivated e.g. in the Hořava-Lifshitz proposal or in the Einstein-aether framework. This gives a single-parameter extension to the standard $Λ$CDM model, which is equivalent to adding a dark energy component, but where the energy density of this component can have either sign. Fitting to data from the Planck satellite, we find that negative contributions are, in fact, preferred. Additionally, we find that roughly one percent weaker superhorizon gravity can somewhat ease the Hubble and clustering tensions in a range of cosmological observations, although at the expense of spoiling fits to the baryonic acoustic oscillation scale in galaxy surveys. Therefore, the extra parametric freedom offered by our model deserves further exploration, and we discuss how future observations may elucidate this potential cosmic glitch in gravity, through a four-fold reduction in statistical uncertainties. △ Less

Submitted 17 May, 2024; v1 submitted 6 November, 2023; originally announced November 2023.

Comments: 20 pages, 10 figures, 4 tables. Revised to match version accepted by JCAP

Journal ref: JCAP03(2024)045

AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

Authors: Qiaoling Chen, Qinghao Hu, Guoteng Wang, Yingtong Xiong, Ting Huang, Xun Chen, Yang Gao, Hang Yan, Yonggang Wen, Tianwei Zhang, Peng Sun

Abstract: Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorp… ▽ More Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorporates three flexible sharding strategies: Full-Replica, Full-Sharding, and Partial-Sharding, and allows each component within the model states (Parameters, Gradients, Optimizer States) to independently choose a sharding strategy as well as the device mesh. We conduct a thorough analysis of communication costs, formulating an optimization problem to discover the optimal sharding strategy. Additionally, AMSP optimizes distributed LLM training by efficiently overlapping communication with computation. Evaluations demonstrate up to 52\% Model FLOPs Utilization (MFU) when training the LLaMA-based model on 1024 GPUs, resulting in a 1.56 times improvement in training throughput compared to newly proposed systems like MiCS and ZeRO++. △ Less

Submitted 13 March, 2024; v1 submitted 31 October, 2023; originally announced November 2023.

Pairwise Similarity Learning is SimPLE

Authors: Yandong Wen, Weiyang Liu, Yao Feng, Bhiksha Raj, Rita Singh, Adrian Weller, Michael J. Black, Bernhard Schölkopf

Abstract: In this paper, we focus on a general yet important learning problem, pairwise similarity learning (PSL). PSL subsumes a wide range of important applications, such as open-set face recognition, speaker verification, image retrieval and person re-identification. The goal of PSL is to learn a pairwise similarity function assigning a higher similarity score to positive pairs (i.e., a pair of samples w… ▽ More In this paper, we focus on a general yet important learning problem, pairwise similarity learning (PSL). PSL subsumes a wide range of important applications, such as open-set face recognition, speaker verification, image retrieval and person re-identification. The goal of PSL is to learn a pairwise similarity function assigning a higher similarity score to positive pairs (i.e., a pair of samples with the same label) than to negative pairs (i.e., a pair of samples with different label). We start by identifying a key desideratum for PSL, and then discuss how existing methods can achieve this desideratum. We then propose a surprisingly simple proxy-free method, called SimPLE, which requires neither feature/proxy normalization nor angular margin and yet is able to generalize well in open-set recognition. We apply the proposed method to three challenging PSL tasks: open-set face recognition, image retrieval and speaker verification. Comprehensive experimental results on large-scale benchmarks show that our method performs significantly better than current state-of-the-art methods. △ Less

Submitted 13 October, 2023; originally announced October 2023.

Comments: Published in ICCV 2023 (Project page: https://simple.is.tue.mpg.de/)

`Maser-in-a-Shoebox': a portable plug-and-play maser device at room-temperature and zero magnetic-field

Authors: Wern Ng, Yongqiang Wen, Max Attwood, Daniel C Jones, Mark Oxborrow, Neil McN. Alford, Daan M. Arroo

Abstract: Masers, the microwave analogues of lasers, have seen a renaissance owing to the discovery of gain media that mase at room-temperature and zero-applied magnetic field. However, despite the ease with which the devices can be demonstrated under ambient conditions, achieving the ubiquity and portability which lasers enjoy has to date remained challenging. We present a maser device with a miniaturized… ▽ More Masers, the microwave analogues of lasers, have seen a renaissance owing to the discovery of gain media that mase at room-temperature and zero-applied magnetic field. However, despite the ease with which the devices can be demonstrated under ambient conditions, achieving the ubiquity and portability which lasers enjoy has to date remained challenging. We present a maser device with a miniaturized maser cavity, gain material and laser pump source that fits within the size of a shoebox. The gain medium used is pentacene-doped in para-terphenyl and it is shown to give a strong masing signal with a peak power of -5 dBm even within a smaller form factor. The device is also shown to mase at different frequencies within a small range of 1.5 MHz away from the resonant frequency. The portability and simplicity of the device, which weighs under 5 kg, paves the way for demonstrators particularly in the areas of low-noise amplifiers, quantum sensors, cavity quantum electrodynamics and long-range communications. △ Less

Submitted 13 October, 2023; originally announced October 2023.

Journal ref: Appl. Phys. Lett. 124, 044004 (2024)

High-speed photonic crystal modulator with non-volatile memory via structurally-engineered strain concentration in a piezo-MEMS platform

Authors: Y. Henry Wen, David Heim, Matthew Zimmermann, Roman A. Shugayev, Mark Dong, Andrew J. Leenheer, Gerald Gilbert, Matt Eichenfield, Mikkel Heuck, Dirk R. Englund

Abstract: Numerous applications in quantum and classical optics require scalable, high-speed modulators that cover visible-NIR wavelengths with low footprint, drive voltage (V) and power dissipation. A critical figure of merit for electro-optic (EO) modulators is the transmission change per voltage, dT/dV. Conventional approaches in wave-guided modulators seek to maximize dT/dV by the selection of a high EO… ▽ More Numerous applications in quantum and classical optics require scalable, high-speed modulators that cover visible-NIR wavelengths with low footprint, drive voltage (V) and power dissipation. A critical figure of merit for electro-optic (EO) modulators is the transmission change per voltage, dT/dV. Conventional approaches in wave-guided modulators seek to maximize dT/dV by the selection of a high EO coefficient or a longer light-material interaction, but are ultimately limited by nonlinear material properties and material losses, respectively. Optical and RF resonances can improve dT/dV, but introduce added challenges in terms of speed and spectral tuning, especially for high-Q photonic cavity resonances. Here, we introduce a cavity-based EO modulator to solve both trade-offs in a piezo-strained photonic crystal cavity. Our approach concentrates the displacement of a piezo-electric actuator of length L and a given piezoelectric coefficient into the PhCC, resulting in dT/dV proportional to L under fixed material loss. Secondly, we employ a material deformation that is programmable under a "read-write" protocol with a continuous, repeatable tuning range of 5 GHz and a maximum non-volatile excursion of 8 GHz. In telecom-band demonstrations, we measure a fundamental mode linewidth = 5.4 GHz, with voltage response 177 MHz/V corresponding to 40 GHz for voltage spanning -120 to 120 V, 3dB-modulation bandwidth of 3.2 MHz broadband DC-AC, and 142 MHz for resonant operation near 2.8 GHz operation, optical extinction down to min(log(T)) = -25 dB via Michelson-type interference, and an energy consumption down to 0.17 nW/GHz. The strain-enhancement methods presented here are applicable to study and control other strain-sensitive systems. △ Less

Submitted 13 October, 2023; v1 submitted 11 October, 2023; originally announced October 2023.

NEFTune: Noisy Embeddings Improve Instruction Finetuning

Authors: Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, Tom Goldstein

Abstract: We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instru… ▽ More We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune. △ Less

Submitted 10 October, 2023; v1 submitted 9 October, 2023; originally announced October 2023.

Comments: 25 pages, Code is available on Github: https://github.com/neelsjain/NEFTune