-
Generalized flow-composed symplectic methods for post-Newtonian Hamiltonian systems
Authors:
Shixiang Huang,
Kaiming Zeng,
Xinghua Niu,
Lijie Mei
Abstract:
Due to the nonseparability of the post-Newtonian (PN) Hamiltonian systems of compact objects, the symplectic methods that admit the linear error growth and the near preservation of first integrals are always implicit as explicit symplectic methods have not been currently found for general nonseparable Hamiltonian systems. Since the PN Hamiltonian has a particular formulation that includes a domina…
▽ More
Due to the nonseparability of the post-Newtonian (PN) Hamiltonian systems of compact objects, the symplectic methods that admit the linear error growth and the near preservation of first integrals are always implicit as explicit symplectic methods have not been currently found for general nonseparable Hamiltonian systems. Since the PN Hamiltonian has a particular formulation that includes a dominant Newtonian part and a perturbation PN part, we present the generalized flow-composed Runge--Kutta (GFCRK) method with a free parameter $λ$ to PN Hamiltonian systems. It is shown that the GFCRK method is symplectic once the underlying RK method is symplectic, and it is symmetric once the underlying RK method is symmetric under the setting $λ=1/2$. Numerical experiments with the 2PN Hamiltonian of spinning compact binaries demonstrate the higher accuracy and efficiency of the symplectic GFCRK method than the underlying symplectic RK. Meanwhile, the numerical results also support higher efficiency of the symplectic GFCRK method than the semi-explicit mixed symplectic method of the same order.
△ Less
Submitted 12 August, 2024;
originally announced August 2024.
-
Text-Augmented Multimodal LLMs for Chemical Reaction Condition Recommendation
Authors:
Yu Zhang,
Ruijie Yu,
Kaipeng Zeng,
Ding Li,
Feng Zhu,
Xiaokang Yang,
Yaohui Jin,
Yanyan Xu
Abstract:
High-throughput reaction condition (RC) screening is fundamental to chemical synthesis. However, current RC screening suffers from laborious and costly trial-and-error workflows. Traditional computer-aided synthesis planning (CASP) tools fail to find suitable RCs due to data sparsity and inadequate reaction representations. Nowadays, large language models (LLMs) are capable of tackling chemistry-r…
▽ More
High-throughput reaction condition (RC) screening is fundamental to chemical synthesis. However, current RC screening suffers from laborious and costly trial-and-error workflows. Traditional computer-aided synthesis planning (CASP) tools fail to find suitable RCs due to data sparsity and inadequate reaction representations. Nowadays, large language models (LLMs) are capable of tackling chemistry-related problems, such as molecule design, and chemical logic Q\&A tasks. However, LLMs have not yet achieved accurate predictions of chemical reaction conditions. Here, we present MM-RCR, a text-augmented multimodal LLM that learns a unified reaction representation from SMILES, reaction graphs, and textual corpus for chemical reaction recommendation (RCR). To train MM-RCR, we construct 1.2 million pair-wised Q\&A instruction datasets. Our experimental results demonstrate that MM-RCR achieves state-of-the-art performance on two open benchmark datasets and exhibits strong generalization capabilities on out-of-domain (OOD) and High-Throughput Experimentation (HTE) datasets. MM-RCR has the potential to accelerate high-throughput condition screening in chemical synthesis.
△ Less
Submitted 21 July, 2024;
originally announced July 2024.
-
Schatten classes on noncommutative tori: Kernel conditions
Authors:
Michael Ruzhansky,
Kai Zeng
Abstract:
In this note, we give criteria on noncommutative integral kernels ensuring that integral operators on quantum torus belong to Schatten classes. With the engagement of a noncommutative Schwartz' kernel theorem on the quantum torus, a specific test for Schatten class properties of bounded operators on the quantum torus is established.
In this note, we give criteria on noncommutative integral kernels ensuring that integral operators on quantum torus belong to Schatten classes. With the engagement of a noncommutative Schwartz' kernel theorem on the quantum torus, a specific test for Schatten class properties of bounded operators on the quantum torus is established.
△ Less
Submitted 22 July, 2024; v1 submitted 12 July, 2024;
originally announced July 2024.
-
LLMAEL: Large Language Models are Good Context Augmenters for Entity Linking
Authors:
Amy Xin,
Yunjia Qi,
Zijun Yao,
Fangwei Zhu,
Kaisheng Zeng,
Xu Bin,
Lei Hou,
Juanzi Li
Abstract:
Entity Linking (EL) models are well-trained at mapping mentions to their corresponding entities according to a given context. However, EL models struggle to disambiguate long-tail entities due to their limited training data. Meanwhile, large language models (LLMs) are more robust at interpreting uncommon mentions. Yet, due to a lack of specialized training, LLMs suffer at generating correct entity…
▽ More
Entity Linking (EL) models are well-trained at mapping mentions to their corresponding entities according to a given context. However, EL models struggle to disambiguate long-tail entities due to their limited training data. Meanwhile, large language models (LLMs) are more robust at interpreting uncommon mentions. Yet, due to a lack of specialized training, LLMs suffer at generating correct entity IDs. Furthermore, training an LLM to perform EL is cost-intensive. Building upon these insights, we introduce LLM-Augmented Entity Linking LLMAEL, a plug-and-play approach to enhance entity linking through LLM data augmentation. We leverage LLMs as knowledgeable context augmenters, generating mention-centered descriptions as additional input, while preserving traditional EL models for task specific processing. Experiments on 6 standard datasets show that the vanilla LLMAEL outperforms baseline EL models in most cases, while the fine-tuned LLMAEL set the new state-of-the-art results across all 6 benchmarks.
△ Less
Submitted 15 July, 2024; v1 submitted 4 July, 2024;
originally announced July 2024.
-
PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
Authors:
Kuo-Hao Zeng,
Zichen Zhang,
Kiana Ehsani,
Rose Hendrix,
Jordi Salvador,
Alvaro Herrasti,
Ross Girshick,
Aniruddha Kembhavi,
Luca Weihs
Abstract:
We present PoliFormer (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that generalizes to the real-world without adaptation despite being trained purely in simulation. PoliFormer uses a foundational vision transformer encoder with a causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of mil…
▽ More
We present PoliFormer (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that generalizes to the real-world without adaptation despite being trained purely in simulation. PoliFormer uses a foundational vision transformer encoder with a causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of millions of interactions across diverse environments, leveraging parallelized, multi-machine rollouts for efficient training with high throughput. PoliFormer is a masterful navigator, producing state-of-the-art results across two distinct embodiments, the LoCoBot and Stretch RE-1 robots, and four navigation benchmarks. It breaks through the plateaus of previous work, achieving an unprecedented 85.5% success rate in object goal navigation on the CHORES-S benchmark, a 28.5% absolute improvement. PoliFormer can also be trivially extended to a variety of downstream applications such as object tracking, multi-object navigation, and open-vocabulary navigation with no finetuning.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
Take a Step Further: Understanding Page Spray in Linux Kernel Exploitation
Authors:
Ziyi Guo,
Dang K Le,
Zhenpeng Lin,
Kyle Zeng,
Ruoyu Wang,
Tiffany Bao,
Yan Shoshitaishvili,
Adam Doupé,
Xinyu Xing
Abstract:
Recently, a novel method known as Page Spray emerges, focusing on page-level exploitation for kernel vulnerabilities. Despite the advantages it offers in terms of exploitability, stability, and compatibility, comprehensive research on Page Spray remains scarce. Questions regarding its root causes, exploitation model, comparative benefits over other exploitation techniques, and possible mitigation…
▽ More
Recently, a novel method known as Page Spray emerges, focusing on page-level exploitation for kernel vulnerabilities. Despite the advantages it offers in terms of exploitability, stability, and compatibility, comprehensive research on Page Spray remains scarce. Questions regarding its root causes, exploitation model, comparative benefits over other exploitation techniques, and possible mitigation strategies have largely remained unanswered. In this paper, we conduct a systematic investigation into Page Spray, providing an in-depth understanding of this exploitation technique. We introduce a comprehensive exploit model termed the \sys model, elucidating its fundamental principles. Additionally, we conduct a thorough analysis of the root causes underlying Page Spray occurrences within the Linux Kernel. We design an analyzer based on the Page Spray analysis model to identify Page Spray callsites. Subsequently, we evaluate the stability, exploitability, and compatibility of Page Spray through meticulously designed experiments. Finally, we propose mitigation principles for addressing Page Spray and introduce our own lightweight mitigation approach. This research aims to assist security researchers and developers in gaining insights into Page Spray, ultimately enhancing our collective understanding of this emerging exploitation technique and making improvements to the community.
△ Less
Submitted 6 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs
Authors:
Haoyu Wang,
Bei Liu,
Hang Shao,
Bo Xiao,
Ke Zeng,
Guanglu Wan,
Yanmin Qian
Abstract:
Parameter quantization for Large Language Models (LLMs) has attracted increasing attentions recently in reducing memory costs and improving computational efficiency. Early approaches have been widely adopted. However, the existing methods suffer from poor performance in low-bit (such as 2 to 3 bits) scenarios. In this paper, we present a novel and effective Column-Level Adaptive weight Quantizatio…
▽ More
Parameter quantization for Large Language Models (LLMs) has attracted increasing attentions recently in reducing memory costs and improving computational efficiency. Early approaches have been widely adopted. However, the existing methods suffer from poor performance in low-bit (such as 2 to 3 bits) scenarios. In this paper, we present a novel and effective Column-Level Adaptive weight Quantization (CLAQ) framework by introducing three different types of adaptive strategies for LLM quantization. Firstly, a K-Means clustering based algorithm is proposed that allows dynamic generation of quantization centroids for each column of a parameter matrix. Secondly, we design an outlier-guided adaptive precision search strategy which can dynamically assign varying bit-widths to different columns. Finally, a dynamic outlier reservation scheme is developed to retain some parameters in their original float point precision, in trade off of boosted model performance. Experiments on various mainstream open source LLMs including LLaMA-1, LLaMA-2 and Yi demonstrate that our methods achieve the state-of-the-art results across different bit settings, especially in extremely low-bit scenarios. Code is available at https://github.com/fayuge/CLAQ.
△ Less
Submitted 2 June, 2024; v1 submitted 27 May, 2024;
originally announced May 2024.
-
SN 2023zaw: the low-energy explosion of an ultra-stripped star, with non-radioactive heating
Authors:
Thomas Moore,
James Gillanders,
Matt Nicholl,
Mark Huber,
Stephen Smartt,
Shubham Srivastav,
Heloise Stevance,
Ting-Wan Chen,
Kenneth Chambers,
Joseph Anderson,
Michael Fulton,
Samantha Oates,
Charlotte Angus,
Giuliano Pignata,
Nicolas Erasmus,
Hua Gao,
Joanna Bulger,
Chien-Cheng Lin,
Thomas Lowe,
Eugene Magnier,
Paloma Minguez,
Chow-Choong Ngeow,
Xinyue Sheng,
Stuart A. Sim,
Ken Smith
, et al. (4 additional authors not shown)
Abstract:
Most stripped envelope supernova progenitors are formed through binary interaction, losing hydrogen and/or helium from their outer layers. An emerging class of supernovae with the highest degree of envelope-stripping are thought to be the product of stripping by a NS companion. However, relatively few examples are known and the outcomes of such systems can be diverse and are poorly understood at p…
▽ More
Most stripped envelope supernova progenitors are formed through binary interaction, losing hydrogen and/or helium from their outer layers. An emerging class of supernovae with the highest degree of envelope-stripping are thought to be the product of stripping by a NS companion. However, relatively few examples are known and the outcomes of such systems can be diverse and are poorly understood at present. Here, we present spectroscopic observations and high cadence multi-band photometry of SN 2023zaw, a low ejecta mass and rapidly evolving supernova. SN 2023zaw was discovered in a nearby spiral galaxy at D = 39.7 Mpc, with significant Milky Way extinction, $E(B-V) = 0.21$, and significant (but uncertain) host extinction. Bayesian evidence comparison reveals that nickel is not the only power source and an additional energy source is required to explain our observations. Our models suggest an ejecta mass of $M_{\rm ej} \sim 0.07\,\rm M_\odot$ and a synthesised nickel mass of $M_{\rm ej} \sim 0.007\,\rm M_\odot$ is required to explain the explosion. However an additional heating from a magnetar or interaction with circumstellar material is required to power the early light curve.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
Swipe2Pair: Secure and Fast In-Band Wireless Device Pairing
Authors:
Yaqi He,
Kai Zeng,
Long Jiao,
Brian L. Mark,
Khaled N. Khasawneh
Abstract:
Wireless device pairing is a critical security mechanism to bootstrap the secure communication between two devices without a pre-shared secret. It has been widely used in many Internet of Things (IoT) applications, such as smart-home and smart-health. Most existing device pairing mechanisms are based on out-of-band channels, e.g., extra sensors or hardware, to validate the proximity of pairing dev…
▽ More
Wireless device pairing is a critical security mechanism to bootstrap the secure communication between two devices without a pre-shared secret. It has been widely used in many Internet of Things (IoT) applications, such as smart-home and smart-health. Most existing device pairing mechanisms are based on out-of-band channels, e.g., extra sensors or hardware, to validate the proximity of pairing devices. However, out-of-band channels are not universal across all wireless devices, so such a scheme is limited to certain application scenarios or conditions. On the other hand, in-band channel-based device pairing seeks universal applicability by only relying on wireless interfaces. Existing in-band channel-based pairing schemes either require multiple antennas separated by a good distance on one pairing device, which is not feasible in certain scenarios, or require users to repeat multiple sweeps, which is not optimal in terms of usability.
Therefore, an in-band wireless device pairing scheme providing high security while maintaining high usability (simple pairing process and minimal user intervention) is highly desired. In this work, we propose an easy-to-use mutual authentication device pairing scheme, named Swipe2Pair, based on the proximity of pairing devices and randomization of wireless transmission power. We conduct extensive security analysis and collect considerable experimental data under various settings across different environments. Experimental results show that Swipe2Pair achieves high security and usability. It only takes less than one second to complete the pairing process with a simple swipe of one device in front of the other.
△ Less
Submitted 5 May, 2024;
originally announced May 2024.
-
MambaMOS: LiDAR-based 3D Moving Object Segmentation with Motion-aware State Space Model
Authors:
Kang Zeng,
Hao Shi,
Jiacheng Lin,
Siyu Li,
Jintao Cheng,
Kaiwei Wang,
Zhiyong Li,
Kailun Yang
Abstract:
LiDAR-based Moving Object Segmentation (MOS) aims to locate and segment moving objects in point clouds of the current scan using motion information from previous scans. Despite the promising results achieved by previous MOS methods, several key issues, such as the weak coupling of temporal and spatial information, still need further study. In this paper, we propose a novel LiDAR-based 3D Moving Ob…
▽ More
LiDAR-based Moving Object Segmentation (MOS) aims to locate and segment moving objects in point clouds of the current scan using motion information from previous scans. Despite the promising results achieved by previous MOS methods, several key issues, such as the weak coupling of temporal and spatial information, still need further study. In this paper, we propose a novel LiDAR-based 3D Moving Object Segmentation with Motion-aware State Space Model, termed MambaMOS. Firstly, we develop a novel embedding module, the Time Clue Bootstrapping Embedding (TCBE), to enhance the coupling of temporal and spatial information in point clouds and alleviate the issue of overlooked temporal clues. Secondly, we introduce the Motion-aware State Space Model (MSSM) to endow the model with the capacity to understand the temporal correlations of the same object across different time steps. Specifically, MSSM emphasizes the motion states of the same object at different time steps through two distinct temporal modeling and correlation steps. We utilize an improved state space model to represent these motion differences, significantly modeling the motion states. Finally, extensive experiments on the SemanticKITTI-MOS and KITTI-Road benchmarks demonstrate that the proposed MambaMOS achieves state-of-the-art performance. The source code is publicly available at https://github.com/Terminal-K/MambaMOS.
△ Less
Submitted 5 August, 2024; v1 submitted 19 April, 2024;
originally announced April 2024.
-
CMNEE: A Large-Scale Document-Level Event Extraction Dataset based on Open-Source Chinese Military News
Authors:
Mengna Zhu,
Zijie Xu,
Kaisheng Zeng,
Kaiming Xiao,
Mao Wang,
Wenjun Ke,
Hongbin Huang
Abstract:
Extracting structured event knowledge, including event triggers and corresponding arguments, from military texts is fundamental to many applications, such as intelligence analysis and decision assistance. However, event extraction in the military field faces the data scarcity problem, which impedes the research of event extraction models in this domain. To alleviate this problem, we propose CMNEE,…
▽ More
Extracting structured event knowledge, including event triggers and corresponding arguments, from military texts is fundamental to many applications, such as intelligence analysis and decision assistance. However, event extraction in the military field faces the data scarcity problem, which impedes the research of event extraction models in this domain. To alleviate this problem, we propose CMNEE, a large-scale, document-level open-source Chinese Military News Event Extraction dataset. It contains 17,000 documents and 29,223 events, which are all manually annotated based on a pre-defined schema for the military domain including 8 event types and 11 argument role types. We designed a two-stage, multi-turns annotation strategy to ensure the quality of CMNEE and reproduced several state-of-the-art event extraction models with a systematic evaluation. The experimental results on CMNEE fall shorter than those on other domain datasets obviously, which demonstrates that event extraction for military domain poses unique challenges and requires further research efforts. Our code and data can be obtained from https://github.com/Mzzzhu/CMNEE.
△ Less
Submitted 18 April, 2024;
originally announced April 2024.
-
Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models
Authors:
Zijin Yang,
Kai Zeng,
Kejiang Chen,
Han Fang,
Weiming Zhang,
Nenghai Yu
Abstract:
Ethical concerns surrounding copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. One effective solution involves watermarking the generated images. However, existing methods often compromise the model performance or require additional training, which is undesirable for operators and users. To address this issue, we propose…
▽ More
Ethical concerns surrounding copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. One effective solution involves watermarking the generated images. However, existing methods often compromise the model performance or require additional training, which is undesirable for operators and users. To address this issue, we propose Gaussian Shading, a diffusion model watermarking technique that is both performance-lossless and training-free, while serving the dual purpose of copyright protection and tracing of offending content. Our watermark embedding is free of model parameter modifications and thus is plug-and-play. We map the watermark to latent representations following a standard Gaussian distribution, which is indistinguishable from latent representations obtained from the non-watermarked diffusion model. Therefore we can achieve watermark embedding with lossless performance, for which we also provide theoretical proof. Furthermore, since the watermark is intricately linked with image semantics, it exhibits resilience to lossy processing and erasure attempts. The watermark can be extracted by Denoising Diffusion Implicit Models (DDIM) inversion and inverse sampling. We evaluate Gaussian Shading on multiple versions of Stable Diffusion, and the results demonstrate that Gaussian Shading not only is performance-lossless but also outperforms existing methods in terms of robustness.
△ Less
Submitted 6 May, 2024; v1 submitted 7 April, 2024;
originally announced April 2024.
-
UAlign: Pushing the Limit of Template-free Retrosynthesis Prediction with Unsupervised SMILES Alignment
Authors:
Kaipeng Zeng,
Bo yang,
Xin Zhao,
Yu Zhang,
Fan Nie,
Xiaokang Yang,
Yaohui Jin,
Yanyan Xu
Abstract:
Motivation: Retrosynthesis planning poses a formidable challenge in the organic chemical industry. Single-step retrosynthesis prediction, a crucial step in the planning process, has witnessed a surge in interest in recent years due to advancements in AI for science. Various deep learning-based methods have been proposed for this task in recent years, incorporating diverse levels of additional chem…
▽ More
Motivation: Retrosynthesis planning poses a formidable challenge in the organic chemical industry. Single-step retrosynthesis prediction, a crucial step in the planning process, has witnessed a surge in interest in recent years due to advancements in AI for science. Various deep learning-based methods have been proposed for this task in recent years, incorporating diverse levels of additional chemical knowledge dependency.
Results: This paper introduces UAlign, a template-free graph-to-sequence pipeline for retrosynthesis prediction. By combining graph neural networks and Transformers, our method can more effectively leverage the inherent graph structure of molecules. Based on the fact that the majority of molecule structures remain unchanged during a chemical reaction, we propose a simple yet effective SMILES alignment technique to facilitate the reuse of unchanged structures for reactant generation. Extensive experiments show that our method substantially outperforms state-of-the-art template-free and semi-template-based approaches. Importantly, our template-free method achieves effectiveness comparable to, or even surpasses, established powerful template-based methods.
Scientific contribution: We present a novel graph-to-sequence template-free retrosynthesis prediction pipeline that overcomes the limitations of Transformer-based methods in molecular representation learning and insufficient utilization of chemical information. We propose an unsupervised learning mechanism for establishing product-atom correspondence with reactant SMILES tokens, achieving even better results than supervised SMILES alignment methods. Extensive experiments demonstrate that UAlign significantly outperforms state-of-the-art template-free methods and rivals or surpasses template-based approaches, with up to 5\% (top-5) and 5.4\% (top-10) increased accuracy over the strongest baseline.
△ Less
Submitted 19 April, 2024; v1 submitted 24 March, 2024;
originally announced April 2024.
-
Distributed Swarm Learning for Edge Internet of Things
Authors:
Yue Wang,
Zhi Tian,
FXin Fan,
Zhipeng Cai,
Cameron Nowzari,
Kai Zeng
Abstract:
The rapid growth of Internet of Things (IoT) has led to the widespread deployment of smart IoT devices at wireless edge for collaborative machine learning tasks, ushering in a new era of edge learning. With a huge number of hardware-constrained IoT devices operating in resource-limited wireless networks, edge learning encounters substantial challenges, including communication and computation bottl…
▽ More
The rapid growth of Internet of Things (IoT) has led to the widespread deployment of smart IoT devices at wireless edge for collaborative machine learning tasks, ushering in a new era of edge learning. With a huge number of hardware-constrained IoT devices operating in resource-limited wireless networks, edge learning encounters substantial challenges, including communication and computation bottlenecks, device and data heterogeneity, security risks, privacy leakages, non-convex optimization, and complex wireless environments. To address these issues, this article explores a novel framework known as distributed swarm learning (DSL), which combines artificial intelligence and biological swarm intelligence in a holistic manner. By harnessing advanced signal processing and communications, DSL provides efficient solutions and robust tools for large-scale IoT at the edge of wireless networks.
△ Less
Submitted 29 March, 2024;
originally announced March 2024.
-
Provably Secure Disambiguating Neural Linguistic Steganography
Authors:
Yuang Qi,
Kejiang Chen,
Kai Zeng,
Weiming Zhang,
Nenghai Yu
Abstract:
Recent research in provably secure neural linguistic steganography has overlooked a crucial aspect: the sender must detokenize stegotexts to avoid raising suspicion from the eavesdropper. The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures in all neural language steganography implementations based on these models. Cur…
▽ More
Recent research in provably secure neural linguistic steganography has overlooked a crucial aspect: the sender must detokenize stegotexts to avoid raising suspicion from the eavesdropper. The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures in all neural language steganography implementations based on these models. Current solutions to this issue involve altering the probability distribution of candidate words, rendering them incompatible with provably secure steganography. We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem. We group all tokens with prefix relationships in the candidate pool before the steganographic embedding algorithm runs to eliminate uncertainty among ambiguous tokens. To enable the receiver to synchronize the sampling process of the sender, a shared cryptographically-secure pseudorandom number generator (CSPRNG) is deployed to select a token from the ambiguity pool. SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods. We provide theoretical proofs and experimentally demonstrate the applicability of our solution to various languages and models, showing its potential to significantly improve the reliability and security of neural linguistic steganography systems.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
Site Symmetry and Multiorbital Flat Bands on Kagome and Pyrochlore Lattices
Authors:
Keyu Zeng,
Ziqiang Wang
Abstract:
Flat bands in electronic band structures are intriguing platforms for strong correlation and topological physics, primarily due to the suppressed kinetic energy of electrons. Various methods have been developed to create flat bands, utilizing lattice geometry or finely tuned parameters. Despite this, the investigation of orbital symmetry in multiorbital materials is a relatively new area of focus.…
▽ More
Flat bands in electronic band structures are intriguing platforms for strong correlation and topological physics, primarily due to the suppressed kinetic energy of electrons. Various methods have been developed to create flat bands, utilizing lattice geometry or finely tuned parameters. Despite this, the investigation of orbital symmetry in multiorbital materials is a relatively new area of focus. In this work, we propose a site symmetry based systematic approach to emerging multiorbital flat bands in lattices made of corner-connecting motifs such as the kagome and pyrochlore lattices. As a conceptual advance, the one-orbital flat bands are shown to originate as mutual eigenstates of isolated molecular motifs. Further developing the mutual eigenstate method for multiorbitals transforming differently under the site symmetries such as mirror and inversion, we derive multiorbital flat bands from the skew-symmetric interorbital Hamiltonian and introduce an isolated molecule enabled group-theoretic description of the flat band wavefunctions. Realizations of the multiorbital flatbands in relevant materials are shown to be possible under the Slater-Koster formalism. Our findings provide new directions for exploring flatband electronic structures for novel correlated and topological quantum states.
△ Less
Submitted 14 March, 2024; v1 submitted 5 March, 2024;
originally announced March 2024.
-
Learning or Self-aligning? Rethinking Instruction Fine-tuning
Authors:
Mengjie Ren,
Boxi Cao,
Hongyu Lin,
Cao Liu,
Xianpei Han,
Ke Zeng,
Guanglu Wan,
Xunliang Cai,
Le Sun
Abstract:
Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potentia…
▽ More
Instruction Fine-tuning~(IFT) is a critical phase in building large language models~(LLMs). Previous works mainly focus on the IFT's role in the transfer of behavioral norms and the learning of additional world knowledge. However, the understanding of the underlying mechanisms of IFT remains significantly limited. In this paper, we design a knowledge intervention framework to decouple the potential underlying factors of IFT, thereby enabling individual analysis of different factors. Surprisingly, our experiments reveal that attempting to learn additional world knowledge through IFT often struggles to yield positive impacts and can even lead to markedly negative effects. Further, we discover that maintaining internal knowledge consistency before and after IFT is a critical factor for achieving successful IFT. Our findings reveal the underlying mechanisms of IFT and provide robust support for some very recent and potential future works.
△ Less
Submitted 11 August, 2024; v1 submitted 28 February, 2024;
originally announced February 2024.
-
Multi-Bit Distortion-Free Watermarking for Large Language Models
Authors:
Massieh Kordi Boroujeny,
Ya Jiang,
Kai Zeng,
Brian Mark
Abstract:
Methods for watermarking large language models have been proposed that distinguish AI-generated text from human-generated text by slightly altering the model output distribution, but they also distort the quality of the text, exposing the watermark to adversarial detection. More recently, distortion-free watermarking methods were proposed that require a secret key to detect the watermark. The prio…
▽ More
Methods for watermarking large language models have been proposed that distinguish AI-generated text from human-generated text by slightly altering the model output distribution, but they also distort the quality of the text, exposing the watermark to adversarial detection. More recently, distortion-free watermarking methods were proposed that require a secret key to detect the watermark. The prior methods generally embed zero-bit watermarks that do not provide additional information beyond tagging a text as being AI-generated. We extend an existing zero-bit distortion-free watermarking method by embedding multiple bits of meta-information as part of the watermark. We also develop a computationally efficient decoder that extracts the embedded information from the watermark with low bit error rate.
△ Less
Submitted 26 February, 2024;
originally announced February 2024.
-
Event-level Knowledge Editing
Authors:
Hao Peng,
Xiaozhi Wang,
Chunyang Li,
Kaisheng Zeng,
Jiangshan Duo,
Yixin Cao,
Lei Hou,
Juanzi Li
Abstract:
Knowledge editing aims at updating knowledge of large language models (LLMs) to prevent them from becoming outdated. Existing work edits LLMs at the level of factual knowledge triplets. However, natural knowledge updates in the real world come from the occurrences of new events rather than direct changes in factual triplets. In this paper, we propose a new task setting: event-level knowledge editi…
▽ More
Knowledge editing aims at updating knowledge of large language models (LLMs) to prevent them from becoming outdated. Existing work edits LLMs at the level of factual knowledge triplets. However, natural knowledge updates in the real world come from the occurrences of new events rather than direct changes in factual triplets. In this paper, we propose a new task setting: event-level knowledge editing, which directly edits new events into LLMs and improves over conventional triplet-level editing on (1) Efficiency. A single event edit leads to updates in multiple entailed knowledge triplets. (2) Completeness. Beyond updating factual knowledge, event-level editing also requires considering the event influences and updating LLMs' knowledge about future trends. We construct a high-quality event-level editing benchmark ELKEN, consisting of 1,515 event edits, 6,449 questions about factual knowledge, and 10,150 questions about future tendencies. We systematically evaluate the performance of various knowledge editing methods and LLMs on this benchmark. We find that ELKEN poses significant challenges to existing knowledge editing approaches. Our codes and dataset are publicly released to facilitate further research.
△ Less
Submitted 21 April, 2024; v1 submitted 20 February, 2024;
originally announced February 2024.
-
MF-MOS: A Motion-Focused Model for Moving Object Segmentation
Authors:
Jintao Cheng,
Kang Zeng,
Zhuoxu Huang,
Xiaoyu Tang,
Jin Wu,
Chengxi Zhang,
Xieyuanli Chen,
Rui Fan
Abstract:
Moving object segmentation (MOS) provides a reliable solution for detecting traffic participants and thus is of great interest in the autonomous driving field. Dynamic capture is always critical in the MOS problem. Previous methods capture motion features from the range images directly. Differently, we argue that the residual maps provide greater potential for motion information, while range image…
▽ More
Moving object segmentation (MOS) provides a reliable solution for detecting traffic participants and thus is of great interest in the autonomous driving field. Dynamic capture is always critical in the MOS problem. Previous methods capture motion features from the range images directly. Differently, we argue that the residual maps provide greater potential for motion information, while range images contain rich semantic guidance. Based on this intuition, we propose MF-MOS, a novel motion-focused model with a dual-branch structure for LiDAR moving object segmentation. Novelly, we decouple the spatial-temporal information by capturing the motion from residual maps and generating semantic features from range images, which are used as movable object guidance for the motion branch. Our straightforward yet distinctive solution can make the most use of both range images and residual maps, thus greatly improving the performance of the LiDAR-based MOS task. Remarkably, our MF-MOS achieved a leading IoU of 76.7% on the MOS leaderboard of the SemanticKITTI dataset upon submission, demonstrating the current state-of-the-art performance. The implementation of our MF-MOS has been released at https://github.com/SCNU-RISLAB/MF-MOS.
△ Less
Submitted 30 January, 2024;
originally announced January 2024.
-
MorphGrower: A Synchronized Layer-by-layer Growing Approach for Plausible Neuronal Morphology Generation
Authors:
Nianzu Yang,
Kaipeng Zeng,
Haotian Lu,
Yexin Wu,
Zexin Yuan,
Danni Chen,
Shengdian Jiang,
Jiaxiang Wu,
Yimin Wang,
Junchi Yan
Abstract:
Neuronal morphology is essential for studying brain functioning and understanding neurodegenerative disorders. As acquiring real-world morphology data is expensive, computational approaches for morphology generation have been studied. Traditional methods heavily rely on expert-set rules and parameter tuning, making it difficult to generalize across different types of morphologies. Recently, MorphV…
▽ More
Neuronal morphology is essential for studying brain functioning and understanding neurodegenerative disorders. As acquiring real-world morphology data is expensive, computational approaches for morphology generation have been studied. Traditional methods heavily rely on expert-set rules and parameter tuning, making it difficult to generalize across different types of morphologies. Recently, MorphVAE was introduced as the sole learning-based method, but its generated morphologies lack plausibility, i.e., they do not appear realistic enough and most of the generated samples are topologically invalid. To fill this gap, this paper proposes MorphGrower, which mimicks the neuron natural growth mechanism for generation. Specifically, MorphGrower generates morphologies layer by layer, with each subsequent layer conditioned on the previously generated structure. During each layer generation, MorphGrower utilizes a pair of sibling branches as the basic generation block and generates branch pairs synchronously. This approach ensures topological validity and allows for fine-grained generation, thereby enhancing the realism of the final generated morphologies. Results on four real-world datasets demonstrate that MorphGrower outperforms MorphVAE by a notable margin. Importantly, the electrophysiological response simulation demonstrates the plausibility of our generated samples from a neuroscience perspective. Our code is available at https://github.com/Thinklab-SJTU/MorphGrower.
△ Less
Submitted 27 May, 2024; v1 submitted 17 January, 2024;
originally announced January 2024.
-
Seeing the Unseen: Visual Common Sense for Semantic Placement
Authors:
Ram Ramrakhya,
Aniruddha Kembhavi,
Dhruv Batra,
Zsolt Kira,
Kuo-Hao Zeng,
Luca Weihs
Abstract:
Computer vision tasks typically involve describing what is present in an image (e.g. classification, detection, segmentation, and captioning). We study a visual common sense task that requires understanding what is not present. Specifically, given an image (e.g. of a living room) and name of an object ("cushion"), a vision system is asked to predict semantically-meaningful regions (masks or boundi…
▽ More
Computer vision tasks typically involve describing what is present in an image (e.g. classification, detection, segmentation, and captioning). We study a visual common sense task that requires understanding what is not present. Specifically, given an image (e.g. of a living room) and name of an object ("cushion"), a vision system is asked to predict semantically-meaningful regions (masks or bounding boxes) in the image where that object could be placed or is likely be placed by humans (e.g. on the sofa). We call this task: Semantic Placement (SP) and believe that such common-sense visual understanding is critical for assitive robots (tidying a house), and AR devices (automatically rendering an object in the user's space). Studying the invisible is hard. Datasets for image description are typically constructed by curating relevant images and asking humans to annotate the contents of the image; neither of those two steps are straightforward for objects not present in the image. We overcome this challenge by operating in the opposite direction: we start with an image of an object in context from web, and then remove that object from the image via inpainting. This automated pipeline converts unstructured web data into a dataset comprising pairs of images with/without the object. Using this, we collect a novel dataset, with ${\sim}1.3$M images across $9$ object categories, and train a SP prediction model called CLIP-UNet. CLIP-UNet outperforms existing VLMs and baselines that combine semantic priors with object detectors on real-world and simulated images. In our user studies, we find that the SP masks predicted by CLIP-UNet are favored $43.7\%$ and $31.3\%$ times when comparing against the $4$ SP baselines on real and simulated images. In addition, we demonstrate leveraging SP mask predictions from CLIP-UNet enables downstream applications like building tidying robots in indoor environments.
△ Less
Submitted 15 January, 2024;
originally announced January 2024.
-
SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
Authors:
Kiana Ehsani,
Tanmay Gupta,
Rose Hendrix,
Jordi Salvador,
Luca Weihs,
Kuo-Hao Zeng,
Kunal Pratap Singh,
Yejin Kim,
Winson Han,
Alvaro Herrasti,
Ranjay Krishna,
Dustin Schwenk,
Eli VanderBilt,
Aniruddha Kembhavi
Abstract:
Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely…
▽ More
Reinforcement learning (RL) with dense rewards and imitation learning (IL) with human-generated trajectories are the most widely used approaches for training modern embodied agents. RL requires extensive reward shaping and auxiliary losses and is often too slow and ineffective for long-horizon tasks. While IL with human supervision is effective, collecting human trajectories at scale is extremely expensive. In this work, we show that imitating shortest-path planners in simulation produces agents that, given a language instruction, can proficiently navigate, explore, and manipulate objects in both simulation and in the real world using only RGB sensors (no depth map or GPS coordinates). This surprising result is enabled by our end-to-end, transformer-based, SPOC architecture, powerful visual encoders paired with extensive image augmentation, and the dramatic scale and diversity of our training data: millions of frames of shortest-path-expert trajectories collected inside approximately 200,000 procedurally generated houses containing 40,000 unique 3D assets. Our models, data, training code, and newly proposed 10-task benchmarking suite CHORES are available in https://spoc-robot.github.io.
△ Less
Submitted 7 August, 2024; v1 submitted 5 December, 2023;
originally announced December 2023.
-
Turning Your Strength into Watermark: Watermarking Large Language Model via Knowledge Injection
Authors:
Shuai Li,
Kejiang Chen,
Kunsheng Tang,
Jie Zhang,
Weiming Zhang,
Nenghai Yu,
Kai Zeng
Abstract:
Large language models (LLMs) have demonstrated outstanding performance, making them valuable digital assets with significant commercial potential. Unfortunately, the LLM and its API are susceptible to intellectual property theft. Watermarking is a classic solution for copyright verification. However, most recent emerging LLM watermarking methods focus on identifying AI-generated texts rather than…
▽ More
Large language models (LLMs) have demonstrated outstanding performance, making them valuable digital assets with significant commercial potential. Unfortunately, the LLM and its API are susceptible to intellectual property theft. Watermarking is a classic solution for copyright verification. However, most recent emerging LLM watermarking methods focus on identifying AI-generated texts rather than watermarking LLM itself. Only a few attempts are based on weight quantification and backdoor watermarking, which are not robust or covert enough, limiting their applicability in practice.
To address this issue, we propose a novel watermarking method for LLMs based on knowledge injection and innovatively use knowledge as the watermark carrier. Specifically, in the watermark embedding stage, we first embed the watermarks into the selected knowledge to obtain the watermarked knowledge, subsequently injected into the to-be-protected LLM. In the watermark extraction stage, questions related to the watermarked knowledge are designed, for querying the suspect LLM and extracting the watermarks from its response. The experiments show that the watermark extraction success rate is close to 100% and demonstrate the effectiveness, fidelity, stealthiness, and robustness of our proposed method.
△ Less
Submitted 24 July, 2024; v1 submitted 15 November, 2023;
originally announced November 2023.
-
MAVEN-Arg: Completing the Puzzle of All-in-One Event Understanding Dataset with Event Argument Annotation
Authors:
Xiaozhi Wang,
Hao Peng,
Yong Guan,
Kaisheng Zeng,
Jianhui Chen,
Lei Hou,
Xu Han,
Yankai Lin,
Zhiyuan Liu,
Ruobing Xie,
Jie Zhou,
Juanzi Li
Abstract:
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg,…
▽ More
Understanding events in texts is a core objective of natural language understanding, which requires detecting event occurrences, extracting event arguments, and analyzing inter-event relationships. However, due to the annotation challenges brought by task complexity, a large-scale dataset covering the full process of event understanding has long been absent. In this paper, we introduce MAVEN-Arg, which augments MAVEN datasets with event argument annotations, making the first all-in-one dataset supporting event detection, event argument extraction (EAE), and event relation extraction. As an EAE benchmark, MAVEN-Arg offers three main advantages: (1) a comprehensive schema covering 162 event types and 612 argument roles, all with expert-written definitions and examples; (2) a large data scale, containing 98,591 events and 290,613 arguments obtained with laborious human annotation; (3) the exhaustive annotation supporting all task variants of EAE, which annotates both entity and non-entity event arguments in document level. Experiments indicate that MAVEN-Arg is quite challenging for both fine-tuned EAE models and proprietary large language models (LLMs). Furthermore, to demonstrate the benefits of an all-in-one dataset, we preliminarily explore a potential application, future event prediction, with LLMs. MAVEN-Arg and codes can be obtained from https://github.com/THU-KEG/MAVEN-Argument.
△ Less
Submitted 18 June, 2024; v1 submitted 15 November, 2023;
originally announced November 2023.
-
When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks
Authors:
Hao Peng,
Xiaozhi Wang,
Jianhui Chen,
Weikai Li,
Yunjia Qi,
Zimu Wang,
Zhili Wu,
Kaisheng Zeng,
Bin Xu,
Lei Hou,
Juanzi Li
Abstract:
In-context learning (ICL) has become the default method for using large language models (LLMs), making the exploration of its limitations and understanding the underlying causes crucial. In this paper, we find that ICL falls short of handling specification-heavy tasks, which are tasks with complicated and extensive task specifications, requiring several hours for ordinary humans to master, such as…
▽ More
In-context learning (ICL) has become the default method for using large language models (LLMs), making the exploration of its limitations and understanding the underlying causes crucial. In this paper, we find that ICL falls short of handling specification-heavy tasks, which are tasks with complicated and extensive task specifications, requiring several hours for ordinary humans to master, such as traditional information extraction tasks. The performance of ICL on these tasks mostly cannot reach half of the state-of-the-art results. To explore the reasons behind this failure, we conduct comprehensive experiments on 18 specification-heavy tasks with various LLMs and identify three primary reasons: inability to specifically understand context, misalignment in task schema comprehension with humans, and inadequate long-text understanding ability. Furthermore, we demonstrate that through fine-tuning, LLMs can achieve decent performance on these tasks, indicating that the failure of ICL is not an inherent flaw of LLMs, but rather a drawback of existing alignment methods that renders LLMs incapable of handling complicated specification-heavy tasks via ICL. To substantiate this, we perform dedicated instruction tuning on LLMs for these tasks and observe a notable improvement. We hope the analyses in this paper could facilitate advancements in alignment methods enabling LLMs to meet more sophisticated human demands.
△ Less
Submitted 15 November, 2023;
originally announced November 2023.
-
FireMatch: A Semi-Supervised Video Fire Detection Network Based on Consistency and Distribution Alignment
Authors:
Qinghua Lin,
Zuoyong Li,
Kun Zeng,
Haoyi Fan,
Wei Li,
Xiaoguang Zhou
Abstract:
Deep learning techniques have greatly enhanced the performance of fire detection in videos. However, video-based fire detection models heavily rely on labeled data, and the process of data labeling is particularly costly and time-consuming, especially when dealing with videos. Considering the limited quantity of labeled video data, we propose a semi-supervised fire detection model called FireMatch…
▽ More
Deep learning techniques have greatly enhanced the performance of fire detection in videos. However, video-based fire detection models heavily rely on labeled data, and the process of data labeling is particularly costly and time-consuming, especially when dealing with videos. Considering the limited quantity of labeled video data, we propose a semi-supervised fire detection model called FireMatch, which is based on consistency regularization and adversarial distribution alignment. Specifically, we first combine consistency regularization with pseudo-label. For unlabeled data, we design video data augmentation to obtain corresponding weakly augmented and strongly augmented samples. The proposed model predicts weakly augmented samples and retains pseudo-label above a threshold, while training on strongly augmented samples to predict these pseudo-labels for learning more robust feature representations. Secondly, we generate video cross-set augmented samples by adversarial distribution alignment to expand the training data and alleviate the decline in classification performance caused by insufficient labeled data. Finally, we introduce a fairness loss to help the model produce diverse predictions for input samples, thereby addressing the issue of high confidence with the non-fire class in fire classification scenarios. The FireMatch achieved an accuracy of 76.92% and 91.81% on two real-world fire datasets, respectively. The experimental results demonstrate that the proposed method outperforms the current state-of-the-art semi-supervised classification methods.
△ Less
Submitted 9 November, 2023;
originally announced November 2023.
-
Selective Visual Representations Improve Convergence and Generalization for Embodied AI
Authors:
Ainaz Eftekhar,
Kuo-Hao Zeng,
Jiafei Duan,
Ali Farhadi,
Ani Kembhavi,
Ranjay Krishna
Abstract:
Embodied AI models often employ off the shelf vision backbones like CLIP to encode their visual observations. Although such general purpose representations encode rich syntactic and semantic information about the scene, much of this information is often irrelevant to the specific task at hand. This introduces noise within the learning process and distracts the agent's focus from task-relevant visu…
▽ More
Embodied AI models often employ off the shelf vision backbones like CLIP to encode their visual observations. Although such general purpose representations encode rich syntactic and semantic information about the scene, much of this information is often irrelevant to the specific task at hand. This introduces noise within the learning process and distracts the agent's focus from task-relevant visual cues. Inspired by selective attention in humans-the process through which people filter their perception based on their experiences, knowledge, and the task at hand-we introduce a parameter-efficient approach to filter visual stimuli for embodied AI. Our approach induces a task-conditioned bottleneck using a small learnable codebook module. This codebook is trained jointly to optimize task reward and acts as a task-conditioned selective filter over the visual observation. Our experiments showcase state-of-the-art performance for object goal navigation and object displacement across 5 benchmarks, ProcTHOR, ArchitecTHOR, RoboTHOR, AI2-iTHOR, and ManipulaTHOR. The filtered representations produced by the codebook are also able generalize better and converge faster when adapted to other simulation environments such as Habitat. Our qualitative analyses show that agents explore their environments more effectively and their representations retain task-relevant information like target object recognition while ignoring superfluous information about other objects. Code and pretrained models are available at our project website: https://embodied-codebook.github.io.
△ Less
Submitted 9 March, 2024; v1 submitted 7 November, 2023;
originally announced November 2023.
-
Mastering the Task of Open Information Extraction with Large Language Models and Consistent Reasoning Environment
Authors:
Ji Qi,
Kaixuan Ji,
Xiaozhi Wang,
Jifan Yu,
Kaisheng Zeng,
Lei Hou,
Juanzi Li,
Bin Xu
Abstract:
Open Information Extraction (OIE) aims to extract objective structured knowledge from natural texts, which has attracted growing attention to build dedicated models with human experience. As the large language models (LLMs) have exhibited remarkable in-context learning capabilities, a question arises as to whether the task of OIE can be effectively tackled with this paradigm? In this paper, we exp…
▽ More
Open Information Extraction (OIE) aims to extract objective structured knowledge from natural texts, which has attracted growing attention to build dedicated models with human experience. As the large language models (LLMs) have exhibited remarkable in-context learning capabilities, a question arises as to whether the task of OIE can be effectively tackled with this paradigm? In this paper, we explore solving the OIE problem by constructing an appropriate reasoning environment for LLMs. Specifically, we first propose a method to effectively estimate the discrepancy of syntactic distribution between a LLM and test samples, which can serve as correlation evidence for preparing positive demonstrations. Upon the evidence, we introduce a simple yet effective mechanism to establish the reasoning environment for LLMs on specific tasks. Without bells and whistles, experimental results on the standard CaRB benchmark demonstrate that our $6$-shot approach outperforms state-of-the-art supervised method, achieving an $55.3$ $F_1$ score. Further experiments on TACRED and ACE05 show that our method can naturally generalize to other information extraction tasks, resulting in improvements of $5.7$ and $6.8$ $F_1$ scores, respectively.
△ Less
Submitted 16 October, 2023;
originally announced October 2023.
-
One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models
Authors:
Hang Shao,
Bei Liu,
Bo Xiao,
Ke Zeng,
Guanglu Wan,
Yanmin Qian
Abstract:
Various Large Language Models~(LLMs) from the Generative Pretrained Transformer(GPT) family have achieved outstanding performances in a wide range of text generation tasks. However, the enormous model sizes have hindered their practical use in real-world applications due to high inference latency. Therefore, improving the efficiencies of LLMs through quantization, pruning, and other means has been…
▽ More
Various Large Language Models~(LLMs) from the Generative Pretrained Transformer(GPT) family have achieved outstanding performances in a wide range of text generation tasks. However, the enormous model sizes have hindered their practical use in real-world applications due to high inference latency. Therefore, improving the efficiencies of LLMs through quantization, pruning, and other means has been a key issue in LLM studies. In this work, we propose a method based on Hessian sensitivity-aware mixed sparsity pruning to prune LLMs to at least 50% sparsity without the need of any retraining. It allocates sparsity adaptively based on sensitivity, allowing us to reduce pruning-induced error while maintaining the overall sparsity level. The advantages of the proposed method exhibit even more when the sparsity is extremely high. Furthermore, our method is compatible with quantization, enabling further compression of LLMs. We have released the available code.
△ Less
Submitted 23 April, 2024; v1 submitted 14 October, 2023;
originally announced October 2023.
-
Open X-Embodiment: Robotic Learning Datasets and RT-X Models
Authors:
Open X-Embodiment Collaboration,
Abby O'Neill,
Abdul Rehman,
Abhinav Gupta,
Abhiram Maddukuri,
Abhishek Gupta,
Abhishek Padalkar,
Abraham Lee,
Acorn Pooley,
Agrim Gupta,
Ajay Mandlekar,
Ajinkya Jain,
Albert Tung,
Alex Bewley,
Alex Herzog,
Alex Irpan,
Alexander Khazatsky,
Anant Rai,
Anchit Gupta,
Andrew Wang,
Andrey Kolobov,
Anikait Singh,
Animesh Garg,
Aniruddha Kembhavi,
Annie Xie
, et al. (267 additional authors not shown)
Abstract:
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning method…
▽ More
Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.
△ Less
Submitted 1 June, 2024; v1 submitted 13 October, 2023;
originally announced October 2023.
-
Exploring Large Language Models for Multi-Modal Out-of-Distribution Detection
Authors:
Yi Dai,
Hao Lang,
Kaisheng Zeng,
Fei Huang,
Yongbin Li
Abstract:
Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descript…
▽ More
Out-of-distribution (OOD) detection is essential for reliable and trustworthy machine learning. Recent multi-modal OOD detection leverages textual information from in-distribution (ID) class names for visual OOD detection, yet it currently neglects the rich contextual information of ID classes. Large language models (LLMs) encode a wealth of world knowledge and can be prompted to generate descriptive features for each class. Indiscriminately using such knowledge causes catastrophic damage to OOD detection due to LLMs' hallucinations, as is observed by our analysis. In this paper, we propose to apply world knowledge to enhance OOD detection performance through selective generation from LLMs. Specifically, we introduce a consistency-based uncertainty calibration method to estimate the confidence score of each generation. We further extract visual objects from each image to fully capitalize on the aforementioned world knowledge. Extensive experiments demonstrate that our method consistently outperforms the state-of-the-art.
△ Less
Submitted 12 October, 2023;
originally announced October 2023.
-
Advective Diffusion Transformers for Topological Generalization in Graph Learning
Authors:
Qitian Wu,
Chenxiao Yang,
Kaipeng Zeng,
Fan Nie,
Michael Bronstein,
Junchi Yan
Abstract:
Graph diffusion equations are intimately related to graph neural networks (GNNs) and have recently attracted attention as a principled framework for analyzing GNN dynamics, formalizing their expressive power, and justifying architectural choices. One key open questions in graph learning is the generalization capabilities of GNNs. A major limitation of current approaches hinges on the assumption th…
▽ More
Graph diffusion equations are intimately related to graph neural networks (GNNs) and have recently attracted attention as a principled framework for analyzing GNN dynamics, formalizing their expressive power, and justifying architectural choices. One key open questions in graph learning is the generalization capabilities of GNNs. A major limitation of current approaches hinges on the assumption that the graph topologies in the training and test sets come from the same distribution. In this paper, we make steps towards understanding the generalization of GNNs by exploring how graph diffusion equations extrapolate and generalize in the presence of varying graph topologies. We first show deficiencies in the generalization capability of existing models built upon local diffusion on graphs, stemming from the exponential sensitivity to topology variation. Our subsequent analysis reveals the promise of non-local diffusion, which advocates for feature propagation over fully-connected latent graphs, under the assumption of a specific data-generating condition. In addition to these findings, we propose a novel graph encoder backbone, Advective Diffusion Transformer (ADiT), inspired by advective graph diffusion equations that have a closed-form solution backed up with theoretical guarantees of desired generalization under topological distribution shifts. The new model, functioning as a versatile graph Transformer, demonstrates superior performance across a wide range of graph learning tasks.
△ Less
Submitted 10 October, 2023;
originally announced October 2023.
-
A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-training
Authors:
Lucen Zhong,
Hengtong Lu,
Caixia Yuan,
Xiaojie Wang,
Jiashen Sun,
Ke Zeng,
Guanglu Wan
Abstract:
Pre-trained conversation models (PCMs) have achieved promising progress in recent years. However, existing PCMs for Task-oriented dialog (TOD) are insufficient for capturing the sequential nature of the TOD-related tasks, as well as for learning dialog policy information. To alleviate these problems, this paper proposes a task-progressive PCM with two policy-aware pre-training tasks. The model is…
▽ More
Pre-trained conversation models (PCMs) have achieved promising progress in recent years. However, existing PCMs for Task-oriented dialog (TOD) are insufficient for capturing the sequential nature of the TOD-related tasks, as well as for learning dialog policy information. To alleviate these problems, this paper proposes a task-progressive PCM with two policy-aware pre-training tasks. The model is pre-trained through three stages where TOD-related tasks are progressively employed according to the task logic of the TOD system. A global policy consistency task is designed to capture the multi-turn dialog policy sequential relation, and an act-based contrastive learning task is designed to capture similarities among samples with the same dialog policy. Our model achieves better results on both MultiWOZ and In-Car end-to-end dialog modeling benchmarks with only 18\% parameters and 25\% pre-training data compared to the previous state-of-the-art PCM, GALAXY.
△ Less
Submitted 1 October, 2023;
originally announced October 2023.
-
OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding
Authors:
Hao Peng,
Xiaozhi Wang,
Feng Yao,
Zimu Wang,
Chuzhao Zhu,
Kaisheng Zeng,
Lei Hou,
Juanzi Li
Abstract:
Event understanding aims at understanding the content and relationship of events within texts, which covers multiple complicated information extraction tasks: event detection, event argument extraction, and event relation extraction. To facilitate related research and application, we present an event understanding toolkit OmniEvent, which features three desiderata: (1) Comprehensive. OmniEvent sup…
▽ More
Event understanding aims at understanding the content and relationship of events within texts, which covers multiple complicated information extraction tasks: event detection, event argument extraction, and event relation extraction. To facilitate related research and application, we present an event understanding toolkit OmniEvent, which features three desiderata: (1) Comprehensive. OmniEvent supports mainstream modeling paradigms of all the event understanding tasks and the processing of 15 widely-used English and Chinese datasets. (2) Fair. OmniEvent carefully handles the inconspicuous evaluation pitfalls reported in Peng et al. (2023), which ensures fair comparisons between different models. (3) Easy-to-use. OmniEvent is designed to be easily used by users with varying needs. We provide off-the-shelf models that can be directly deployed as web services. The modular framework also enables users to easily implement and evaluate new event understanding models with OmniEvent. The toolkit (https://github.com/THU-KEG/OmniEvent) is publicly released along with the demonstration website and video (https://omnievent.xlore.cn/).
△ Less
Submitted 25 September, 2023;
originally announced September 2023.
-
Recovering lossless propagation of polaritons with synthesized complex frequency excitation
Authors:
Fuxin Guan,
Xiangdong Guo,
Shu Zhang,
Kebo Zeng,
Yue Hu,
Chenchen Wu,
Shaobo Zhou,
Yuanjiang Xiang,
Xiaoxia Yang,
Qing Dai,
Shuang Zhang
Abstract:
Surface plasmon polaritons and phonon polaritons offer a means of surpassing the diffraction limit of conventional optics and facilitate efficient energy storage, local field enhancement, high sensitivities, benefitting from their subwavelength confinement of light. Unfortunately, losses severely limit the propagation decay length, thus restricting the practical use of polaritons. While optimizing…
▽ More
Surface plasmon polaritons and phonon polaritons offer a means of surpassing the diffraction limit of conventional optics and facilitate efficient energy storage, local field enhancement, high sensitivities, benefitting from their subwavelength confinement of light. Unfortunately, losses severely limit the propagation decay length, thus restricting the practical use of polaritons. While optimizing the fabrication technique can help circumvent the scattering loss of imperfect structures, the intrinsic absorption channel leading to heat production cannot be eliminated. Here, we utilize synthetic optical excitation of complex frequency with virtual gain, synthesized by combining the measurements taken at multiple real frequencies, to restore the lossless propagations of phonon polaritons with significantly reduced intrinsic losses. The concept of synthetic complex frequency excitation represents a viable solution to compensate for loss and would benefit applications including photonic circuits, waveguiding and plasmonic/phononic structured illumination microscopy.
△ Less
Submitted 18 September, 2023; v1 submitted 28 August, 2023;
originally announced August 2023.
-
Location Privacy and Spectrum Efficiency Enhancement in Spectrum Sharing Systems
Authors:
Long Jiao,
Yao Ge,
Kai Zeng,
B. C. Hilburn
Abstract:
In this work, we investigate the benefits of secondary user (SU) network beamforming on improving primary user (PU) location privacy in spectrum sharing systems, where the beamformer in the SU network is designed to suppress the aggregate interference to improve the location privacy of PUs. We consider two problems: improving SU network communication throughput subject to the specified PU location…
▽ More
In this work, we investigate the benefits of secondary user (SU) network beamforming on improving primary user (PU) location privacy in spectrum sharing systems, where the beamformer in the SU network is designed to suppress the aggregate interference to improve the location privacy of PUs. We consider two problems: improving SU network communication throughput subject to the specified PU location privacy requirements, and enhancing PU location privacy given the quality of service (QoS) requirements of SU networks. In the first problem, we provide an algorithm to achieve high data rates with the constrained PU location privacy level. Numerical results show that for a given PU location privacy requirement, the proposed scheme is able to interfere/exclude only a few SU nodes from the PU band and the network throughput can be greatly improved. In the second problem, to fully explore the potential of SU network beamforming for enhancing PU location privacy, we propose a two-step scheme to decouple the beamforming and privacy zone design so that the PU location privacy can be improved while satisfying the SU network throughput requirement. According to numerical evaluations, the proposed scheme can maintain/achieve higher PU location privacy than the benchmark beamforming schemes while satisfying a QoS requirement for the SU network.
△ Less
Submitted 26 August, 2023;
originally announced August 2023.
-
Optically levitated gyroscopes with a MHz rotating micro-rotor
Authors:
Kai Zeng,
Xiangming Xu,
Yulie Wu,
Xuezhong Wu,
Dingbang Xiao
Abstract:
The optically levitated particles have been driven to rotate at an ultra-high speed of GHz, and the gyroscopic application of these levitated particles to measure angular motion have long been explored. However, this gyroscope has not been proven either theoretically or experimentally. Here, a rotor gyroscope based on optically levitated high-speed rotating particles is proposed. In vacuum, an ell…
▽ More
The optically levitated particles have been driven to rotate at an ultra-high speed of GHz, and the gyroscopic application of these levitated particles to measure angular motion have long been explored. However, this gyroscope has not been proven either theoretically or experimentally. Here, a rotor gyroscope based on optically levitated high-speed rotating particles is proposed. In vacuum, an ellipsoidal vaterite particle with 3.58 $μ$m average diameter is driven to rotate at MHz, and the optical axis orientation of the particle is measured by the particle rotational signal. The external inputted angular velocity makes the optical axis deviate from the initial position, which changes the frequency and amplitude of the rotational signal. The inputted angular velocity is hence detected by the rotational signal, and the angular rate bias instability of the prototype is measured to be $0.08^o/s$. It is the smallest rotor gyroscope in the world, and the bias instability can be further improved up to $10^{-9o}/h$ theoretically by cooling the motion and increasing the angular moment of the levitated particle. Our work opens a new application paradigm of the levitated optomechanical systems and possibly bring the rotor gyroscope to the quantum realm.
△ Less
Submitted 17 August, 2023;
originally announced August 2023.
-
Loday-Quillen-Tsygan theorem on Quivers
Authors:
Keyou Zeng
Abstract:
The well-known Loday-Quillen-Tsygan theorem calculates the Lie algebra homology of the infinite general linear Lie algebra $\mathfrak{gl}(A)$ over an unital associative algebra $A$. We generalize the Loday-Quillen-Tsygan theorem to an infinite Lie algebra associated with a (framed) quiver, where we assign to each vertex $v$ an infinite general linear Lie algebra $\mathfrak{gl}(A_v)$, to each edge…
▽ More
The well-known Loday-Quillen-Tsygan theorem calculates the Lie algebra homology of the infinite general linear Lie algebra $\mathfrak{gl}(A)$ over an unital associative algebra $A$. We generalize the Loday-Quillen-Tsygan theorem to an infinite Lie algebra associated with a (framed) quiver, where we assign to each vertex $v$ an infinite general linear Lie algebra $\mathfrak{gl}(A_v)$, to each edge $e$ an infinite matrix module and to each framed vertex a (anti)-fundamental representation. Given this data, each loop or path ending on framed vertices of the quiver defined a stratified factorization algebra over $S^1$ or $[0,1]$ respectively. We show that the corresponding Lie algebra homology can be expressed as summing the factorization homology over all loops and framed paths of the quiver.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
Wide-band Unambiguous Quantum Sensing via Geodesic Evolution
Authors:
Ke Zeng,
Xiaohui Yu,
Martin B. Plenio,
Zhen-Yu Wang
Abstract:
We present a quantum sensing technique that utilizes a sequence of $π$ pulses to cyclically drive the qubit dynamics along a geodesic path of adiabatic evolution. This approach effectively suppresses the effects of both decoherence noise and control errors while simultaneously removing unwanted resonance terms, such as higher harmonics and spurious responses commonly encountered in dynamical decou…
▽ More
We present a quantum sensing technique that utilizes a sequence of $π$ pulses to cyclically drive the qubit dynamics along a geodesic path of adiabatic evolution. This approach effectively suppresses the effects of both decoherence noise and control errors while simultaneously removing unwanted resonance terms, such as higher harmonics and spurious responses commonly encountered in dynamical decoupling control. As a result, our technique offers robust, wide-band, unambiguous, and high-resolution quantum sensing capabilities for signal detection and individual addressing of quantum systems, including spins. To demonstrate its versatility, we showcase successful applications of our method in both low-frequency and high-frequency sensing scenarios. The significance of this quantum sensing technique extends to the detection of complex signals and the control of intricate quantum environments. By enhancing detection accuracy and enabling precise manipulation of quantum systems, our method holds considerable promise for a variety of practical applications.
△ Less
Submitted 19 July, 2023;
originally announced July 2023.
-
Synthesized complex-frequency excitation for ultrasensitive molecular sensing
Authors:
Kebo Zeng,
Chenchen Wu,
Xiangdong Guo,
Fuxin Guan,
Yu Duan,
Lauren L Zhang,
Xiaoxia Yang,
Na Liu,
Qing Dai,
Shuang Zhang
Abstract:
Detecting trace molecules remains a significant challenge. Surface-enhanced infrared absorption (SEIRA) based on plasmonic nanostructures, particularly graphene, has emerged as a promising approach to enhance sensing sensitivity. While graphene-based SEIRA offers advantages such as ultrahigh sensitivity and active tunability, intrinsic molecular damping weakens the interaction between vibrational…
▽ More
Detecting trace molecules remains a significant challenge. Surface-enhanced infrared absorption (SEIRA) based on plasmonic nanostructures, particularly graphene, has emerged as a promising approach to enhance sensing sensitivity. While graphene-based SEIRA offers advantages such as ultrahigh sensitivity and active tunability, intrinsic molecular damping weakens the interaction between vibrational modes and plasmons. Here, we demonstrate ultrahigh-sensitive molecular sensing based on synthesized complex-frequency waves (CFW). Our experiment shows that CFW can amplify the molecular signals (~1.2-nm-thick silk protein layer) detected by graphene-based sensor by at least an order of magnitude and can be universally applied to molecular sensing in different phases. Our approach is highly scalable and can facilitate the investigation of light-matter interactions, enabling diverse potential applications in fields such as optical spectroscopy, metasurfaces, optoelectronics, biomedicine and pharmaceutics.
△ Less
Submitted 18 July, 2023;
originally announced July 2023.
-
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Authors:
Jifan Yu,
Xiaozhi Wang,
Shangqing Tu,
Shulin Cao,
Daniel Zhang-Li,
Xin Lv,
Hao Peng,
Zijun Yao,
Xiaohan Zhang,
Hanming Li,
Chunyang Li,
Zheyuan Zhang,
Yushi Bai,
Yantao Liu,
Amy Xin,
Nianyi Lin,
Kaifeng Yun,
Linlu Gong,
Jianhui Chen,
Zhili Wu,
Yunjia Qi,
Weikai Li,
Yong Guan,
Kaisheng Zeng,
Ji Qi
, et al. (10 additional authors not shown)
Abstract:
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we…
▽ More
The unprecedented performance of large language models (LLMs) necessitates improvements in evaluations. Rather than merely exploring the breadth of LLM abilities, we believe meticulous and thoughtful designs are essential to thorough, unbiased, and applicable evaluations. Given the importance of world knowledge to LLMs, we construct a Knowledge-oriented LLM Assessment benchmark (KoLA), in which we carefully design three crucial factors: (1) For \textbf{ability modeling}, we mimic human cognition to form a four-level taxonomy of knowledge-related abilities, covering $19$ tasks. (2) For \textbf{data}, to ensure fair comparisons, we use both Wikipedia, a corpus prevalently pre-trained by LLMs, along with continuously collected emerging corpora, aiming to evaluate the capacity to handle unseen data and evolving knowledge. (3) For \textbf{evaluation criteria}, we adopt a contrastive system, including overall standard scores for better numerical comparability across tasks and models and a unique self-contrast metric for automatically evaluating knowledge-creating ability. We evaluate $28$ open-source and commercial LLMs and obtain some intriguing findings. The KoLA dataset and open-participation leaderboard are publicly released at https://kola.xlore.cn and will be continuously updated to provide references for developing LLMs and knowledge-related systems.
△ Less
Submitted 30 June, 2024; v1 submitted 15 June, 2023;
originally announced June 2023.
-
The Devil is in the Details: On the Pitfalls of Event Extraction Evaluation
Authors:
Hao Peng,
Xiaozhi Wang,
Feng Yao,
Kaisheng Zeng,
Lei Hou,
Juanzi Li,
Zhiyuan Liu,
Weixing Shen
Abstract:
Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing…
▽ More
Event extraction (EE) is a crucial task aiming at extracting events from texts, which includes two subtasks: event detection (ED) and event argument extraction (EAE). In this paper, we check the reliability of EE evaluations and identify three major pitfalls: (1) The data preprocessing discrepancy makes the evaluation results on the same dataset not directly comparable, but the data preprocessing details are not widely noted and specified in papers. (2) The output space discrepancy of different model paradigms makes different-paradigm EE models lack grounds for comparison and also leads to unclear mapping issues between predictions and annotations. (3) The absence of pipeline evaluation of many EAE-only works makes them hard to be directly compared with EE works and may not well reflect the model performance in real-world pipeline scenarios. We demonstrate the significant influence of these pitfalls through comprehensive meta-analyses of recent papers and empirical experiments. To avoid these pitfalls, we suggest a series of remedies, including specifying data preprocessing, standardizing outputs, and providing pipeline evaluation results. To help implement these remedies, we develop a consistent evaluation framework OMNIEVENT, which can be obtained from https://github.com/THU-KEG/OmniEvent.
△ Less
Submitted 15 June, 2023; v1 submitted 12 June, 2023;
originally announced June 2023.
-
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Authors:
Yushi Bai,
Jiahao Ying,
Yixin Cao,
Xin Lv,
Yuze He,
Xiaozhi Wang,
Jifan Yu,
Kaisheng Zeng,
Yijia Xiao,
Haozhe Lyu,
Jiayin Zhang,
Juanzi Li,
Lei Hou
Abstract:
Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and…
▽ More
Numerous benchmarks have been established to assess the performance of foundation models on open-ended question answering, which serves as a comprehensive test of a model's ability to understand and generate language in a manner similar to humans. Most of these works focus on proposing new datasets, however, we see two main issues within previous benchmarking pipelines, namely testing leakage and evaluation automation. In this paper, we propose a novel benchmarking framework, Language-Model-as-an-Examiner, where the LM serves as a knowledgeable examiner that formulates questions based on its knowledge and evaluates responses in a reference-free manner. Our framework allows for effortless extensibility as various LMs can be adopted as the examiner, and the questions can be constantly updated given more diverse trigger topics. For a more comprehensive and equitable evaluation, we devise three strategies: (1) We instruct the LM examiner to generate questions across a multitude of domains to probe for a broad acquisition, and raise follow-up questions to engage in a more in-depth assessment. (2) Upon evaluation, the examiner combines both scoring and ranking measurements, providing a reliable result as it aligns closely with human annotations. (3) We additionally propose a decentralized Peer-examination method to address the biases in a single examiner. Our data and benchmarking results are available at: http://lmexam.xlore.cn.
△ Less
Submitted 4 November, 2023; v1 submitted 7 June, 2023;
originally announced June 2023.
-
Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction
Authors:
Ji Qi,
Chuchun Zhang,
Xiaozhi Wang,
Kaisheng Zeng,
Jifan Yu,
Jinxin Liu,
Jiuding Sun,
Yuxiang Chen,
Lei Hou,
Juanzi Li,
Bin Xu
Abstract:
The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open inf…
▽ More
The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code are available at https://github.com/qijimrc/ROBUST.
△ Less
Submitted 24 October, 2023; v1 submitted 23 May, 2023;
originally announced May 2023.
-
Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awareness
Authors:
Zeyuan Tan,
Xiulong Yuan,
Congjie He,
Man-Kit Sit,
Guo Li,
Xiaoze Liu,
Baole Ai,
Kai Zeng,
Peter Pietzuch,
Luo Mai
Abstract:
Systems for serving inference requests on graph neural networks (GNN) must combine low latency with high throughout, but they face irregular computation due to skew in the number of sampled graph nodes and aggregated GNN features. This makes it challenging to exploit GPUs effectively: using GPUs to sample only a few graph nodes yields lower performance than CPU-based sampling; and aggregating many…
▽ More
Systems for serving inference requests on graph neural networks (GNN) must combine low latency with high throughout, but they face irregular computation due to skew in the number of sampled graph nodes and aggregated GNN features. This makes it challenging to exploit GPUs effectively: using GPUs to sample only a few graph nodes yields lower performance than CPU-based sampling; and aggregating many features exhibits high data movement costs between GPUs and CPUs. Therefore, current GNN serving systems use CPUs for graph sampling and feature aggregation, limiting throughput.
We describe Quiver, a distributed GPU-based GNN serving system with low-latency and high-throughput. Quiver's key idea is to exploit workload metrics for predicting the irregular computation of GNN requests, and governing the use of GPUs for graph sampling and feature aggregation: (1) for graph sampling, Quiver calculates the probabilistic sampled graph size, a metric that predicts the degree of parallelism in graph sampling. Quiver uses this metric to assign sampling tasks to GPUs only when the performance gains surpass CPU-based sampling; and (2) for feature aggregation, Quiver relies on the feature access probability to decide which features to partition and replicate across a distributed GPU NUMA topology. We show that Quiver achieves up to 35 times lower latency with an 8 times higher throughput compared to state-of-the-art GNN approaches (DGL and PyG).
△ Less
Submitted 18 May, 2023;
originally announced May 2023.
-
Autoencoders for discovering manifold dimension and coordinates in data from complex dynamical systems
Authors:
Kevin Zeng,
Carlos E. Pérez De Jesús,
Andrew J. Fox,
Michael D. Graham
Abstract:
While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and $L_2$ regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal…
▽ More
While many phenomena in physics and engineering are formally high-dimensional, their long-time dynamics often live on a lower-dimensional manifold. The present work introduces an autoencoder framework that combines implicit regularization with internal linear layers and $L_2$ regularization (weight decay) to automatically estimate the underlying dimensionality of a data set, produce an orthogonal manifold coordinate system, and provide the mapping functions between the ambient space and manifold space, allowing for out-of-sample projections. We validate our framework's ability to estimate the manifold dimension for a series of datasets from dynamical systems of varying complexities and compare to other state-of-the-art estimators. We analyze the training dynamics of the network to glean insight into the mechanism of low-rank learning and find that collectively each of the implicit regularizing layers compound the low-rank representation and even self-correct during training. Analysis of gradient descent dynamics for this architecture in the linear case reveals the role of the internal linear layers in leading to faster decay of a "collective weight variable" incorporating all layers, and the role of weight decay in breaking degeneracies and thus driving convergence along directions in which no decay would occur in its absence. We show that this framework can be naturally extended for applications of state-space modeling and forecasting by generating a data-driven dynamic model of a spatiotemporally chaotic partial differential equation using only the manifold coordinates. Finally, we demonstrate that our framework is robust to hyperparameter choices.
△ Less
Submitted 6 December, 2023; v1 submitted 1 May, 2023;
originally announced May 2023.
-
Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics
Authors:
Kuo-Hao Zeng,
Luca Weihs,
Roozbeh Mottaghi,
Ali Farhadi
Abstract:
A common assumption when training embodied agents is that the impact of taking an action is stable; for instance, executing the "move ahead" action will always move the agent forward by a fixed distance, perhaps with some small amount of actuator-induced noise. This assumption is limiting; an agent may encounter settings that dramatically alter the impact of actions: a move ahead action on a wet f…
▽ More
A common assumption when training embodied agents is that the impact of taking an action is stable; for instance, executing the "move ahead" action will always move the agent forward by a fixed distance, perhaps with some small amount of actuator-induced noise. This assumption is limiting; an agent may encounter settings that dramatically alter the impact of actions: a move ahead action on a wet floor may send the agent twice as far as it expects and using the same action with a broken wheel might transform the expected translation into a rotation. Instead of relying that the impact of an action stably reflects its pre-defined semantic meaning, we propose to model the impact of actions on-the-fly using latent embeddings. By combining these latent action embeddings with a novel, transformer-based, policy head, we design an Action Adaptive Policy (AAP). We evaluate our AAP on two challenging visual navigation tasks in the AI2-THOR and Habitat environments and show that our AAP is highly performant even when faced, at inference-time with missing actions and, previously unseen, perturbed action space. Moreover, we observe significant improvement in robustness against these actions when evaluating in real-world scenarios.
△ Less
Submitted 24 April, 2023;
originally announced April 2023.
-
Assessing the Socio-economic Impacts of Secure Texting and Anti-Jamming Technologies in Non-Cooperative Networks
Authors:
Osoro B Ogutu,
Edward J Oughton,
Kai Zeng,
Brian L. Mark
Abstract:
Operating securely over 5G (and legacy) infrastructure is a challenge. In non-cooperative networks, malicious actors may try to decipher, block encrypted messages, or specifically jam wireless radio systems. Such activities can disrupt operations, from causing minor inconvenience, through to fully paralyzing the functionality of critical infrastructure. While technological mitigation measures do e…
▽ More
Operating securely over 5G (and legacy) infrastructure is a challenge. In non-cooperative networks, malicious actors may try to decipher, block encrypted messages, or specifically jam wireless radio systems. Such activities can disrupt operations, from causing minor inconvenience, through to fully paralyzing the functionality of critical infrastructure. While technological mitigation measures do exist, there are very few methods capable of assessing the socio-economic impacts from different mitigation strategies. This leads to a lack of robust evidence to inform cost-benefit analysis, and thus support decision makers in industry and government. Consequently, this paper presents two open-source simulation models for assessing the socio-economic impacts of operating in untrusted non-cooperative networks. The first focuses on using multiple non-cooperative networks to transmit a message. The second model simulates a case where a message is converted into alternative plain language to avoid detection, separated into different portions and then transmitted over multiple non-cooperative networks. A probabilistic simulation of the two models is performed for a 15 km by 15 km spatial grid with 5 untrusted non-cooperative networks and intercepting agents. The results are used to estimate economic losses for private, commercial, government and military sectors. The highest probabilistic total losses for military applications include US$300, US$150, and US$75, incurred for a 1, 3 and 5 site multi-transmission approach, respectively, for non-cooperative networks when considering 1,000 texts being sent. These results form a framework for deterministic socio-economic impact analysis of using non-cooperative networks and secure texting as protection against radio network attacks. The simulation data and the open-source codebase is provided for reproducibility.
△ Less
Submitted 10 April, 2023; v1 submitted 29 March, 2023;
originally announced March 2023.
-
Overcoming losses in superlenses with synthetic waves of complex frequency
Authors:
Fuxin Guan,
Kebo Zeng,
Zhaoyu Nie,
Xiangdong Guo,
Shaojie Ma,
Qing Dai,
John B. Pendry,
Xiang Zhang,
Shuang Zhang
Abstract:
Superlenses made of plasmonic materials and metamaterials have been exploited to image features of sub-diffractional scale. However, their intrinsic losses impose a serious restriction on the imaging resolution, which is a long-standing problem that has hindered wide-spread applications of superlenses. Optical waves of complex frequency exhibiting a temporally attenuating behavior have been propos…
▽ More
Superlenses made of plasmonic materials and metamaterials have been exploited to image features of sub-diffractional scale. However, their intrinsic losses impose a serious restriction on the imaging resolution, which is a long-standing problem that has hindered wide-spread applications of superlenses. Optical waves of complex frequency exhibiting a temporally attenuating behavior have been proposed to offset the intrinsic losses in superlenses via virtual gain, but the experimental realization has been missing due to the challenge involved in preparing the illumination with temporal decay. Here, by employing multi-frequency measurement, we successfully implement a synthetic optical wave of complex frequency to experimentally observe deep-subwavelength superimaging patterns enabled by the virtual gain. Our work represents a practical approach to overcoming the intrinsic losses of plasmonic systems for imaging and sensing applications.
△ Less
Submitted 22 March, 2023;
originally announced March 2023.