-
GS-Octree: Octree-based 3D Gaussian Splatting for Robust Object-level 3D Reconstruction Under Strong Lighting
Authors:
Jiaze Li,
Zhengyu Wen,
Luo Zhang,
Jiangbei Hu,
Fei Hou,
Zhebin Zhang,
Ying He
Abstract:
The 3D Gaussian Splatting technique has significantly advanced the construction of radiance fields from multi-view images, enabling real-time rendering. While point-based rasterization effectively reduces computational demands for rendering, it often struggles to accurately reconstruct the geometry of the target object, especially under strong lighting. To address this challenge, we introduce a no…
▽ More
The 3D Gaussian Splatting technique has significantly advanced the construction of radiance fields from multi-view images, enabling real-time rendering. While point-based rasterization effectively reduces computational demands for rendering, it often struggles to accurately reconstruct the geometry of the target object, especially under strong lighting. To address this challenge, we introduce a novel approach that combines octree-based implicit surface representations with Gaussian splatting. Our method consists of four stages. Initially, it reconstructs a signed distance field (SDF) and a radiance field through volume rendering, encoding them in a low-resolution octree. The initial SDF represents the coarse geometry of the target object. Subsequently, it introduces 3D Gaussians as additional degrees of freedom, which are guided by the SDF. In the third stage, the optimized Gaussians further improve the accuracy of the SDF, allowing it to recover finer geometric details compared to the initial SDF obtained in the first stage. Finally, it adopts the refined SDF to further optimize the 3D Gaussians via splatting, eliminating those that contribute little to visual appearance. Experimental results show that our method, which leverages the distribution of 3D Gaussians with SDFs, reconstructs more accurate geometry, particularly in images with specular highlights caused by strong lighting.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
Shimo Lab at "Discharge Me!": Discharge Summarization by Prompt-Driven Concatenation of Electronic Health Record Sections
Authors:
Yunzhen He,
Hiroaki Yamagiwa,
Hidetoshi Shimodaira
Abstract:
In this paper, we present our approach to the shared task "Discharge Me!" at the BioNLP Workshop 2024. The primary goal of this task is to reduce the time and effort clinicians spend on writing detailed notes in the electronic health record (EHR). Participants develop a pipeline to generate the "Brief Hospital Course" and "Discharge Instructions" sections from the EHR. Our approach involves a firs…
▽ More
In this paper, we present our approach to the shared task "Discharge Me!" at the BioNLP Workshop 2024. The primary goal of this task is to reduce the time and effort clinicians spend on writing detailed notes in the electronic health record (EHR). Participants develop a pipeline to generate the "Brief Hospital Course" and "Discharge Instructions" sections from the EHR. Our approach involves a first step of extracting the relevant sections from the EHR. We then add explanatory prompts to these sections and concatenate them with separate tokens to create the input text. To train a text generation model, we perform LoRA fine-tuning on the ClinicalT5-large model. On the final test data, our approach achieved a ROUGE-1 score of $0.394$, which is comparable to the top solutions.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptation
Authors:
Baoqi Pei,
Guo Chen,
Jilan Xu,
Yuping He,
Yicheng Liu,
Kanghua Pan,
Yifei Huang,
Yali Wang,
Tong Lu,
Limin Wang,
Yu Qiao
Abstract:
In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the uniqu…
▽ More
In this report, we present our solutions to the EgoVis Challenges in CVPR 2024, including five tracks in the Ego4D challenge and three tracks in the EPIC-Kitchens challenge. Building upon the video-language two-tower model and leveraging our meticulously organized egocentric video data, we introduce a novel foundation model called EgoVideo. This model is specifically designed to cater to the unique characteristics of egocentric videos and provides strong support for our competition submissions. In the Ego4D challenges, we tackle various tasks including Natural Language Queries, Step Grounding, Moment Queries, Short-term Object Interaction Anticipation, and Long-term Action Anticipation. In addition, we also participate in the EPIC-Kitchens challenge, where we engage in the Action Recognition, Multiple Instance Retrieval, and Domain Adaptation for Action Recognition tracks. By adapting EgoVideo to these diverse tasks, we showcase its versatility and effectiveness in different egocentric video analysis scenarios, demonstrating the powerful representation ability of EgoVideo as an egocentric foundation model. Our codebase and pretrained models are publicly available at https://github.com/OpenGVLab/EgoVideo.
△ Less
Submitted 30 June, 2024; v1 submitted 26 June, 2024;
originally announced June 2024.
-
Leveraging Pre-trained Models for FF-to-FFPE Histopathological Image Translation
Authors:
Qilai Zhang,
Jiawen Li,
Peiran Liao,
Jiali Hu,
Tian Guan,
Anjia Han,
Yonghong He
Abstract:
The two primary types of Hematoxylin and Eosin (H&E) slides in histopathology are Formalin-Fixed Paraffin-Embedded (FFPE) and Fresh Frozen (FF). FFPE slides offer high quality histopathological images but require a labor-intensive acquisition process. In contrast, FF slides can be prepared quickly, but the image quality is relatively poor. Our task is to translate FF images into FFPE style, thereb…
▽ More
The two primary types of Hematoxylin and Eosin (H&E) slides in histopathology are Formalin-Fixed Paraffin-Embedded (FFPE) and Fresh Frozen (FF). FFPE slides offer high quality histopathological images but require a labor-intensive acquisition process. In contrast, FF slides can be prepared quickly, but the image quality is relatively poor. Our task is to translate FF images into FFPE style, thereby improving the image quality for diagnostic purposes. In this paper, we propose Diffusion-FFPE, a method for FF-to-FFPE histopathological image translation using a pre-trained diffusion model. Specifically, we employ a one-step diffusion model as the generator and fine-tune it with LoRA adapters using adversarial learning objectives. To ensure that the model effectively captures both global structural information and local details, we propose a multi-scale feature fusion (MFF) module. This module utilizes two VAE encoders to extract features of varying image sizes and performs feature fusion before feeding them into the UNet. Furthermore, we utilize a pre-trained vision-language model for histopathology as the backbone for the discriminator to further improve performance We conducted FF-to-FFPE translation experiments on the TCGA-NSCLC datasets, and our method achieved better performance compared to other methods. The code and models are released at https://github.com/QilaiZhang/Diffusion-FFPE.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective
Authors:
Hanqi Yan,
Yanzheng Xiang,
Guangyi Chen,
Yifei Wang,
Lin Gui,
Yulan He
Abstract:
To better interpret the intrinsic mechanism of large language models (LLMs), recent studies focus on monosemanticity on its basic units. A monosemantic neuron is dedicated to a single and specific concept, which forms a one-to-one correlation between neurons and concepts. Despite extensive research in monosemanticity probing, it remains unclear whether monosemanticity is beneficial or harmful to m…
▽ More
To better interpret the intrinsic mechanism of large language models (LLMs), recent studies focus on monosemanticity on its basic units. A monosemantic neuron is dedicated to a single and specific concept, which forms a one-to-one correlation between neurons and concepts. Despite extensive research in monosemanticity probing, it remains unclear whether monosemanticity is beneficial or harmful to model capacity. To explore this question, we revisit monosemanticity from the feature decorrelation perspective and advocate for its encouragement. We experimentally observe that the current conclusion by wang2024learning, which suggests that decreasing monosemanticity enhances model performance, does not hold when the model changes. Instead, we demonstrate that monosemanticity consistently exhibits a positive correlation with model capacity, in the preference alignment process. Consequently, we apply feature correlation as a proxy for monosemanticity and incorporate a feature decorrelation regularizer into the dynamic preference optimization process. The experiments show that our method not only enhances representation diversity and activation sparsity but also improves preference alignment performance.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
End-to-End Autonomous Driving without Costly Modularization and 3D Manual Annotation
Authors:
Mingzhe Guo,
Zhipeng Zhang,
Yuan He,
Ke Wang,
Liping Jing
Abstract:
We propose UAD, a method for vision-based end-to-end autonomous driving (E2EAD), achieving the best open-loop evaluation performance in nuScenes, meanwhile showing robust closed-loop driving quality in CARLA. Our motivation stems from the observation that current E2EAD models still mimic the modular architecture in typical driving stacks, with carefully designed supervised perception and predictio…
▽ More
We propose UAD, a method for vision-based end-to-end autonomous driving (E2EAD), achieving the best open-loop evaluation performance in nuScenes, meanwhile showing robust closed-loop driving quality in CARLA. Our motivation stems from the observation that current E2EAD models still mimic the modular architecture in typical driving stacks, with carefully designed supervised perception and prediction subtasks to provide environment information for oriented planning. Although achieving groundbreaking progress, such design has certain drawbacks: 1) preceding subtasks require massive high-quality 3D annotations as supervision, posing a significant impediment to scaling the training data; 2) each submodule entails substantial computation overhead in both training and inference. To this end, we propose UAD, an E2EAD framework with an unsupervised proxy to address all these issues. Firstly, we design a novel Angular Perception Pretext to eliminate the annotation requirement. The pretext models the driving scene by predicting the angular-wise spatial objectness and temporal dynamics, without manual annotation. Secondly, a self-supervised training strategy, which learns the consistency of the predicted trajectories under different augment views, is proposed to enhance the planning robustness in steering scenarios. Our UAD achieves 38.7% relative improvements over UniAD on the average collision rate in nuScenes and surpasses VAD for 41.32 points on the driving score in CARLA's Town05 Long benchmark. Moreover, the proposed method only consumes 44.3% training resources of UniAD and runs 3.4 times faster in inference. Our innovative design not only for the first time demonstrates unarguable performance advantages over supervised counterparts, but also enjoys unprecedented efficiency in data, training, and inference. Code and models will be released at https://github.com/KargoBot_Research/UAD.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
DMF-Net: Image-Guided Point Cloud Completion with Dual-Channel Modality Fusion and Shape-Aware Upsampling Transformer
Authors:
Aihua Mao,
Yuxuan Tang,
Jiangtao Huang,
Ying He
Abstract:
In this paper we study the task of a single-view image-guided point cloud completion. Existing methods have got promising results by fusing the information of image into point cloud explicitly or implicitly. However, given that the image has global shape information and the partial point cloud has rich local details, We believe that both modalities need to be given equal attention when performing…
▽ More
In this paper we study the task of a single-view image-guided point cloud completion. Existing methods have got promising results by fusing the information of image into point cloud explicitly or implicitly. However, given that the image has global shape information and the partial point cloud has rich local details, We believe that both modalities need to be given equal attention when performing modality fusion. To this end, we propose a novel dual-channel modality fusion network for image-guided point cloud completion(named DMF-Net), in a coarse-to-fine manner. In the first stage, DMF-Net takes a partial point cloud and corresponding image as input to recover a coarse point cloud. In the second stage, the coarse point cloud will be upsampled twice with shape-aware upsampling transformer to get the dense and complete point cloud. Extensive quantitative and qualitative experimental results show that DMF-Net outperforms the state-of-the-art unimodal and multimodal point cloud completion works on ShapeNet-ViPC dataset.
△ Less
Submitted 25 June, 2024;
originally announced June 2024.
-
FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models
Authors:
Haonan Qiu,
Zhaoxi Chen,
Zhouxia Wang,
Yingqing He,
Menghan Xia,
Ziwei Liu
Abstract:
Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introd…
▽ More
Diffusion model has demonstrated remarkable capability in video generation, which further sparks interest in introducing trajectory control into the generation process. While existing works mainly focus on training-based methods (e.g., conditional adapter), we argue that diffusion model itself allows decent control over the generated content without requiring any training. In this study, we introduce a tuning-free framework to achieve trajectory-controllable video generation, by imposing guidance on both noise construction and attention computation. Specifically, 1) we first show several instructive phenomenons and analyze how initial noises influence the motion trajectory of generated content. 2) Subsequently, we propose FreeTraj, a tuning-free approach that enables trajectory control by modifying noise sampling and attention mechanisms. 3) Furthermore, we extend FreeTraj to facilitate longer and larger video generation with controllable trajectories. Equipped with these designs, users have the flexibility to provide trajectories manually or opt for trajectories automatically generated by the LLM trajectory planner. Extensive experiments validate the efficacy of our approach in enhancing the trajectory controllability of video diffusion models.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling
Authors:
Yuxin Dai,
Qi Wang,
Jingsen Zhu,
Dianbing Xi,
Yuchi Huo,
Chen Qian,
Ying He
Abstract:
We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based…
▽ More
We present MIRReS, a novel two-stage inverse rendering framework that jointly reconstructs and optimizes the explicit geometry, material, and lighting from multi-view images. Unlike previous methods that rely on implicit irradiance fields or simplified path tracing algorithms, our method extracts an explicit geometry (triangular mesh) in stage one, and introduces a more realistic physically-based inverse rendering model that utilizes multi-bounce path tracing and Monte Carlo integration. By leveraging multi-bounce path tracing, our method effectively estimates indirect illumination, including self-shadowing and internal reflections, which improves the intrinsic decomposition of shape, material, and lighting. Moreover, we incorporate reservoir sampling into our framework to address the noise in Monte Carlo integration, enhancing convergence and facilitating gradient-based optimization with low sample counts. Through qualitative and quantitative evaluation of several scenarios, especially in challenging scenarios with complex shadows, we demonstrate that our method achieves state-of-the-art performance on decomposition results. Additionally, our optimized explicit geometry enables applications such as scene editing, relighting, and material editing with modern graphics engines or CAD software. The source code is available at https://brabbitdousha.github.io/MIRReS/
△ Less
Submitted 24 June, 2024; v1 submitted 24 June, 2024;
originally announced June 2024.
-
Additive Manufacturing of functionalised atomic vapour cells for next-generation quantum technologies
Authors:
Feiran Wang,
Nathan Cooper,
Yinfeng He,
Benjamin Hopton,
David Johnson,
Peng Zhao,
T. Mark Fromhold,
Christopher J. Tuck,
Richard Hague,
Ricky D. Wildman,
Lyudmila Turyanska,
Lucia Hackermüller
Abstract:
Atomic vapour cells are an indispensable tool for quantum technologies (QT), but potential improvements are limited by the capacities of conventional manufacturing methods. Using an additive manufacturing (AM) technique - vat polymerisation by digital light processing - we demonstrate, for the first time, a 3D-printed glass vapour cell. The exploitation of AM capacities allows intricate internal a…
▽ More
Atomic vapour cells are an indispensable tool for quantum technologies (QT), but potential improvements are limited by the capacities of conventional manufacturing methods. Using an additive manufacturing (AM) technique - vat polymerisation by digital light processing - we demonstrate, for the first time, a 3D-printed glass vapour cell. The exploitation of AM capacities allows intricate internal architectures, overprinting of 2D optoelectronical materials to create integrated sensors and surface functionalisation, while also showing the ability to tailor the optical properties of the AM glass by in-situ growth of gold nanoparticles. The produced cells achieve ultra-high vacuum of $2 \times 10^{-9}$ mbar and enable Doppler-free spectroscopy; we demonstrate laser frequency stabilisation as a QT application. These results highlight the transformative role that AM can play for QT in enabling compact, optimised and integrated multi-material components and devices.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
Rapid and Accurate Diagnosis of Acute Aortic Syndrome using Non-contrast CT: A Large-scale, Retrospective, Multi-center and AI-based Study
Authors:
Yujian Hu,
Yilang Xiang,
Yan-Jie Zhou,
Yangyan He,
Shifeng Yang,
Xiaolong Du,
Chunlan Den,
Youyao Xu,
Gaofeng Wang,
Zhengyao Ding,
Jingyong Huang,
Wenjun Zhao,
Xuejun Wu,
Donglin Li,
Qianqian Zhu,
Zhenjiang Li,
Chenyang Qiu,
Ziheng Wu,
Yunjun He,
Chen Tian,
Yihui Qiu,
Zuodong Lin,
Xiaolong Zhang,
Yuan He,
Zhenpeng Yuan
, et al. (15 additional authors not shown)
Abstract:
Chest pain symptoms are highly prevalent in emergency departments (EDs), where acute aortic syndrome (AAS) is a catastrophic cardiovascular emergency with a high fatality rate, especially when timely and accurate treatment is not administered. However, current triage practices in the ED can cause up to approximately half of patients with AAS to have an initially missed diagnosis or be misdiagnosed…
▽ More
Chest pain symptoms are highly prevalent in emergency departments (EDs), where acute aortic syndrome (AAS) is a catastrophic cardiovascular emergency with a high fatality rate, especially when timely and accurate treatment is not administered. However, current triage practices in the ED can cause up to approximately half of patients with AAS to have an initially missed diagnosis or be misdiagnosed as having other acute chest pain conditions. Subsequently, these AAS patients will undergo clinically inaccurate or suboptimal differential diagnosis. Fortunately, even under these suboptimal protocols, nearly all these patients underwent non-contrast CT covering the aorta anatomy at the early stage of differential diagnosis. In this study, we developed an artificial intelligence model (DeepAAS) using non-contrast CT, which is highly accurate for identifying AAS and provides interpretable results to assist in clinical decision-making. Performance was assessed in two major phases: a multi-center retrospective study (n = 20,750) and an exploration in real-world emergency scenarios (n = 137,525). In the multi-center cohort, DeepAAS achieved a mean area under the receiver operating characteristic curve of 0.958 (95% CI 0.950-0.967). In the real-world cohort, DeepAAS detected 109 AAS patients with misguided initial suspicion, achieving 92.6% (95% CI 76.2%-97.5%) in mean sensitivity and 99.2% (95% CI 99.1%-99.3%) in mean specificity. Our AI model performed well on non-contrast CT at all applicable early stages of differential diagnosis workflows, effectively reduced the overall missed diagnosis and misdiagnosis rate from 48.8% to 4.8% and shortened the diagnosis time for patients with misguided initial suspicion from an average of 681.8 (74-11,820) mins to 68.5 (23-195) mins. DeepAAS could effectively fill the gap in the current clinical workflow without requiring additional tests.
△ Less
Submitted 16 July, 2024; v1 submitted 13 June, 2024;
originally announced June 2024.
-
Gaussian-Informed Continuum for Physical Property Identification and Simulation
Authors:
Junhao Cai,
Yuji Yang,
Weihao Yuan,
Yisheng He,
Zilong Dong,
Liefeng Bo,
Hui Cheng,
Qifeng Chen
Abstract:
This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to deduce implicit shapes during training. We propose a…
▽ More
This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to deduce implicit shapes during training. We propose a new dynamic 3D Gaussian framework based on motion factorization to recover the object as 3D Gaussian point sets across different time states. Furthermore, we develop a coarse-to-fine filling strategy to generate the density fields of the object from the Gaussian reconstruction, allowing for the extraction of object continuums along with their surfaces and the integration of Gaussian attributes into these continuums. In addition to the extracted object surfaces, the Gaussian-informed continuum also enables the rendering of object masks during simulations, serving as implicit shape guidance for physical property estimation. Extensive experimental evaluations demonstrate that our pipeline achieves state-of-the-art performance across multiple benchmarks and metrics. Additionally, we illustrate the effectiveness of the proposed method through real-world demonstrations, showcasing its practical utility. Our project page is at https://jukgei.github.io/project/gic.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
Older and Wiser: The Marriage of Device Aging and Intellectual Property Protection of Deep Neural Networks
Authors:
Ning Lin,
Shaocong Wang,
Yue Zhang,
Yangu He,
Kwunhang Wong,
Arindam Basu,
Dashan Shang,
Xiaoming Chen,
Zhongrui Wang
Abstract:
Deep neural networks (DNNs), such as the widely-used GPT-3 with billions of parameters, are often kept secret due to high training costs and privacy concerns surrounding the data used to train them. Previous approaches to securing DNNs typically require expensive circuit redesign, resulting in additional overheads such as increased area, energy consumption, and latency. To address these issues, we…
▽ More
Deep neural networks (DNNs), such as the widely-used GPT-3 with billions of parameters, are often kept secret due to high training costs and privacy concerns surrounding the data used to train them. Previous approaches to securing DNNs typically require expensive circuit redesign, resulting in additional overheads such as increased area, energy consumption, and latency. To address these issues, we propose a novel hardware-software co-design approach for DNN intellectual property (IP) protection that capitalizes on the inherent aging characteristics of circuits and a novel differential orientation fine-tuning (DOFT) to ensure effective protection. Hardware-wise, we employ random aging to produce authorized chips. This process circumvents the need for chip redesign, thereby eliminating any additional hardware overhead during the inference procedure of DNNs. Moreover, the authorized chips demonstrate a considerable disparity in DNN inference performance when compared to unauthorized chips. Software-wise, we propose a novel DOFT, which allows pre-trained DNNs to maintain their original accuracy on authorized chips with minimal fine-tuning, while the model's performance on unauthorized chips is reduced to random guessing. Extensive experiments on various models, including MLP, VGG, ResNet, Mixer, and SwinTransformer, with lightweight binary and practical multi-bit weights demonstrate that the proposed method achieves effective IP protection, with only 10\% accuracy on unauthorized chips, while preserving nearly the original accuracy on authorized ones.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models
Authors:
Shilong Li,
Yancheng He,
Hangyu Guo,
Xingyuan Bu,
Ge Bai,
Jie Liu,
Jiaheng Liu,
Xingwei Qu,
Yangguang Li,
Wanli Ouyang,
Wenbo Su,
Bo Zheng
Abstract:
Long-context capabilities are essential for large language models (LLMs) to tackle complex and long-input tasks. Despite numerous efforts made to optimize LLMs for long contexts, challenges persist in robustly processing long inputs. In this paper, we introduce GraphReader, a graph-based agent system designed to handle long texts by structuring them into a graph and employing an agent to explore t…
▽ More
Long-context capabilities are essential for large language models (LLMs) to tackle complex and long-input tasks. Despite numerous efforts made to optimize LLMs for long contexts, challenges persist in robustly processing long inputs. In this paper, we introduce GraphReader, a graph-based agent system designed to handle long texts by structuring them into a graph and employing an agent to explore this graph autonomously. Upon receiving a question, the agent first undertakes a step-by-step analysis and devises a rational plan. It then invokes a set of predefined functions to read node content and neighbors, facilitating a coarse-to-fine exploration of the graph. Throughout the exploration, the agent continuously records new insights and reflects on current circumstances to optimize the process until it has gathered sufficient information to generate an answer. Experimental results on the LV-Eval dataset reveal that GraphReader, using a 4k context window, consistently outperforms GPT-4-128k across context lengths from 16k to 256k by a large margin. Additionally, our approach demonstrates superior performance on four challenging single-hop and multi-hop benchmarks.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Energy Mapping of Existing Building Stock in Cambridge using Energy Performance Certificates and Thermal Infrared Imagery
Authors:
Yinglong He,
Jiayu Pan,
Ramit Debnath,
Ronita Bardhan,
Luke Cullen,
Marco Gomez Jenkins,
Erik Mackie,
George Hawker,
Ian Parry
Abstract:
Energy performance certificate (EPC) and thermal infrared (TIR) images both play a key role in the energy performance mapping of the urban building stock. In this paper, we developed parametric building archetypes using an EPC database and conducted temperature clustering on TIR images acquired through drones and satellite datasets. We evaluated 1725 EPCs of existing building stock in Cambridge, U…
▽ More
Energy performance certificate (EPC) and thermal infrared (TIR) images both play a key role in the energy performance mapping of the urban building stock. In this paper, we developed parametric building archetypes using an EPC database and conducted temperature clustering on TIR images acquired through drones and satellite datasets. We evaluated 1725 EPCs of existing building stock in Cambridge, UK to generate energy consumption profiles. Drone-based TIR images of individual buildings in two Cambridge University colleges were processed using a machine learning pipeline for thermal anomaly detection and investigated the influence of two specific factors that affect the reliability of TIR for energy management applications: ground sample distance (GSD) and angle of view (AOV). The EPC-level results suggest that the construction year of the buildings influences their energy consumption. For example, modern buildings were over 30% more energy-efficient than older ones. Parallelly, older buildings were found to show almost double the energy savings potential through retrofitting than newly constructed buildings. TIR imaging results showed that thermal anomalies can only be properly identified in images with a GSD of 1m/pixel or less. A GSD of 1-6m/pixel can detect hot areas of building surfaces. We found that a GSD > 6m/pixel cannot characterise individual buildings but does help identify urban heat island effects. Additional sensitivity analysis showed that building thermal anomaly detection is more sensitive to AOV than to GSD. Our study informs newer approaches to building energy diagnostics using thermography and supports decision-making for retrofitting at a large scale.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts
Authors:
Zhili Shen,
Zihang Xi,
Ying He,
Wei Tong,
Jingyu Hua,
Sheng Zhong
Abstract:
The rapid adoption of online chatbots represents a significant advancement in artificial intelligence. However, this convenience brings considerable privacy concerns, as prompts can inadvertently contain sensitive information exposed to large language models (LLMs). Limited by high computational costs, reduced task usability, and excessive system modifications, previous works based on local deploy…
▽ More
The rapid adoption of online chatbots represents a significant advancement in artificial intelligence. However, this convenience brings considerable privacy concerns, as prompts can inadvertently contain sensitive information exposed to large language models (LLMs). Limited by high computational costs, reduced task usability, and excessive system modifications, previous works based on local deployment, embedding perturbation, and homomorphic encryption are inapplicable to online prompt-based LLM applications.
To address these issues, this paper introduces Prompt Privacy Sanitizer (i.e., ProSan), an end-to-end prompt privacy protection framework that can produce anonymized prompts with contextual privacy removed while maintaining task usability and human readability. It can also be seamlessly integrated into the online LLM service pipeline. To achieve high usability and dynamic anonymity, ProSan flexibly adjusts its protection targets and strength based on the importance of the words and the privacy leakage risk of the prompts. Additionally, ProSan is capable of adapting to diverse computational resource conditions, ensuring privacy protection even for mobile devices with limited computing power. Our experiments demonstrate that ProSan effectively removes private information across various tasks, including question answering, text summarization, and code generation, with minimal reduction in task performance.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Causal Inference with Latent Variables: Recent Advances and Future Prospectives
Authors:
Yaochen Zhu,
Yinhan He,
Jing Ma,
Mengxuan Hu,
Sheng Li,
Jundong Li
Abstract:
Causality lays the foundation for the trajectory of our world. Causal inference (CI), which aims to infer intrinsic causal relations among variables of interest, has emerged as a crucial research topic. Nevertheless, the lack of observation of important variables (e.g., confounders, mediators, exogenous variables, etc.) severely compromises the reliability of CI methods. The issue may arise from t…
▽ More
Causality lays the foundation for the trajectory of our world. Causal inference (CI), which aims to infer intrinsic causal relations among variables of interest, has emerged as a crucial research topic. Nevertheless, the lack of observation of important variables (e.g., confounders, mediators, exogenous variables, etc.) severely compromises the reliability of CI methods. The issue may arise from the inherent difficulty in measuring the variables. Additionally, in observational studies where variables are passively recorded, certain covariates might be inadvertently omitted by the experimenter. Depending on the type of unobserved variables and the specific CI task, various consequences can be incurred if these latent variables are carelessly handled, such as biased estimation of causal effects, incomplete understanding of causal mechanisms, lack of individual-level causal consideration, etc. In this survey, we provide a comprehensive review of recent developments in CI with latent variables. We start by discussing traditional CI techniques when variables of interest are assumed to be fully observed. Afterward, under the taxonomy of circumvention and inference-based methods, we provide an in-depth discussion of various CI strategies to handle latent variables, covering the tasks of causal effect estimation, mediation analysis, counterfactual reasoning, and causal discovery. Furthermore, we generalize the discussion to graph data where interference among units may exist. Finally, we offer fresh aspects for further advancement of CI with latent variables, especially new opportunities in the era of large language models (LLMs).
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
FastPersist: Accelerating Model Checkpointing in Deep Learning
Authors:
Guanhua Wang,
Olatunji Ruwase,
Bing Xie,
Yuxiong He
Abstract:
Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we prop…
▽ More
Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
In-Context Former: Lightning-fast Compressing Context for Large Language Model
Authors:
Xiangfeng Wang,
Zaiyi Chen,
Zheyong Xie,
Tong Xu,
Yongyi He,
Enhong Chen
Abstract:
With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach is to compress the long input contexts. Existing methods typically leverage the self-attention mechanism of the LLM itself for context compression. While these methods have achieved notable results, the compression process…
▽ More
With the rising popularity of Transformer-based large language models (LLMs), reducing their high inference costs has become a significant research focus. One effective approach is to compress the long input contexts. Existing methods typically leverage the self-attention mechanism of the LLM itself for context compression. While these methods have achieved notable results, the compression process still involves quadratic time complexity, which limits their applicability. To mitigate this limitation, we propose the In-Context Former (IC-Former). Unlike previous methods, IC-Former does not depend on the target LLMs. Instead, it leverages the cross-attention mechanism and a small number of learnable digest tokens to directly condense information from the contextual word embeddings. This approach significantly reduces inference time, which achieves linear growth in time complexity within the compression range. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times while achieving over 90% of the baseline performance on evaluation metrics. Overall, our model effectively reduces compression costs and makes real-time compression scenarios feasible.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Act Better by Timing: A timing-Aware Reinforcement Learning for Autonomous Driving
Authors:
Guanzhou Li,
Jianping Wu,
Yujing He
Abstract:
Coping with intensively interactive scenarios is one of the significant challenges in the development of autonomous driving. Reinforcement learning (RL) offers an ideal solution for such scenarios through its self-evolution mechanism via interaction with the environment. However, the lack of sufficient safety mechanisms in common RL leads to the fact that agent often find it difficult to interact…
▽ More
Coping with intensively interactive scenarios is one of the significant challenges in the development of autonomous driving. Reinforcement learning (RL) offers an ideal solution for such scenarios through its self-evolution mechanism via interaction with the environment. However, the lack of sufficient safety mechanisms in common RL leads to the fact that agent often find it difficult to interact well in highly dynamic environment and may collide in pursuit of short-term rewards. Much of the existing safe RL methods require environment modeling to generate reliable safety boundaries that constrain agent behavior. Nevertheless, acquiring such safety boundaries is not always feasible in dynamic environments. Inspired by the driver's behavior of acting when uncertainty is minimal, this study introduces the concept of action timing to replace explicit safety boundary modeling. We define "actor" as an agent to decide optimal action at each step. By imaging the actor take opportunity to act as a timing-dependent gradual process, the other agent called "timing taker" can evaluate the optimal action execution time, and relate the optimal timing to each action moment as a dynamic safety factor to constrain the actor's action. In the experiment involving a complex, unsignaled intersection interaction, this framework achieved superior safety performance compared to all benchmark models.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
AutoFirm: Automatically Identifying Reused Libraries inside IoT Firmware at Large-Scale
Authors:
YongLe Chen,
Feng Ma,
Ying Zhang,
YongZhong He,
Haining Wang,
Qiang Li
Abstract:
The Internet of Things (IoT) has become indispensable to our daily lives and work. Unfortunately, developers often reuse software libraries in the IoT firmware, leading to a major security concern. If vulnerabilities or insecure versions of these libraries go unpatched, a massive number of IoT devices can be impacted. In this paper, we propose the AutoFirm, an automated tool for detecting reused l…
▽ More
The Internet of Things (IoT) has become indispensable to our daily lives and work. Unfortunately, developers often reuse software libraries in the IoT firmware, leading to a major security concern. If vulnerabilities or insecure versions of these libraries go unpatched, a massive number of IoT devices can be impacted. In this paper, we propose the AutoFirm, an automated tool for detecting reused libraries in IoT firmware at a large scale. Specifically, AutoFirm leverages the syntax information (library name and version) to determine whether IoT firmware reuses the libraries. We conduct a large-scale empirical study of reused libraries of IoT firmware, investigating more than 6,900+ firmware and 2,700+ distinct vulnerabilities affecting 11,300+ vulnerable versions from 349 open-source software libraries. Leveraging this diverse information set, we conduct a qualitative assessment of vulnerable library versions to understand security gaps and the misplaced trust of libraries in IoT firmware. Our research reveals that: manufacturers neglected to update outdated libraries for IoT firmware in 67.3\% of cases; on average, outdated libraries persisted for over 1.34 years prior to remediation; vulnerabilities of software libraries have posed server threats to widespread IoT devices.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Evaluating the design space of diffusion-based generative models
Authors:
Yuqing Wang,
Ye He,
Molei Tao
Abstract:
Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a…
▽ More
Most existing theoretical investigations of the accuracy of diffusion models, albeit significant, assume the score function has been approximated to a certain accuracy, and then use this a priori bound to control the error of generation. This article instead provides a first quantitative understanding of the whole generation process, i.e., both training and sampling. More precisely, it conducts a non-asymptotic convergence analysis of denoising score matching under gradient descent. In addition, a refined sampling error analysis for variance exploding models is also provided. The combination of these two results yields a full error analysis, which elucidates (again, but this time theoretically) how to design the training and sampling processes for effective generation. For instance, our theory implies a preference toward noise distribution and loss weighting in training that qualitatively agree with the ones used in [Karras et al. 2022]. It also provides perspectives on the choices of time and variance schedules in sampling: when the score is well trained, the design in [Song et al. 2020] is more preferable, but when it is less trained, the design in [Karras et al. 2022] becomes more preferable.
△ Less
Submitted 25 July, 2024; v1 submitted 18 June, 2024;
originally announced June 2024.
-
Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Ba
Authors:
Ruiqi He,
Yushu He,
Longju Bai,
Jiarui Liu,
Zhenjie Sun,
Zenghao Tang,
He Wang,
Hanchen Xia,
Naihao Deng
Abstract:
Existing humor datasets and evaluations predominantly focus on English, lacking resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing intellectually challenging and culturally specific jokes. We annotate explanations for each joke and evalua…
▽ More
Existing humor datasets and evaluations predominantly focus on English, lacking resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing intellectually challenging and culturally specific jokes. We annotate explanations for each joke and evaluate human explanations against two state-of-the-art LLMs, GPT-4o and ERNIE Bot, through A/B testing by native Chinese speakers. Our evaluation shows that Chumor is challenging even for SOTA LLMs, and the human explanations for Chumor jokes are significantly better than explanations generated by the LLMs.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Deploying scalable traffic prediction models for efficient management in real-world large transportation networks during hurricane evacuations
Authors:
Qinhua Jiang,
Brian Yueshuai He,
Changju Lee,
Jiaqi Ma
Abstract:
Accurate traffic prediction is vital for effective traffic management during hurricane evacuation. This paper proposes a predictive modeling system that integrates Multilayer Perceptron (MLP) and Long-Short Term Memory (LSTM) models to capture both long-term congestion patterns and short-term speed patterns. Leveraging various input variables, including archived traffic data, spatial-temporal road…
▽ More
Accurate traffic prediction is vital for effective traffic management during hurricane evacuation. This paper proposes a predictive modeling system that integrates Multilayer Perceptron (MLP) and Long-Short Term Memory (LSTM) models to capture both long-term congestion patterns and short-term speed patterns. Leveraging various input variables, including archived traffic data, spatial-temporal road network information, and hurricane forecast data, the framework is designed to address challenges posed by heterogeneous human behaviors, limited evacuation data, and hurricane event uncertainties. Deployed in a real-world traffic prediction system in Louisiana, the model achieved an 82% accuracy in predicting long-term congestion states over a 6-hour period during a 7-day hurricane-impacted duration. The short-term speed prediction model exhibited Mean Absolute Percentage Errors (MAPEs) ranging from 7% to 13% across evacuation horizons from 1 to 6 hours. Evaluation results underscore the model's potential to enhance traffic management during hurricane evacuations, and real-world deployment highlights its adaptability and scalability in diverse hurricane scenarios within extensive transportation networks.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
Multi-Layer Ranking with Large Language Models for News Source Recommendation
Authors:
Wenjia Zhang,
Lin Gui,
Rob Procter,
Yulan He
Abstract:
To seek reliable information sources for news events, we introduce a novel task of expert recommendation, which aims to identify trustworthy sources based on their previously quoted statements. To achieve this, we built a novel dataset, called NewsQuote, consisting of 23,571 quote-speaker pairs sourced from a collection of news articles. We formulate the recommendation task as the retrieval of exp…
▽ More
To seek reliable information sources for news events, we introduce a novel task of expert recommendation, which aims to identify trustworthy sources based on their previously quoted statements. To achieve this, we built a novel dataset, called NewsQuote, consisting of 23,571 quote-speaker pairs sourced from a collection of news articles. We formulate the recommendation task as the retrieval of experts based on their likelihood of being associated with a given query. We also propose a multi-layer ranking framework employing Large Language Models to improve the recommendation performance. Our results show that employing an in-context learning based LLM ranker and a multi-layer ranking-based filter significantly improve both the predictive quality and behavioural quality of the recommender system.
△ Less
Submitted 17 June, 2024;
originally announced June 2024.
-
SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field
Authors:
Yuhang He,
Shitong Xu,
Jia-Xing Zhong,
Sangyun Shin,
Niki Trigoni,
Andrew Markham
Abstract:
We present SPEAR, a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction in an acoustic 3D space with a single stationary audio source. Unlike traditional source-to-receiver modelling methods that require prior space acoustic properties knowledge to rigorously model audio propagation from source to receiver, we propose to predict by warping the spat…
▽ More
We present SPEAR, a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction in an acoustic 3D space with a single stationary audio source. Unlike traditional source-to-receiver modelling methods that require prior space acoustic properties knowledge to rigorously model audio propagation from source to receiver, we propose to predict by warping the spatial acoustic effects from one reference receiver position to another target receiver position, so that the warped audio essentially accommodates all spatial acoustic effects belonging to the target position. SPEAR can be trained in a data much more readily accessible manner, in which we simply ask two robots to independently record spatial audio at different positions. We further theoretically prove the universal existence of the warping field if and only if one audio source presents. Three physical principles are incorporated to guide SPEAR network design, leading to the learned warping field physically meaningful. We demonstrate SPEAR superiority on both synthetic, photo-realistic and real-world dataset, showing the huge potential of SPEAR to various down-stream robotic tasks.
△ Less
Submitted 16 June, 2024;
originally announced June 2024.
-
Ontology Embedding: A Survey of Methods, Applications and Resources
Authors:
Jiaoyan Chen,
Olga Mashkova,
Fernando Zhapa-Camacho,
Robert Hoehndorf,
Yuan He,
Ian Horrocks
Abstract:
Ontologies are widely used for representing domain knowledge and meta data, playing an increasingly important role in Information Systems, the Semantic Web, Bioinformatics and many other domains. However, logical reasoning that ontologies can directly support are quite limited in learning, approximation and prediction. One straightforward solution is to integrate statistical analysis and machine l…
▽ More
Ontologies are widely used for representing domain knowledge and meta data, playing an increasingly important role in Information Systems, the Semantic Web, Bioinformatics and many other domains. However, logical reasoning that ontologies can directly support are quite limited in learning, approximation and prediction. One straightforward solution is to integrate statistical analysis and machine learning. To this end, automatically learning vector representation for knowledge of an ontology i.e., ontology embedding has been widely investigated in recent years. Numerous papers have been published on ontology embedding, but a lack of systematic reviews hinders researchers from gaining a comprehensive understanding of this field. To bridge this gap, we write this survey paper, which first introduces different kinds of semantics of ontologies, and formally defines ontology embedding from the perspectives of both mathematics and machine learning, as well as its property of faithfulness. Based on this, it systematically categorises and analyses a relatively complete set of over 80 papers, according to the ontologies and semantics that they aim at, and their technical solutions including geometric modeling, sequence modeling and graph propagation. This survey also introduces the applications of ontology embedding in ontology engineering, machine learning augmentation and life sciences, presents a new library mOWL, and discusses the challenges and future directions.
△ Less
Submitted 16 June, 2024;
originally announced June 2024.
-
Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence
Authors:
Junru Lu,
Jiazheng Li,
Siyu An,
Meng Zhao,
Yulan He,
Di Yin,
Xing Sun
Abstract:
Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: "verbosity", a common over-optimization phenomenon also observ…
▽ More
Direct Preference Optimization (DPO) has emerged as a prominent algorithm for the direct and robust alignment of Large Language Models (LLMs) with human preferences, offering a more straightforward alternative to the complex Reinforcement Learning from Human Feedback (RLHF). Despite its promising efficacy, DPO faces a notable drawback: "verbosity", a common over-optimization phenomenon also observed in RLHF. While previous studies mainly attributed verbosity to biased labels within the data, we propose that the issue also stems from an inherent algorithmic length reliance in DPO. Specifically, we suggest that the discrepancy between sequence-level Kullback-Leibler (KL) divergences between chosen and rejected sequences, used in DPO, results in overestimated or underestimated rewards due to varying token lengths. Empirically, we utilize datasets with different label lengths to demonstrate the presence of biased rewards. We then introduce an effective downsampling approach, named SamPO, to eliminate potential length reliance. Our experimental evaluations, conducted across three LLMs of varying scales and a diverse array of conditional and open-ended benchmarks, highlight the efficacy of SamPO in mitigating verbosity, achieving improvements of 5% to 12% over DPO through debaised rewards. Our codes can be accessed at: https://github.com/LuJunru/SamPO/.
△ Less
Submitted 14 August, 2024; v1 submitted 16 June, 2024;
originally announced June 2024.
-
Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognition
Authors:
Wenhan Yao,
Jiangkun Yang,
Yongqiang He,
Jia Liu,
Weiping Wen
Abstract:
Speech recognition is an essential start ring of human-computer interaction, and recently, deep learning models have achieved excellent success in this task. However, when the model training and private data provider are always separated, some security threats that make deep neural networks (DNNs) abnormal deserve to be researched. In recent years, the typical backdoor attacks have been researched…
▽ More
Speech recognition is an essential start ring of human-computer interaction, and recently, deep learning models have achieved excellent success in this task. However, when the model training and private data provider are always separated, some security threats that make deep neural networks (DNNs) abnormal deserve to be researched. In recent years, the typical backdoor attacks have been researched in speech recognition systems. The existing backdoor methods are based on data poisoning. The attacker adds some incorporated changes to benign speech spectrograms or changes the speech components, such as pitch and timbre. As a result, the poisoned data can be detected by human hearing or automatic deep algorithms. To improve the stealthiness of data poisoning, we propose a non-neural and fast algorithm called Random Spectrogram Rhythm Transformation (RSRT) in this paper. The algorithm combines four steps to generate stealthy poisoned utterances. From the perspective of rhythm component transformation, our proposed trigger stretches or squeezes the mel spectrograms and recovers them back to signals. The operation keeps timbre and content unchanged for good stealthiness. Our experiments are conducted on two kinds of speech recognition tasks, including testing the stealthiness of poisoned samples by speaker verification and automatic speech recognition. The results show that our method has excellent effectiveness and stealthiness. The rhythm trigger needs a low poisoning rate and gets a very high attack success rate.
△ Less
Submitted 21 August, 2024; v1 submitted 16 June, 2024;
originally announced June 2024.
-
Impurities with a cusp: general theory and 3d Ising
Authors:
Gabriel Cuomo,
Yin-Chen He,
Zohar Komargodski
Abstract:
In CFTs, the partition function of a line defect with a cusp depends logarithmically on the size of the line with an angle-dependent coefficient: the cusp anomalous dimension. In the first part of this work, we study the general properties of the cusp anomalous dimension. We relate the small cusp angle limit to the effective field theory of defect fusion, making predictions for the first couple of…
▽ More
In CFTs, the partition function of a line defect with a cusp depends logarithmically on the size of the line with an angle-dependent coefficient: the cusp anomalous dimension. In the first part of this work, we study the general properties of the cusp anomalous dimension. We relate the small cusp angle limit to the effective field theory of defect fusion, making predictions for the first couple of terms in the expansion. Using a concavity property of the cusp anomalous dimension we argue that the Casimir energy between a line defect and its orientation reversal is always negative ("opposites attract"). We use these results to determine the fusion algebra of Wilson lines in $\mathcal{N}=4$ SYM as well as pinning field defects in the Wilson-Fisher fixed points. In the second part of the paper we obtain nonperturbative numerical results for the cusp anomalous dimension of pinning field defects in the Ising model in $d=3$, using the recently developed fuzzy-sphere regularization. We also compute the pinning field cusp anomalous dimension in the $O(N)$ model at one-loop in the $\varepsilon$-expansion. Our results are in agreement with the general theory developed in the first part of the work, and we make several predictions for impurities in magnets.
△ Less
Submitted 15 July, 2024; v1 submitted 14 June, 2024;
originally announced June 2024.
-
Enhancing Question Answering on Charts Through Effective Pre-training Tasks
Authors:
Ashim Gupta,
Vivek Gupta,
Shuo Zhang,
Yujie He,
Ning Zhang,
Shalin Shah
Abstract:
To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the li…
▽ More
To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7% over the baseline model.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
Advancing Graph Generation through Beta Diffusion
Authors:
Yilin He,
Xinyang Liu,
Bo Chen,
Mingyuan Zhou
Abstract:
Diffusion models have demonstrated effectiveness in generating natural images and have been extended to generate diverse data types, including graphs. This new generation of diffusion-based graph generative models has demonstrated significant performance improvements over methods that rely on variational autoencoders or generative adversarial networks. It's important to recognize, however, that mo…
▽ More
Diffusion models have demonstrated effectiveness in generating natural images and have been extended to generate diverse data types, including graphs. This new generation of diffusion-based graph generative models has demonstrated significant performance improvements over methods that rely on variational autoencoders or generative adversarial networks. It's important to recognize, however, that most of these models employ Gaussian or categorical diffusion processes, which can struggle with sparse and long-tailed data distributions. In our work, we introduce Graph Beta Diffusion (GBD), a diffusion-based generative model particularly adept at capturing diverse graph structures. GBD utilizes a beta diffusion process, tailored for the sparse and range-bounded characteristics of graph adjacency matrices. Furthermore, we have developed a modulation technique that enhances the realism of the generated graphs by stabilizing the generation of critical graph structures, while preserving flexibility elsewhere. The outstanding performance of GBD across three general graph benchmarks and two biochemical graph benchmarks highlights its capability to effectively capture the complexities of real-world graph data. The code will be made available at https://github.com/YH-UtMSB/Graph_Beta_Diffusion
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
Authors:
Jing Liu,
Ruihao Gong,
Mingyang Zhang,
Yefei He,
Jianfei Cai,
Bohan Zhuang
Abstract:
The typical process for developing LLMs involves pre-training a general foundation model on massive data, followed by fine-tuning on task-specific data to create specialized experts. Serving these experts poses challenges, as loading all experts onto devices is impractical, and frequent switching between experts in response to user requests incurs substantial I/O costs, increasing latency and expe…
▽ More
The typical process for developing LLMs involves pre-training a general foundation model on massive data, followed by fine-tuning on task-specific data to create specialized experts. Serving these experts poses challenges, as loading all experts onto devices is impractical, and frequent switching between experts in response to user requests incurs substantial I/O costs, increasing latency and expenses. Previous approaches decompose expert weights into pre-trained model weights and residual delta weights, then quantize the delta weights to reduce model size. However, these methods often lead to significant quantization errors at extremely low bitwidths and assume the appropriate model for a user request is known in advance, which is not practical. To address these issues, we introduce ME-Switch, a memory-efficient expert switching framework for LLM serving. ME-Switch uses mixed-precision quantization, selectively quantizing non-salient input channels of delta weights to extremely low bits while keeping salient ones intact, significantly reducing storage demands while maintaining performance. Additionally, we develop a routing method that efficiently directs user queries to the most suitable expert by transforming the model selection problem into a domain classification problem. Extensive experiments show ME-Switch's promising memory efficiency and routing performance. For example, when serving three models from the Mistral-7B family, ME-Switch reduces model size by 1.74x while maintaining nearly lossless performance on instruction, mathematical reasoning, and code generation tasks. Furthermore, ME-Switch can efficiently serve 16 models from the Mistral-7B family on a single NVIDIA A100 GPU.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Kinematics and star formation of hub-filament systems in W49A
Authors:
WenJun Zhang,
Jianjun Zhou,
Jarken Esimbek,
Willem Baan,
Yuxin He,
Xindi Tang,
Dalei Li,
Weiguang Ji,
Gang Wu,
Yingxiu Ma,
Jiasheng Li,
Dongdong Zhou,
Kadirya Tursun,
Toktarkhan Komesh
Abstract:
W49A is a prominent giant molecular cloud (GMC) that exhibits strong star formation activities, yet its structural and kinematic properties remain uncertain. Our study aims to investigate the large-scale structure and kinematics of W49A, and elucidate the role of filaments and hub-filament systems (HFSs) in its star formation activity. We utilized continuum data from Herschel and the James Clerk M…
▽ More
W49A is a prominent giant molecular cloud (GMC) that exhibits strong star formation activities, yet its structural and kinematic properties remain uncertain. Our study aims to investigate the large-scale structure and kinematics of W49A, and elucidate the role of filaments and hub-filament systems (HFSs) in its star formation activity. We utilized continuum data from Herschel and the James Clerk Maxwell Telescope (JCMT) as well as the molecular lines 12CO (3-2), 13CO (3-2), and C18O (3-2) to identify filaments and HFS structures within W49A. Further analysis focused on the physical properties, kinematics, and mass transport within these structures. Additionally, recombination line emission from the H I/OH/Recombination (THOR) line survey was employed to trace the central H II region and ionized gas. Our findings reveal that W49A comprises one blue-shifted (B-S) HFS and one red-shifted (R-S) HFS, each with multiple filaments and dense hubs. Notably, significant velocity gradients were detected along these filaments, indicative of material transport toward the hubs. High mass accretion rates along the filaments facilitate the formation of massive stars in the HFSs. Furthermore, the presence of V-shaped structures around clumps in position-velocity diagrams suggests ongoing gravitational collapse and local star formation within the filaments. Our results indicate that W49A consists of one R-S HFS and one B-S HFS, and that the material transport from filaments to the hub promotes the formation of massive stars in the hub. These findings underscore the significance of HFSs in shaping the star formation history of W49A.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Constraints on Ultra Heavy Dark Matter Properties from Dwarf Spheroidal Galaxies with LHAASO Observations
Authors:
Zhen Cao,
F. Aharonian,
Q. An,
Axikegu,
Y. X. Bai,
Y. W. Bao,
D. Bastieri,
X. J. Bi,
Y. J. Bi,
J. T. Cai,
Q. Cao,
W. Y. Cao,
Zhe Cao,
J. Chang,
J. F. Chang,
A. M. Chen,
E. S. Chen,
Liang Chen,
Lin Chen,
Long Chen,
M. J. Chen,
M. L. Chen,
Q. H. Chen,
S. H. Chen,
S. Z. Chen
, et al. (255 additional authors not shown)
Abstract:
In this work we try to search for signals generated by ultra-heavy dark matter at the Large High Altitude Air Shower Observatory (LHAASO) data. We look for possible gamma-ray by dark matter annihilation or decay from 16 dwarf spheroidal galaxies in the field of view of LHAASO. Dwarf spheroidal galaxies are among the most promising targets for indirect detection of dark matter which have low fluxes…
▽ More
In this work we try to search for signals generated by ultra-heavy dark matter at the Large High Altitude Air Shower Observatory (LHAASO) data. We look for possible gamma-ray by dark matter annihilation or decay from 16 dwarf spheroidal galaxies in the field of view of LHAASO. Dwarf spheroidal galaxies are among the most promising targets for indirect detection of dark matter which have low fluxes of astrophysical $γ$-ray background while large amount of dark matter. By analyzing more than 700 days observational data at LHAASO, no significant dark matter signal from 1 TeV to 1 EeV is detected. Accordingly we derive the most stringent constraints on the ultra-heavy dark matter annihilation cross-section up to EeV. The constraints on the lifetime of dark matter in decay mode are also derived.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Security of AI Agents
Authors:
Yifeng He,
Ethan Wang,
Yuyang Rong,
Zifei Cheng,
Hao Chen
Abstract:
The study and development of AI agents have been boosted by large language models. AI agents can function as intelligent assistants and complete tasks on behalf of their users with access to tools and the ability to execute commands in their environments, Through studying and experiencing the workflow of typical AI agents, we have raised several concerns regarding their security. These potential v…
▽ More
The study and development of AI agents have been boosted by large language models. AI agents can function as intelligent assistants and complete tasks on behalf of their users with access to tools and the ability to execute commands in their environments, Through studying and experiencing the workflow of typical AI agents, we have raised several concerns regarding their security. These potential vulnerabilities are not addressed by the frameworks used to build the agents, nor by research aimed at improving the agents. In this paper, we identify and describe these vulnerabilities in detail from a system security perspective, emphasizing their causes and severe effects. Furthermore, we introduce defense mechanisms corresponding to each vulnerability with meticulous design and experiments to evaluate their viability. Altogether, this paper contextualizes the security issues in the current development of AI agents and delineates methods to make AI agents safer and more reliable.
△ Less
Submitted 20 June, 2024; v1 submitted 12 June, 2024;
originally announced June 2024.
-
Exploring Fuzzing as Data Augmentation for Neural Test Generation
Authors:
Yifeng He,
Jicheng Wang,
Yuyang Rong,
Hao Chen
Abstract:
Testing is an essential part of modern software engineering to build reliable programs. As testing the software is important but expensive, automatic test case generation methods have become popular in software development. Unlike traditional search-based coverage-guided test generation like fuzzing, neural test generation backed by large language models can write tests that are semantically meani…
▽ More
Testing is an essential part of modern software engineering to build reliable programs. As testing the software is important but expensive, automatic test case generation methods have become popular in software development. Unlike traditional search-based coverage-guided test generation like fuzzing, neural test generation backed by large language models can write tests that are semantically meaningful and can be understood by other maintainers. However, compared to regular code corpus, unit tests in the datasets are limited in amount and diversity. In this paper, we present a novel data augmentation technique **FuzzAug**, that combines the advantages of fuzzing and large language models. FuzzAug not only keeps valid program semantics in the augmented data, but also provides more diverse inputs to the function under test, helping the model to associate correct inputs embedded with the function's dynamic behaviors with the function under test. We evaluate FuzzAug's benefits by using it on a neural test generation dataset to train state-of-the-art code generation models. By augmenting the training set, our model generates test cases with $11\%$ accuracy increases. Models trained with FuzzAug generate unit test functions with double the branch coverage compared to those without it. FuzzAug can be used across various datasets to train advanced code generation models, enhancing their utility in automated software testing. Our work shows the benefits of using dynamic analysis results to enhance neural test generation. Code and data will be publicly available.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
Authors:
Qingyun Li,
Zhe Chen,
Weiyun Wang,
Wenhai Wang,
Shenglong Ye,
Zhenjiang Jin,
Guanzhou Chen,
Yinan He,
Zhangwei Gao,
Erfei Cui,
Jiashuo Yu,
Hao Tian,
Jiasheng Zhou,
Chao Xu,
Bin Wang,
Xingjian Wei,
Wei Li,
Wenjian Zhang,
Bo Zhang,
Pinlong Cai,
Licheng Wen,
Xiangchao Yan,
Zhenxiang Li,
Pei Chu,
Yi Wang
, et al. (15 additional authors not shown)
Abstract:
Image-text interleaved data, consisting of multiple images and texts arranged in a natural document format, aligns with the presentation paradigm of internet data and closely resembles human reading habits. Recent studies have shown that such data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning. However, the limited scale an…
▽ More
Image-text interleaved data, consisting of multiple images and texts arranged in a natural document format, aligns with the presentation paradigm of internet data and closely resembles human reading habits. Recent studies have shown that such data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning. However, the limited scale and diversity of current image-text interleaved data restrict the development of multimodal large language models. In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Using an efficient data engine, we filter and extract large-scale high-quality documents, which contain 8.6 billion images and 1,696 billion text tokens. Compared to counterparts (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while maintaining good data quality; 2) features more diverse sources, including both English and non-English websites as well as video-centric websites; 3) is more flexible, easily degradable from an image-text interleaved format to pure text corpus and image-text pairs. Through comprehensive analysis and experiments, we validate the quality, usability, and effectiveness of the proposed dataset. We hope this could provide a solid data foundation for future multimodal model research. Code and data are released at https://github.com/OpenGVLab/OmniCorpus.
△ Less
Submitted 12 July, 2024; v1 submitted 12 June, 2024;
originally announced June 2024.
-
Continuous-Time Digital Twin with Analogue Memristive Neural Ordinary Differential Equation Solver
Authors:
Hegan Chen,
Jichang Yang,
Jia Chen,
Songqi Wang,
Shaocong Wang,
Dingchen Wang,
Xinyu Tian,
Yifei Yu,
Xi Chen,
Yinan Lin,
Yangu He,
Xiaoshan Wu,
Yi Li,
Xinyuan Zhang,
Ning Lin,
Meng Xu,
Yi Li,
Xumeng Zhang,
Zhongrui Wang,
Han Wang,
Dashan Shang,
Qi Liu,
Kwang-Ting Cheng,
Ming Liu
Abstract:
Digital twins, the cornerstone of Industry 4.0, replicate real-world entities through computer models, revolutionising fields such as manufacturing management and industrial automation. Recent advances in machine learning provide data-driven methods for developing digital twins using discrete-time data and finite-depth models on digital computers. However, this approach fails to capture the underl…
▽ More
Digital twins, the cornerstone of Industry 4.0, replicate real-world entities through computer models, revolutionising fields such as manufacturing management and industrial automation. Recent advances in machine learning provide data-driven methods for developing digital twins using discrete-time data and finite-depth models on digital computers. However, this approach fails to capture the underlying continuous dynamics and struggles with modelling complex system behaviour. Additionally, the architecture of digital computers, with separate storage and processing units, necessitates frequent data transfers and Analogue-Digital (A/D) conversion, thereby significantly increasing both time and energy costs. Here, we introduce a memristive neural ordinary differential equation (ODE) solver for digital twins, which is capable of capturing continuous-time dynamics and facilitates the modelling of complex systems using an infinite-depth model. By integrating storage and computation within analogue memristor arrays, we circumvent the von Neumann bottleneck, thus enhancing both speed and energy efficiency. We experimentally validate our approach by developing a digital twin of the HP memristor, which accurately extrapolates its nonlinear dynamics, achieving a 4.2-fold projected speedup and a 41.4-fold projected decrease in energy consumption compared to state-of-the-art digital hardware, while maintaining an acceptable error margin. Additionally, we demonstrate scalability through experimentally grounded simulations of Lorenz96 dynamics, exhibiting projected performance improvements of 12.6-fold in speed and 189.7-fold in energy efficiency relative to traditional digital approaches. By harnessing the capabilities of fully analogue computing, our breakthrough accelerates the development of digital twins, offering an efficient and rapid solution to meet the demands of Industry 4.0.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing
Authors:
Hongxiang Zhang,
Yuyang Rong,
Yifeng He,
Hao Chen
Abstract:
Greybox fuzzing has achieved success in revealing bugs and vulnerabilities in programs. However, randomized mutation strategies have limited the fuzzer's performance on structured data. Specialized fuzzers can handle complex structured data, but require additional efforts in grammar and suffer from low throughput.
In this paper, we explore the potential of utilizing the Large Language Model to e…
▽ More
Greybox fuzzing has achieved success in revealing bugs and vulnerabilities in programs. However, randomized mutation strategies have limited the fuzzer's performance on structured data. Specialized fuzzers can handle complex structured data, but require additional efforts in grammar and suffer from low throughput.
In this paper, we explore the potential of utilizing the Large Language Model to enhance greybox fuzzing for structured data. We utilize the pre-trained knowledge of LLM about data conversion and format to generate new valid inputs. We further fine-tuned it with paired mutation seeds to learn structured format and mutation strategies effectively. Our LLM-based fuzzer, LLAMAFUZZ, integrates the power of LLM to understand and mutate structured data to fuzzing. We conduct experiments on the standard bug-based benchmark Magma and a wide variety of real-world programs. LLAMAFUZZ outperforms our top competitor by 41 bugs on average. We also identified 47 unique bugs across all trials. Moreover, LLAMAFUZZ demonstrated consistent performance on both bug trigger and bug reached. Compared to AFL++, LLAMAFUZZ achieved 27.19% more branches in real-world program sets on average. We also demonstrate a case study to explain how LLMs enhance the fuzzing process in terms of code coverage.
△ Less
Submitted 13 June, 2024; v1 submitted 11 June, 2024;
originally announced June 2024.
-
RefGaussian: Disentangling Reflections from 3D Gaussian Splatting for Realistic Rendering
Authors:
Rui Zhang,
Tianyue Luo,
Weidong Yang,
Ben Fei,
Jingyi Xu,
Qingyuan Zhou,
Keyi Liu,
Ying He
Abstract:
3D Gaussian Splatting (3D-GS) has made a notable advancement in the field of neural rendering, 3D scene reconstruction, and novel view synthesis. Nevertheless, 3D-GS encounters the main challenge when it comes to accurately representing physical reflections, especially in the case of total reflection and semi-reflection that are commonly found in real-world scenes. This limitation causes reflectio…
▽ More
3D Gaussian Splatting (3D-GS) has made a notable advancement in the field of neural rendering, 3D scene reconstruction, and novel view synthesis. Nevertheless, 3D-GS encounters the main challenge when it comes to accurately representing physical reflections, especially in the case of total reflection and semi-reflection that are commonly found in real-world scenes. This limitation causes reflections to be mistakenly treated as independent elements with physical presence, leading to imprecise reconstructions. Herein, to tackle this challenge, we propose RefGaussian to disentangle reflections from 3D-GS for realistically modeling reflections. Specifically, we propose to split a scene into transmitted and reflected components and represent these components using two Spherical Harmonics (SH). Given that this decomposition is not fully determined, we employ local regularization techniques to ensure local smoothness for both the transmitted and reflected components, thereby achieving more plausible decomposition outcomes than 3D-GS. Experimental results demonstrate that our approach achieves superior novel view synthesis and accurate depth estimation outcomes. Furthermore, it enables the utilization of scene editing applications, ensuring both high-quality results and physical coherence.
△ Less
Submitted 9 June, 2024;
originally announced June 2024.
-
Janus graphene nanoribbons with a single ferromagnetic zigzag edge
Authors:
Shaotang Song,
Yu Teng,
Weichen Tang,
Zhen Xu,
Yuanyuan He,
Jiawei Ruan,
Takahiro Kojima,
Wenping Hu,
Franz J Giessibl,
Hiroshi Sakaguchi,
Steven G Louie,
Jiong Lu
Abstract:
Topological design of pi-electrons in zigzag-edged graphene nanoribbons (ZGNRs) leads to a wealth of magnetic quantum phenomena and exotic quantum phases. Symmetric ZGNRs typically exhibit antiferromagnetically coupled spin-ordered edge states. Eliminating cross-edge magnetic coupling in ZGNRs not only enables the realization of a new class of ferromagnetic quantum spin chains, enabling the explor…
▽ More
Topological design of pi-electrons in zigzag-edged graphene nanoribbons (ZGNRs) leads to a wealth of magnetic quantum phenomena and exotic quantum phases. Symmetric ZGNRs typically exhibit antiferromagnetically coupled spin-ordered edge states. Eliminating cross-edge magnetic coupling in ZGNRs not only enables the realization of a new class of ferromagnetic quantum spin chains, enabling the exploration of quantum spin physics and entanglement of multiple qubits in the 1D limit, but also establishes a long-sought carbon-based ferromagnetic transport channel, pivotal for ultimate scaling of GNR-based quantum electronics. However, designing such GNRs entails overcoming daunting challenges, including simultaneous breaking of structural and spin symmetries, and designing elegant precursors for asymmetric fabrication of reactive zigzag edges. Here, we report a general approach for designing and fabricating such ferromagnetic GNRs in the form of Janus GNRs with two distinct edge configurations. Guided by Lieb's theorem and topological classification theory, we devised two JGNRs by asymmetrically introduced a topological defect array of benzene motifs to one zigzag edge, while keeping the opposing zigzag edge unchanged. This breaks structural symmetry and creates a sublattice imbalance within each unit cell, initiating a spin symmetry breaking. Three Z-shape precursors are designed to fabricate one parent ZGNR and two JGNRs with an optimal lattice spacing of the defect array for a complete quench of the magnetic edge states at the defective edge. Characterization via scanning probe microscopy/spectroscopy and first-principles density functional theory confirms the successful fabrication of Janus GNRs with ferromagnetic ground state delocalised along the pristine zigzag edge.
△ Less
Submitted 8 June, 2024;
originally announced June 2024.
-
VISTA3D: Versatile Imaging SegmenTation and Annotation model for 3D Computed Tomography
Authors:
Yufan He,
Pengfei Guo,
Yucheng Tang,
Andriy Myronenko,
Vishwesh Nath,
Ziyue Xu,
Dong Yang,
Can Zhao,
Benjamin Simon,
Mason Belue,
Stephanie Harmon,
Baris Turkbey,
Daguang Xu,
Wenqi Li
Abstract:
Medical image segmentation is a core component of precision medicine, and 3D computed tomography (CT) is one of the most important imaging techniques. A highly accurate and clinically applicable segmentation foundation model will greatly facilitate clinicians and researchers using CT images. Although existing foundation models have attracted great interest, none are adequate for 3D CT, either beca…
▽ More
Medical image segmentation is a core component of precision medicine, and 3D computed tomography (CT) is one of the most important imaging techniques. A highly accurate and clinically applicable segmentation foundation model will greatly facilitate clinicians and researchers using CT images. Although existing foundation models have attracted great interest, none are adequate for 3D CT, either because they lack accurate automatic segmentation for large cohort analysis or the ability to segment novel classes. An ideal segmentation solution should possess two features: accurate out-of-the-box performance covering major organ classes, and effective adaptation or zero-shot ability to novel structures. To achieve this goal, we introduce Versatile Imaging SegmenTation and Annotation model (VISTA3D). VISTA3D is trained systematically on 11454 volumes and provides accurate out-of-the-box segmentation for 127 common types of human anatomical structures and various lesions. Additionally, VISTA3D supports 3D interactive segmentation, allowing convenient editing of automatic results and achieving state-of-the-art annotation results on unseen classes. The novel model design and training recipe represent a promising step toward developing a versatile medical image foundation model and will serve as a valuable foundation for CT image analysis. Code and model weights are available at https://github.com/Project-MONAI/VISTA
△ Less
Submitted 7 August, 2024; v1 submitted 7 June, 2024;
originally announced June 2024.
-
Diverse Intra- and Inter-Domain Activity Style Fusion for Cross-Person Generalization in Activity Recognition
Authors:
Junru Zhang,
Lang Feng,
Zhidan Liu,
Yuhan Wu,
Yang He,
Yabo Dong,
Duanqing Xu
Abstract:
Existing domain generalization (DG) methods for cross-person generalization tasks often face challenges in capturing intra- and inter-domain style diversity, resulting in domain gaps with the target domain. In this study, we explore a novel perspective to tackle this problem, a process conceptualized as domain padding. This proposal aims to enrich the domain diversity by synthesizing intra- and in…
▽ More
Existing domain generalization (DG) methods for cross-person generalization tasks often face challenges in capturing intra- and inter-domain style diversity, resulting in domain gaps with the target domain. In this study, we explore a novel perspective to tackle this problem, a process conceptualized as domain padding. This proposal aims to enrich the domain diversity by synthesizing intra- and inter-domain style data while maintaining robustness to class labels. We instantiate this concept using a conditional diffusion model and introduce a style-fused sampling strategy to enhance data generation diversity. In contrast to traditional condition-guided sampling, our style-fused sampling strategy allows for the flexible use of one or more random styles to guide data synthesis. This feature presents a notable advancement: it allows for the maximum utilization of possible permutations and combinations among existing styles to generate a broad spectrum of new style instances. Empirical evaluations on a broad range of datasets demonstrate that our generated data achieves remarkable diversity within the domain space. Both intra- and inter-domain generated data have proven to be significant and valuable, contributing to varying degrees of performance enhancements. Notably, our approach outperforms state-of-the-art DG methods in all human activity recognition tasks.
△ Less
Submitted 28 June, 2024; v1 submitted 6 June, 2024;
originally announced June 2024.
-
Explainable Machine Learning Identification of Superconductivity from Single-Particle Spectral Functions
Authors:
Xu Chen,
Yuanjie Sun,
Eugen Hruska,
Vivek Dixit,
Jinming Yang,
Yu He,
Yao Wang,
Fang Liu
Abstract:
The traditional method of identifying symmetry-breaking phase transitions through the emergence of a single-particle gap encounters significant challenges in quantum materials with strong fluctuations. To address this, we have developed a data-driven approach using a domain-adversarial neural network trained on simulated spectra of cuprates. This model compensates for the scarcity of experimental…
▽ More
The traditional method of identifying symmetry-breaking phase transitions through the emergence of a single-particle gap encounters significant challenges in quantum materials with strong fluctuations. To address this, we have developed a data-driven approach using a domain-adversarial neural network trained on simulated spectra of cuprates. This model compensates for the scarcity of experimental data -- a significant barrier to the wide deployment of machine learning in physical research -- by leveraging the abundance of theoretically simulated data. When applied to unlabeled experimental spectra, our model successfully distinguishes the true superconducting states from gapped fluctuating states, without the need for fine temperature sampling across the transition. Further, the explanation of our machine learning model reveals the crucial role of the Fermi-surface spectral intensity even in gapped states. It paves the way for robust and direct spectroscopic identification of fluctuating orders, particularly in low-dimensional, strongly correlated materials.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Ghost imaging-based Non-contact Heart Rate Detection
Authors:
Jianming Yu,
Yuchen He,
Bin Li,
Hui Chen,
Huaibin Zheng,
Jianbin Liu,
Zhuo Xu
Abstract:
Remote heart rate measurement is an increasingly concerned research field, usually using remote photoplethysmography (rPPG) to collect heart rate information through video data collection. However, in certain specific scenarios (such as low light conditions, intense lighting, and non-line-of-sight situations), traditional imaging methods fail to capture image information effectively, that may lead…
▽ More
Remote heart rate measurement is an increasingly concerned research field, usually using remote photoplethysmography (rPPG) to collect heart rate information through video data collection. However, in certain specific scenarios (such as low light conditions, intense lighting, and non-line-of-sight situations), traditional imaging methods fail to capture image information effectively, that may lead to difficulty or inability in measuring heart rate. To address these limitations, this study proposes using ghost imaging as a substitute for traditional imaging in the aforementioned scenarios. The mean absolute error between experimental measurements and reference true values is 4.24 bpm.Additionally, the bucket signals obtained by the ghost imaging system can be directly processed using digital signal processing techniques, thereby enhancing personal privacy protection.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Experimental single-photon quantum key distribution surpassing the fundamental coherent-state rate limit
Authors:
Yang Zhang,
Xing Ding,
Yang Li,
Likang Zhang,
Yong-Peng Guo,
Gao-Qiang Wang,
Zhen Ning,
Mo-Chi Xu,
Run-Ze Liu,
Jun-Yi Zhao,
Geng-Yan Zou,
Hui Wang,
Yuan Cao,
Yu-Ming He,
Cheng-Zhi Peng,
Yong-Heng Huo,
Sheng-Kai Liao,
Chao-Yang Lu,
Feihu Xu,
Jian-Wei Pan
Abstract:
Single-photon sources are essential for quantum networks, enabling applications ranging from quantum key distribution (QKD) to the burgeoning quantum internet. Despite the remarkable advancements, the current reliance of QKD on attenuated coherent (laser) light sources has imposed a fundamental limit on the secret key rate (SKR). This constraint is primarily attributable to the scarcity of single-…
▽ More
Single-photon sources are essential for quantum networks, enabling applications ranging from quantum key distribution (QKD) to the burgeoning quantum internet. Despite the remarkable advancements, the current reliance of QKD on attenuated coherent (laser) light sources has imposed a fundamental limit on the secret key rate (SKR). This constraint is primarily attributable to the scarcity of single-photon components within coherent light, confined by an inherent upper bound of 1/e. Here, we report high-rate QKD using a high-efficiency single-photon source, enabling an SKR transcending the fundamental rate limit of coherent light. We developed an on-demand, bright semiconductor quantum-dot single-photon source with an efficiency of 0.71(2), exceeding the inherent bound of coherent light by approximately 2.87 dB. Implementing narrow-bandwidth filtering and random polarization modulation, we conducted a field QKD trial over a 14.6(1.1)-dB-loss free-space urban channel, achieving an SKR of 0.00108 bits per pulse. This surpasses the practical limit of coherent-light-based QKD by 2.53 dB. Our findings conclusively demonstrate the superior performance of nanotechnology-based single-photon sources over coherent light for QKD applications, marking a pivotal stride towards the realization of a global quantum internet.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation
Authors:
Yue Ma,
Hongyu Liu,
Hongfa Wang,
Heng Pan,
Yingqing He,
Junkun Yuan,
Ailing Zeng,
Chengfei Cai,
Heung-Yeung Shum,
Wei Liu,
Qifeng Chen
Abstract:
We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences. The main challenge of portrait animation is to preserve the identity of the reference portrait and transfer the target expression to this portrait while maintaining temporal consistency and fidelity. To address these challenges, Follow-Your-Emoji equ…
▽ More
We present Follow-Your-Emoji, a diffusion-based framework for portrait animation, which animates a reference portrait with target landmark sequences. The main challenge of portrait animation is to preserve the identity of the reference portrait and transfer the target expression to this portrait while maintaining temporal consistency and fidelity. To address these challenges, Follow-Your-Emoji equipped the powerful Stable Diffusion model with two well-designed technologies. Specifically, we first adopt a new explicit motion signal, namely expression-aware landmark, to guide the animation process. We discover this landmark can not only ensure the accurate motion alignment between the reference portrait and target motion during inference but also increase the ability to portray exaggerated expressions (i.e., large pupil movements) and avoid identity leakage. Then, we propose a facial fine-grained loss to improve the model's ability of subtle expression perception and reference portrait appearance reconstruction by using both expression and facial masks. Accordingly, our method demonstrates significant performance in controlling the expression of freestyle portraits, including real humans, cartoons, sculptures, and even animals. By leveraging a simple and effective progressive generation strategy, we extend our model to stable long-term animation, thus increasing its potential application value. To address the lack of a benchmark for this field, we introduce EmojiBench, a comprehensive benchmark comprising diverse portrait images, driving videos, and landmarks. We show extensive evaluations on EmojiBench to verify the superiority of Follow-Your-Emoji.
△ Less
Submitted 6 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Antiferromagnetic order in a layered magnetic topological insulator MnBi$_2$Se$_4$ probed by resonant soft x-ray scattering
Authors:
Xiang Chen,
Alejandro Ruiz,
Alexander J. Bishop,
Brandon Gunn,
Rourav Basak,
Tiancong Zhu,
Yu He,
Mayia Vranas,
Eugen Weschke,
Roland K. Kawakami,
Robert J. Birgeneau,
Alex Frano
Abstract:
The quasi-two-dimensional magnetic topological insulator MnBi$_2$Se$_4$, stabilized via non-equilibrium molecular beam epitaxy, is investigated by resonant soft x-ray scattering. Kiessig fringes are observed, confirming a high sample quality and a thin film thickness of 10 septuple layers ($\sim$13 nm). An antiferromagnetic Bragg peak is observed at the structurally forbidden reflection, whose mag…
▽ More
The quasi-two-dimensional magnetic topological insulator MnBi$_2$Se$_4$, stabilized via non-equilibrium molecular beam epitaxy, is investigated by resonant soft x-ray scattering. Kiessig fringes are observed, confirming a high sample quality and a thin film thickness of 10 septuple layers ($\sim$13 nm). An antiferromagnetic Bragg peak is observed at the structurally forbidden reflection, whose magnetic nature is validated by studying its temperature, energy, and polarization dependence. Through a detailed analysis, an A-type antiferromagetic order with in-plane moments is implied. This alternative spin structure in MnBi$_2$Se$_4$, in contrast to the Ising antiferromagnetic states in other magnetic topological insulators, might be relevant for hosting new topological states.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Task-oriented Embedding Counts: Heuristic Clustering-driven Feature Fine-tuning for Whole Slide Image Classification
Authors:
Xuenian Wang,
Shanshan Shi,
Renao Yan,
Qiehe Sun,
Lianghui Zhu,
Tian Guan,
Yonghong He
Abstract:
In the field of whole slide image (WSI) classification, multiple instance learning (MIL) serves as a promising approach, commonly decoupled into feature extraction and aggregation. In this paradigm, our observation reveals that discriminative embeddings are crucial for aggregation to the final prediction. Among all feature updating strategies, task-oriented ones can capture characteristics specifi…
▽ More
In the field of whole slide image (WSI) classification, multiple instance learning (MIL) serves as a promising approach, commonly decoupled into feature extraction and aggregation. In this paradigm, our observation reveals that discriminative embeddings are crucial for aggregation to the final prediction. Among all feature updating strategies, task-oriented ones can capture characteristics specifically for certain tasks. However, they can be prone to overfitting and contaminated by samples assigned with noisy labels. To address this issue, we propose a heuristic clustering-driven feature fine-tuning method (HC-FT) to enhance the performance of multiple instance learning by providing purified positive and hard negative samples. Our method first employs a well-trained MIL model to evaluate the confidence of patches. Then, patches with high confidence are marked as positive samples, while the remaining patches are used to identify crucial negative samples. After two rounds of heuristic clustering and selection, purified positive and hard negative samples are obtained to facilitate feature fine-tuning. The proposed method is evaluated on both CAMELYON16 and BRACS datasets, achieving an AUC of 97.13% and 85.85%, respectively, consistently outperforming all compared methods.
△ Less
Submitted 2 June, 2024;
originally announced June 2024.