-
Advancing Multi-talker ASR Performance with Large Language Models
Authors:
Mohan Shi,
Zengrui Jin,
Yaoxun Xu,
Yong Xu,
Shi-Xiong Zhang,
Kun Wei,
Yiwen Shao,
Chunlei Zhang,
Dong Yu
Abstract:
Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output training (SOT) is a classic method to address multi-talker ASR, with the idea of concatenating transcriptions from multiple speakers according to the emission times of their speech for training. However, SOT-style transcr…
▽ More
Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output training (SOT) is a classic method to address multi-talker ASR, with the idea of concatenating transcriptions from multiple speakers according to the emission times of their speech for training. However, SOT-style transcriptions, derived from concatenating multiple related utterances in a conversation, depend significantly on modeling long contexts. Therefore, compared to traditional methods that primarily emphasize encoder performance in attention-based encoder-decoder (AED) architectures, a novel approach utilizing large language models (LLMs) that leverages the capabilities of pre-trained decoders may be better suited for such complex and challenging scenarios. In this paper, we propose an LLM-based SOT approach for multi-talker ASR, leveraging pre-trained speech encoder and LLM, fine-tuning them on multi-talker dataset using appropriate strategies. Experimental results demonstrate that our approach surpasses traditional AED-based methods on the simulated dataset LibriMix and achieves state-of-the-art performance on the evaluation set of the real-world dataset AMI, outperforming the AED model trained with 1000 times more supervised data in previous works.
△ Less
Submitted 30 August, 2024;
originally announced August 2024.
-
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Authors:
Min Shi,
Fuxiao Liu,
Shihao Wang,
Shijia Liao,
Subhashree Radhakrishnan,
De-An Huang,
Hongxu Yin,
Karan Sapra,
Yaser Yacoob,
Humphrey Shi,
Bryan Catanzaro,
Andrew Tao,
Jan Kautz,
Zhiding Yu,
Guilin Liu
Abstract:
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vis…
▽ More
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks. Models and code: https://github.com/NVlabs/Eagle
△ Less
Submitted 28 August, 2024;
originally announced August 2024.
-
Triorthogonal Codes and Self-dual Codes
Authors:
Minjia Shi,
Haodong Lu,
Jon-Lark Kim,
Patrick Sole
Abstract:
Triorthogonal matrices were introduced in Quantum Information Theory in connection with distillation of magic states (Bravyi and Haah (2012)). We give an algorithm to construct binary triorthogonal matrices from binary self-dual codes. Further, we generalize to this setting the classical coding techniques of shortening and extending. We also give some simple propagation rules.
Triorthogonal matrices were introduced in Quantum Information Theory in connection with distillation of magic states (Bravyi and Haah (2012)). We give an algorithm to construct binary triorthogonal matrices from binary self-dual codes. Further, we generalize to this setting the classical coding techniques of shortening and extending. We also give some simple propagation rules.
△ Less
Submitted 18 August, 2024;
originally announced August 2024.
-
Low frequency communication based on Rydberg-atom receiver
Authors:
Yipeng Xie,
Mingwei Lei,
Meng Shi
Abstract:
Low frequency communication has a wide range of applications in the fields of satellite detection, underground mining, disaster relief. Rydberg atom sensor has rapidly developed in recent years, capitalizing on its calibration-free SI-traceability, large polarizabilities and transition dipole moments. A Rydberg atom sensor is capable of sensitively detecting electric field signals from DC to THz.…
▽ More
Low frequency communication has a wide range of applications in the fields of satellite detection, underground mining, disaster relief. Rydberg atom sensor has rapidly developed in recent years, capitalizing on its calibration-free SI-traceability, large polarizabilities and transition dipole moments. A Rydberg atom sensor is capable of sensitively detecting electric field signals from DC to THz. In this work, we demonstrate low frequency communication using Rydberg atoms in a vapor cell with two parallel electrode plates inside. Three modulations, BPSK, OOK, and 2FSK, are used for the communication by Rydberg atom receiver near 100kHz. We have measured the SNR of the modulated low frequency signal received by Rydberg atoms at various emission voltages. Meanwhile, we have demonstrated IQ constellation diagram, EVM and eye diagram of the demodulated signal at different symbol rate. The EVM is measured to be 8.8% when the symbol rate is 2Kbps, 9.4% when the symbol rate is 4Kbps, and 13.7% when the symbol rate is 8Kbps. The high-fidelity digital color image transmission resulted in a peak signal-to-noise ratio of 70dB. This study proves that Rydberg-atom receiver can finely work in low frequency communication.
△ Less
Submitted 18 August, 2024;
originally announced August 2024.
-
How a simple pendulum inside a running elevator oscillates
Authors:
Mingyuan Shi,
Yu Shi
Abstract:
We propose to effectively realize a time-dependent gravitational acceleration by using a running elevator, so that a simple pendulum inside it effectively becomes one with a time-dependent gravitational acceleration. We did such an experiment using a realistic elevator, and analyzed the data. The acceleration of an elevator is much smaller than the gravitational acceleration, and is time-dependent…
▽ More
We propose to effectively realize a time-dependent gravitational acceleration by using a running elevator, so that a simple pendulum inside it effectively becomes one with a time-dependent gravitational acceleration. We did such an experiment using a realistic elevator, and analyzed the data. The acceleration of an elevator is much smaller than the gravitational acceleration, and is time-dependent only when the elevator starts and stops. However, we have managed to establish the effect on the oscillation of the pendulum. The effect becomes pronounced if the simple pendulum is put in a container vertically accelerating, and the acceleration is time-dependent, while its magnitude is comparable with that of the gravitational acceleration.
△ Less
Submitted 16 August, 2024;
originally announced August 2024.
-
Measuring the acceleration of an elevator by using the apparent weight of an object inside it
Authors:
Mingyuan Shi,
Yu Shi
Abstract:
An accelerating elevator changes the apparent weight of any object inside it from the original weight, as measured inside the elevator, because the acceleration causes an inertial force on it. For any object in a running elevator, the variation of the acceleration of the elevator causes the variation of the apparent weight of the object. We have studied the time dependence of the apparent weight o…
▽ More
An accelerating elevator changes the apparent weight of any object inside it from the original weight, as measured inside the elevator, because the acceleration causes an inertial force on it. For any object in a running elevator, the variation of the acceleration of the elevator causes the variation of the apparent weight of the object. We have studied the time dependence of the apparent weight of the object and thus the acceleration of the elevator. For chosen initial and final floors, we measured the apparent weight of an object by using an electronic scale inside the elevator, and shot the readings of the scale and a watch during the movement of the elevator. Then we analyzed the data collected from the recorded video. If the initial and final floors are exchanged, the variations of the weight and acceleration are, respectively, same in magnitudes and opposite in signs. The experiments indicate that for the elevator to go directly from a floor to another, the process consists of periods with variable acceleration, constant acceleration, uniform motion, variable deceleration, constant deceleration and variable deceleration consecutively. If there are pauses during the movement, each pause causes an additional process consisting of periods with deceleration, stop and acceleration, replacing the original period of constant motion. Depending on the distance to the destination, the elevator reduces or diminishes the periods of constant acceleration and uniform motion.
△ Less
Submitted 31 July, 2024;
originally announced August 2024.
-
An investigation into the causes of race bias in AI-based cine CMR segmentation
Authors:
Tiarna Lee,
Esther Puyol-Anton,
Bram Ruijsink,
Sebastien Roujol,
Theodore Barfoot,
Shaheim Ogbomo-Harmitt,
Miaojing Shi,
Andrew P. King
Abstract:
Artificial intelligence (AI) methods are being used increasingly for the automated segmentation of cine cardiac magnetic resonance (CMR) imaging. However, these methods have been shown to be subject to race bias, i.e. they exhibit different levels of performance for different races depending on the (im)balance of the data used to train the AI model. In this paper we investigate the source of this…
▽ More
Artificial intelligence (AI) methods are being used increasingly for the automated segmentation of cine cardiac magnetic resonance (CMR) imaging. However, these methods have been shown to be subject to race bias, i.e. they exhibit different levels of performance for different races depending on the (im)balance of the data used to train the AI model. In this paper we investigate the source of this bias, seeking to understand its root cause(s) so that it can be effectively mitigated. We perform a series of classification and segmentation experiments on short-axis cine CMR images acquired from Black and White subjects from the UK Biobank and apply AI interpretability methods to understand the results. In the classification experiments, we found that race can be predicted with high accuracy from the images alone, but less accurately from ground truth segmentations, suggesting that the distributional shift between races, which is often the cause of AI bias, is mostly image-based rather than segmentation-based. The interpretability methods showed that most attention in the classification models was focused on non-heart regions, such as subcutaneous fat. Cropping the images tightly around the heart reduced classification accuracy to around chance level. Similarly, race can be predicted from the latent representations of a biased segmentation model, suggesting that race information is encoded in the model. Cropping images tightly around the heart reduced but did not eliminate segmentation bias. We also investigate the influence of possible confounders on the bias observed.
△ Less
Submitted 5 August, 2024;
originally announced August 2024.
-
Persistence of small polarons into the superconducting phase of Ba$_{1-x}$K$_x$BiO$_3$
Authors:
Muntaser Naamneh,
Eugenio Paris,
Daniel McNally,
Yi Tseng,
Wojciech R. Pudelko,
Dariusz J. Gawryluk,
J. Shamblin,
Eric OQuinn,
Benjamin Cohen-Stead,
Ming Shi,
Milan Radovic,
M. Lang,
Thorsten Schmitt,
Steven Johnston,
Nicholas C. Plumb
Abstract:
Bipolaronic superconductivity is an exotic pairing mechanism proposed for materials like Ba$_{1-x}$K$_x$BiO$_3$ (BKBO); however, conclusive experimental evidence for a (bi)polaron metallic state in this material remains elusive. Here, we combine resonant inelastic x-ray and neutron total scattering techniques with advanced modelling to study the local lattice distortions, electronic structure, and…
▽ More
Bipolaronic superconductivity is an exotic pairing mechanism proposed for materials like Ba$_{1-x}$K$_x$BiO$_3$ (BKBO); however, conclusive experimental evidence for a (bi)polaron metallic state in this material remains elusive. Here, we combine resonant inelastic x-ray and neutron total scattering techniques with advanced modelling to study the local lattice distortions, electronic structure, and electron-phonon coupling ($e$-ph) in BKBO as a function of doping. Data for the parent compound ($x = 0$) indicates that the electronic gap opens in predominantly oxygen-derived states strongly coupled to a long-range ordered breathing distortion of the oxygen sublattice. Upon doping, short-range breathing distortions and sizable ($e$-ph) coupling persist into the superconducting regime ($x = 0.4$). Comparisons with exact diagonalization and determinant quantum Monte Carlo calculations further support this conclusion. Our results provide compelling evidence that BKBO's metallic phase hosts a liquid of small (bi)polarons derived from local breathing distortions of the lattice, with implications for understanding the low-temperature superconducting instability
△ Less
Submitted 1 August, 2024;
originally announced August 2024.
-
High efficient 120W 1018nm single-frequency narrow linewidth amplification based on wide-tunable DBR fiber seed source
Authors:
Pan Li,
Linfeng Li,
Mingze Wang,
KaiMing Cao,
Ruihong Gao,
Heshan Liu,
Meng Shi,
Ziren Luo
Abstract:
This paper reports the achievement of 120W single-frequency narrow linewidth 1018nm laser based on wide-tunable DBR fiber seed source. The DBR structure seed source uses 8mm long doped optical fibers with a line width of 3.25k. The wavelength tuning range of this seed source exceeds 1.5 nm with the temperature range from 1°C to 95°C. The tuning wavelength and temperature show extremely high linear…
▽ More
This paper reports the achievement of 120W single-frequency narrow linewidth 1018nm laser based on wide-tunable DBR fiber seed source. The DBR structure seed source uses 8mm long doped optical fibers with a line width of 3.25k. The wavelength tuning range of this seed source exceeds 1.5 nm with the temperature range from 1°C to 95°C. The tuning wavelength and temperature show extremely high linearity, and there is no mode hopping during the tuning process. By adopting a multi-level fiber amplification structure, selecting appropriate doped fibers and optimizing their length, an output power exceeding 120W of 1018nm laser has been achieved. Measurement results indicate that the slope efficiency of the main amplification 77.3%, with an amplified spontaneous emission (ASE) suppression ratio greater than 60 dB. he output linewidth is 10.3 kHz, and the beam quality factor M2 is less than 1.3.
△ Less
Submitted 28 July, 2024;
originally announced July 2024.
-
Triangle decompositions of PG(n,2)
Authors:
Minjia Shi,
Xiaoxiao Li,
Denis S. Krotov
Abstract:
We define a triangle design as a partition of the set of $2$-dimensional subspaces of an $n$-dimensional vector space into triangles, where a triangle consists of three subspaces with the trivial, $0$-dimensional, intersection and $1$-dimensional mutual intersections. A triangle design is balanced if all nonzero vectors are involved in the same number of triangles. Over the binary field GF$(2)$, w…
▽ More
We define a triangle design as a partition of the set of $2$-dimensional subspaces of an $n$-dimensional vector space into triangles, where a triangle consists of three subspaces with the trivial, $0$-dimensional, intersection and $1$-dimensional mutual intersections. A triangle design is balanced if all nonzero vectors are involved in the same number of triangles. Over the binary field GF$(2)$, we construct balanced triangle designs for all admissible $n$ (congruent to $1$ modulo $6$) and an infinite class of balanced block-divisible triangle designs. We also prove that the existence of a triangle design over GF$(2)$ invariant under the action of the Singer cycle group is equivalent to the existence of a partition of $Z_{2^n-1}\backslash\{0\}$ into special $18$-subsets and find such designs for $n=7$, $13$, $19$.
△ Less
Submitted 26 July, 2024;
originally announced July 2024.
-
A Labeled Ophthalmic Ultrasound Dataset with Medical Report Generation Based on Cross-modal Deep Learning
Authors:
Jing Wang,
Junyan Fan,
Meng Zhou,
Yanzhu Zhang,
Mingyu Shi
Abstract:
Ultrasound imaging reveals eye morphology and aids in diagnosing and treating eye diseases. However, interpreting diagnostic reports requires specialized physicians. We present a labeled ophthalmic dataset for the precise analysis and the automated exploration of medical images along with their associated reports. It collects three modal data, including the ultrasound images, blood flow informatio…
▽ More
Ultrasound imaging reveals eye morphology and aids in diagnosing and treating eye diseases. However, interpreting diagnostic reports requires specialized physicians. We present a labeled ophthalmic dataset for the precise analysis and the automated exploration of medical images along with their associated reports. It collects three modal data, including the ultrasound images, blood flow information and examination reports from 2,417 patients at an ophthalmology hospital in Shenyang, China, during the year 2018, in which the patient information is de-identified for privacy protection. To the best of our knowledge, it is the only ophthalmic dataset that contains the three modal information simultaneously. It incrementally consists of 4,858 images with the corresponding free-text reports, which describe 15 typical imaging findings of intraocular diseases and the corresponding anatomical locations. Each image shows three kinds of blood flow indices at three specific arteries, i.e., nine parameter values to describe the spectral characteristics of blood flow distribution. The reports were written by ophthalmologists during the clinical care. The proposed dataset is applied to generate medical report based on the cross-modal deep learning model. The experimental results demonstrate that our dataset is suitable for training supervised models concerning cross-modal medical data.
△ Less
Submitted 26 July, 2024;
originally announced July 2024.
-
Some $3$-designs invariant under $2.PΣL(2,49).$
Authors:
Minjia Shi,
Ruowen Liu,
Patrick Solé
Abstract:
We construct a ternary [49,25,7] code from the row span of a Jacobsthal matrix. It is equivalent to a Generalized Quadratic Residue (GQR) code in the sense of van Lint and MacWilliams (1978). These codes are the abelian generalizations of the quadratic residue (QR) codes which are cyclic. The union of the [50,25,8] extension of the said code and its dual supports a 3-(50,14,1248) design. The autom…
▽ More
We construct a ternary [49,25,7] code from the row span of a Jacobsthal matrix. It is equivalent to a Generalized Quadratic Residue (GQR) code in the sense of van Lint and MacWilliams (1978). These codes are the abelian generalizations of the quadratic residue (QR) codes which are cyclic. The union of the [50,25,8] extension of the said code and its dual supports a 3-(50,14,1248) design. The automorphism group of the latter design is a double cover of the permutation part of the automorphism group of the [50,25,8] code, which is isomorphic to $PΣL(2,49).$ Other weights in this code, other GQR codes, and other QR codes yield other 3-designs by the same process. A simple group action argument is provided to explain this behaviour of isodual codes.
△ Less
Submitted 23 July, 2024;
originally announced July 2024.
-
Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation
Authors:
Xinghao Wu,
Jianwei Niu,
Xuefeng Liu,
Mingjia Shi,
Guogang Zhu,
Shaojie Tang
Abstract:
In traditional Federated Learning approaches like FedAvg, the global model underperforms when faced with data heterogeneity. Personalized Federated Learning (PFL) enables clients to train personalized models to fit their local data distribution better. However, we surprisingly find that the feature extractor in FedAvg is superior to those in most PFL methods. More interestingly, by applying a line…
▽ More
In traditional Federated Learning approaches like FedAvg, the global model underperforms when faced with data heterogeneity. Personalized Federated Learning (PFL) enables clients to train personalized models to fit their local data distribution better. However, we surprisingly find that the feature extractor in FedAvg is superior to those in most PFL methods. More interestingly, by applying a linear transformation on local features extracted by the feature extractor to align with the classifier, FedAvg can surpass the majority of PFL methods. This suggests that the primary cause of FedAvg's inadequate performance stems from the mismatch between the locally extracted features and the classifier. While current PFL methods mitigate this issue to some extent, their designs compromise the quality of the feature extractor, thus limiting the full potential of PFL. In this paper, we propose a new PFL framework called FedPFT to address the mismatch problem while enhancing the quality of the feature extractor. FedPFT integrates a feature transformation module, driven by personalized prompts, between the global feature extractor and classifier. In each round, clients first train prompts to transform local features to match the global classifier, followed by training model parameters. This approach can also align the training objectives of clients, reducing the impact of data heterogeneity on model collaboration. Moreover, FedPFT's feature transformation module is highly scalable, allowing for the use of different prompts to tailor local features to various tasks. Leveraging this, we introduce a collaborative contrastive learning task to further refine feature extractor quality. Our experiments demonstrate that FedPFT outperforms state-of-the-art methods by up to 7.08%.
△ Less
Submitted 22 July, 2024;
originally announced July 2024.
-
OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Models
Authors:
Zijian Zhou,
Zheng Zhu,
Holger Caesar,
Miaojing Shi
Abstract:
Panoptic Scene Graph Generation (PSG) aims to segment objects and recognize their relations, enabling the structured understanding of an image. Previous methods focus on predicting predefined object and relation categories, hence limiting their applications in the open world scenarios. With the rapid development of large multimodal models (LMMs), significant progress has been made in open-set obje…
▽ More
Panoptic Scene Graph Generation (PSG) aims to segment objects and recognize their relations, enabling the structured understanding of an image. Previous methods focus on predicting predefined object and relation categories, hence limiting their applications in the open world scenarios. With the rapid development of large multimodal models (LMMs), significant progress has been made in open-set object detection and segmentation, yet open-set relation prediction in PSG remains unexplored. In this paper, we focus on the task of open-set relation prediction integrated with a pretrained open-set panoptic segmentation model to achieve true open-set panoptic scene graph generation (OpenPSG). Our OpenPSG leverages LMMs to achieve open-set relation prediction in an autoregressive manner. We introduce a relation query transformer to efficiently extract visual features of object pairs and estimate the existence of relations between them. The latter can enhance the prediction efficiency by filtering irrelevant pairs. Finally, we design the generation and judgement instructions to perform open-set relation prediction in PSG autoregressively. To our knowledge, we are the first to propose the open-set PSG task. Extensive experiments demonstrate that our method achieves state-of-the-art performance in open-set relation prediction and panoptic scene graph generation. Code is available at \url{https://github.com/franciszzj/OpenPSG}.
△ Less
Submitted 15 July, 2024;
originally announced July 2024.
-
FairDomain: Achieving Fairness in Cross-Domain Medical Image Segmentation and Classification
Authors:
Yu Tian,
Congcong Wen,
Min Shi,
Muhammad Muneeb Afzal,
Hao Huang,
Muhammad Osama Khan,
Yan Luo,
Yi Fang,
Mengyu Wang
Abstract:
Addressing fairness in artificial intelligence (AI), particularly in medical AI, is crucial for ensuring equitable healthcare outcomes. Recent efforts to enhance fairness have introduced new methodologies and datasets in medical AI. However, the fairness issue under the setting of domain transfer is almost unexplored, while it is common that clinics rely on different imaging technologies (e.g., di…
▽ More
Addressing fairness in artificial intelligence (AI), particularly in medical AI, is crucial for ensuring equitable healthcare outcomes. Recent efforts to enhance fairness have introduced new methodologies and datasets in medical AI. However, the fairness issue under the setting of domain transfer is almost unexplored, while it is common that clinics rely on different imaging technologies (e.g., different retinal imaging modalities) for patient diagnosis. This paper presents FairDomain, a pioneering systemic study into algorithmic fairness under domain shifts, employing state-of-the-art domain adaptation (DA) and generalization (DG) algorithms for both medical segmentation and classification tasks to understand how biases are transferred between different domains. We also introduce a novel plug-and-play fair identity attention (FIA) module that adapts to various DA and DG algorithms to improve fairness by using self-attention to adjust feature importance based on demographic attributes. Additionally, we curate the first fairness-focused dataset with two paired imaging modalities for the same patient cohort on medical segmentation and classification tasks, to rigorously assess fairness in domain-shift scenarios. Excluding the confounding impact of demographic distribution variation between source and target domains will allow clearer quantification of the performance of domain transfer models. Our extensive evaluations reveal that the proposed FIA significantly enhances both model performance accounted for fairness across all domain shift settings (i.e., DA and DG) with respect to different demographics, which outperforms existing methods on both segmentation and classification. The code and data can be accessed at https://ophai.hms.harvard.edu/datasets/harvard-fairdomain20k.
△ Less
Submitted 18 July, 2024; v1 submitted 11 July, 2024;
originally announced July 2024.
-
Bootstrapping Vision-language Models for Self-supervised Remote Physiological Measurement
Authors:
Zijie Yue,
Miaojing Shi,
Hanli Wang,
Shuai Ding,
Qijun Chen,
Shanlin Yang
Abstract:
Facial video-based remote physiological measurement is a promising research area for detecting human vital signs (e.g., heart rate, respiration frequency) in a non-contact way. Conventional approaches are mostly supervised learning, requiring extensive collections of facial videos and synchronously recorded photoplethysmography (PPG) signals. To tackle it, self-supervised learning has recently gai…
▽ More
Facial video-based remote physiological measurement is a promising research area for detecting human vital signs (e.g., heart rate, respiration frequency) in a non-contact way. Conventional approaches are mostly supervised learning, requiring extensive collections of facial videos and synchronously recorded photoplethysmography (PPG) signals. To tackle it, self-supervised learning has recently gained attentions; due to the lack of ground truth PPG signals, its performance is however limited. In this paper, we propose a novel self-supervised framework that successfully integrates the popular vision-language models (VLMs) into the remote physiological measurement task. Given a facial video, we first augment its positive and negative video samples with varying rPPG signal frequencies. Next, we introduce a frequency-oriented vision-text pair generation method by carefully creating contrastive spatio-temporal maps from positive and negative samples and designing proper text prompts to describe their relative ratios of signal frequencies. A pre-trained VLM is employed to extract features for these formed vision-text pairs and estimate rPPG signals thereafter. We develop a series of generative and contrastive learning mechanisms to optimize the VLM, including the text-guided visual map reconstruction task, the vision-text contrastive learning task, and the frequency contrastive and ranking task. Overall, our method for the first time adapts VLMs to digest and align the frequency-related knowledge in vision and text modalities. Extensive experiments on four benchmark datasets demonstrate that it significantly outperforms state of the art self-supervised methods.
△ Less
Submitted 11 July, 2024;
originally announced July 2024.
-
A Pre-trained Deep Potential Model for Sulfide Solid Electrolytes with Broad Coverage and High Accuracy
Authors:
Ruoyu Wang,
Mingyu Guo,
Yuxiang Gao,
Xiaoxu Wang,
Yuzhi Zhang,
Bin Deng,
Xin Chen,
Mengchao Shi,
Linfeng Zhang,
Zhicheng Zhong
Abstract:
Solid electrolytes with fast ion transport are one of the key challenges for solid state lithium metal batteries. To improve ion conductivity, chemical doping has been the most effective strategy, and atomistic simulation with machine-learning potential helps find optimized doping by predicting ion conductivity for arbitrary composition. Yet most existing machine-learning models are trained on nar…
▽ More
Solid electrolytes with fast ion transport are one of the key challenges for solid state lithium metal batteries. To improve ion conductivity, chemical doping has been the most effective strategy, and atomistic simulation with machine-learning potential helps find optimized doping by predicting ion conductivity for arbitrary composition. Yet most existing machine-learning models are trained on narrow chemistry, and new model has to be trained for each system, wasting transferable knowledge and incurring significant cost. Here, we propose a pre-trained deep potential model purpose-built for sulfide electrolytes with attention mechanism, known as DPA-SSE. The training set encompasses 15 elements and consists of both equilibrium and extensive out-of-equilibrium configurations. DPA-SSE achieves a high energy resolution of less than 2 meV/atom for dynamical trajectories up to 1150 K, and reproduces experimental ion conductivity of sulfide electrolytes with remarkable accuracy. DPA-SSE exhibits good transferability, covering a range of complex electrolytes with mixes of cation and anion atoms. Highly efficient dynamical simulation with DPA-SSE can be realized by model distillation which generates a faster model for given systems. DPA-SSE also serves as a platform for continuous learning, and the model fine-tune requires only a portion of downstream data. These results demonstrate the possibility of a new pathway for AI-driven development of solid electrolytes with exceptional performance.
△ Less
Submitted 24 July, 2024; v1 submitted 26 June, 2024;
originally announced June 2024.
-
Spin order and dynamics in the topological rare-earth germanide semimetals
Authors:
Yuhao Wang,
Zhixuan Zhen,
Jing Meng,
Igor Plokhikh,
Delong Wu,
Dariusz J. Gawryluk,
Yang Xu,
Qingfeng Zhan,
Ming Shi,
Ekaterina Pomjakushina,
Toni Shiroka,
Tian Shang
Abstract:
The $RE$Al(Si,Ge) ($RE$ = rare earth) family, known to break both the inversion- and time-reversal symmetries, represents one of the most suitable platforms for investigating the interplay between correlated-electron phenomena and topologically nontrivial bands. Here, we report on systematic magnetic, transport, and muon-spin rotation and relaxation ($μ$SR) measurements on (Nd,Sm)AlGe single cryst…
▽ More
The $RE$Al(Si,Ge) ($RE$ = rare earth) family, known to break both the inversion- and time-reversal symmetries, represents one of the most suitable platforms for investigating the interplay between correlated-electron phenomena and topologically nontrivial bands. Here, we report on systematic magnetic, transport, and muon-spin rotation and relaxation ($μ$SR) measurements on (Nd,Sm)AlGe single crystals, which exhibit antiferromagnetic (AFM) transitions at $T_\mathrm{N} = 6.1$ and 5.9 K, respectively. In addition, NdAlGe undergoes also an incommensurate-to-commensurate ferrimagnetic transition at 4.5 K. Weak transverse-field $μ$SR measurements confirm the AFM transitions, featuring a $\sim$90 % magnetic volume fraction. In both cases, zero-field (ZF) $μ$SR measurements reveal a more disordered internal field distribution in NdAlGe than in SmAlGe, reflected in a larger transverse muon-spin relaxation rate $λ^\mathrm{T}$ at $T \ll T_\mathrm{N}$. This may be due to the complex magnetic structure of NdAlGe, which undergoes a series of metamagnetic transitions in an external magnetic field, while SmAlGe shows only a robust AFM order. In NdAlGe, the topological Hall effect (THE) appears between the first and the second metamagnetic transitions for $H \parallel c$, while it is absent in SmAlGe. Such THE in NdAlGe is most likely attributed to the field-induced topological spin textures. The longitudinal muon-spin relaxation rate $λ^\mathrm{L}(T)$, diverges near the AFM order, followed by a clear drop at $T < T_\mathrm{N}$. In the magnetically ordered state, spin fluctuations are significantly stronger in NdAlGe than in SmAlGe. In general, our longitudinal-field $μ$SR data indicate vigorous spin fluctuations in NdAlGe, thus providing valuable insights into the origin of THE and of the possible topological spin textures in $RE$Al(Si,Ge) Weyl semimetals.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
CholecInstanceSeg: A Tool Instance Segmentation Dataset for Laparoscopic Surgery
Authors:
Oluwatosin Alabi,
Ko Ko Zayar Toe,
Zijian Zhou,
Charlie Budd,
Nicholas Raison,
Miaojing Shi,
Tom Vercauteren
Abstract:
In laparoscopic and robotic surgery, precise tool instance segmentation is an essential technology for advanced computer-assisted interventions. Although publicly available procedures of routine surgeries exist, they often lack comprehensive annotations for tool instance segmentation. Additionally, the majority of standard datasets for tool segmentation are derived from porcine(pig) surgeries. To…
▽ More
In laparoscopic and robotic surgery, precise tool instance segmentation is an essential technology for advanced computer-assisted interventions. Although publicly available procedures of routine surgeries exist, they often lack comprehensive annotations for tool instance segmentation. Additionally, the majority of standard datasets for tool segmentation are derived from porcine(pig) surgeries. To address this gap, we introduce CholecInstanceSeg, the largest open-access tool instance segmentation dataset to date. Derived from the existing CholecT50 and Cholec80 datasets, CholecInstanceSeg provides novel annotations for laparoscopic cholecystectomy procedures in patients. Our dataset comprises 41.9k annotated frames extracted from 85 clinical procedures and 64.4k tool instances, each labelled with semantic masks and instance IDs. To ensure the reliability of our annotations, we perform extensive quality control, conduct label agreement statistics, and benchmark the segmentation results with various instance segmentation baselines. CholecInstanceSeg aims to advance the field by offering a comprehensive and high-quality open-access dataset for the development and evaluation of tool instance segmentation algorithms.
△ Less
Submitted 23 June, 2024;
originally announced June 2024.
-
CMDS: Cross-layer Dataflow Optimization for DNN Accelerators Exploiting Multi-bank Memories
Authors:
Man Shi,
Steven Colleman,
Charlotte VanDeMieroop,
Antony Joseph,
Maurice Meijer,
Wim Dehaene,
Marian Verhelst
Abstract:
Deep neural networks (DNN) use a wide range of network topologies to achieve high accuracy within diverse applications. This model diversity makes it impossible to identify a single "dataflow" (execution schedule) to perform optimally across all possible layers and network topologies. Several frameworks support the exploration of the best dataflow for a given DNN layer and hardware. However, switc…
▽ More
Deep neural networks (DNN) use a wide range of network topologies to achieve high accuracy within diverse applications. This model diversity makes it impossible to identify a single "dataflow" (execution schedule) to perform optimally across all possible layers and network topologies. Several frameworks support the exploration of the best dataflow for a given DNN layer and hardware. However, switching the dataflow from one layer to the next layer within one DNN model can result in hardware inefficiencies stemming from memory data layout mismatch among the layers. Unfortunately, all existing frameworks treat each layer independently and typically model memories as black boxes (one large monolithic wide memory), which ignores the data layout and can not deal with the data layout dependencies of sequential layers. These frameworks are not capable of doing dataflow cross-layer optimization. This work, hence, aims at cross-layer dataflow optimization, taking the data dependency and data layout reshuffling overheads among layers into account. Additionally, we propose to exploit the multibank memories typically present in modern DNN accelerators towards efficiently reshuffling data to support more dataflow at low overhead. These innovations are supported through the Cross-layer Memory-aware Dataflow Scheduler (CMDS). CMDS can model DNN execution energy/latency while considering the different data layout requirements due to the varied optimal dataflow of layers. Compared with the state-of-the-art (SOTA), which performs layer-optimized memory-unaware scheduling, CMDS achieves up to 5.5X energy reduction and 1.35X latency reduction with negligible hardware cost.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
COAC: Cross-layer Optimization of Accelerator Configurability for Efficient CNN Processing
Authors:
Steven Colleman,
Man Shi,
Marian Verhelst
Abstract:
To achieve high accuracy, convolutional neural networks (CNNs) are increasingly growing in complexity and diversity in layer types and topologies. This makes it very challenging to efficiently deploy such networks on custom processor architectures for resource-scarce edge devices. Existing mapping exploration frameworks enable searching for the optimal execution schedules or hardware mappings of i…
▽ More
To achieve high accuracy, convolutional neural networks (CNNs) are increasingly growing in complexity and diversity in layer types and topologies. This makes it very challenging to efficiently deploy such networks on custom processor architectures for resource-scarce edge devices. Existing mapping exploration frameworks enable searching for the optimal execution schedules or hardware mappings of individual network layers, by optimizing each layer's spatial (dataflow parallelization) and temporal unrolling (execution order). However, these tools fail to take into account the overhead of supporting different unrolling schemes within a common hardware architecture. Using a fixed unrolling scheme across all layers is also not ideal, as this misses significant opportunities for energy and latency savings from optimizing the mapping of diverse layer types. A balanced approach assesses the right amount of mapping flexibility needed across target neural networks, while taking into account the overhead to support multiple unrollings. This paper, therefore, presents COAC, a cross-layer design space exploration and mapping framework to optimize the flexibility of neural processing architectures by balancing configurability overhead against resulting energy and latency savings for end-to-end inference. COAC does not only provide a systematical analysis of the architectural overhead in function of the supported spatial unrollings, but also builds an automated flow to find the best unrolling combination(s) for efficient end-to-end inference with limited hardware overhead. Results demonstrate that architectures with carefully optimized flexibility can achieve up to 38% EDP (energy-delay-product) savings for a set of six neural networks at the expense of a relative area increase of 9.5%.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Pitch-Aware RNN-T for Mandarin Chinese Mispronunciation Detection and Diagnosis
Authors:
Xintong Wang,
Mingqian Shi,
Ye Wang
Abstract:
Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD,…
▽ More
Mispronunciation Detection and Diagnosis (MDD) systems, leveraging Automatic Speech Recognition (ASR), face two main challenges in Mandarin Chinese: 1) The two-stage models create an information gap between the phoneme or tone classification stage and the MDD stage. 2) The scarcity of Mandarin MDD datasets limits model training. In this paper, we introduce a stateless RNN-T model for Mandarin MDD, utilizing HuBERT features with pitch embedding through a Pitch Fusion Block. Our model, trained solely on native speaker data, shows a 3% improvement in Phone Error Rate and a 7% increase in False Acceptance Rate over the state-of-the-art baseline in non-native scenarios
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Memory-guided Network with Uncertainty-based Feature Augmentation for Few-shot Semantic Segmentation
Authors:
Xinyue Chen,
Miaojing Shi
Abstract:
The performance of supervised semantic segmentation methods highly relies on the availability of large-scale training data. To alleviate this dependence, few-shot semantic segmentation (FSS) is introduced to leverage the model trained on base classes with sufficient data into the segmentation of novel classes with few data. FSS methods face the challenge of model generalization on novel classes du…
▽ More
The performance of supervised semantic segmentation methods highly relies on the availability of large-scale training data. To alleviate this dependence, few-shot semantic segmentation (FSS) is introduced to leverage the model trained on base classes with sufficient data into the segmentation of novel classes with few data. FSS methods face the challenge of model generalization on novel classes due to the distribution shift between base and novel classes. To overcome this issue, we propose a class-shared memory (CSM) module consisting of a set of learnable memory vectors. These memory vectors learn elemental object patterns from base classes during training whilst re-encoding query features during both training and inference, thereby improving the distribution alignment between base and novel classes. Furthermore, to cope with the performance degradation resulting from the intra-class variance across images, we introduce an uncertainty-based feature augmentation (UFA) module to produce diverse query features during training for improving the model's robustness. We integrate CSM and UFA into representative FSS works, with experimental results on the widely-used PASCAL-5$^i$ and COCO-20$^i$ datasets demonstrating the superior performance of ours over state of the art.
△ Less
Submitted 9 June, 2024; v1 submitted 1 June, 2024;
originally announced June 2024.
-
A Closer Look at Time Steps is Worthy of Triple Speed-Up for Diffusion Model Training
Authors:
Kai Wang,
Yukun Zhou,
Mingjia Shi,
Zhihang Yuan,
Yuzhang Shang,
Xiaojiang Peng,
Hanwang Zhang,
Yang You
Abstract:
Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment. ii) These time steps are imbalanced, with many…
▽ More
Training diffusion models is always a computation-intensive task. In this paper, we introduce a novel speed-up method for diffusion model training, called, which is based on a closer look at time steps. Our key findings are: i) Time steps can be empirically divided into acceleration, deceleration, and convergence areas based on the process increment. ii) These time steps are imbalanced, with many concentrated in the convergence area. iii) The concentrated steps provide limited benefits for diffusion training. To address this, we design an asymmetric sampling strategy that reduces the frequency of steps from the convergence area while increasing the sampling probability for steps from other areas. Additionally, we propose a weighting strategy to emphasize the importance of time steps with rapid-change process increments. As a plug-and-play and architecture-agnostic approach, SpeeD consistently achieves 3-times acceleration across various diffusion architectures, datasets, and tasks. Notably, due to its simple design, our approach significantly reduces the cost of diffusion model training with minimal overhead. Our research enables more researchers to train diffusion models at a lower cost.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
Charge transfer and Spin-Valley locking in 4Hb-TaS$_{2}$
Authors:
Avior Almoalem,
Roni Gofman,
Yuval Nitzav,
Ilay Mangel,
Irena Feldman,
Jahyun Koo,
Federico Mazzola,
Jun Fujii,
Ivana Vobornik,
J. Sanchez-Barriga,
Oliver J. Clark,
Nicholas Clark Plumb,
Ming Shi,
Binghai Yan,
Amit Kanigel
Abstract:
4Hb-TaS$_2$ is a superconductor that exhibits unique characteristics such as time-reversal symmetry breaking, hidden magnetic memory, and topological edge modes. It is a naturally occurring heterostructure comprising of alternating layers of 1H-TaS$_2$ and 1T-TaS$_2$. The former is a well-known superconductor, while the latter is a correlated insulator with a possible non-trivial magnetic ground s…
▽ More
4Hb-TaS$_2$ is a superconductor that exhibits unique characteristics such as time-reversal symmetry breaking, hidden magnetic memory, and topological edge modes. It is a naturally occurring heterostructure comprising of alternating layers of 1H-TaS$_2$ and 1T-TaS$_2$. The former is a well-known superconductor, while the latter is a correlated insulator with a possible non-trivial magnetic ground state. In this study, we use angle resolved photoemission spectroscopy to investigate the normal state electronic structure of this unconventional superconductor. Our findings reveal that the band structure of 4H-TaS$_2$ fundamentally differs from that of its constituent materials. Specifically, we observe a significant charge transfer from the 1T layers to the 1H layers that drives the 1T layers away from half-filling. In addition, we find a substantial reduction in inter-layer coupling in 4Hb-TaS$_2$ compared to the coupling in 2H-TaS$_2$ that results in a pronounced spin-valley locking within 4Hb-TaS$_2$
△ Less
Submitted 26 May, 2024;
originally announced May 2024.
-
Three-dimensional mapping and electronic origin of large altermagnetic splitting near Fermi level in CrSb
Authors:
Guowei Yang,
Zhanghuan Li,
Sai Yang,
Jiyuan Li,
Hao Zheng,
Weifan Zhu,
Saizheng Cao,
Wenxuan Zhao,
Jiawen Zhang,
Mao Ye,
Yu Song,
Lun-Hui Hu,
Lexian Yang,
Ming Shi,
Huiqiu Yuan,
Yongjun Zhang,
Yuanfeng Xu,
Yang Liu
Abstract:
Recently, a new kind of collinear magnetism, dubbed altermagnetism, has attracted considerable interests. A key characteristic of altermagnet is the momentum-dependent band and spin splitting without net magnetization. However, finding altermagnetic materials with large splitting near the Fermi level, which necessarily requires three-dimensional k-space mapping and is crucial for spintronic applic…
▽ More
Recently, a new kind of collinear magnetism, dubbed altermagnetism, has attracted considerable interests. A key characteristic of altermagnet is the momentum-dependent band and spin splitting without net magnetization. However, finding altermagnetic materials with large splitting near the Fermi level, which necessarily requires three-dimensional k-space mapping and is crucial for spintronic applications and emergent phenomena, remains challenging. Here by employing synchrotron-based angle-resolved photoemission spectroscopy (ARPES) and model calculations, we uncover a large altermagnetic splitting, up to ~1.0 eV, near the Fermi level in CrSb. We verify its bulk-type g-wave altermagnetism through systematic three-dimensional kspace mapping, which unambiguously reveals the altermagnetic symmetry and associated nodal planes. The ARPES results are well captured by density functional theory calculations. In addition, tight-binding model analysis indicate that the large altermagnetic splitting arises from strong third-nearest-neighbor hopping mediated by Sb ions, which breaks both the space-time reversal symmetry and the translational spin-rotation symmetry. The large band/spin splitting near Fermi level in metallic CrSb, together with its high TN (up to 705 K) and simple spin configuration, paves the way for exploring emergent phenomena and spintronic applications based on altermagnets.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
InterAct: Capture and Modelling of Realistic, Expressive and Interactive Activities between Two Persons in Daily Scenarios
Authors:
Yinghao Huang,
Leo Ho,
Dafei Qin,
Mingyi Shi,
Taku Komura
Abstract:
We address the problem of accurate capture and expressive modelling of interactive behaviors happening between two persons in daily scenarios. Different from previous works which either only consider one person or focus on conversational gestures, we propose to simultaneously model the activities of two persons, and target objective-driven, dynamic, and coherent interactions which often span long…
▽ More
We address the problem of accurate capture and expressive modelling of interactive behaviors happening between two persons in daily scenarios. Different from previous works which either only consider one person or focus on conversational gestures, we propose to simultaneously model the activities of two persons, and target objective-driven, dynamic, and coherent interactions which often span long duration. To this end, we capture a new dataset dubbed InterAct, which is composed of 241 motion sequences where two persons perform a realistic scenario over the whole sequence. The audios, body motions, and facial expressions of both persons are all captured in our dataset. We also demonstrate the first diffusion model based approach that directly estimates the interactive motions between two persons from their audios alone. All the data and code will be available at: https://hku-cg.github.io/interact.
△ Less
Submitted 27 May, 2024; v1 submitted 19 May, 2024;
originally announced May 2024.
-
Quantum Metrology with Higher-order Exceptional Points in Atom-cavity Magnonics
Authors:
Minwei Shi,
Guzhi Bao,
Jinxian Guo,
Weiping Zhang
Abstract:
Exceptional points (EPs), early arising from non-Hermitian physics, significantly amplify the system's response to minor perturbations, and act as a useful concept to enhance measurement in metrology. In particular, such a metrological enhancement grows dramatically with the EP's order. However, the Langevin noises intrinsically existing in the non-Hermitian systems diminish this enhancement. In t…
▽ More
Exceptional points (EPs), early arising from non-Hermitian physics, significantly amplify the system's response to minor perturbations, and act as a useful concept to enhance measurement in metrology. In particular, such a metrological enhancement grows dramatically with the EP's order. However, the Langevin noises intrinsically existing in the non-Hermitian systems diminish this enhancement. In this study, we propose a protocol for quantum metrology with the construction of higher-order EPs (HOEPs) in atom-cavity system through Hermitian magnon-photon interaction. The construction of HOEPs utilizes the atom-cavity non-Hermitian-like dynamical behavior but avoids the external Langevin noises via the Hermitian interaction. A general analysis is exhibited for the construction of arbitrary $n$-th order EP (EPn). As a demonstration of the superiority of these HOEPs in quantum metrology, we work out an EP3/4-based atomic sensor with sensitivity being orders of magnitude higher than that achievable in an EP2-based one. We further unveil the mechanism behind the sensitivity enhancement from HOEPs. The experimental establishment for this proposal is suggested with potential candidates. This EP-based atomic sensor, taking advantage of the atom-light interface, offers new insight into quantum metrology with HOEPs.
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
High sensitivity measurement of ULF, VLF and LF fields with Rydberg-atom sensor
Authors:
Mingwei Lei,
Meng Shi
Abstract:
Fields with frequencies below megahertz are challenging for Rydberg-atom-based measurements, due to the low-frequency electric field screening effect that is caused by the alkali-metal atoms adsorbed on the inner surface of the container. In this paper, we investigate on electric fields measurements in the ULF, VLF and LF bands in a Cs vapor cell with built-in parallel electrodes. With optimizatio…
▽ More
Fields with frequencies below megahertz are challenging for Rydberg-atom-based measurements, due to the low-frequency electric field screening effect that is caused by the alkali-metal atoms adsorbed on the inner surface of the container. In this paper, we investigate on electric fields measurements in the ULF, VLF and LF bands in a Cs vapor cell with built-in parallel electrodes. With optimization of the applied DC field, we achieve high-sensitive detection of the electric field at frequencies of 1kHz, 10kHz and 100kHz based on Rydberg-atom sensor, with the minimum electric field strength down to 18.0μV/cm, 6.9μV/cm and 3.0μV/cm, respectively. The corresponding sensitivity is 5.7 μV/cm/{\sqrt{Hz}}, 2.2μV/cm/{\sqrt{Hz}} and 0.95μV/cm/{\sqrt{Hz}} for ULF, VLF and LF fields, which is better than 1-cm dipole antenna. Besides, the linear dynamic range of Rydberg-atom sensor is over 50 dB. This work presents the potential to enable more applications that utilize atomic sensing technology in ULF, VLF and LF fields.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation
Authors:
Ye Liu,
Yue Xue,
Daoyuan Wu,
Yuqiang Sun,
Yi Li,
Miaolei Shi,
Yang Liu
Abstract:
With recent advances in large language models (LLMs), this paper explores the potential of leveraging state-of-the-art LLMs, such as GPT-4, to transfer existing human-written properties (e.g., those from Certora auditing reports) and automatically generate customized properties for unknown code. To this end, we embed existing properties into a vector database and retrieve a reference property for…
▽ More
With recent advances in large language models (LLMs), this paper explores the potential of leveraging state-of-the-art LLMs, such as GPT-4, to transfer existing human-written properties (e.g., those from Certora auditing reports) and automatically generate customized properties for unknown code. To this end, we embed existing properties into a vector database and retrieve a reference property for LLM-based in-context learning to generate a new prop- erty for a given code. While this basic process is relatively straight- forward, ensuring that the generated properties are (i) compilable, (ii) appropriate, and (iii) runtime-verifiable presents challenges. To address (i), we use the compilation and static analysis feedback as an external oracle to guide LLMs in iteratively revising the generated properties. For (ii), we consider multiple dimensions of similarity to rank the properties and employ a weighted algorithm to identify the top-K properties as the final result. For (iii), we design a dedicated prover to formally verify the correctness of the generated prop- erties. We have implemented these strategies into a novel system called PropertyGPT, with 623 human-written properties collected from 23 Certora projects. Our experiments show that PropertyGPT can generate comprehensive and high-quality properties, achieving an 80% recall compared to the ground truth. It successfully detected 26 CVEs/attack incidents out of 37 tested and also uncovered 12 zero-day vulnerabilities, resulting in $8,256 bug bounty rewards.
△ Less
Submitted 4 May, 2024;
originally announced May 2024.
-
OmniDrive: A Holistic LLM-Agent Framework for Autonomous Driving with 3D Perception, Reasoning and Planning
Authors:
Shihao Wang,
Zhiding Yu,
Xiaohui Jiang,
Shiyi Lan,
Min Shi,
Nadine Chang,
Jan Kautz,
Ying Li,
Jose M. Alvarez
Abstract:
The advances in multimodal large language models (MLLMs) have led to growing interests in LLM-based autonomous driving agents to leverage their strong reasoning capabilities. However, capitalizing on MLLMs' strong reasoning capabilities for improved planning behavior is challenging since planning requires full 3D situational awareness beyond 2D reasoning. To address this challenge, our work propos…
▽ More
The advances in multimodal large language models (MLLMs) have led to growing interests in LLM-based autonomous driving agents to leverage their strong reasoning capabilities. However, capitalizing on MLLMs' strong reasoning capabilities for improved planning behavior is challenging since planning requires full 3D situational awareness beyond 2D reasoning. To address this challenge, our work proposes a holistic framework for strong alignment between agent models and 3D driving tasks. Our framework starts with a novel 3D MLLM architecture that uses sparse queries to lift and compress visual representations into 3D before feeding them into an LLM. This query-based representation allows us to jointly encode dynamic objects and static map elements (e.g., traffic lanes), providing a condensed world model for perception-action alignment in 3D. We further propose OmniDrive-nuScenes, a new visual question-answering dataset challenging the true 3D situational awareness of a model with comprehensive visual question-answering (VQA) tasks, including scene description, traffic regulation, 3D grounding, counterfactual reasoning, decision making and planning. Extensive studies show the effectiveness of the proposed architecture as well as the importance of the VQA tasks for reasoning and planning in complex 3D scenes.
△ Less
Submitted 2 May, 2024;
originally announced May 2024.
-
Geometry-aware Reconstruction and Fusion-refined Rendering for Generalizable Neural Radiance Fields
Authors:
Tianqi Liu,
Xinyi Ye,
Min Shi,
Zihao Huang,
Zhiyu Pan,
Zhan Peng,
Zhiguo Cao
Abstract:
Generalizable NeRF aims to synthesize novel views for unseen scenes. Common practices involve constructing variance-based cost volumes for geometry reconstruction and encoding 3D descriptors for decoding novel views. However, existing methods show limited generalization ability in challenging conditions due to inaccurate geometry, sub-optimal descriptors, and decoding strategies. We address these…
▽ More
Generalizable NeRF aims to synthesize novel views for unseen scenes. Common practices involve constructing variance-based cost volumes for geometry reconstruction and encoding 3D descriptors for decoding novel views. However, existing methods show limited generalization ability in challenging conditions due to inaccurate geometry, sub-optimal descriptors, and decoding strategies. We address these issues point by point. First, we find the variance-based cost volume exhibits failure patterns as the features of pixels corresponding to the same point can be inconsistent across different views due to occlusions or reflections. We introduce an Adaptive Cost Aggregation (ACA) approach to amplify the contribution of consistent pixel pairs and suppress inconsistent ones. Unlike previous methods that solely fuse 2D features into descriptors, our approach introduces a Spatial-View Aggregator (SVA) to incorporate 3D context into descriptors through spatial and inter-view interaction. When decoding the descriptors, we observe the two existing decoding strategies excel in different areas, which are complementary. A Consistency-Aware Fusion (CAF) strategy is proposed to leverage the advantages of both. We incorporate the above ACA, SVA, and CAF into a coarse-to-fine framework, termed Geometry-aware Reconstruction and Fusion-refined Rendering (GeFu). GeFu attains state-of-the-art performance across multiple datasets. Code is available at https://github.com/TQTQliu/GeFu .
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
Decentralized Multi-Agent Trajectory Planning in Dynamic Environments with Spatiotemporal Occupancy Grid Maps
Authors:
Siyuan Wu,
Gang Chen,
Moji Shi,
Javier Alonso-Mora
Abstract:
This paper proposes a decentralized trajectory planning framework for the collision avoidance problem of multiple micro aerial vehicles (MAVs) in environments with static and dynamic obstacles. The framework utilizes spatiotemporal occupancy grid maps (SOGM), which forecast the occupancy status of neighboring space in the near future, as the environment representation. Based on this representation…
▽ More
This paper proposes a decentralized trajectory planning framework for the collision avoidance problem of multiple micro aerial vehicles (MAVs) in environments with static and dynamic obstacles. The framework utilizes spatiotemporal occupancy grid maps (SOGM), which forecast the occupancy status of neighboring space in the near future, as the environment representation. Based on this representation, we extend the kinodynamic A* and the corridor-constrained trajectory optimization algorithms to efficiently tackle static and dynamic obstacles with arbitrary shapes. Collision avoidance between communicating robots is integrated by sharing planned trajectories and projecting them onto the SOGM. The simulation results show that our method achieves competitive performance against state-of-the-art methods in dynamic environments with different numbers and shapes of obstacles. Finally, the proposed method is validated in real experiments.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Taming Diffusion Probabilistic Models for Character Control
Authors:
Rui Chen,
Mingyi Shi,
Shaoli Huang,
Ping Tan,
Taku Komura,
Xuelin Chen
Abstract:
We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM), which takes as input the character's his…
▽ More
We present a novel character control framework that effectively utilizes motion diffusion probabilistic models to generate high-quality and diverse character animations, responding in real-time to a variety of dynamic user-supplied control signals. At the heart of our method lies a transformer-based Conditional Autoregressive Motion Diffusion Model (CAMDM), which takes as input the character's historical motion and can generate a range of diverse potential future motions conditioned on high-level, coarse user control. To meet the demands for diversity, controllability, and computational efficiency required by a real-time controller, we incorporate several key algorithmic designs. These include separate condition tokenization, classifier-free guidance on past motion, and heuristic future trajectory extension, all designed to address the challenges associated with taming motion diffusion probabilistic models for character control. As a result, our work represents the first model that enables real-time generation of high-quality, diverse character animations based on user interactive control, supporting animating the character in multiple styles with a single unified model. We evaluate our method on a diverse set of locomotion skills, demonstrating the merits of our method over existing character controllers. Project page and source codes: https://aiganimation.github.io/CAMDM/
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Evaluating Dynamic Environment Difficulty for Obstacle Avoidance Benchmarking
Authors:
Moji Shi,
Gang Chen,
Álvaro Serra Gómez,
Siyuan Wu,
Javier Alonso-Mora
Abstract:
Dynamic obstacle avoidance is a popular research topic for autonomous systems, such as micro aerial vehicles and service robots. Accurately evaluating the performance of dynamic obstacle avoidance methods necessitates the establishment of a metric to quantify the environment's difficulty, a crucial aspect that remains unexplored. In this paper, we propose four metrics to measure the difficulty of…
▽ More
Dynamic obstacle avoidance is a popular research topic for autonomous systems, such as micro aerial vehicles and service robots. Accurately evaluating the performance of dynamic obstacle avoidance methods necessitates the establishment of a metric to quantify the environment's difficulty, a crucial aspect that remains unexplored. In this paper, we propose four metrics to measure the difficulty of dynamic environments. These metrics aim to comprehensively capture the influence of obstacles' number, size, velocity, and other factors on the difficulty. We compare the proposed metrics with existing static environment difficulty metrics and validate them through over 1.5 million trials in a customized simulator. This simulator excludes the effects of perception and control errors and supports different motion and gaze planners for obstacle avoidance. The results indicate that the survivability metric outperforms and establishes a monotonic relationship between the success rate, with a Spearman's Rank Correlation Coefficient (SRCC) of over 0.9. Specifically, for every planner, lower survivability leads to a higher success rate. This metric not only facilitates fair and comprehensive benchmarking but also provides insights for refining collision avoidance methods, thereby furthering the evolution of autonomous systems in dynamic environments.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Measuring Geographic Diversity of Foundation Models with a Natural Language--based Geo-guessing Experiment on GPT-4
Authors:
Zilong Liu,
Krzysztof Janowicz,
Kitty Currier,
Meilin Shi
Abstract:
Generative AI based on foundation models provides a first glimpse into the world represented by machines trained on vast amounts of multimodal data ingested by these models during training. If we consider the resulting models as knowledge bases in their own right, this may open up new avenues for understanding places through the lens of machines. In this work, we adopt this thinking and select GPT…
▽ More
Generative AI based on foundation models provides a first glimpse into the world represented by machines trained on vast amounts of multimodal data ingested by these models during training. If we consider the resulting models as knowledge bases in their own right, this may open up new avenues for understanding places through the lens of machines. In this work, we adopt this thinking and select GPT-4, a state-of-the-art representative in the family of multimodal large language models, to study its geographic diversity regarding how well geographic features are represented. Using DBpedia abstracts as a ground-truth corpus for probing, our natural language--based geo-guessing experiment shows that GPT-4 may currently encode insufficient knowledge about several geographic feature types on a global level. On a local level, we observe not only this insufficiency but also inter-regional disparities in GPT-4's geo-guessing performance on UNESCO World Heritage Sites that carry significance to both local and global populations, and the inter-regional disparities may become smaller as the geographic scale increases. Morever, whether assessing the geo-guessing performance on a global or local level, we find inter-model disparities in GPT-4's geo-guessing performance when comparing its unimodal and multimodal variants. We hope this work can initiate a discussion on geographic diversity as an ethical principle within the GIScience community in the face of global socio-technical challenges.
△ Less
Submitted 11 April, 2024;
originally announced April 2024.
-
SCAResNet: A ResNet Variant Optimized for Tiny Object Detection in Transmission and Distribution Towers
Authors:
Weile Li,
Muqing Shi,
Zhonghua Hong
Abstract:
Traditional deep learning-based object detection networks often resize images during the data preprocessing stage to achieve a uniform size and scale in the feature map. Resizing is done to facilitate model propagation and fully connected classification. However, resizing inevitably leads to object deformation and loss of valuable information in the images. This drawback becomes particularly prono…
▽ More
Traditional deep learning-based object detection networks often resize images during the data preprocessing stage to achieve a uniform size and scale in the feature map. Resizing is done to facilitate model propagation and fully connected classification. However, resizing inevitably leads to object deformation and loss of valuable information in the images. This drawback becomes particularly pronounced for tiny objects like distribution towers with linear shapes and few pixels. To address this issue, we propose abandoning the resizing operation. Instead, we introduce Positional-Encoding Multi-head Criss-Cross Attention. This allows the model to capture contextual information and learn from multiple representation subspaces, effectively enriching the semantics of distribution towers. Additionally, we enhance Spatial Pyramid Pooling by reshaping three pooled feature maps into a new unified one while also reducing the computational burden. This approach allows images of different sizes and scales to generate feature maps with uniform dimensions and can be employed in feature map propagation. Our SCAResNet incorporates these aforementioned improvements into the backbone network ResNet. We evaluated our SCAResNet using the Electric Transmission and Distribution Infrastructure Imagery dataset from Duke University. Without any additional tricks, we employed various object detection models with Gaussian Receptive Field based Label Assignment as the baseline. When incorporating the SCAResNet into the baseline model, we achieved a 2.1% improvement in mAPs. This demonstrates the advantages of our SCAResNet in detecting transmission and distribution towers and its value in tiny object detection. The source code is available at https://github.com/LisavilaLee/SCAResNet_mmdet.
△ Less
Submitted 5 April, 2024;
originally announced April 2024.
-
Some bounds on the cardinality of the $b$-symbol weight spectrum of codes
Authors:
Hongwei Zhu,
Shitao Li,
Minjia Shi,
Shu-Tao Xia,
Patrick Sole
Abstract:
The size of the Hamming distance spectrum of a code has received great attention in recent research. The main objective of this paper is to extend these significant theories to the $b$-symbol distance spectrum. We examine this question for various types of codes, including unrestricted codes, additive codes, linear codes, and cyclic codes, successively. For the first three cases, we determine the…
▽ More
The size of the Hamming distance spectrum of a code has received great attention in recent research. The main objective of this paper is to extend these significant theories to the $b$-symbol distance spectrum. We examine this question for various types of codes, including unrestricted codes, additive codes, linear codes, and cyclic codes, successively. For the first three cases, we determine the maximum size of the $b$-symbol distance spectra of these codes smoothly. For the case of cyclic codes, we introduce three approaches to characterize the upper bound for the cardinality of the $b$-symbol weight spectrum of cyclic codes, namely the period distribution approach, the primitive idempotent approach, and the $b$-symbol weight formula approach. As two by-products of this paper, the maximum number of symplectic weights of linear codes is determined, and a basic inequality among the parameters $[n,k,d_H(\C)]_q$ of cyclic codes is provided.
△ Less
Submitted 3 April, 2024;
originally announced April 2024.
-
Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge
Authors:
Haoxiang Ma,
Modi Shi,
Boyang Gao,
Di Huang
Abstract:
We focus on the generalization ability of the 6-DoF grasp detection method in this paper. While learning-based grasp detection methods can predict grasp poses for unseen objects using the grasp distribution learned from the training set, they often exhibit a significant performance drop when encountering objects with diverse shapes and structures. To enhance the grasp detection methods' generaliza…
▽ More
We focus on the generalization ability of the 6-DoF grasp detection method in this paper. While learning-based grasp detection methods can predict grasp poses for unseen objects using the grasp distribution learned from the training set, they often exhibit a significant performance drop when encountering objects with diverse shapes and structures. To enhance the grasp detection methods' generalization ability, we incorporate domain prior knowledge of robotic grasping, enabling better adaptation to objects with significant shape and structure differences. More specifically, we employ the physical constraint regularization during the training phase to guide the model towards predicting grasps that comply with the physical rule on grasping. For the unstable grasp poses predicted on novel objects, we design a contact-score joint optimization using the projection contact map to refine these poses in cluttered scenarios. Extensive experiments conducted on the GraspNet-1billion benchmark demonstrate a substantial performance gain on the novel object set and the real-world grasping experiments also demonstrate the effectiveness of our generalizing 6-DoF grasp detection method.
△ Less
Submitted 2 April, 2024;
originally announced April 2024.
-
FairCLIP: Harnessing Fairness in Vision-Language Learning
Authors:
Yan Luo,
Min Shi,
Muhammad Osama Khan,
Muhammad Muneeb Afzal,
Hao Huang,
Shuaihang Yuan,
Yu Tian,
Luo Song,
Ava Kouhana,
Tobias Elze,
Yi Fang,
Mengyu Wang
Abstract:
Fairness is a critical concern in deep learning, especially in healthcare, where these models influence diagnoses and treatment decisions. Although fairness has been investigated in the vision-only domain, the fairness of medical vision-language (VL) models remains unexplored due to the scarcity of medical VL datasets for studying fairness. To bridge this research gap, we introduce the first fair…
▽ More
Fairness is a critical concern in deep learning, especially in healthcare, where these models influence diagnoses and treatment decisions. Although fairness has been investigated in the vision-only domain, the fairness of medical vision-language (VL) models remains unexplored due to the scarcity of medical VL datasets for studying fairness. To bridge this research gap, we introduce the first fair vision-language medical dataset Harvard-FairVLMed that provides detailed demographic attributes, ground-truth labels, and clinical notes to facilitate an in-depth examination of fairness within VL foundation models. Using Harvard-FairVLMed, we conduct a comprehensive fairness analysis of two widely-used VL models (CLIP and BLIP2), pre-trained on both natural and medical domains, across four different protected attributes. Our results highlight significant biases in all VL models, with Asian, Male, Non-Hispanic, and Spanish being the preferred subgroups across the protected attributes of race, gender, ethnicity, and language, respectively. In order to alleviate these biases, we propose FairCLIP, an optimal-transport-based approach that achieves a favorable trade-off between performance and fairness by reducing the Sinkhorn distance between the overall sample distribution and the distributions corresponding to each demographic group. As the first VL dataset of its kind, Harvard-FairVLMed holds the potential to catalyze advancements in the development of machine learning models that are both ethically aware and clinically effective. Our dataset and code are available at https://ophai.hms.harvard.edu/datasets/harvard-fairvlmed10k.
△ Less
Submitted 5 April, 2024; v1 submitted 28 March, 2024;
originally announced March 2024.
-
Ac$_3$Ni$_2$O$_7$ and La$_2$$Ae$Ni$_2$O$_6$F ($Ae$ = Sr, Ba): Benchmark Materials for Bilayer Nickelate Superconductivity
Authors:
Siqi Wu,
Zihan Yang,
Xin Ma,
Jianhui Dai,
Ming Shi,
Hui-Qiu Yuan,
Hai-Qing Lin,
Chao Cao
Abstract:
We theoretically propose Ac$_3$Ni$_2$O$_7$, La$_2$BaNi$_2$O$_6$F, and La$_2$SrNi$_2$O$_6$F compounds to be benchmark materials for bilayer nickelate superconductivity. The stable phase of Ac$_3$Ni$_2$O$_7$ and La$_2$BaNi$_2$O$_6$F are found to be $I4/mmm$ without the lattice distortion caused by octahedra rotation at ambient pressure, where as the lattice distortion in La$_2$SrNi$_2$O$_6$F can be…
▽ More
We theoretically propose Ac$_3$Ni$_2$O$_7$, La$_2$BaNi$_2$O$_6$F, and La$_2$SrNi$_2$O$_6$F compounds to be benchmark materials for bilayer nickelate superconductivity. The stable phase of Ac$_3$Ni$_2$O$_7$ and La$_2$BaNi$_2$O$_6$F are found to be $I4/mmm$ without the lattice distortion caused by octahedra rotation at ambient pressure, where as the lattice distortion in La$_2$SrNi$_2$O$_6$F can be suppressed with relatively small external pressure of 4 GPa. The magnetism, electronic structure and spin susceptibilities of Ac$_3$Ni$_2$O$_7$ are extremely close to those of La$_3$Ni$_2$O$_7$ at 30 GPa. The ground state of La$_2$BaNi$_2$O$_6$F and La$_2$SrNi$_2$O$_6$F are antiferromagnetically coupled checkerboard bilayer with sizable magnetic moment on Ni. In addition, the inter-layer coupling $J_{\perp}$ between Ni-bilayers in La$_2$BaNi$_2$O$_6$F or La$_2$SrNi$_2$O$_6$F is only $\sim$ 1/10 of that in Ac$_3$Ni$_2$O$_7$ or La$_3$Ni$_2$O$_7$ at 30 GPa. We argue that these compounds may serve as superconducting candidates at ambient pressure and can be employed to testify theoretical proposals for bilayer nickelate superconductivity.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Sim-to-Real Grasp Detection with Global-to-Local RGB-D Adaptation
Authors:
Haoxiang Ma,
Ran Qin,
Modi shi,
Boyang Gao,
Di Huang
Abstract:
This paper focuses on the sim-to-real issue of RGB-D grasp detection and formulates it as a domain adaptation problem. In this case, we present a global-to-local method to address hybrid domain gaps in RGB and depth data and insufficient multi-modal feature alignment. First, a self-supervised rotation pre-training strategy is adopted to deliver robust initialization for RGB and depth networks. We…
▽ More
This paper focuses on the sim-to-real issue of RGB-D grasp detection and formulates it as a domain adaptation problem. In this case, we present a global-to-local method to address hybrid domain gaps in RGB and depth data and insufficient multi-modal feature alignment. First, a self-supervised rotation pre-training strategy is adopted to deliver robust initialization for RGB and depth networks. We then propose a global-to-local alignment pipeline with individual global domain classifiers for scene features of RGB and depth images as well as a local one specifically working for grasp features in the two modalities. In particular, we propose a grasp prototype adaptation module, which aims to facilitate fine-grained local feature alignment by dynamically updating and matching the grasp prototypes from the simulation and real-world scenarios throughout the training process. Due to such designs, the proposed method substantially reduces the domain shift and thus leads to consistent performance improvements. Extensive experiments are conducted on the GraspNet-Planar benchmark and physical environment, and superior results are achieved which demonstrate the effectiveness of our method.
△ Less
Submitted 18 March, 2024;
originally announced March 2024.
-
Large Model driven Radiology Report Generation with Clinical Quality Reinforcement Learning
Authors:
Zijian Zhou,
Miaojing Shi,
Meng Wei,
Oluwatosin Alabi,
Zijie Yue,
Tom Vercauteren
Abstract:
Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. Current RRG approaches are still unsatisfactory against clinical standards. This paper introduces a novel RRG method, \textbf{LM-RRG}, that integrates large models (LMs) with clinical quality reinforcement learning to generate accurate and comprehensive chest X-ray rad…
▽ More
Radiology report generation (RRG) has attracted significant attention due to its potential to reduce the workload of radiologists. Current RRG approaches are still unsatisfactory against clinical standards. This paper introduces a novel RRG method, \textbf{LM-RRG}, that integrates large models (LMs) with clinical quality reinforcement learning to generate accurate and comprehensive chest X-ray radiology reports. Our method first designs a large language model driven feature extractor to analyze and interpret different regions of the chest X-ray image, emphasizing specific regions with medical significance. Next, based on the large model's decoder, we develop a multimodal report generator that leverages multimodal prompts from visual features and textual instruction to produce the radiology report in an auto-regressive way. Finally, to better reflect the clinical significant and insignificant errors that radiologists would normally assign in the report, we introduce a novel clinical quality reinforcement learning strategy. It utilizes the radiology report clinical quality (RadCliQ) metric as a reward function in the learning process. Extensive experiments on the MIMIC-CXR and IU-Xray datasets demonstrate the superiority of our method over the state of the art.
△ Less
Submitted 11 March, 2024;
originally announced March 2024.
-
3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors
Authors:
Fangzhou Hong,
Jiaxiang Tang,
Ziang Cao,
Min Shi,
Tong Wu,
Zhaoxi Chen,
Shuai Yang,
Tengfei Wang,
Liang Pan,
Dahua Lin,
Ziwei Liu
Abstract:
We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The sec…
▽ More
We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The second stage utilizes 2D diffusion priors to further refine the texture of coarse 3D models from the first stage. The refinement consists of both latent and pixel space optimization for high-quality texture generation. To facilitate the training of the proposed system, we clean and caption the largest open-source 3D dataset, Objaverse, by combining the power of vision language models and large language models. Experiment results are reported qualitatively and quantitatively to show the performance of the proposed system. Our codes and models are available at https://github.com/3DTopia/3DTopia
△ Less
Submitted 6 May, 2024; v1 submitted 4 March, 2024;
originally announced March 2024.
-
Efficiency-improved doubly robust estimation with non-confounding predictive covariates
Authors:
Shanshan Luo,
Mengchen Shi,
Wei Li,
Xueli Wang,
Zhi Geng
Abstract:
In observational studies, covariates with substantial missing data are often omitted, despite their strong predictive capabilities. These excluded covariates are generally believed not to simultaneously affect both treatment and outcome, indicating that they are not genuine confounders and do not impact the identification of the average treatment effect (ATE). In this paper, we introduce an altern…
▽ More
In observational studies, covariates with substantial missing data are often omitted, despite their strong predictive capabilities. These excluded covariates are generally believed not to simultaneously affect both treatment and outcome, indicating that they are not genuine confounders and do not impact the identification of the average treatment effect (ATE). In this paper, we introduce an alternative doubly robust (DR) estimator that fully leverages non-confounding predictive covariates to enhance efficiency, while also allowing missing values in such covariates. Beyond the double robustness property, our proposed estimator is designed to be more efficient than the standard DR estimator. Specifically, when the propensity score model is correctly specified, it achieves the smallest asymptotic variance among the class of DR estimators, and brings additional efficiency gains by further integrating predictive covariates. Simulation studies demonstrate the notable performance of the proposed estimator over current popular methods. An illustrative example is provided to assess the effectiveness of right heart catheterization (RHC) for critically ill patients.
△ Less
Submitted 22 February, 2024;
originally announced February 2024.
-
An Elementary Predictor Obtaining $2\sqrt{T}$ Distance to Calibration
Authors:
Eshwar Ram Arunachaleswaran,
Natalie Collina,
Aaron Roth,
Mirah Shi
Abstract:
Blasiok et al. [2023] proposed distance to calibration as a natural measure of calibration error that unlike expected calibration error (ECE) is continuous. Recently, Qiao and Zheng [2024] gave a non-constructive argument establishing the existence of an online predictor that can obtain $O(\sqrt{T})$ distance to calibration in the adversarial setting, which is known to be impossible for ECE. They…
▽ More
Blasiok et al. [2023] proposed distance to calibration as a natural measure of calibration error that unlike expected calibration error (ECE) is continuous. Recently, Qiao and Zheng [2024] gave a non-constructive argument establishing the existence of an online predictor that can obtain $O(\sqrt{T})$ distance to calibration in the adversarial setting, which is known to be impossible for ECE. They leave as an open problem finding an explicit, efficient algorithm. We resolve this problem and give an extremely simple, efficient, deterministic algorithm that obtains distance to calibration error at most $2\sqrt{T}$.
△ Less
Submitted 17 February, 2024;
originally announced February 2024.
-
Horizontally Polarized Kink Oscillations Supported by Solar Coronal Loops in an Asymmetric Environment
Authors:
Mijie Shi,
Bo Li,
Shengju Yuan
Abstract:
Kink oscillations are ubiquitously observed in solar coronal loops, their understanding being crucial in the contexts of coronal seismology and atmospheric heating. We study kink modes supported by a straight coronal loop embeded in an asymmetric environment using three-dimensional magnetohydrodynamic (MHD) simulations. We implement the asymmetric effect by setting different exterior densities bel…
▽ More
Kink oscillations are ubiquitously observed in solar coronal loops, their understanding being crucial in the contexts of coronal seismology and atmospheric heating. We study kink modes supported by a straight coronal loop embeded in an asymmetric environment using three-dimensional magnetohydrodynamic (MHD) simulations. We implement the asymmetric effect by setting different exterior densities below and above the loop interior, and initiate the simulation using a kink-like velocity perturbation perpendicular to the loop plane, mimicking the frequently measured horizontally polarized kink modes. We find that the external velocity fields show fan blade structures propagating in the azimuthal direction as a result of the successive excitation of higher azimuthal Fourier modes. Resonant absorption and phase mixing can still occur despite an asymmetric environment, leading to the development of small scales at loop boundaries. These small scales nonetheless develop asymmetrically at the upper and lower boundaries due to the different gradients of the Alfven speed. These findings enrich our understanding of kink modes in coronal loops embedded within an asymmetric environment, providing insights helpful for future high-resolution observations.
△ Less
Submitted 16 February, 2024;
originally announced February 2024.
-
AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators
Authors:
Jingwei Ni,
Minjing Shi,
Dominik Stammbach,
Mrinmaya Sachan,
Elliott Ash,
Markus Leippold
Abstract:
With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To addre…
▽ More
With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.
△ Less
Submitted 2 June, 2024; v1 submitted 16 February, 2024;
originally announced February 2024.
-
Forecasting for Swap Regret for All Downstream Agents
Authors:
Aaron Roth,
Mirah Shi
Abstract:
We study the problem of making predictions so that downstream agents who best respond to them will be guaranteed diminishing swap regret, no matter what their utility functions are. It has been known since Foster and Vohra (1997) that agents who best-respond to calibrated forecasts have no swap regret. Unfortunately, the best known algorithms for guaranteeing calibrated forecasts in sequential adv…
▽ More
We study the problem of making predictions so that downstream agents who best respond to them will be guaranteed diminishing swap regret, no matter what their utility functions are. It has been known since Foster and Vohra (1997) that agents who best-respond to calibrated forecasts have no swap regret. Unfortunately, the best known algorithms for guaranteeing calibrated forecasts in sequential adversarial environments do so at rates that degrade exponentially with the dimension of the prediction space. In this work, we show that by making predictions that are not calibrated, but are unbiased subject to a carefully selected collection of events, we can guarantee arbitrary downstream agents diminishing swap regret at rates that substantially improve over the rates that result from calibrated forecasts -- while maintaining the appealing property that our forecasts give guarantees for any downstream agent, without our forecasting algorithm needing to know their utility function.
We give separate results in the ``low'' (1 or 2) dimensional setting and the ``high'' ($> 2$) dimensional setting. In the low dimensional setting, we show how to make predictions such that all agents who best respond to our predictions have diminishing swap regret -- in 1 dimension, at the optimal $O(\sqrt{T})$ rate. In the high dimensional setting we show how to make forecasts that guarantee regret scaling at a rate of $O(T^{2/3})$ (crucially, a dimension independent exponent), under the assumption that downstream agents smoothly best respond. Our results stand in contrast to rates that derive from agents who best respond to calibrated forecasts, which have an exponential dependence on the dimension of the prediction space.
△ Less
Submitted 15 June, 2024; v1 submitted 13 February, 2024;
originally announced February 2024.
-
AdaTreeFormer: Few Shot Domain Adaptation for Tree Counting from a Single High-Resolution Image
Authors:
Hamed Amini Amirkolaee,
Miaojing Shi,
Lianghua He,
Mark Mulligan
Abstract:
The process of estimating and counting tree density using only a single aerial or satellite image is a difficult task in the fields of photogrammetry and remote sensing. However, it plays a crucial role in the management of forests. The huge variety of trees in varied topography severely hinders tree counting models to perform well. The purpose of this paper is to propose a framework that is learn…
▽ More
The process of estimating and counting tree density using only a single aerial or satellite image is a difficult task in the fields of photogrammetry and remote sensing. However, it plays a crucial role in the management of forests. The huge variety of trees in varied topography severely hinders tree counting models to perform well. The purpose of this paper is to propose a framework that is learnt from the source domain with sufficient labeled trees and is adapted to the target domain with only a limited number of labeled trees. Our method, termed as AdaTreeFormer, contains one shared encoder with a hierarchical feature extraction scheme to extract robust features from the source and target domains. It also consists of three subnets: two for extracting self-domain attention maps from source and target domains respectively and one for extracting cross-domain attention maps. For the latter, an attention-to-adapt mechanism is introduced to distill relevant information from different domains while generating tree density maps; a hierarchical cross-domain feature alignment scheme is proposed that progressively aligns the features from the source and target domains. We also adopt adversarial learning into the framework to further reduce the gap between source and target domains. Our AdaTreeFormer is evaluated on six designed domain adaptation tasks using three tree counting datasets, \ie Jiangsu, Yosemite, and London. Experimental results show that AdaTreeFormer significantly surpasses the state of the art, \eg in the cross domain from the Yosemite to Jiangsu dataset, it achieves a reduction of 15.9 points in terms of the absolute counting errors and an increase of 10.8\% in the accuracy of the detected trees' locations. The codes and datasets are available at https://github.com/HAAClassic/AdaTreeFormer.
△ Less
Submitted 30 June, 2024; v1 submitted 5 February, 2024;
originally announced February 2024.