-
MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting
Authors:
Zhiqi Ai,
Zhiyong Chen,
Shugong Xu
Abstract:
In this paper, we propose MM-KWS, a novel approach to user-defined keyword spotting leveraging multi-modal enrollments of text and speech templates. Unlike previous methods that focus solely on either text or speech features, MM-KWS extracts phoneme, text, and speech embeddings from both modalities. These embeddings are then compared with the query speech embedding to detect the target keywords. T…
▽ More
In this paper, we propose MM-KWS, a novel approach to user-defined keyword spotting leveraging multi-modal enrollments of text and speech templates. Unlike previous methods that focus solely on either text or speech features, MM-KWS extracts phoneme, text, and speech embeddings from both modalities. These embeddings are then compared with the query speech embedding to detect the target keywords. To ensure the applicability of MM-KWS across diverse languages, we utilize a feature extractor incorporating several multilingual pre-trained models. Subsequently, we validate its effectiveness on Mandarin and English tasks. In addition, we have integrated advanced data augmentation tools for hard case mining to enhance MM-KWS in distinguishing confusable words. Experimental results on the LibriPhrase and WenetPhrase datasets demonstrate that MM-KWS outperforms prior methods significantly.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
DeformNet: Latent Space Modeling and Dynamics Prediction for Deformable Object Manipulation
Authors:
Chenchang Li,
Zihao Ai,
Tong Wu,
Xiaosa Li,
Wenbo Ding,
Huazhe Xu
Abstract:
Manipulating deformable objects is a ubiquitous task in household environments, demanding adequate representation and accurate dynamics prediction due to the objects' infinite degrees of freedom. This work proposes DeformNet, which utilizes latent space modeling with a learned 3D representation model to tackle these challenges effectively. The proposed representation model combines a PointNet enco…
▽ More
Manipulating deformable objects is a ubiquitous task in household environments, demanding adequate representation and accurate dynamics prediction due to the objects' infinite degrees of freedom. This work proposes DeformNet, which utilizes latent space modeling with a learned 3D representation model to tackle these challenges effectively. The proposed representation model combines a PointNet encoder and a conditional neural radiance field (NeRF), facilitating a thorough acquisition of object deformations and variations in lighting conditions. To model the complex dynamics, we employ a recurrent state-space model (RSSM) that accurately predicts the transformation of the latent representation over time. Extensive simulation experiments with diverse objectives demonstrate the generalization capabilities of DeformNet for various deformable object manipulation tasks, even in the presence of previously unseen goals. Finally, we deploy DeformNet on an actual UR5 robotic arm to demonstrate its capability in real-world scenarios.
△ Less
Submitted 12 February, 2024;
originally announced February 2024.
-
GPT-Lab: Next Generation Of Optimal Chemistry Discovery By GPT Driven Robotic Lab
Authors:
Xiaokai Qin,
Mingda Song,
Yangguan Chen,
Zhehong Ai,
Jing Jiang
Abstract:
The integration of robots in chemical experiments has enhanced experimental efficiency, but lacking the human intelligence to comprehend literature, they seldom provide assistance in experimental design. Therefore, achieving full-process autonomy from experiment design to validation in self-driven laboratories (SDL) remains a challenge. The introduction of Generative Pre-trained Transformers (GPT)…
▽ More
The integration of robots in chemical experiments has enhanced experimental efficiency, but lacking the human intelligence to comprehend literature, they seldom provide assistance in experimental design. Therefore, achieving full-process autonomy from experiment design to validation in self-driven laboratories (SDL) remains a challenge. The introduction of Generative Pre-trained Transformers (GPT), particularly GPT-4, into robotic experimentation offers a solution. We introduce GPT-Lab, a paradigm that employs GPT models to give robots human-like intelligence. With our robotic experimentation platform, GPT-Lab mines literature for materials and methods and validates findings through high-throughput synthesis. As a demonstration, GPT-Lab analyzed 500 articles, identified 18 potential reagents, and successfully produced an accurate humidity colorimetric sensor with a root mean square error (RMSE) of 2.68%. This showcases the rapid materials discovery and validation potential of our system.
△ Less
Submitted 15 September, 2023;
originally announced September 2023.
-
On the Optimization of Margin Distribution
Authors:
Meng-Zhang Qian,
Zheng Ai,
Teng Zhang,
Wei Gao
Abstract:
Margin has played an important role on the design and analysis of learning algorithms during the past years, mostly working with the maximization of the minimum margin. Recent years have witnessed the increasing empirical studies on the optimization of margin distribution according to different statistics such as medium margin, average margin, margin variance, etc., whereas there is a relative pau…
▽ More
Margin has played an important role on the design and analysis of learning algorithms during the past years, mostly working with the maximization of the minimum margin. Recent years have witnessed the increasing empirical studies on the optimization of margin distribution according to different statistics such as medium margin, average margin, margin variance, etc., whereas there is a relative paucity of theoretical understanding. In this work, we take one step on this direction by providing a new generalization error bound, which is heavily relevant to margin distribution by incorporating ingredients such as average margin and semi-variance, a new margin statistics for the characterization of margin distribution. Inspired by the theoretical findings, we propose the MSVMAv, an efficient approach to achieve better performance by optimizing margin distribution in terms of its empirical average margin and semi-variance. We finally conduct extensive experiments to show the superiority of the proposed MSVMAv approach.
△ Less
Submitted 31 May, 2022; v1 submitted 29 April, 2022;
originally announced April 2022.
-
FedCoin: A Peer-to-Peer Payment System for Federated Learning
Authors:
Yuan Liu,
Shuai Sun,
Zhengpeng Ai,
Shuangfeng Zhang,
Zelei Liu,
Han Yu
Abstract:
Federated learning (FL) is an emerging collaborative machine learning method to train models on distributed datasets with privacy concerns. To properly incentivize data owners to contribute their efforts, Shapley Value (SV) is often adopted to fairly assess their contribution. However, the calculation of SV is time-consuming and computationally costly. In this paper, we propose FedCoin, a blockcha…
▽ More
Federated learning (FL) is an emerging collaborative machine learning method to train models on distributed datasets with privacy concerns. To properly incentivize data owners to contribute their efforts, Shapley Value (SV) is often adopted to fairly assess their contribution. However, the calculation of SV is time-consuming and computationally costly. In this paper, we propose FedCoin, a blockchain-based peer-to-peer payment system for FL to enable a feasible SV based profit distribution. In FedCoin, blockchain consensus entities calculate SVs and a new block is created based on the proof of Shapley (PoSap) protocol. It is in contrast to the popular BitCoin network where consensus entities "mine" new blocks by solving meaningless puzzles. Based on the computed SVs, a scheme for dividing the incentive payoffs among FL clients with nonrepudiation and tamper-resistance properties is proposed. Experimental results based on real-world data show that FedCoin can promote high-quality data from FL clients through accurately computing SVs with an upper bound on the computational resources required for reaching consensus. It opens opportunities for non-data owners to play a role in FL.
△ Less
Submitted 25 February, 2020;
originally announced February 2020.
-
User interface design for military AR applications
Authors:
Mark A. Livingston,
Zhuming Ai,
Kevin Karsch,
Gregory O. Gibson
Abstract:
Designing a user interface for military situation awareness presents challenges for managing information in a useful and usable manner. We present an integrated set of functions for the presentation of and interaction with information for a mobile augmented reality application for military applications. Our research has concentrated on four areas. We filter information based on relevance to the us…
▽ More
Designing a user interface for military situation awareness presents challenges for managing information in a useful and usable manner. We present an integrated set of functions for the presentation of and interaction with information for a mobile augmented reality application for military applications. Our research has concentrated on four areas. We filter information based on relevance to the user (in turn based on location), evaluate methods for presenting information that represents entities occluded from the user's view, enable interaction through a top-down map view metaphor akin to current techniques used in the military, and facilitate collaboration with other mobile users and/or a command center. In addition, we refined the user interface architecture to conform to requirements from subject matter experts. We discuss the lessons learned in our work and directions for future research.
△ Less
Submitted 20 April, 2019;
originally announced April 2019.
-
Boundary Evolution Algorithm for SAT-NP
Authors:
Zhaoyang Ai,
Chaodong Fan,
Yingjie Zhang,
Huigui Rong,
Ze'an Tian,
Haibing Fu
Abstract:
A boundary evolution Algorithm (BEA) is proposed by simultaneously taking into account the bottom and the high-level crossover and mutation, ie., the boundary of the hierarchical genetic algorithm. Operators and optimal individuals based on optional annealing are designed. Based on the numerous versions of genetic algorithm, the boundary evolution approach with crossover and mutation has been test…
▽ More
A boundary evolution Algorithm (BEA) is proposed by simultaneously taking into account the bottom and the high-level crossover and mutation, ie., the boundary of the hierarchical genetic algorithm. Operators and optimal individuals based on optional annealing are designed. Based on the numerous versions of genetic algorithm, the boundary evolution approach with crossover and mutation has been tested on the SAT problem and compared with two competing methods: a traditional genetic algorithm and another traditional hierarchical genetic algorithm, and among some others. The results of the comparative experiments in solving SAT problem have proved that the new hierarchical genetic algorithm based on simulated annealing and optimal individuals (BEA) can improve the success rate and convergence speed considerably for SAT problem due to its avoidance of both divergence and loss of optimal individuals, and by coronary, conducive to NP problem. Though more extensive comparisons are to be made on more algorithms, the consideration of the boundary elasticity of hierarchical genetic algorithm is an implication of evolutionary algorithm.
△ Less
Submitted 22 December, 2018;
originally announced March 2019.
