-
One-Shot Learning Meets Depth Diffusion in Multi-Object Videos
Authors:
Anisha Jain
Abstract:
Creating editable videos that depict complex interactions between multiple objects in various artistic styles has long been a challenging task in filmmaking. Progress is often hampered by the scarcity of data sets that contain paired text descriptions and corresponding videos that showcase these interactions. This paper introduces a novel depth-conditioning approach that significantly advances thi…
▽ More
Creating editable videos that depict complex interactions between multiple objects in various artistic styles has long been a challenging task in filmmaking. Progress is often hampered by the scarcity of data sets that contain paired text descriptions and corresponding videos that showcase these interactions. This paper introduces a novel depth-conditioning approach that significantly advances this field by enabling the generation of coherent and diverse videos from just a single text-video pair using a pre-trained depth-aware Text-to-Image (T2I) model. Our method fine-tunes the pre-trained model to capture continuous motion by employing custom-designed spatial and temporal attention mechanisms. During inference, we use the DDIM inversion to provide structural guidance for video generation. This innovative technique allows for continuously controllable depth in videos, facilitating the generation of multiobject interactions while maintaining the concept generation and compositional strengths of the original T2I model across various artistic styles, such as photorealism, animation, and impressionism.
△ Less
Submitted 29 August, 2024;
originally announced August 2024.
-
DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving
Authors:
Yongjie Fu,
Anmol Jain,
Xuan Di,
Xu Chen,
Zhaobin Mo
Abstract:
The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achie…
▽ More
The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achieve this, we employ a video generation framework grounded in denoising diffusion probabilistic models (DDPM) aimed at predicting real-world video sequences. We then explore the adequacy of our generated videos for use in VLMs by employing a pre-trained model known as Efficient In-context Learning on Egocentric Videos (EILEV). The diffusion model is trained with the Waymo open dataset and evaluated using the Fréchet Video Distance (FVD) score to ensure the quality and realism of the generated videos. Corresponding narrations are provided by EILEV for these generated videos, which may be beneficial in the autonomous driving domain. These narrations can enhance traffic scene understanding, aid in navigation, and improve planning capabilities. The integration of video generation with VLMs in the DriveGenVLM framework represents a significant step forward in leveraging advanced AI models to address complex challenges in autonomous driving.
△ Less
Submitted 29 August, 2024;
originally announced August 2024.
-
Rooftop and Community Solar Adoption with Income Heterogeneity
Authors:
Swapnil Rayal,
Apurva Jain,
Matthew Lorig
Abstract:
Each household in a population characterized by income heterogeneity faces random demand for electricity and decides if and when it should adopt a solar product, rooftop solar or community solar. A central planner, aiming to meet an adoption level target within a set time, offers net metering and subsidy on solar products and minimizes its total cost. Our focus is on analyzing the interactions of…
▽ More
Each household in a population characterized by income heterogeneity faces random demand for electricity and decides if and when it should adopt a solar product, rooftop solar or community solar. A central planner, aiming to meet an adoption level target within a set time, offers net metering and subsidy on solar products and minimizes its total cost. Our focus is on analyzing the interactions of three new features we add to the literature: income diversity, availability of community solar, and consideration of adoption timing. {Methodology and results:} We develop a bilevel optimization formulation to derive the optimal subsidy policy. The upper level (planner's) problem is a constrained non-linear optimization model in which the planner aims to minimize the average subsidy cost. The lower level (household's) problem is an optimal stopping formulation, which captures the adoption decisions of the households. We derive a closed-form expression for the distribution of optimal adoption time of households for a given subsidy policy. We show that the planner's problem is convex in the case of homogeneous subsidy for the two products. {Managerial implications:} Our results underscore the importance for planners to consider three factors - adoption level target, time target, and subsidy budget - simultaneously as they work in tandem to influence the adoption outcome. The planners must also consider the inclusion of community solar in their plans because, as we show, community and rooftop solar attract households from different sides of the income spectrum. In the presence of income inequality, the availability of community makes it easier to meet solar adoption targets.
△ Less
Submitted 21 August, 2024;
originally announced August 2024.
-
On Missing Scores in Evolving Multibiometric Systems
Authors:
Melissa R Dale,
Anil Jain,
Arun Ross
Abstract:
The use of multiple modalities (e.g., face and fingerprint) or multiple algorithms (e.g., three face comparators) has shown to improve the recognition accuracy of an operational biometric system. Over time a biometric system may evolve to add new modalities, retire old modalities, or be merged with other biometric systems. This can lead to scenarios where there are missing scores corresponding to…
▽ More
The use of multiple modalities (e.g., face and fingerprint) or multiple algorithms (e.g., three face comparators) has shown to improve the recognition accuracy of an operational biometric system. Over time a biometric system may evolve to add new modalities, retire old modalities, or be merged with other biometric systems. This can lead to scenarios where there are missing scores corresponding to the input probe set. Previous work on this topic has focused on either the verification or identification tasks, but not both. Further, the proportion of missing data considered has been less than 50%. In this work, we study the impact of missing score data for both the verification and identification tasks. We show that the application of various score imputation methods along with simple sum fusion can improve recognition accuracy, even when the proportion of missing scores increases to 90%. Experiments show that fusion after score imputation outperforms fusion with no imputation. Specifically, iterative imputation with K nearest neighbors consistently surpasses other imputation methods in both the verification and identification tasks, regardless of the amount of scores missing, and provides imputed values that are consistent with the ground truth complete dataset.
△ Less
Submitted 20 August, 2024;
originally announced August 2024.
-
A Comparison of Imitation Learning Algorithms for Bimanual Manipulation
Authors:
Michael Drolet,
Simon Stepputtis,
Siva Kailas,
Ajinkya Jain,
Jan Peters,
Stefan Schaal,
Heni Ben Amor
Abstract:
Amidst the wide popularity of imitation learning algorithms in robotics, their properties regarding hyperparameter sensitivity, ease of training, data efficiency, and performance have not been well-studied in high-precision industry-inspired environments. In this work, we demonstrate the limitations and benefits of prominent imitation learning approaches and analyze their capabilities regarding th…
▽ More
Amidst the wide popularity of imitation learning algorithms in robotics, their properties regarding hyperparameter sensitivity, ease of training, data efficiency, and performance have not been well-studied in high-precision industry-inspired environments. In this work, we demonstrate the limitations and benefits of prominent imitation learning approaches and analyze their capabilities regarding these properties. We evaluate each algorithm on a complex bimanual manipulation task involving an over-constrained dynamics system in a setting involving multiple contacts between the manipulated object and the environment. While we find that imitation learning is well suited to solve such complex tasks, not all algorithms are equal in terms of handling environmental and hyperparameter perturbations, training requirements, performance, and ease of use. We investigate the empirical influence of these key characteristics by employing a carefully designed experimental procedure and learning environment. Paper website: https://bimanual-imitation.github.io/
△ Less
Submitted 24 August, 2024; v1 submitted 12 August, 2024;
originally announced August 2024.
-
CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
Authors:
Kushal Kumar Jain,
Steve Grosz,
Anoop M. Namboodiri,
Anil K. Jain
Abstract:
Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recogni…
▽ More
Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.
△ Less
Submitted 13 August, 2024; v1 submitted 2 August, 2024;
originally announced August 2024.
-
The Llama 3 Herd of Models
Authors:
Abhimanyu Dubey,
Abhinav Jauhri,
Abhinav Pandey,
Abhishek Kadian,
Ahmad Al-Dahle,
Aiesha Letman,
Akhil Mathur,
Alan Schelten,
Amy Yang,
Angela Fan,
Anirudh Goyal,
Anthony Hartshorn,
Aobo Yang,
Archi Mitra,
Archie Sravankumar,
Artem Korenev,
Arthur Hinsvark,
Arun Rao,
Aston Zhang,
Aurelien Rodriguez,
Austen Gregerson,
Ava Spataru,
Baptiste Roziere,
Bethany Biron,
Binh Tang
, et al. (510 additional authors not shown)
Abstract:
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical…
▽ More
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.
△ Less
Submitted 15 August, 2024; v1 submitted 31 July, 2024;
originally announced July 2024.
-
Apple Intelligence Foundation Language Models
Authors:
Tom Gunter,
Zirui Wang,
Chong Wang,
Ruoming Pang,
Andy Narayanan,
Aonan Zhang,
Bowen Zhang,
Chen Chen,
Chung-Cheng Chiu,
David Qiu,
Deepak Gopinath,
Dian Ang Yap,
Dong Yin,
Feng Nan,
Floris Weers,
Guoli Yin,
Haoshuo Huang,
Jianyu Wang,
Jiarui Lu,
John Peebles,
Ke Ye,
Mark Lee,
Nan Du,
Qibin Chen,
Quentin Keunebroek
, et al. (130 additional authors not shown)
Abstract:
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute. These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly. This report describes the model architecture, the data used…
▽ More
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large server-based language model designed for Private Cloud Compute. These models are designed to perform a wide range of tasks efficiently, accurately, and responsibly. This report describes the model architecture, the data used to train the model, the training process, how the models are optimized for inference, and the evaluation results. We highlight our focus on Responsible AI and how the principles are applied throughout the model development.
△ Less
Submitted 29 July, 2024;
originally announced July 2024.
-
Knowledge Models for Cancer Clinical Practice Guidelines : Construction, Management and Usage in Question Answering
Authors:
Pralaypati Ta,
Bhumika Gupta,
Arihant Jain,
Sneha Sree C,
Keerthi Ram,
Mohanasankar Sivaprakasam
Abstract:
An automated knowledge modeling algorithm for Cancer Clinical Practice Guidelines (CPGs) extracts the knowledge contained in the CPG documents and transforms it into a programmatically interactable, easy-to-update structured model with minimal human intervention. The existing automated algorithms have minimal scope and cannot handle the varying complexity of the knowledge content in the CPGs for d…
▽ More
An automated knowledge modeling algorithm for Cancer Clinical Practice Guidelines (CPGs) extracts the knowledge contained in the CPG documents and transforms it into a programmatically interactable, easy-to-update structured model with minimal human intervention. The existing automated algorithms have minimal scope and cannot handle the varying complexity of the knowledge content in the CPGs for different cancer types. This work proposes an improved automated knowledge modeling algorithm to create knowledge models from the National Comprehensive Cancer Network (NCCN) CPGs in Oncology for different cancer types. The proposed algorithm has been evaluated with NCCN CPGs for four different cancer types. We also proposed an algorithm to compare the knowledge models for different versions of a guideline to discover the specific changes introduced in the treatment protocol of a new version. We created a question-answering (Q&A) framework with the guideline knowledge models as the augmented knowledge base to study our ability to query the knowledge models. We compiled a set of 32 question-answer pairs derived from two reliable data sources for the treatment of Non-Small Cell Lung Cancer (NSCLC) to evaluate the Q&A framework. The framework was evaluated against the question-answer pairs from one data source, and it can generate the answers with 54.5% accuracy from the treatment algorithm and 81.8% accuracy from the discussion part of the NCCN NSCLC guideline knowledge model.
△ Less
Submitted 23 July, 2024;
originally announced July 2024.
-
Application State Management (ASM) in the Modern Web and Mobile Applications: A Comprehensive Review
Authors:
Anujkumarsinh Donvir,
Apeksha Jain,
Pradeep Kumar Saraswathi
Abstract:
The rapid evolution of web and mobile applications has necessitated robust mechanisms for managing application state to ensure consistency, performance, and user-friendliness. This comprehensive review examines the most effective Application State Management (ASM) techniques, categorized into Local State Management, State Management Libraries, and Server-Side State Management. By analyzing popular…
▽ More
The rapid evolution of web and mobile applications has necessitated robust mechanisms for managing application state to ensure consistency, performance, and user-friendliness. This comprehensive review examines the most effective Application State Management (ASM) techniques, categorized into Local State Management, State Management Libraries, and Server-Side State Management. By analyzing popular front end frameworks the study delves into local state management mechanisms. It also evaluates the state of front end management libraries, highlighting their implementations, benefits, and limitations. Server-side state management techniques, particularly caching, are discussed for their roles in enhancing data retrieval efficiency. This paper offers actionable insights for developers to build scalable, responsive applications, aiming to bridge the gap between theoretical knowledge and practical application. This study's critical analysis and recommendations aim to guide future research and development in ASM, contributing to the advancement of modern application architecture.
△ Less
Submitted 27 July, 2024;
originally announced July 2024.
-
Ultrahigh pyroelectricity in monoelemental 2D tellurium
Authors:
Hari Krishna Mishra,
Ayushi Jain,
Dalip Saini,
Bidya Mondal,
Chandan Bera,
Shanker Ram,
Dipankar Mandal
Abstract:
We report an ultrahigh pyroelectric response in van der Waals bonded layers of two-dimensional (2D) tellurium (Te) nanosheets (thickness, d = 4 to 5 nm) at periodic on-off temperature oscillations. For the first time a large pyroelectric coefficient, Pc ~ 3 mC.m-2.K-1, is observed which is eightfold higher than the traditional state-of-the-art pyroelectrics (lead zirconate titanate, PZT). The firs…
▽ More
We report an ultrahigh pyroelectric response in van der Waals bonded layers of two-dimensional (2D) tellurium (Te) nanosheets (thickness, d = 4 to 5 nm) at periodic on-off temperature oscillations. For the first time a large pyroelectric coefficient, Pc ~ 3 mC.m-2.K-1, is observed which is eightfold higher than the traditional state-of-the-art pyroelectrics (lead zirconate titanate, PZT). The first-principles calculations point out that the breakdown of centro-symmetry in the 1-3 Te-layers (P-3m1 space group) of a non-centrosymmetry (higher-order symmetry of C2 space group) on an angular twist in the Te-Te bonds of an exotic electronic state in 2D Te. The angular Te-Te twisting elicits a surface-enhanced Raman band at 101 cm-1 (absent in bulk Te). The stimulation of the Born effective charge, in-plane piezoelectricity and thermal expansion coefficient are shown to tailor the large pyroelectricity. Thus, 2D Te nanosheets present a new paradigm for the wide application of pyroelectric materials for developing thermal energy-based flexible electronics.
△ Less
Submitted 26 July, 2024;
originally announced July 2024.
-
Open-Set Biometrics: Beyond Good Closed-Set Models
Authors:
Yiyang Su,
Minchul Kim,
Feng Liu,
Anil Jain,
Xiaoming Liu
Abstract:
Biometric recognition has primarily addressed closed-set identification, assuming all probe subjects are in the gallery. However, most practical applications involve open-set biometrics, where probe subjects may or may not be present in the gallery. This poses distinct challenges in effectively distinguishing individuals in the gallery while minimizing false detections. While it is commonly believ…
▽ More
Biometric recognition has primarily addressed closed-set identification, assuming all probe subjects are in the gallery. However, most practical applications involve open-set biometrics, where probe subjects may or may not be present in the gallery. This poses distinct challenges in effectively distinguishing individuals in the gallery while minimizing false detections. While it is commonly believed that powerful biometric models can excel in both closed- and open-set scenarios, existing loss functions are inconsistent with open-set evaluation. They treat genuine (mated) and imposter (non-mated) similarity scores symmetrically and neglect the relative magnitudes of imposter scores. To address these issues, we simulate open-set evaluation using minibatches during training and introduce novel loss functions: (1) the identification-detection loss optimized for open-set performance under selective thresholds and (2) relative threshold minimization to reduce the maximum negative score for each probe. Across diverse biometric tasks, including face recognition, gait recognition, and person re-identification, our experiments demonstrate the effectiveness of the proposed loss functions, significantly enhancing open-set performance while positively impacting closed-set performance. Our code and models are available at https://github.com/prevso1088/open-set-biometrics.
△ Less
Submitted 22 July, 2024;
originally announced July 2024.
-
An explicit solution to the spinning ring problem
Authors:
Aradhya Jain
Abstract:
A ring may be regarded as a torus with r << R, where R is the major radius and r is the minor radius. When such a ring is placed on a rough rod and released with some angular velocity, it may continue to vertically spin around the rod for some time instead of falling down immediately. It was observed that two different kinds of motion for the rod exist, which are referred to as single point and do…
▽ More
A ring may be regarded as a torus with r << R, where R is the major radius and r is the minor radius. When such a ring is placed on a rough rod and released with some angular velocity, it may continue to vertically spin around the rod for some time instead of falling down immediately. It was observed that two different kinds of motion for the rod exist, which are referred to as single point and double point contact motion, based on the number of contact points of the ring that are in contact with the rod. Single point contact motion was observed for rings and double point contact motion was observed in the case of a washer. We investigated the characteristics of the single point contact motion. An explanation is provided for the single point contact motion of the ring and an analysis of the forces on the ring is made. We observe a hyperbolic decay in the angular velocity of the ring. An explicit solution is determined for the single point contact motion of the ring and the obtained solutions match the observed motion of the ring.
△ Less
Submitted 16 July, 2024;
originally announced July 2024.
-
Weight Block Sparsity: Training, Compilation, and AI Engine Accelerators
Authors:
Paolo D'Alberto,
Taehee Jeong,
Akshai Jain,
Shreyas Manjunath,
Mrinal Sarmah,
Samuel Hsu,
Yaswanth Raparti,
Nitesh Pipralia
Abstract:
Nowadays, increasingly larger Deep Neural Networks (DNNs) are being developed, trained, and utilized. These networks require significant computational resources, putting a strain on both advanced and limited devices. Our solution is to implement {\em weight block sparsity}, which is a structured sparsity that is friendly to hardware. By zeroing certain sections of the convolution and fully connect…
▽ More
Nowadays, increasingly larger Deep Neural Networks (DNNs) are being developed, trained, and utilized. These networks require significant computational resources, putting a strain on both advanced and limited devices. Our solution is to implement {\em weight block sparsity}, which is a structured sparsity that is friendly to hardware. By zeroing certain sections of the convolution and fully connected layers parameters of pre-trained DNN models, we can efficiently speed up the DNN's inference process. This results in a smaller memory footprint, faster communication, and fewer operations.
Our work presents a vertical system that allows for the training of convolution and matrix multiplication weights to exploit 8x8 block sparsity on a single GPU within a reasonable amount of time. Compilers recognize this sparsity and use it for both data compaction and computation splitting into threads. Blocks like these take full advantage of both spatial and temporal locality, paving the way for fast vector operations and memory reuse. By using this system on a Resnet50 model, we were able to reduce the weight by half with minimal accuracy loss, resulting in a two-times faster inference speed. We will present performance estimates using accurate and complete code generation for AIE2 configuration sets (AMD Versal FPGAs) with Resnet50, Inception V3, and VGG16 to demonstrate the necessary synergy between hardware overlay designs and software stacks for compiling and executing machine learning applications.
△ Less
Submitted 12 July, 2024;
originally announced July 2024.
-
Calibrated Forecasting and Persuasion
Authors:
Atulya Jain,
Vianney Perchet
Abstract:
How should an expert send forecasts to maximize her utility subject to passing a calibration test? We consider a dynamic game where an expert sends probabilistic forecasts to a decision maker. The decision maker uses a calibration test based on past outcomes to verify the expert's forecasts. We characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion proble…
▽ More
How should an expert send forecasts to maximize her utility subject to passing a calibration test? We consider a dynamic game where an expert sends probabilistic forecasts to a decision maker. The decision maker uses a calibration test based on past outcomes to verify the expert's forecasts. We characterize the optimal forecasting strategy by reducing the dynamic game to a static persuasion problem. A distribution of forecasts is implementable by a calibrated strategy if and only if it is a mean-preserving contraction of the distribution of conditionals (honest forecasts). We characterize the value of information by comparing what an informed and uninformed expert can attain. Moreover, we consider a decision maker who uses regret minimization, instead of the calibration test, to take actions. We show that the expert can achieve the same payoff against a regret minimizer as under the calibration test, and in some instances, she can achieve strictly more.
△ Less
Submitted 21 June, 2024;
originally announced June 2024.
-
A machine learning pipeline for automated insect monitoring
Authors:
Aditya Jain,
Fagner Cunha,
Michael Bunsen,
Léonard Pasi,
Anna Viklund,
Maxim Larrivée,
David Rolnick
Abstract:
Climate change and other anthropogenic factors have led to a catastrophic decline in insects, endangering both biodiversity and the ecosystem services on which human society depends. Data on insect abundance, however, remains woefully inadequate. Camera traps, conventionally used for monitoring terrestrial vertebrates, are now being modified for insects, especially moths. We describe a complete, o…
▽ More
Climate change and other anthropogenic factors have led to a catastrophic decline in insects, endangering both biodiversity and the ecosystem services on which human society depends. Data on insect abundance, however, remains woefully inadequate. Camera traps, conventionally used for monitoring terrestrial vertebrates, are now being modified for insects, especially moths. We describe a complete, open-source machine learning-based software pipeline for automated monitoring of moths via camera traps, including object detection, moth/non-moth classification, fine-grained identification of moth species, and tracking individuals. We believe that our tools, which are already in use across three continents, represent the future of massively scalable data collection in entomology.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Insect Identification in the Wild: The AMI Dataset
Authors:
Aditya Jain,
Fagner Cunha,
Michael James Bunsen,
Juan Sebastián Cañas,
Léonard Pasi,
Nathan Pinoy,
Flemming Helsing,
JoAnne Russo,
Marc Botham,
Michael Sabourin,
Jonathan Fréchette,
Alexandre Anctil,
Yacksecari Lopez,
Eduardo Navarro,
Filonila Perez Pimentel,
Ana Cecilia Zamora,
José Alejandro Ramirez Silva,
Jonathan Gagnon,
Tom August,
Kim Bjerge,
Alba Gomez Segura,
Marc Bélisle,
Yves Basset,
Kent P. McFarland,
David Roy
, et al. (3 additional authors not shown)
Abstract:
Insects represent half of all global biodiversity, yet many of the world's insects are disappearing, with severe implications for ecosystems and agriculture. Despite this crisis, data on insect diversity and abundance remain woefully inadequate, due to the scarcity of human experts and the lack of scalable tools for monitoring. Ecologists have started to adopt camera traps to record and study inse…
▽ More
Insects represent half of all global biodiversity, yet many of the world's insects are disappearing, with severe implications for ecosystems and agriculture. Despite this crisis, data on insect diversity and abundance remain woefully inadequate, due to the scarcity of human experts and the lack of scalable tools for monitoring. Ecologists have started to adopt camera traps to record and study insects, and have proposed computer vision algorithms as an answer for scalable data processing. However, insect monitoring in the wild poses unique challenges that have not yet been addressed within computer vision, including the combination of long-tailed data, extremely similar classes, and significant distribution shifts. We provide the first large-scale machine learning benchmarks for fine-grained insect recognition, designed to match real-world tasks faced by ecologists. Our contributions include a curated dataset of images from citizen science platforms and museums, and an expert-annotated dataset drawn from automated camera traps across multiple continents, designed to test out-of-distribution generalization under field conditions. We train and evaluate a variety of baseline algorithms and introduce a combination of data augmentation techniques that enhance generalization across geographies and hardware setups. Code and datasets are made publicly available.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
ESI-GAL: EEG Source Imaging-based Kinematics Parameter Estimation for Grasp and Lift Task
Authors:
Anant Jain,
Lalan Kumar
Abstract:
Electroencephalogram (EEG) signals-based motor kinematics prediction (MKP) has been an active area of research to develop brain-computer interface (BCI) systems such as exosuits, prostheses, and rehabilitation devices. However, EEG source imaging (ESI) based kinematics prediction is sparsely explored in the literature. In this study, pre-movement EEG features are utilized to predict three-dimensio…
▽ More
Electroencephalogram (EEG) signals-based motor kinematics prediction (MKP) has been an active area of research to develop brain-computer interface (BCI) systems such as exosuits, prostheses, and rehabilitation devices. However, EEG source imaging (ESI) based kinematics prediction is sparsely explored in the literature. In this study, pre-movement EEG features are utilized to predict three-dimensional (3D) hand kinematics for the grasp-and-lift motor task. A public dataset, WAY-EEG-GAL, is utilized for MKP analysis. In particular, sensor-domain (EEG data) and source-domain (ESI data) based features from the frontoparietal region are explored for MKP. Deep learning-based models are explored to achieve efficient kinematics decoding. Various time-lagged and window sizes are analyzed for hand kinematics prediction. Subsequently, intra-subject and inter-subject MKP analysis is performed to investigate the subject-specific and subject-independent motor-learning capabilities of the neural decoders. The Pearson correlation coefficient (PCC) is used as the performance metric for kinematics trajectory decoding. The rEEGNet neural decoder achieved the best performance with sensor-domain and source-domain features with a time lag and window size of 100 ms and 450 ms, respectively. The highest mean PCC values of 0.790, 0.795, and 0.637 are achieved using sensor-domain features, while 0.769, 0.777, and 0.647 are achieved using source-domain features in x, y, and z-directions, respectively. This study explores the feasibility of trajectory prediction using EEG sensor-domain and source-domain EEG features for the grasp-and-lift task. Furthermore, inter-subject trajectory estimation is performed using the proposed deep learning decoder with EEG source domain features.
△ Less
Submitted 4 July, 2024; v1 submitted 17 June, 2024;
originally announced June 2024.
-
Gate voltage modulation of the superconducting state in a degenerate semiconductor
Authors:
Bikash C. Barik,
Himadri Chakraborti,
Buddhadeb Pal,
Aditya K. Jain,
Swagata Bhunia,
Sounak Samanta,
Apurba Laha,
Suddhasatta Mahapatra,
K. Das Gupta
Abstract:
In this work, we demonstrate that the modulation of carrier density can alter the superconducting transition temperature by up to $204$ mK in epitaxial Indium Nitride on Gallium Nitride, accounting for the $10$% of the transition temperature in ungated conditions. Our samples are likely free from strong localization effects and significant granularity, as indicated by $( k_{f l} \gg 1 )$, suggesti…
▽ More
In this work, we demonstrate that the modulation of carrier density can alter the superconducting transition temperature by up to $204$ mK in epitaxial Indium Nitride on Gallium Nitride, accounting for the $10$% of the transition temperature in ungated conditions. Our samples are likely free from strong localization effects and significant granularity, as indicated by $( k_{f l} \gg 1 )$, suggesting that the primary determinant of the transition temperature in InN is carrier density, rather than disorder scattering. The observed behavior is consistent with BCS s-wave superconductivity, corroborated by the superconducting parameters we measured. Furthermore, we observed a $60$% bipolar suppression of the supercurrent in our experiments.
△ Less
Submitted 13 June, 2024;
originally announced June 2024.
-
Primordial magnetic relics and their signatures
Authors:
Arka Banerjee,
Lalit Singh Bhandari,
Ashwat Jain,
Arun M. Thalapillil
Abstract:
Primordial black holes bearing magnetic charges may bypass the constraints imposed by Hawking radiation, thereby enabling reasonable present-day populations, even for masses below $10^{15}\,\text{g}$ -- a range previously considered improbable. They could, therefore, conceivably contribute to a component of dark matter. We investigate novel Faraday rotation signatures exhibited by primordial magne…
▽ More
Primordial black holes bearing magnetic charges may bypass the constraints imposed by Hawking radiation, thereby enabling reasonable present-day populations, even for masses below $10^{15}\,\text{g}$ -- a range previously considered improbable. They could, therefore, conceivably contribute to a component of dark matter. We investigate novel Faraday rotation signatures exhibited by primordial magnetic black holes while also establishing new Parker-type bounds on their populations. For the latter, we bound the dark matter fraction from intergalactic magnetic fields in cosmic voids $\left(f_{\text{DM}} \lesssim 10^{-8}\right)$ and cosmic web filaments $\left(f_{\text{ DM}} \lesssim 10^{-4}\right)$, notably eclipsing previous bounds. Exploring Faraday rotation effects, we discern a pronounced rotation of the polarization angle and the rotation measure values for extremal primordial magnetic black holes with masses $M^{\text{ ex.}}_{\text{ BH}}\gtrsim 10^{-6}~ \text{M}_\odot$. This makes them potentially detectable in current observations. A comparative investigation finds that the effects are notably greater than for a neutron star, like a Magnetar, with a similar magnetic field at the surface. Moreover, the polarization angle maps for primordial magnetic black holes exhibit unique features, notably absent in other astrophysical magnetic configurations. In this context, we also introduce a simple integral measure, offering a quantitative measure for their discrimination in many scenarios. These traits potentially suggest a robust avenue for their observational detection and differentiation.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
Authors:
Benjamin Biggs,
Arjun Seshadri,
Yang Zou,
Achin Jain,
Aditya Golatkar,
Yusheng Xie,
Alessandro Achille,
Ashwin Swaminathan,
Stefano Soatto
Abstract:
We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data. By construction, our approach enables training-free continual learning and unlearning with no additional memory or inference costs, since models corresponding to data shards can be added or removed by re-averaging. We show that Diffusion Soup…
▽ More
We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data. By construction, our approach enables training-free continual learning and unlearning with no additional memory or inference costs, since models corresponding to data shards can be added or removed by re-averaging. We show that Diffusion Soup samples from a point in weight space that approximates the geometric mean of the distributions of constituent datasets, which offers anti-memorization guarantees and enables zero-shot style mixing. Empirically, Diffusion Soup outperforms a paragon model trained on the union of all data shards and achieves a 30% improvement in Image Reward (.34 $\to$ .44) on domain sharded data, and a 59% improvement in IR (.37 $\to$ .59) on aesthetic data. In both cases, souping also prevails in TIFA score (respectively, 85.5 $\to$ 86.5 and 85.6 $\to$ 86.8). We demonstrate robust unlearning -- removing any individual domain shard only lowers performance by 1% in IR (.45 $\to$ .44) -- and validate our theoretical insights on anti-memorization using real data. Finally, we showcase Diffusion Soup's ability to blend the distinct styles of models finetuned on different shards, resulting in the zero-shot generation of hybrid styles.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Fractonic solids
Authors:
Akash Jain
Abstract:
Fractons are exotic quasiparticles whose mobility in space is restricted by symmetries. In potential real-world realisations, fractons are likely lodged to a physical material rather than absolute space. Motivated by this, we propose and explore a new symmetry principle that restricts the motion of fractons relative to a physical solid. Unlike models with restricted mobility in absolute space, the…
▽ More
Fractons are exotic quasiparticles whose mobility in space is restricted by symmetries. In potential real-world realisations, fractons are likely lodged to a physical material rather than absolute space. Motivated by this, we propose and explore a new symmetry principle that restricts the motion of fractons relative to a physical solid. Unlike models with restricted mobility in absolute space, these fractonic solids admit gauge-invariant momentum density, are compatible with boost symmetry, and can consistently be coupled to gravity. We also propose a holographic model for fractonic solids.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Fast White-Box Adversarial Streaming Without a Random Oracle
Authors:
Ying Feng,
Aayush Jain,
David P. Woodruff
Abstract:
Recently, the question of adversarially robust streaming, where the stream is allowed to depend on the randomness of the streaming algorithm, has gained a lot of attention. In this work, we consider a strong white-box adversarial model (Ajtai et al. PODS 2022), in which the adversary has access to all past random coins and the parameters used by the streaming algorithm. We focus on the sparse reco…
▽ More
Recently, the question of adversarially robust streaming, where the stream is allowed to depend on the randomness of the streaming algorithm, has gained a lot of attention. In this work, we consider a strong white-box adversarial model (Ajtai et al. PODS 2022), in which the adversary has access to all past random coins and the parameters used by the streaming algorithm. We focus on the sparse recovery problem and extend our result to other tasks such as distinct element estimation and low-rank approximation of matrices and tensors. The main drawback of previous work is that it requires a random oracle, which is especially problematic in the streaming model since the amount of randomness is counted in the space complexity of a streaming algorithm. Also, the previous work suffers from large update time. We construct a near-optimal solution for the sparse recovery problem in white-box adversarial streams, based on the subexponentially secure Learning with Errors assumption. Importantly, our solution does not require a random oracle and has a polylogarithmic per item processing time. We also give results in a related white-box adversarially robust distributed model. Our constructions are based on homomorphic encryption schemes satisfying very mild structural properties that are currently satisfied by most known schemes.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
GenPalm: Contactless Palmprint Generation with Diffusion Models
Authors:
Steven A. Grosz,
Anil K. Jain
Abstract:
The scarcity of large-scale palmprint databases poses a significant bottleneck to advancements in contactless palmprint recognition. To address this, researchers have turned to synthetic data generation. While Generative Adversarial Networks (GANs) have been widely used, they suffer from instability and mode collapse. Recently, diffusion probabilistic models have emerged as a promising alternative…
▽ More
The scarcity of large-scale palmprint databases poses a significant bottleneck to advancements in contactless palmprint recognition. To address this, researchers have turned to synthetic data generation. While Generative Adversarial Networks (GANs) have been widely used, they suffer from instability and mode collapse. Recently, diffusion probabilistic models have emerged as a promising alternative, offering stable training and better distribution coverage. This paper introduces a novel palmprint generation method using diffusion probabilistic models, develops an end-to-end framework for synthesizing multiple palm identities, and validates the realism and utility of the generated palmprints. Experimental results demonstrate the effectiveness of our approach in generating palmprint images which enhance contactless palmprint recognition performance across several test databases utilizing challenging cross-database and time-separated evaluation protocols.
△ Less
Submitted 31 May, 2024;
originally announced June 2024.
-
A Comparative Study of CNN, ResNet, and Vision Transformers for Multi-Classification of Chest Diseases
Authors:
Ananya Jain,
Aviral Bhardwaj,
Kaushik Murali,
Isha Surani
Abstract:
Large language models, notably utilizing Transformer architectures, have emerged as powerful tools due to their scalability and ability to process large amounts of data. Dosovitskiy et al. expanded this architecture to introduce Vision Transformers (ViT), extending its applicability to image processing tasks. Motivated by this advancement, we fine-tuned two variants of ViT models, one pre-trained…
▽ More
Large language models, notably utilizing Transformer architectures, have emerged as powerful tools due to their scalability and ability to process large amounts of data. Dosovitskiy et al. expanded this architecture to introduce Vision Transformers (ViT), extending its applicability to image processing tasks. Motivated by this advancement, we fine-tuned two variants of ViT models, one pre-trained on ImageNet and another trained from scratch, using the NIH Chest X-ray dataset containing over 100,000 frontal-view X-ray images. Our study evaluates the performance of these models in the multi-label classification of 14 distinct diseases, while using Convolutional Neural Networks (CNNs) and ResNet architectures as baseline models for comparison. Through rigorous assessment based on accuracy metrics, we identify that the pre-trained ViT model surpasses CNNs and ResNet in this multilabel classification task, highlighting its potential for accurate diagnosis of various lung conditions from chest X-ray images.
△ Less
Submitted 31 May, 2024;
originally announced June 2024.
-
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Authors:
Anchit Jain,
Rozhin Nobahari,
Aristide Baratin,
Stefano Sarao Mannelli
Abstract:
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student s…
▽ More
Machine learning systems often acquire biases by leveraging undesired features in the data, impacting accuracy variably across different sub-populations. Current understanding of bias formation mostly focuses on the initial and final stages of learning, leaving a gap in knowledge regarding the transient dynamics. To address this gap, this paper explores the evolution of bias in a teacher-student setup modeling different data sub-populations with a Gaussian-mixture model. We provide an analytical description of the stochastic gradient descent dynamics of a linear classifier in this setting, which we prove to be exact in high dimension. Notably, our analysis reveals how different properties of sub-populations influence bias at different timescales, showing a shifting preference of the classifier during training. Applying our findings to fairness and robustness, we delineate how and when heterogeneous data and spurious features can generate and amplify bias. We empirically validate our results in more complex scenarios by training deeper networks on synthetic and real datasets, including CIFAR10, MNIST, and CelebA.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt Adaptation
Authors:
Abhinav Jain,
Swarat Chaudhuri,
Thomas Reps,
Chris Jermaine
Abstract:
Parameter-Efficient Fine-Tuning (PEFT) has become the standard for customising Foundation Models (FMs) to user-specific downstream tasks. However, typical PEFT methods require storing multiple task-specific adapters, creating scalability issues as these adapters must be housed and run at the FM server. Traditional prompt tuning offers a potential solution by customising them through task-specific…
▽ More
Parameter-Efficient Fine-Tuning (PEFT) has become the standard for customising Foundation Models (FMs) to user-specific downstream tasks. However, typical PEFT methods require storing multiple task-specific adapters, creating scalability issues as these adapters must be housed and run at the FM server. Traditional prompt tuning offers a potential solution by customising them through task-specific input prefixes, but it under-performs compared to other PEFT methods like LoRA. To address this gap, we propose Low-Rank Prompt Adaptation (LOPA), a prompt-tuning-based approach that performs on par with state-of-the-art PEFT methods and full fine-tuning while being more parameter-efficient and not requiring a server-based adapter. LOPA generates soft prompts by balancing between sharing task-specific information across instances and customization for each instance. It uses a low-rank decomposition of the soft-prompt component encoded for each instance to achieve parameter efficiency. We provide a comprehensive evaluation on multiple natural language understanding and code generation and understanding tasks across a wide range of foundation models with varying sizes.
△ Less
Submitted 24 May, 2024;
originally announced May 2024.
-
AlabOS: A Python-based Reconfigurable Workflow Management Framework for Autonomous Laboratories
Authors:
Yuxing Fei,
Bernardus Rendy,
Rishi Kumar,
Olympia Dartsi,
Hrushikesh P. Sahasrabuddhe,
Matthew J. McDermott,
Zheren Wang,
Nathan J. Szymanski,
Lauren N. Walters,
David Milsted,
Yan Zeng,
Anubhav Jain,
Gerbrand Ceder
Abstract:
The recent advent of autonomous laboratories, coupled with algorithms for high-throughput screening and active learning, promises to accelerate materials discovery and innovation. As these autonomous systems grow in complexity, the demand for robust and efficient workflow management software becomes increasingly critical. In this paper, we introduce AlabOS, a general-purpose software framework for…
▽ More
The recent advent of autonomous laboratories, coupled with algorithms for high-throughput screening and active learning, promises to accelerate materials discovery and innovation. As these autonomous systems grow in complexity, the demand for robust and efficient workflow management software becomes increasingly critical. In this paper, we introduce AlabOS, a general-purpose software framework for orchestrating experiments and managing resources, with an emphasis on automated laboratories for materials synthesis and characterization. We demonstrate the implementation of AlabOS in a prototype autonomous materials laboratory. AlabOS features a reconfigurable experiment workflow model, enabling the simultaneous execution of varied workflows composed of modular tasks. Therefore, AlabOS is well-suited to handle the rapidly changing experimental protocols defining the progress of self-driving laboratory development for materials research.
△ Less
Submitted 22 May, 2024;
originally announced May 2024.
-
Downlink Power Control based UE-Sided Initial Access for Tactical 5G NR
Authors:
Akshay Jain,
Karthik Upadhya,
Mikko A. Uusitalo,
Harish Viswanathan
Abstract:
Communication technologies play a crucial role in battlefields. They are an inalienable part of any tactical response, whether at the battlefront or inland. Such scenarios require that the communication technologies be versatile, scalable, cost-effective, and stealthy. While multiple studies and past products have tried to address these requirements, none of them have been able to solve all the fo…
▽ More
Communication technologies play a crucial role in battlefields. They are an inalienable part of any tactical response, whether at the battlefront or inland. Such scenarios require that the communication technologies be versatile, scalable, cost-effective, and stealthy. While multiple studies and past products have tried to address these requirements, none of them have been able to solve all the four challenges simultaneously. Hence, in this paper, we propose a tactical solution that is based on the versatile, scalable, and cost effective 5G NR system. Our focus is on the initial-access phase which is subject to a high probability of detection by an eavesdropper. To address this issue, we propose a novel approach that involves some modifications to the initial access procedure that lowers the probability of detection while not affecting standards compliance and not requiring any modifications to the user equipment chipset implementation. Further, we demonstrate that with a simple downlink power control algorithm, we reduce the probability of detection at an eavesdropper. The result is a 5G NR based initial-access method that improves stealthiness when compared with a vanilla 5G NR implementation.
△ Less
Submitted 14 June, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
-
SmartFlow: Robotic Process Automation using LLMs
Authors:
Arushi Jain,
Shubham Paliwal,
Monika Sharma,
Lovekesh Vig,
Gautam Shroff
Abstract:
Robotic Process Automation (RPA) systems face challenges in handling complex processes and diverse screen layouts that require advanced human-like decision-making capabilities. These systems typically rely on pixel-level encoding through drag-and-drop or automation frameworks such as Selenium to create navigation workflows, rather than visual understanding of screen elements. In this context, we p…
▽ More
Robotic Process Automation (RPA) systems face challenges in handling complex processes and diverse screen layouts that require advanced human-like decision-making capabilities. These systems typically rely on pixel-level encoding through drag-and-drop or automation frameworks such as Selenium to create navigation workflows, rather than visual understanding of screen elements. In this context, we present SmartFlow, an AI-based RPA system that uses pre-trained large language models (LLMs) coupled with deep-learning based image understanding. Our system can adapt to new scenarios, including changes in the user interface and variations in input data, without the need for human intervention. SmartFlow uses computer vision and natural language processing to perceive visible elements on the graphical user interface (GUI) and convert them into a textual representation. This information is then utilized by LLMs to generate a sequence of actions that are executed by a scripting engine to complete an assigned task. To assess the effectiveness of SmartFlow, we have developed a dataset that includes a set of generic enterprise applications with diverse layouts, which we are releasing for research use. Our evaluations on this dataset demonstrate that SmartFlow exhibits robustness across different layouts and applications. SmartFlow can automate a wide range of business processes such as form filling, customer service, invoice processing, and back-office operations. SmartFlow can thus assist organizations in enhancing productivity by automating an even larger fraction of screen-based workflows. The demo-video and dataset are available at https://smartflow-4c5a0a.webflow.io/.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
Multi-Subject Personalization
Authors:
Arushi Jain,
Shubham Paliwal,
Monika Sharma,
Vikram Jamwal,
Lovekesh Vig
Abstract:
Creative story illustration requires a consistent interplay of multiple characters or objects. However, conventional text-to-image models face significant challenges while producing images featuring multiple personalized subjects. For example, they distort the subject rendering, or the text descriptions fail to render coherent subject interactions. We present Multi-Subject Personalization (MSP) to…
▽ More
Creative story illustration requires a consistent interplay of multiple characters or objects. However, conventional text-to-image models face significant challenges while producing images featuring multiple personalized subjects. For example, they distort the subject rendering, or the text descriptions fail to render coherent subject interactions. We present Multi-Subject Personalization (MSP) to alleviate some of these challenges. We implement MSP using Stable Diffusion and assess our approach against other text-to-image models, showcasing its consistent generation of good-quality images representing intended subjects and interactions.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
LPD-Aware Uplink CSI-based 5G NR Downlink Synchronization for Tactical Networks
Authors:
Karthik Upadhya,
Akshay Jain,
Mikko A. Uusitalo,
Harish Viswanathan
Abstract:
5G NR is touted to be an attractive candidate for tactical networks owing to its versatility, scalability, and low cost. However, tactical networks need to be stealthy, where an adversary is not able to detect or intercept the tactical communication. In this paper, we investigate the stealthiness of 5G NR by looking at the probability with which an adversary that monitors the downlink synchronizat…
▽ More
5G NR is touted to be an attractive candidate for tactical networks owing to its versatility, scalability, and low cost. However, tactical networks need to be stealthy, where an adversary is not able to detect or intercept the tactical communication. In this paper, we investigate the stealthiness of 5G NR by looking at the probability with which an adversary that monitors the downlink synchronization signals can detect the presence of the network. We simulate a single-cell single-eavesdropper scenario and evaluate the probability with which the eavesdropper can detect the synchronization signal block when using either a correlator or an energy detector. We show that this probability is close to $ 100\% $ suggesting that 5G out-of-the-box is not suitable for a tactical network. We then propose utilizing the uplink channel-state-information to beamform the downlink synchronization-signals towards the tactical user-equipment (UE) to lower the eavesdropper detection probability while not compromising the performance of the legitimate tactical UE.
△ Less
Submitted 14 June, 2024; v1 submitted 21 May, 2024;
originally announced May 2024.
-
CustomText: Customized Textual Image Generation using Diffusion Models
Authors:
Shubham Paliwal,
Arushi Jain,
Monika Sharma,
Vikram Jamwal,
Lovekesh Vig
Abstract:
Textual image generation spans diverse fields like advertising, education, product packaging, social media, information visualization, and branding. Despite recent strides in language-guided image synthesis using diffusion models, current models excel in image generation but struggle with accurate text rendering and offer limited control over font attributes. In this paper, we aim to enhance the s…
▽ More
Textual image generation spans diverse fields like advertising, education, product packaging, social media, information visualization, and branding. Despite recent strides in language-guided image synthesis using diffusion models, current models excel in image generation but struggle with accurate text rendering and offer limited control over font attributes. In this paper, we aim to enhance the synthesis of high-quality images with precise text customization, thereby contributing to the advancement of image generation models. We call our proposed method CustomText. Our implementation leverages a pre-trained TextDiffuser model to enable control over font color, background, and types. Additionally, to address the challenge of accurately rendering small-sized fonts, we train the ControlNet model for a consistency decoder, significantly enhancing text-generation performance. We assess the performance of CustomText in comparison to previous methods of textual image generation on the publicly available CTW-1500 dataset and a self-curated dataset for small-text generation, showcasing superior results.
△ Less
Submitted 21 May, 2024;
originally announced May 2024.
-
Hydrodynamics of thermal active matter
Authors:
Jay Armas,
Akash Jain,
Ruben Lier
Abstract:
Active matter concerns many-body systems comprised of living or self-driven agents that collectively exhibit macroscopic phenomena distinct from conventional passive matter. Using Schwinger-Keldysh effective field theory, we develop a novel hydrodynamic framework for thermal active matter that accounts for local temperature variations and the ensuing stochastic effects. This framework provides a d…
▽ More
Active matter concerns many-body systems comprised of living or self-driven agents that collectively exhibit macroscopic phenomena distinct from conventional passive matter. Using Schwinger-Keldysh effective field theory, we develop a novel hydrodynamic framework for thermal active matter that accounts for local temperature variations and the ensuing stochastic effects. This framework provides a deeper understanding of energy balance, second law of thermodynamics, and thermostated steady states in active matter, while also addressing the systematic violations of fluctuation-dissipation theorem and detailed balance. We use our framework of active hydrodynamics to develop effective field theory actions for active superfluids and active nematics that offer a first-principle derivation of various active transport coefficients and feature activity-induced phase transitions.
△ Less
Submitted 17 May, 2024;
originally announced May 2024.
-
Adaptive Exploration for Data-Efficient General Value Function Evaluations
Authors:
Arushi Jain,
Josiah P. Hanna,
Doina Precup
Abstract:
General Value Functions (GVFs) (Sutton et al, 2011) are an established way to represent predictive knowledge in reinforcement learning. Each GVF computes the expected return for a given policy, based on a unique pseudo-reward. Multiple GVFs can be estimated in parallel using off-policy learning from a single stream of data, often sourced from a fixed behavior policy or pre-collected dataset. This…
▽ More
General Value Functions (GVFs) (Sutton et al, 2011) are an established way to represent predictive knowledge in reinforcement learning. Each GVF computes the expected return for a given policy, based on a unique pseudo-reward. Multiple GVFs can be estimated in parallel using off-policy learning from a single stream of data, often sourced from a fixed behavior policy or pre-collected dataset. This leaves an open question: how can behavior policy be chosen for data-efficient GVF learning? To address this gap, we propose GVFExplorer, which aims at learning a behavior policy that efficiently gathers data for evaluating multiple GVFs in parallel. This behavior policy selects actions in proportion to the total variance in the return across all GVFs, reducing the number of environmental interactions. To enable accurate variance estimation, we use a recently proposed temporal-difference-style variance estimator. We prove that each behavior policy update reduces the mean squared error in the summed predictions over all GVFs. We empirically demonstrate our method's performance in both tabular representations and nonlinear function approximation.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Identifying Hate Speech Peddlers in Online Platforms. A Bayesian Social Learning Approach for Large Language Model Driven Decision-Makers
Authors:
Adit Jain,
Vikram Krishnamurthy
Abstract:
This paper studies the problem of autonomous agents performing Bayesian social learning for sequential detection when the observations of the state belong to a high-dimensional space and are expensive to analyze. Specifically, when the observations are textual, the Bayesian agent can use a large language model (LLM) as a map to get a low-dimensional private observation. The agent performs Bayesian…
▽ More
This paper studies the problem of autonomous agents performing Bayesian social learning for sequential detection when the observations of the state belong to a high-dimensional space and are expensive to analyze. Specifically, when the observations are textual, the Bayesian agent can use a large language model (LLM) as a map to get a low-dimensional private observation. The agent performs Bayesian learning and takes an action that minimizes the expected cost and is visible to subsequent agents. We prove that a sequence of such Bayesian agents herd in finite time to the public belief and take the same action disregarding the private observations. We propose a stopping time formulation for quickest time herding in social learning and optimally balance privacy and herding. Structural results are shown on the threshold nature of the optimal policy to the stopping time problem. We illustrate the application of our framework when autonomous Bayesian detectors aim to sequentially identify if a user is a hate speech peddler on an online platform by parsing text observations using an LLM. We numerically validate our results on real-world hate speech datasets. We show that autonomous Bayesian agents designed to flag hate speech peddlers in online platforms herd and misclassify the users when the public prior is strong. We also numerically show the effect of a threshold policy in delaying herding.
△ Less
Submitted 12 May, 2024;
originally announced May 2024.
-
Structured Reinforcement Learning for Incentivized Stochastic Covert Optimization
Authors:
Adit Jain,
Vikram Krishnamurthy
Abstract:
This paper studies how a stochastic gradient algorithm (SG) can be controlled to hide the estimate of the local stationary point from an eavesdropper. Such problems are of significant interest in distributed optimization settings like federated learning and inventory management. A learner queries a stochastic oracle and incentivizes the oracle to obtain noisy gradient measurements and perform SG.…
▽ More
This paper studies how a stochastic gradient algorithm (SG) can be controlled to hide the estimate of the local stationary point from an eavesdropper. Such problems are of significant interest in distributed optimization settings like federated learning and inventory management. A learner queries a stochastic oracle and incentivizes the oracle to obtain noisy gradient measurements and perform SG. The oracle probabilistically returns either a noisy gradient of the function} or a non-informative measurement, depending on the oracle state and incentive. The learner's query and incentive are visible to an eavesdropper who wishes to estimate the stationary point. This paper formulates the problem of the learner performing covert optimization by dynamically incentivizing the stochastic oracle and obfuscating the eavesdropper as a finite-horizon Markov decision process (MDP). Using conditions for interval-dominance on the cost and transition probability structure, we show that the optimal policy for the MDP has a monotone threshold structure. We propose searching for the optimal stationary policy with the threshold structure using a stochastic approximation algorithm and a multi-armed bandit approach. The effectiveness of our methods is numerically demonstrated on a covert federated learning hate-speech classification task.
△ Less
Submitted 12 May, 2024;
originally announced May 2024.
-
Mobius Transformation-Based Circular Motion Control for Unicycle Robots in Nonconcentric Circular Geofences
Authors:
Shubham Singh,
Anoop Jain
Abstract:
Nonuniform motion constraints are ubiquitous in robotic applications. Geofencing control is one such paradigm where the motion of a robot must be constrained within a predefined boundary. This paper addresses the problem of stabilizing a unicycle robot around a desired circular orbit while confining its motion within a nonconcentric external circular boundary. Our solution approach relies on the c…
▽ More
Nonuniform motion constraints are ubiquitous in robotic applications. Geofencing control is one such paradigm where the motion of a robot must be constrained within a predefined boundary. This paper addresses the problem of stabilizing a unicycle robot around a desired circular orbit while confining its motion within a nonconcentric external circular boundary. Our solution approach relies on the concept of the so-called Mobius transformation that, under certain practical conditions, maps two nonconcentric circles to a pair of concentric circles, and hence, results in uniform spatial motion constraints. The choice of such a Mobius transformation is governed by the roots of a quadratic equation in the post-design analysis that decides how the regions enclosed by the two circles are mapped onto the two planes. We show that the problem can be formulated either as a trajectory-constraining problem or an obstacle-avoidance problem in the transformed plane, depending on these roots. Exploiting the idea of the barrier Lyapunov function, we propose a unique control law that solves both these contrasting problems in the transformed plane and renders a solution to the original problem in the actual plane. By relating parameters of two planes under Mobius transformation and its inverse map, we further establish a connection between the control laws in two planes and determine the control law to be applied in the actual plane. Simulation and experimental results are provided to illustrate the key theoretical developments.
△ Less
Submitted 17 July, 2024; v1 submitted 11 May, 2024;
originally announced May 2024.
-
Uncovering implementable dormant pruning decisions from three different stakeholder perspectives
Authors:
Deanna Flynn,
Abhinav Jain,
Heather Knight,
Cristina G. Wilson,
Cindy Grimm
Abstract:
Dormant pruning, or the removal of unproductive portions of a tree while a tree is not actively growing, is an important orchard task to help maintain yield, requiring years to build expertise. Because of long training periods and an increasing labor shortage in agricultural jobs, pruning could benefit from robotic automation. However, to program robots to prune branches, we first need to understa…
▽ More
Dormant pruning, or the removal of unproductive portions of a tree while a tree is not actively growing, is an important orchard task to help maintain yield, requiring years to build expertise. Because of long training periods and an increasing labor shortage in agricultural jobs, pruning could benefit from robotic automation. However, to program robots to prune branches, we first need to understand how pruning decisions are made, and what variables in the environment (e.g., branch size and thickness) we need to capture. Working directly with three pruning stakeholders -- horticulturists, growers, and pruners -- we find that each group of human experts approaches pruning decision-making differently. To capture this knowledge, we present three studies and two extracted pruning protocols from field work conducted in Prosser, Washington in January 2022 and 2023. We interviewed six stakeholders (two in each group) and observed pruning across three cultivars -- Bing Cherries, Envy Apples, and Jazz Apples -- and two tree architectures -- Upright Fruiting Offshoot and V-Trellis. Leveraging participant interviews and video data, this analysis uses grounded coding to extract pruning terminology, discover horticultural contexts that influence pruning decisions, and find implementable pruning heuristics for autonomous systems. The results include a validated terminology set, which we offer for use by both pruning stakeholders and roboticists, to communicate general pruning concepts and heuristics. The results also highlight seven pruning heuristics utilizing this terminology set that would be relevant for use by future autonomous robot pruning systems, and characterize three discovered horticultural contexts (i.e., environmental management, crop-load management, and replacement wood) across all three cultivars.
△ Less
Submitted 7 May, 2024;
originally announced May 2024.
-
Diabetic Retinopathy Detection Using Quantum Transfer Learning
Authors:
Ankush Jain,
Rinav Gupta,
Jai Singhal
Abstract:
Diabetic Retinopathy (DR), a prevalent complication in diabetes patients, can lead to vision impairment due to lesions formed on the retina. Detecting DR at an advanced stage often results in irreversible blindness. The traditional process of diagnosing DR through retina fundus images by ophthalmologists is not only time-intensive but also expensive. While classical transfer learning models have b…
▽ More
Diabetic Retinopathy (DR), a prevalent complication in diabetes patients, can lead to vision impairment due to lesions formed on the retina. Detecting DR at an advanced stage often results in irreversible blindness. The traditional process of diagnosing DR through retina fundus images by ophthalmologists is not only time-intensive but also expensive. While classical transfer learning models have been widely adopted for computer-aided detection of DR, their high maintenance costs can hinder their detection efficiency. In contrast, Quantum Transfer Learning offers a more effective solution to this challenge. This approach is notably advantageous because it operates on heuristic principles, making it highly optimized for the task. Our proposed methodology leverages this hybrid quantum transfer learning technique to detect DR. To construct our model, we utilize the APTOS 2019 Blindness Detection dataset, available on Kaggle. We employ the ResNet-18, ResNet34, ResNet50, ResNet101, ResNet152 and Inception V3, pre-trained classical neural networks, for the initial feature extraction. For the classification stage, we use a Variational Quantum Classifier. Our hybrid quantum model has shown remarkable results, achieving an accuracy of 97% for ResNet-18. This demonstrates that quantum computing, when integrated with quantum machine learning, can perform tasks with a level of power and efficiency unattainable by classical computers alone. By harnessing these advanced technologies, we can significantly improve the detection and diagnosis of Diabetic Retinopathy, potentially saving many from the risk of blindness.
Keywords: Diabetic Retinopathy, Quantum Transfer Learning, Deep Learning
△ Less
Submitted 2 May, 2024;
originally announced May 2024.
-
Hide and Seek: How Does Watermarking Impact Face Recognition?
Authors:
Yuguang Yao,
Steven Grosz,
Sijia Liu,
Anil Jain
Abstract:
The recent progress in generative models has revolutionized the synthesis of highly realistic images, including face images. This technological development has undoubtedly helped face recognition, such as training data augmentation for higher recognition accuracy and data privacy. However, it has also introduced novel challenges concerning the responsible use and proper attribution of computer gen…
▽ More
The recent progress in generative models has revolutionized the synthesis of highly realistic images, including face images. This technological development has undoubtedly helped face recognition, such as training data augmentation for higher recognition accuracy and data privacy. However, it has also introduced novel challenges concerning the responsible use and proper attribution of computer generated images. We investigate the impact of digital watermarking, a technique for embedding ownership signatures into images, on the effectiveness of face recognition models. We propose a comprehensive pipeline that integrates face image generation, watermarking, and face recognition to systematically examine this question. The proposed watermarking scheme, based on an encoder-decoder architecture, successfully embeds and recovers signatures from both real and synthetic face images while preserving their visual fidelity. Through extensive experiments, we unveil that while watermarking enables robust image attribution, it results in a slight decline in face recognition accuracy, particularly evident for face images with challenging poses and expressions. Additionally, we find that directly training face recognition models on watermarked images offers only a limited alleviation of this performance decline. Our findings underscore the intricate trade off between watermarking and face recognition accuracy. This work represents a pivotal step towards the responsible utilization of generative models in face recognition and serves to initiate discussions regarding the broader implications of watermarking in biometrics.
△ Less
Submitted 29 April, 2024;
originally announced April 2024.
-
Impact of Traffic-Following on Order of Autonomous Airspace Operations
Authors:
Anahita Jain,
Husni R. Idris,
John-Paul Clarke
Abstract:
In this paper, we investigate the dynamic emergence of traffic order in a distributed multi-agent system, aiming to minimize inefficiencies that stem from unnecessary structural impositions. We introduce a methodology for developing a dynamically-updating traffic pattern map of the airspace by leveraging information about the consistency and frequency of flow directions used by current as well as…
▽ More
In this paper, we investigate the dynamic emergence of traffic order in a distributed multi-agent system, aiming to minimize inefficiencies that stem from unnecessary structural impositions. We introduce a methodology for developing a dynamically-updating traffic pattern map of the airspace by leveraging information about the consistency and frequency of flow directions used by current as well as preceding traffic. Informed by this map, an agent can discern the degree to which it is advantageous to follow traffic by trading off utilities such as time and order. We show that for the traffic levels studied, for low degrees of traffic-following behavior, there is minimal penalty in terms of aircraft travel times while improving the overall orderliness of the airspace. On the other hand, heightened traffic-following behavior may result in increased aircraft travel times, while marginally reducing the overall entropy of the airspace. Ultimately, the methods and metrics presented in this paper can be used to optimally and dynamically adjust an agent's traffic-following behavior based on these trade-offs.
△ Less
Submitted 3 June, 2024; v1 submitted 26 April, 2024;
originally announced April 2024.
-
How Could AI Support Design Education? A Study Across Fields Fuels Situating Analytics
Authors:
Ajit Jain,
Andruid Kerne,
Hannah Fowler,
Jinsil Seo,
Galen Newman,
Nic Lupfer,
Aaron Perrine
Abstract:
We use the process and findings from a case study of design educators' practices of assessment and feedback to fuel theorizing about how to make AI useful in service of human experience. We build on Suchman's theory of situated actions. We perform a qualitative study of 11 educators in 5 fields, who teach design processes situated in project-based learning contexts. Through qualitative data gather…
▽ More
We use the process and findings from a case study of design educators' practices of assessment and feedback to fuel theorizing about how to make AI useful in service of human experience. We build on Suchman's theory of situated actions. We perform a qualitative study of 11 educators in 5 fields, who teach design processes situated in project-based learning contexts. Through qualitative data gathering and analysis, we derive codes: design process; assessment and feedback challenges; and computational support.
We twice invoke creative cognition's family resemblance principle. First, to explain how design instructors already use assessment rubrics and second, to explain the analogous role for design creativity analytics: no particular trait is necessary or sufficient; each only tends to indicate good design work. Human teachers remain essential. We develop a set of situated design creativity analytics--Fluency, Flexibility, Visual Consistency, Multiscale Organization, and Legible Contrast--to support instructors' efforts, by providing on-demand, learning objectives-based assessment and feedback to students.
We theorize a methodology, which we call situating analytics, firstly because making AI support living human activity depends on aligning what analytics measure with situated practices. Further, we realize that analytics can become most significant to users by situating them through interfaces that integrate them into the material contexts of their use. Here, this means situating design creativity analytics into actual design environments. Through the case study, we identify situating analytics as a methodology for explaining analytics to users, because the iterative process of alignment with practice has the potential to enable data scientists to derive analytics that make sense as part of and support situated human experiences.
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
Universal Fingerprint Generation: Controllable Diffusion Model with Multimodal Conditions
Authors:
Steven A. Grosz,
Anil K. Jain
Abstract:
The utilization of synthetic data for fingerprint recognition has garnered increased attention due to its potential to alleviate privacy concerns surrounding sensitive biometric data. However, current methods for generating fingerprints have limitations in creating impressions of the same finger with useful intra-class variations. To tackle this challenge, we present GenPrint, a framework to produ…
▽ More
The utilization of synthetic data for fingerprint recognition has garnered increased attention due to its potential to alleviate privacy concerns surrounding sensitive biometric data. However, current methods for generating fingerprints have limitations in creating impressions of the same finger with useful intra-class variations. To tackle this challenge, we present GenPrint, a framework to produce fingerprint images of various types while maintaining identity and offering humanly understandable control over different appearance factors such as fingerprint class, acquisition type, sensor device, and quality level. Unlike previous fingerprint generation approaches, GenPrint is not confined to replicating style characteristics from the training dataset alone: it enables the generation of novel styles from unseen devices without requiring additional fine-tuning. To accomplish these objectives, we developed GenPrint using latent diffusion models with multimodal conditions (text and image) for consistent generation of style and identity. Our experiments leverage a variety of publicly available datasets for training and evaluation. Results demonstrate the benefits of GenPrint in terms of identity preservation, explainable control, and universality of generated images. Importantly, the GenPrint-generated images yield comparable or even superior accuracy to models trained solely on real data and further enhances performance when augmenting the diversity of existing real fingerprint datasets.
△ Less
Submitted 21 April, 2024;
originally announced April 2024.
-
GenCHiP: Generating Robot Policy Code for High-Precision and Contact-Rich Manipulation Tasks
Authors:
Kaylee Burns,
Ajinkya Jain,
Keegan Go,
Fei Xia,
Michael Stark,
Stefan Schaal,
Karol Hausman
Abstract:
Large Language Models (LLMs) have been successful at generating robot policy code, but so far these results have been limited to high-level tasks that do not require precise movement. It is an open question how well such approaches work for tasks that require reasoning over contact forces and working within tight success tolerances. We find that, with the right action space, LLMs are capable of su…
▽ More
Large Language Models (LLMs) have been successful at generating robot policy code, but so far these results have been limited to high-level tasks that do not require precise movement. It is an open question how well such approaches work for tasks that require reasoning over contact forces and working within tight success tolerances. We find that, with the right action space, LLMs are capable of successfully generating policies for a variety of contact-rich and high-precision manipulation tasks, even under noisy conditions, such as perceptual errors or grasping inaccuracies. Specifically, we reparameterize the action space to include compliance with constraints on the interaction forces and stiffnesses involved in reaching a target pose. We validate this approach on subtasks derived from the Functional Manipulation Benchmark (FMB) and NIST Task Board Benchmarks. Exposing this action space alongside methods for estimating object poses improves policy generation with an LLM by greater than 3x and 4x when compared to non-compliant action spaces
△ Less
Submitted 9 April, 2024;
originally announced April 2024.
-
CoBT: Collaborative Programming of Behaviour Trees from One Demonstration for Robot Manipulation
Authors:
Aayush Jain,
Philip Long,
Valeria Villani,
John D. Kelleher,
Maria Chiara Leva
Abstract:
Mass customization and shorter manufacturing cycles are becoming more important among small and medium-sized companies. However, classical industrial robots struggle to cope with product variation and dynamic environments. In this paper, we present CoBT, a collaborative programming by demonstration framework for generating reactive and modular behavior trees. CoBT relies on a single demonstration…
▽ More
Mass customization and shorter manufacturing cycles are becoming more important among small and medium-sized companies. However, classical industrial robots struggle to cope with product variation and dynamic environments. In this paper, we present CoBT, a collaborative programming by demonstration framework for generating reactive and modular behavior trees. CoBT relies on a single demonstration and a combination of data-driven machine learning methods with logic-based declarative learning to learn a task, thus eliminating the need for programming expertise or long development times. The proposed framework is experimentally validated on 7 manipulation tasks and we show that CoBT achieves approx. 93% success rate overall with an average of 7.5s programming time. We conduct a pilot study with non-expert users to provide feedback regarding the usability of CoBT.
△ Less
Submitted 10 April, 2024; v1 submitted 8 April, 2024;
originally announced April 2024.
-
Indexing Analytics to Instances: How Integrating a Dashboard can Support Design Education
Authors:
Ajit Jain,
Andruid Kerne,
Nic Lupfer,
Gabriel Britain,
Aaron Perrine,
Yoonsuck Choe,
John Keyser,
Ruihong Huang,
Jinsil Seo,
Annie Sungkajun,
Robert Lightfoot,
Timothy McGuire
Abstract:
We investigate how to use AI-based analytics to support design education. The analytics at hand measure multiscale design, that is, students' use of space and scale to visually and conceptually organize their design work. With the goal of making the analytics intelligible to instructors, we developed a research artifact integrating a design analytics dashboard with design instances, and the design…
▽ More
We investigate how to use AI-based analytics to support design education. The analytics at hand measure multiscale design, that is, students' use of space and scale to visually and conceptually organize their design work. With the goal of making the analytics intelligible to instructors, we developed a research artifact integrating a design analytics dashboard with design instances, and the design environment that students use to create them. We theorize about how Suchman's notion of mutual intelligibility requires contextualized investigation of AI in order to develop findings about how analytics work for people. We studied the research artifact in 5 situated course contexts, in 3 departments. A total of 236 students used the multiscale design environment. The 9 instructors who taught those students experienced the analytics via the new research artifact.
We derive findings from a qualitative analysis of interviews with instructors regarding their experiences. Instructors reflected on how the analytics and their presentation in the dashboard have the potential to affect design education. We develop research implications addressing: (1) how indexing design analytics in the dashboard to actual design work instances helps design instructors reflect on what they mean and, more broadly, is a technique for how AI-based design analytics can support instructors' assessment and feedback experiences in situated course contexts; and (2) how multiscale design analytics, in particular, have the potential to support design education. By indexing, we mean linking which provides context, here connecting the numbers of the analytics with visually annotated design work instances.
△ Less
Submitted 8 April, 2024;
originally announced April 2024.
-
Passivity-based Attack Identification and Mitigation with Event-triggered Observer Feedback and Switching Controller
Authors:
Pushkal Purohit,
Anoop Jain
Abstract:
This paper addresses the problem of output consensus in linear passive multi-agent systems under a False Data Injection (FDI) attack, considering the unavailability of complete state information. Our formulation relies on an event-based cryptographic authentication scheme for sensor integrity and considers FDI attacks at the actuator end, inspired by their practical nature and usages. For secure c…
▽ More
This paper addresses the problem of output consensus in linear passive multi-agent systems under a False Data Injection (FDI) attack, considering the unavailability of complete state information. Our formulation relies on an event-based cryptographic authentication scheme for sensor integrity and considers FDI attacks at the actuator end, inspired by their practical nature and usages. For secure consensus, we propose (i) a passivity-based approach for detecting FDI attacks on the system and (ii) a Zeno-free event-triggered observer-based switching controller, which switches between the normal and the defense modes following an attack detection. We show that the closed-loop system achieves practical consensus under the controller's action in the defense mode. Simulation examples are provided to support the theoretical findings.
△ Less
Submitted 22 March, 2024;
originally announced March 2024.
-
KeyPoint Relative Position Encoding for Face Recognition
Authors:
Minchul Kim,
Yiyang Su,
Feng Liu,
Anil Jain,
Xiaoming Liu
Abstract:
In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin…
▽ More
In this paper, we address the challenge of making ViT models more robust to unseen affine transformations. Such robustness becomes useful in various recognition tasks such as face recognition when image alignment failures occur. We propose a novel method called KP-RPE, which leverages key points (e.g.~facial landmarks) to make ViT more resilient to scale, translation, and pose variations. We begin with the observation that Relative Position Encoding (RPE) is a good way to bring affine transform generalization to ViTs. RPE, however, can only inject the model with prior knowledge that nearby pixels are more important than far pixels. Keypoint RPE (KP-RPE) is an extension of this principle, where the significance of pixels is not solely dictated by their proximity but also by their relative positions to specific keypoints within the image. By anchoring the significance of pixels around keypoints, the model can more effectively retain spatial relationships, even when those relationships are disrupted by affine transformations. We show the merit of KP-RPE in face and gait recognition. The experimental results demonstrate the effectiveness in improving face recognition performance from low-quality images, particularly where alignment is prone to failure. Code and pre-trained models are available.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
Authors:
Alexander Khazatsky,
Karl Pertsch,
Suraj Nair,
Ashwin Balakrishna,
Sudeep Dasari,
Siddharth Karamcheti,
Soroush Nasiriany,
Mohan Kumar Srirama,
Lawrence Yunliang Chen,
Kirsty Ellis,
Peter David Fagan,
Joey Hejna,
Masha Itkina,
Marion Lepert,
Yecheng Jason Ma,
Patrick Tree Miller,
Jimmy Wu,
Suneel Belkhale,
Shivin Dass,
Huy Ha,
Arhan Jain,
Abraham Lee,
Youngwoon Lee,
Marius Memmel,
Sungjae Park
, et al. (74 additional authors not shown)
Abstract:
The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a resu…
▽ More
The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.
△ Less
Submitted 19 March, 2024;
originally announced March 2024.