-
LUWA Dataset: Learning Lithic Use-Wear Analysis on Microscopic Images
Authors:
Jing Zhang,
Irving Fang,
Juexiao Zhang,
Hao Wu,
Akshat Kaushik,
Alice Rodriguez,
Hanwen Zhao,
Zhuo Zheng,
Radu Iovita,
Chen Feng
Abstract:
Lithic Use-Wear Analysis (LUWA) using microscopic images is an underexplored vision-for-science research area. It seeks to distinguish the worked material, which is critical for understanding archaeological artifacts, material interactions, tool functionalities, and dental records. However, this challenging task goes beyond the well-studied image classification problem for common objects. It is af…
▽ More
Lithic Use-Wear Analysis (LUWA) using microscopic images is an underexplored vision-for-science research area. It seeks to distinguish the worked material, which is critical for understanding archaeological artifacts, material interactions, tool functionalities, and dental records. However, this challenging task goes beyond the well-studied image classification problem for common objects. It is affected by many confounders owing to the complex wear mechanism and microscopic imaging, which makes it difficult even for human experts to identify the worked material successfully. In this paper, we investigate the following three questions on this unique vision task for the first time:(i) How well can state-of-the-art pre-trained models (like DINOv2) generalize to the rarely seen domain? (ii) How can few-shot learning be exploited for scarce microscopic images? (iii) How do the ambiguous magnification and sensing modality influence the classification accuracy? To study these, we collaborated with archaeologists and built the first open-source and the largest LUWA dataset containing 23,130 microscopic images with different magnifications and sensing modalities. Extensive experiments show that existing pre-trained models notably outperform human experts but still leave a large gap for improvements. Most importantly, the LUWA dataset provides an underexplored opportunity for vision and learning communities and complements existing image classification problems on common objects.
△ Less
Submitted 27 March, 2024; v1 submitted 19 March, 2024;
originally announced March 2024.
-
EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction
Authors:
Irving Fang,
Yuzhong Chen,
Yifan Wang,
Jianghan Zhang,
Qiushi Zhang,
Jiali Xu,
Xibo He,
Weibo Gao,
Hao Su,
Yiming Li,
Chen Feng
Abstract:
A robot's ability to anticipate the 3D action target location of a hand's movement from egocentric videos can greatly improve safety and efficiency in human-robot interaction (HRI). While previous research predominantly focused on semantic action classification or 2D target region prediction, we argue that predicting the action target's 3D coordinate could pave the way for more versatile downstrea…
▽ More
A robot's ability to anticipate the 3D action target location of a hand's movement from egocentric videos can greatly improve safety and efficiency in human-robot interaction (HRI). While previous research predominantly focused on semantic action classification or 2D target region prediction, we argue that predicting the action target's 3D coordinate could pave the way for more versatile downstream robotics tasks, especially given the increasing prevalence of headset devices. This study expands EgoPAT3D, the sole dataset dedicated to egocentric 3D action target prediction. We augment both its size and diversity, enhancing its potential for generalization. Moreover, we substantially enhance the baseline algorithm by introducing a large pre-trained model and human prior knowledge. Remarkably, our novel algorithm can now achieve superior prediction outcomes using solely RGB images, eliminating the previous need for 3D point clouds and IMU input. Furthermore, we deploy our enhanced baseline algorithm on a real-world robotic platform to illustrate its practical utility in straightforward HRI tasks. The demonstrations showcase the real-world applicability of our advancements and may inspire more HRI use cases involving egocentric vision. All code and data are open-sourced and can be found on the project website.
△ Less
Submitted 7 March, 2024;
originally announced March 2024.
-
Medium. Permeation: SARS-COV-2 Painting Creation by Generative Model
Authors:
Yuan-Fu Yang,
Iuan-Kai Fang,
Min Sun,
Su-Chu Hsu
Abstract:
Airborne particles are the medium for SARS-CoV-2 to invade the human body. Light also reflects through suspended particles in the air, allowing people to see a colorful world. Impressionism is the most prominent art school that explores the spectrum of color created through color reflection of light. We find similarities of color structure and color stacking in the Impressionist paintings and the…
▽ More
Airborne particles are the medium for SARS-CoV-2 to invade the human body. Light also reflects through suspended particles in the air, allowing people to see a colorful world. Impressionism is the most prominent art school that explores the spectrum of color created through color reflection of light. We find similarities of color structure and color stacking in the Impressionist paintings and the illustrations of the novel coronavirus by artists around the world. With computerized data analysis through the main tones, the way of color layout, and the way of color stacking in the paintings of the Impressionists, we train computers to draw the novel coronavirus in an Impressionist style using a Generative Adversarial Network to create our artwork "Medium. Permeation". This artwork is composed of 196 randomly generated viral pictures arranged in a 14 by 14 matrix to form a large-scale painting. In addition, we have developed an extended work: Gradual Change, which is presented as video art. We use Graph Neural Network to present 196 paintings of the new coronavirus to the audience one by one in a gradual manner. In front of LED TV screen, audience will find 196 virus paintings whose colors will change continuously. This large video painting symbolizes that worldwide 196 countries have been invaded by the epidemic, and every nation continuously pops up mutant viruses. The speed of vaccine development cannot keep up with the speed of virus mutation. This is also the first generative art in the world based on the common features and a metaphorical symbiosis between Impressionist art and the novel coronavirus. This work warns us of the unprecedented challenges posed by the SARS-CoV-2, implying that the world should not ignore the invisible enemy who uses air as a medium.
△ Less
Submitted 22 April, 2023;
originally announced April 2023.
-
Metric-Free Exploration for Topological Mapping by Task and Motion Imitation in Feature Space
Authors:
Yuhang He,
Irving Fang,
Yiming Li,
Rushi Bhavesh Shah,
Chen Feng
Abstract:
We propose DeepExplorer, a simple and lightweight metric-free exploration method for topological mapping of unknown environments. It performs task and motion planning (TAMP) entirely in image feature space. The task planner is a recurrent network using the latest image observation sequence to hallucinate a feature as the next-best exploration goal. The motion planner then utilizes the current and…
▽ More
We propose DeepExplorer, a simple and lightweight metric-free exploration method for topological mapping of unknown environments. It performs task and motion planning (TAMP) entirely in image feature space. The task planner is a recurrent network using the latest image observation sequence to hallucinate a feature as the next-best exploration goal. The motion planner then utilizes the current and the hallucinated features to generate an action taking the agent towards that goal. The two planners are jointly trained via deeply-supervised imitation learning from expert demonstrations. During exploration, we iteratively call the two planners to predict the next action, and the topological map is built by constantly appending the latest image observation and action to the map and using visual place recognition (VPR) for loop closing. The resulting topological map efficiently represents an environment's connectivity and traversability, so it can be used for tasks such as visual navigation. We show DeepExplorer's exploration efficiency and strong sim2sim generalization capability on large-scale simulation datasets like Gibson and MP3D. Its effectiveness is further validated via the image-goal navigation performance on the resulting topological map. We further show its strong zero-shot sim2real generalization capability in real-world experiments. The source code is available at \url{https://ai4ce.github.io/DeepExplorer/}.
△ Less
Submitted 16 March, 2023;
originally announced March 2023.
-
Dynamic Placement of Rapidly Deployable Mobile Sensor Robots Using Machine Learning and Expected Value of Information
Authors:
Alice Agogino,
Hae Young Jang,
Vivek Rao,
Ritik Batra,
Felicity Liao,
Rohan Sood,
Irving Fang,
R. Lily Hu,
Emerson Shoichet-Bartus,
John Matranga
Abstract:
Although the Industrial Internet of Things has increased the number of sensors permanently installed in industrial plants, there will be gaps in coverage due to broken sensors or sparse density in very large plants, such as in the petrochemical industry. Modern emergency response operations are beginning to use Small Unmanned Aerial Systems (sUAS) that have the ability to drop sensor robots to pre…
▽ More
Although the Industrial Internet of Things has increased the number of sensors permanently installed in industrial plants, there will be gaps in coverage due to broken sensors or sparse density in very large plants, such as in the petrochemical industry. Modern emergency response operations are beginning to use Small Unmanned Aerial Systems (sUAS) that have the ability to drop sensor robots to precise locations. sUAS can provide longer-term persistent monitoring that aerial drones are unable to provide. Despite the relatively low cost of these assets, the choice of which robotic sensing systems to deploy to which part of an industrial process in a complex plant environment during emergency response remains challenging.
This paper describes a framework for optimizing the deployment of emergency sensors as a preliminary step towards realizing the responsiveness of robots in disaster circumstances. AI techniques (Long short-term memory, 1-dimensional convolutional neural network, logistic regression, and random forest) identify regions where sensors would be most valued without requiring humans to enter the potentially dangerous area. In the case study described, the cost function for optimization considers costs of false-positive and false-negative errors. Decisions on mitigation include implementing repairs or shutting down the plant. The Expected Value of Information (EVI) is used to identify the most valuable type and location of physical sensors to be deployed to increase the decision-analytic value of a sensor network. This method is applied to a case study using the Tennessee Eastman process data set of a chemical plant, and we discuss implications of our findings for operation, distribution, and decision-making of sensors in plant emergency and resilience scenarios.
△ Less
Submitted 15 November, 2021;
originally announced November 2021.
-
Multi-modal Ensemble Models for Predicting Video Memorability
Authors:
Tony Zhao,
Irving Fang,
Jeffrey Kim,
Gerald Friedland
Abstract:
Modeling media memorability has been a consistent challenge in the field of machine learning. The Predicting Media Memorability task in MediaEval2020 is the latest benchmark among similar challenges addressing this topic. Building upon techniques developed in previous iterations of the challenge, we developed ensemble methods with the use of extracted video, image, text, and audio features. Critic…
▽ More
Modeling media memorability has been a consistent challenge in the field of machine learning. The Predicting Media Memorability task in MediaEval2020 is the latest benchmark among similar challenges addressing this topic. Building upon techniques developed in previous iterations of the challenge, we developed ensemble methods with the use of extracted video, image, text, and audio features. Critically, in this work we introduce and demonstrate the efficacy and high generalizability of extracted audio embeddings as a feature for the task of predicting media memorability.
△ Less
Submitted 1 February, 2021;
originally announced February 2021.
-
An Off-Chip Attack on Hardware Enclaves via the Memory Bus
Authors:
Dayeol Lee,
Dongha Jung,
Ian T. Fang,
Chia-Che Tsai,
Raluca Ada Popa
Abstract:
This paper shows how an attacker can break the confidentiality of a hardware enclave with Membuster, an off-chip attack based on snooping the memory bus. An attacker with physical access can observe an unencrypted address bus and extract fine-grained memory access patterns of the victim. Membuster is qualitatively different from prior on-chip attacks to enclaves and is more difficult to thwart.…
▽ More
This paper shows how an attacker can break the confidentiality of a hardware enclave with Membuster, an off-chip attack based on snooping the memory bus. An attacker with physical access can observe an unencrypted address bus and extract fine-grained memory access patterns of the victim. Membuster is qualitatively different from prior on-chip attacks to enclaves and is more difficult to thwart.
We highlight several challenges for Membuster. First, DRAM requests are only visible on the memory bus at last-level cache misses. Second, the attack needs to incur minimal interference or overhead to the victim to prevent the detection of the attack. Lastly, the attacker needs to reverse-engineer the translation between virtual, physical, and DRAM addresses to perform a robust attack. We introduce three techniques, critical page whitelisting, cache squeezing, and oracle-based fuzzy matching algorithm to increase cache misses for memory accesses that are useful for the attack, with no detectable interference to the victim, and to convert memory accesses to sensitive data. We demonstrate Membuster on an Intel SGX CPU to leak confidential data from two applications: Hunspell and Memcached. We show that a single uninterrupted run of the victim can leak most of the sensitive data with high accuracy.
△ Less
Submitted 3 December, 2019;
originally announced December 2019.
-
Self-Contained Stylization via Steganography for Reverse and Serial Style Transfer
Authors:
Hung-Yu Chen,
I-Sheng Fang,
Wei-Chen Chiu
Abstract:
Style transfer has been widely applied to give real-world images a new artistic look. However, given a stylized image, the attempts to use typical style transfer methods for de-stylization or transferring it again into another style usually lead to artifacts or undesired results. We realize that these issues are originated from the content inconsistency between the original image and its stylized…
▽ More
Style transfer has been widely applied to give real-world images a new artistic look. However, given a stylized image, the attempts to use typical style transfer methods for de-stylization or transferring it again into another style usually lead to artifacts or undesired results. We realize that these issues are originated from the content inconsistency between the original image and its stylized output. Therefore, in this paper we advance to keep the content information of the input image during the process of style transfer by the power of steganography, with two approaches proposed: a two-stage model and an end-to-end model. We conduct extensive experiments to successfully verify the capacity of our models, in which both of them are able to not only generate stylized images of quality comparable with the ones produced by typical style transfer methods, but also effectively eliminate the artifacts introduced in reconstructing original input from a stylized image as well as performing multiple times of style transfer in series.
△ Less
Submitted 9 January, 2020; v1 submitted 10 December, 2018;
originally announced December 2018.
