-
VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation
Authors:
Xuan He,
Dongfu Jiang,
Ge Zhang,
Max Ku,
Achint Soni,
Sherman Siu,
Haonan Chen,
Abhranil Chandra,
Ziyan Jiang,
Aaran Arulraj,
Kai Wang,
Quy Duc Do,
Yuansheng Ni,
Bohan Lyu,
Yaswanth Narsupalli,
Rongqi Fan,
Zhiheng Lyu,
Yuchen Lin,
Wenhu Chen
Abstract:
The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-prov…
▽ More
The recent years have witnessed great advances in video generation. However, the development of automatic video metrics is lagging significantly behind. None of the existing metric is able to provide reliable scores over generated videos. The main barrier is the lack of large-scale human-annotated dataset. In this paper, we release VideoFeedback, the first large-scale dataset containing human-provided multi-aspect score over 37.6K synthesized videos from 11 existing video generative models. We train VideoScore (initialized from Mantis) based on VideoFeedback to enable automatic video quality assessment. Experiments show that the Spearman correlation between VideoScore and humans can reach 77.1 on VideoFeedback-test, beating the prior best metrics by about 50 points. Further result on other held-out EvalCrafter, GenAI-Bench, and VBench show that VideoScore has consistently much higher correlation with human judges than other metrics. Due to these results, we believe VideoScore can serve as a great proxy for human raters to (1) rate different video models to track progress (2) simulate fine-grained human feedback in Reinforcement Learning with Human Feedback (RLHF) to improve current video generation models.
△ Less
Submitted 24 June, 2024; v1 submitted 21 June, 2024;
originally announced June 2024.
-
A Versatile Data Fabric for Advanced IoT-Based Remote Health Monitoring
Authors:
Italo Buleje,
Vince S. Siu,
Kuan Yu Hsieh,
Nigel Hinds,
Bing Dang,
Erhan Bilal,
Thanhnha Nguyen,
Ellen E. Lee,
Colin A. Depp,
Jeffrey L. Rogers
Abstract:
This paper presents a data-centric and security-focused data fabric designed for digital health applications. With the increasing interest in digital health research, there has been a surge in the volume of Internet of Things (IoT) data derived from smartphones, wearables, and ambient sensors. Managing this vast amount of data, encompassing diverse data types and varying time scales, is crucial. M…
▽ More
This paper presents a data-centric and security-focused data fabric designed for digital health applications. With the increasing interest in digital health research, there has been a surge in the volume of Internet of Things (IoT) data derived from smartphones, wearables, and ambient sensors. Managing this vast amount of data, encompassing diverse data types and varying time scales, is crucial. Moreover, compliance with regulatory and contractual obligations is essential. The proposed data fabric comprises an architecture and a toolkit that facilitate the integration of heterogeneous data sources, across different environments, to provide a unified view of the data in dashboards. Furthermore, the data fabric supports the development of reusable and configurable data integration components, which can be shared as open-source or inner-source software. These components are used to generate data pipelines that can be deployed and scheduled to run either in the cloud or on-premises. Additionally, we present the implementation of our data fabric in a home-based telemonitoring research project involving older adults, conducted in collaboration with the University of California, San Diego (UCSD). The study showcases the streamlined integration of data collected from various IoT sensors and mobile applications to create a unified view of older adults' health for further analysis and research.
△ Less
Submitted 2 October, 2023;
originally announced October 2023.
-
Towards automating Codenames spymasters with deep reinforcement learning
Authors:
Sherman Siu
Abstract:
Although most reinforcement learning research has centered on competitive games, little work has been done on applying it to co-operative multiplayer games or text-based games. Codenames is a board game that involves both asymmetric co-operation and natural language processing, which makes it an excellent candidate for advancing RL research. To my knowledge, this work is the first to formulate Cod…
▽ More
Although most reinforcement learning research has centered on competitive games, little work has been done on applying it to co-operative multiplayer games or text-based games. Codenames is a board game that involves both asymmetric co-operation and natural language processing, which makes it an excellent candidate for advancing RL research. To my knowledge, this work is the first to formulate Codenames as a Markov Decision Process and apply some well-known reinforcement learning algorithms such as SAC, PPO, and A2C to the environment. Although none of the above algorithms converge for the Codenames environment, neither do they converge for a simplified environment called ClickPixel, except when the board size is small.
△ Less
Submitted 28 December, 2022;
originally announced December 2022.
-
Health Guardian Platform: A technology stack to accelerate discovery in Digital Health research
Authors:
Bo Wen,
Vince S. Siu,
Italo Buleje,
Kuan Yu Hsieh,
Takashi Itoh,
Lukas Zimmerli,
Nigel Hinds,
Elif Eyigoz,
Bing Dang,
Stefan von Cavallar,
Jeffrey L. Rogers
Abstract:
This paper highlights the design philosophy and architecture of the Health Guardian, a platform developed by the IBM Digital Health team to accelerate discoveries of new digital biomarkers and development of digital health technologies. The Health Guardian allows for rapid translation of artificial intelligence (AI) research into cloud-based microservices that can be tested with data from clinical…
▽ More
This paper highlights the design philosophy and architecture of the Health Guardian, a platform developed by the IBM Digital Health team to accelerate discoveries of new digital biomarkers and development of digital health technologies. The Health Guardian allows for rapid translation of artificial intelligence (AI) research into cloud-based microservices that can be tested with data from clinical cohorts to understand disease and enable early prevention. The platform can be connected to mobile applications, wearables, or Internet of things (IoT) devices to collect health-related data into a secure database. When the analytics are created, the researchers can containerize and deploy their code on the cloud using pre-defined templates, and validate the models using the data collected from one or more sensing devices. The Health Guardian platform currently supports time-series, text, audio, and video inputs with 70+ analytic capabilities and is used for non-commercial scientific research. We provide an example of the Alzheimer's disease (AD) assessment microservice which uses AI methods to extract linguistic features from audio recordings to evaluate an individual's mini-mental state, the likelihood of having AD, and to predict the onset of AD before turning the age of 85. Today, IBM research teams across the globe use the Health Guardian internally as a test bed for early-stage research ideas, and externally with collaborators to support and enhance AI model development and clinical study efforts.
△ Less
Submitted 10 November, 2022;
originally announced November 2022.
