On Leakage in Machine Learning Pipelines
Authors:
Leonard Sasse,
Eliana Nicolaisen-Sobesky,
Juergen Dukart,
Simon B. Eickhoff,
Michael Götz,
Sami Hamdan,
Vera Komeyer,
Abhijit Kulkarni,
Juha Lahnakoski,
Bradley C. Love,
Federico Raimondo,
Kaustubh R. Patil
Abstract:
Machine learning (ML) provides powerful tools for predictive modeling. ML's popularity stems from the promise of sample-level prediction with applications across a variety of fields from physics and marketing to healthcare. However, if not properly implemented and evaluated, ML pipelines may contain leakage typically resulting in overoptimistic performance estimates and failure to generalize to ne…
▽ More
Machine learning (ML) provides powerful tools for predictive modeling. ML's popularity stems from the promise of sample-level prediction with applications across a variety of fields from physics and marketing to healthcare. However, if not properly implemented and evaluated, ML pipelines may contain leakage typically resulting in overoptimistic performance estimates and failure to generalize to new data. This can have severe negative financial and societal implications. Our aim is to expand understanding associated with causes leading to leakage when designing, implementing, and evaluating ML pipelines. Illustrated by concrete examples, we provide a comprehensive overview and discussion of various types of leakage that may arise in ML pipelines.
△ Less
Submitted 5 March, 2024; v1 submitted 7 November, 2023;
originally announced November 2023.
JTrack: A Digital Biomarker Platform for Remote Monitoring in Neurological and Psychiatric Diseases
Authors:
Mehran Sahandi Far,
Michael Stolz,
Jona M. Fischer,
Simon B. Eickhoff,
Juergen Dukart
Abstract:
Objective: Health-related data being collected by smartphones offer a promising complementary approach to in-clinic assessments. Here we introduce the JTrack platform as a secure, reliable and extendable open-source solution for remote monitoring in daily-life and digital phenotyping. Method: JTrack consists of an Android-based smartphone application and a web-based project management dashboard. A…
▽ More
Objective: Health-related data being collected by smartphones offer a promising complementary approach to in-clinic assessments. Here we introduce the JTrack platform as a secure, reliable and extendable open-source solution for remote monitoring in daily-life and digital phenotyping. Method: JTrack consists of an Android-based smartphone application and a web-based project management dashboard. A wide range of anonymized measurements from motion-sensors, social and physical activities and geolocation information can be collected in either active or passive modes. The dashboard also provides management tools to monitor and manage data collection across studies. To facilitate scaling, reproducibility, data management and sharing we integrated DataLad as a data management infrastructure. JTrack was developed to comply with security, privacy and the General Data Protection Regulation (GDPR) requirements. Results: JTrack is an open-source (released under open-source Apache 2.0 licenses) platform for remote assessment of digital biomarkers (DB) in neurological, psychiatric and other indications. The main components of the JTrack platform and examples of data being collected using JTrack are presented here. Conclusion: Smartphone-based Digital Biomarker data may provide valuable insight into daily life behaviour in health and disease. JTrack provides an easy and reliable open-source solution for collection of such data.
△ Less
Submitted 2 February, 2021; v1 submitted 18 January, 2021;
originally announced January 2021.
