-
Slideflow: Deep Learning for Digital Histopathology with Real-Time Whole-Slide Visualization
Authors:
James M. Dolezal,
Sara Kochanny,
Emma Dyer,
Andrew Srisuwananukorn,
Matteo Sacco,
Frederick M. Howard,
Anran Li,
Prajval Mohan,
Alexander T. Pearson
Abstract:
Deep learning methods have emerged as powerful tools for analyzing histopathological images, but current methods are often specialized for specific domains and software environments, and few open-source options exist for deploying models in an interactive interface. Experimenting with different deep learning approaches typically requires switching software libraries and reprocessing data, reducing…
▽ More
Deep learning methods have emerged as powerful tools for analyzing histopathological images, but current methods are often specialized for specific domains and software environments, and few open-source options exist for deploying models in an interactive interface. Experimenting with different deep learning approaches typically requires switching software libraries and reprocessing data, reducing the feasibility and practicality of experimenting with new architectures. We developed a flexible deep learning library for histopathology called Slideflow, a package which supports a broad array of deep learning methods for digital pathology and includes a fast whole-slide interface for deploying trained models. Slideflow includes unique tools for whole-slide image data processing, efficient stain normalization and augmentation, weakly-supervised whole-slide classification, uncertainty quantification, feature generation, feature space analysis, and explainability. Whole-slide image processing is highly optimized, enabling whole-slide tile extraction at 40X magnification in 2.5 seconds per slide. The framework-agnostic data processing pipeline enables rapid experimentation with new methods built with either Tensorflow or PyTorch, and the graphical user interface supports real-time visualization of slides, predictions, heatmaps, and feature space characteristics on a variety of hardware devices, including ARM-based devices such as the Raspberry Pi.
△ Less
Submitted 8 April, 2023;
originally announced April 2023.
-
Deep Learning Generates Synthetic Cancer Histology for Explainability and Education
Authors:
James M. Dolezal,
Rachelle Wolk,
Hanna M. Hieromnimon,
Frederick M. Howard,
Andrew Srisuwananukorn,
Dmitry Karpeyev,
Siddhi Ramesh,
Sara Kochanny,
Jung Woo Kwon,
Meghana Agni,
Richard C. Simon,
Chandni Desai,
Raghad Kherallah,
Tung D. Nguyen,
Jefree J. Schulte,
Kimberly Cole,
Galina Khramtsova,
Marina Chiara Garassino,
Aliya N. Husain,
Huihua Li,
Robert Grossman,
Nicole A. Cipriani,
Alexander T. Pearson
Abstract:
Artificial intelligence methods including deep neural networks (DNN) can provide rapid molecular classification of tumors from routine histology with accuracy that matches or exceeds human pathologists. Discerning how neural networks make their predictions remains a significant challenge, but explainability tools help provide insights into what models have learned when corresponding histologic fea…
▽ More
Artificial intelligence methods including deep neural networks (DNN) can provide rapid molecular classification of tumors from routine histology with accuracy that matches or exceeds human pathologists. Discerning how neural networks make their predictions remains a significant challenge, but explainability tools help provide insights into what models have learned when corresponding histologic features are poorly defined. Here, we present a method for improving explainability of DNN models using synthetic histology generated by a conditional generative adversarial network (cGAN). We show that cGANs generate high-quality synthetic histology images that can be leveraged for explaining DNN models trained to classify molecularly-subtyped tumors, exposing histologic features associated with molecular state. Fine-tuning synthetic histology through class and layer blending illustrates nuanced morphologic differences between tumor subtypes. Finally, we demonstrate the use of synthetic histology for augmenting pathologist-in-training education, showing that these intuitive visualizations can reinforce and improve understanding of histologic manifestations of tumor biology.
△ Less
Submitted 9 December, 2022; v1 submitted 11 November, 2022;
originally announced November 2022.
-
Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images
Authors:
Alexander Partin,
Thomas Brettin,
Yitan Zhu,
James M. Dolezal,
Sara Kochanny,
Alexander T. Pearson,
Maulik Shukla,
Yvonne A. Evrard,
James H. Doroshow,
Rick L. Stevens
Abstract:
Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies because the in vivo environment of PDXs helps preserve tumor heterogeneity and usually better mimics drug response of patients with cancer compared to CCLs. We investigate multimodal neural network (MM-Net) and data augmentation for drug response prediction in PDXs. The MM-Net learns to predict response using…
▽ More
Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies because the in vivo environment of PDXs helps preserve tumor heterogeneity and usually better mimics drug response of patients with cancer compared to CCLs. We investigate multimodal neural network (MM-Net) and data augmentation for drug response prediction in PDXs. The MM-Net learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs) where the multi-modality refers to the tumor features. We explore whether the integration of WSIs with GE improves predictions as compared with models that use GE alone. We use two methods to address the limited number of response values: 1) homogenize drug representations which allows to combine single-drug and drug-pairs treatments into a single dataset, 2) augment drug-pair samples by switching the order of drug features which doubles the sample size of all drug-pair samples. These methods enable us to combine single-drug and drug-pair treatments, allowing us to train multimodal and unimodal neural networks (NNs) without changing architectures or the dataset. Prediction performance of three unimodal NNs which use GE are compared to assess the contribution of data augmentation methods. NN that uses the full dataset which includes the original and the augmented drug-pair treatments as well as single-drug treatments significantly outperforms NNs that ignore either the augmented drug-pairs or the single-drug treatments. In assessing the contribution of multimodal learning based on the MCC metric, MM-Net statistically significantly outperforms all the baselines. Our results show that data augmentation and integration of histology images with GE can improve prediction performance of drug response in PDXs.
△ Less
Submitted 25 April, 2022;
originally announced April 2022.
-
Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathology
Authors:
James M Dolezal,
Andrew Srisuwananukorn,
Dmitry Karpeyev,
Siddhi Ramesh,
Sara Kochanny,
Brittany Cody,
Aaron Mansfield,
Sagar Rakshit,
Radhika Bansa,
Melanie Bois,
Aaron O Bungum,
Jefree J Schulte,
Everett E Vokes,
Marina Chiara Garassino,
Aliya N Husain,
Alexander T Pearson
Abstract:
A model's ability to express its own predictive uncertainty is an essential attribute for maintaining clinical user confidence as computational biomarkers are deployed into real-world medical settings. In the domain of cancer digital histopathology, we describe a novel, clinically-oriented approach to uncertainty quantification (UQ) for whole-slide images, estimating uncertainty using dropout and…
▽ More
A model's ability to express its own predictive uncertainty is an essential attribute for maintaining clinical user confidence as computational biomarkers are deployed into real-world medical settings. In the domain of cancer digital histopathology, we describe a novel, clinically-oriented approach to uncertainty quantification (UQ) for whole-slide images, estimating uncertainty using dropout and calculating thresholds on training data to establish cutoffs for low- and high-confidence predictions. We train models to identify lung adenocarcinoma vs. squamous cell carcinoma and show that high-confidence predictions outperform predictions without UQ, in both cross-validation and testing on two large external datasets spanning multiple institutions. Our testing strategy closely approximates real-world application, with predictions generated on unsupervised, unannotated slides using predetermined thresholds. Furthermore, we show that UQ thresholding remains reliable in the setting of domain shift, with accurate high-confidence predictions of adenocarcinoma vs. squamous cell carcinoma for out-of-distribution, non-lung cancer cohorts.
△ Less
Submitted 9 April, 2022;
originally announced April 2022.