-
Is user feedback always informative? Retrieval Latent Defending for Semi-Supervised Domain Adaptation without Source Data
Authors:
Junha Song,
Tae Soo Kim,
Junha Kim,
Gunhee Nam,
Thijs Kooi,
Jaegul Choo
Abstract:
This paper aims to adapt the source model to the target environment, leveraging small user feedback (i.e., labeled target data) readily available in real-world applications. We find that existing semi-supervised domain adaptation (SemiSDA) methods often suffer from poorly improved adaptation performance when directly utilizing such feedback data, as shown in Figure 1. We analyze this phenomenon vi…
▽ More
This paper aims to adapt the source model to the target environment, leveraging small user feedback (i.e., labeled target data) readily available in real-world applications. We find that existing semi-supervised domain adaptation (SemiSDA) methods often suffer from poorly improved adaptation performance when directly utilizing such feedback data, as shown in Figure 1. We analyze this phenomenon via a novel concept called Negatively Biased Feedback (NBF), which stems from the observation that user feedback is more likely for data points where the model produces incorrect predictions. To leverage this feedback while avoiding the issue, we propose a scalable adapting approach, Retrieval Latent Defending. This approach helps existing SemiSDA methods to adapt the model with a balanced supervised signal by utilizing latent defending samples throughout the adaptation process. We demonstrate the problem caused by NBF and the efficacy of our approach across various benchmarks, including image classification, semantic segmentation, and a real-world medical imaging application. Our extensive experiments reveal that integrating our approach with multiple state-of-the-art SemiSDA methods leads to significant performance improvements.
△ Less
Submitted 22 July, 2024;
originally announced July 2024.
-
ELVIS: Empowering Locality of Vision Language Pre-training with Intra-modal Similarity
Authors:
Sumin Seo,
JaeWoong Shin,
Jaewoo Kang,
Tae Soo Kim,
Thijs Kooi
Abstract:
Deep learning has shown great potential in assisting radiologists in reading chest X-ray (CXR) images, but its need for expensive annotations for improving performance prevents widespread clinical application. Visual language pre-training (VLP) can alleviate the burden and cost of annotation by leveraging routinely generated reports for radiographs, which exist in large quantities as well as in pa…
▽ More
Deep learning has shown great potential in assisting radiologists in reading chest X-ray (CXR) images, but its need for expensive annotations for improving performance prevents widespread clinical application. Visual language pre-training (VLP) can alleviate the burden and cost of annotation by leveraging routinely generated reports for radiographs, which exist in large quantities as well as in paired form (image-text pairs). Additionally, extensions to localization-aware VLPs are being proposed to address the needs for accurate localization of abnormalities for computer-aided diagnosis (CAD) in CXR. However, we find that the formulation proposed by locality-aware VLP literature actually leads to a loss in spatial relationships required for downstream localization tasks. Therefore, we propose Empowering Locality of VLP with Intra-modal Similarity, ELVIS, a VLP aware of intra-modal locality, to better preserve the locality within radiographs or reports, which enhances the ability to comprehend location references in text reports. Our locality-aware VLP method significantly outperforms state-of-the art baselines in multiple segmentation tasks and the MS-CXR phrase grounding task. Qualitatively, we show that ELVIS focuses well on regions of interest described in the report text compared to prior approaches, allowing for enhanced interpretability.
△ Less
Submitted 23 July, 2023; v1 submitted 11 April, 2023;
originally announced April 2023.
-
Enhancing Breast Cancer Risk Prediction by Incorporating Prior Images
Authors:
Hyeonsoo Lee,
Junha Kim,
Eunkyung Park,
Minjeong Kim,
Taesoo Kim,
Thijs Kooi
Abstract:
Recently, deep learning models have shown the potential to predict breast cancer risk and enable targeted screening strategies, but current models do not consider the change in the breast over time. In this paper, we present a new method, PRIME+, for breast cancer risk prediction that leverages prior mammograms using a transformer decoder, outperforming a state-of-the-art risk prediction method th…
▽ More
Recently, deep learning models have shown the potential to predict breast cancer risk and enable targeted screening strategies, but current models do not consider the change in the breast over time. In this paper, we present a new method, PRIME+, for breast cancer risk prediction that leverages prior mammograms using a transformer decoder, outperforming a state-of-the-art risk prediction method that only uses mammograms from a single time point. We validate our approach on a dataset with 16,113 exams and further demonstrate that it effectively captures patterns of changes from prior mammograms, such as changes in breast density, resulting in improved short-term and long-term breast cancer risk prediction. Experimental results show that our model achieves a statistically significant improvement in performance over the state-of-the-art based model, with a C-index increase from 0.68 to 0.73 (p < 0.05) on held-out test sets.
△ Less
Submitted 28 August, 2023; v1 submitted 27 March, 2023;
originally announced March 2023.
-
Understanding metric-related pitfalls in image analysis validation
Authors:
Annika Reinke,
Minu D. Tizabi,
Michael Baumgartner,
Matthias Eisenmann,
Doreen Heckmann-Nötzel,
A. Emre Kavur,
Tim Rädsch,
Carole H. Sudre,
Laura Acion,
Michela Antonelli,
Tal Arbel,
Spyridon Bakas,
Arriel Benis,
Matthew Blaschko,
Florian Buettner,
M. Jorge Cardoso,
Veronika Cheplygina,
Jianxu Chen,
Evangelia Christodoulou,
Beth A. Cimini,
Gary S. Collins,
Keyvan Farahani,
Luciana Ferrer,
Adrian Galdran,
Bram van Ginneken
, et al. (53 additional authors not shown)
Abstract:
Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibilit…
▽ More
Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.
△ Less
Submitted 23 February, 2024; v1 submitted 3 February, 2023;
originally announced February 2023.
-
OOOE: Only-One-Object-Exists Assumption to Find Very Small Objects in Chest Radiographs
Authors:
Gunhee Nam,
Taesoo Kim,
Sanghyup Lee,
Thijs Kooi
Abstract:
The accurate localization of inserted medical tubes and parts of human anatomy is a common problem when analyzing chest radiographs and something deep neural networks could potentially automate. However, many foreign objects like tubes and various anatomical structures are small in comparison to the entire chest X-ray, which leads to severely unbalanced data and makes training deep neural networks…
▽ More
The accurate localization of inserted medical tubes and parts of human anatomy is a common problem when analyzing chest radiographs and something deep neural networks could potentially automate. However, many foreign objects like tubes and various anatomical structures are small in comparison to the entire chest X-ray, which leads to severely unbalanced data and makes training deep neural networks difficult. In this paper, we present a simple yet effective `Only-One-Object-Exists' (OOOE) assumption to improve the deep network's ability to localize small landmarks in chest radiographs. The OOOE enables us to recast the localization problem as a classification problem and we can replace commonly used continuous regression techniques with a multi-class discrete objective. We validate our approach using a large scale proprietary dataset of over 100K radiographs as well as publicly available RANZCR-CLiP Kaggle Challenge dataset and show that our method consistently outperforms commonly used regression-based detection models as well as commonly used pixel-wise classification methods. Additionally, we find that the method using the OOOE assumption generalizes to multiple detection problems in chest X-rays and the resulting model shows state-of-the-art performance on detecting various tube tips inserted to the patient as well as patient anatomy.
△ Less
Submitted 13 October, 2022;
originally announced October 2022.
-
Did You Get What You Paid For? Rethinking Annotation Cost of Deep Learning Based Computer Aided Detection in Chest Radiographs
Authors:
Tae Soo Kim,
Geonwoon Jang,
Sanghyup Lee,
Thijs Kooi
Abstract:
As deep networks require large amounts of accurately labeled training data, a strategy to collect sufficiently large and accurate annotations is as important as innovations in recognition methods. This is especially true for building Computer Aided Detection (CAD) systems for chest X-rays where domain expertise of radiologists is required to annotate the presence and location of abnormalities on X…
▽ More
As deep networks require large amounts of accurately labeled training data, a strategy to collect sufficiently large and accurate annotations is as important as innovations in recognition methods. This is especially true for building Computer Aided Detection (CAD) systems for chest X-rays where domain expertise of radiologists is required to annotate the presence and location of abnormalities on X-ray images. However, there lacks concrete evidence that provides guidance on how much resource to allocate for data annotation such that the resulting CAD system reaches desired performance. Without this knowledge, practitioners often fall back to the strategy of collecting as much detail as possible on as much data as possible which is cost inefficient. In this work, we investigate how the cost of data annotation ultimately impacts the CAD model performance on classification and segmentation of chest abnormalities in frontal-view X-ray images. We define the cost of annotation with respect to the following three dimensions: quantity, quality and granularity of labels. Throughout this study, we isolate the impact of each dimension on the resulting CAD model performance on detecting 10 chest abnormalities in X-rays. On a large scale training data with over 120K X-ray images with gold-standard annotations, we find that cost-efficient annotations provide great value when collected in large amounts and lead to competitive performance when compared to models trained with only gold-standard annotations. We also find that combining large amounts of cost efficient annotations with only small amounts of expensive labels leads to competitive CAD models at a much lower cost.
△ Less
Submitted 30 September, 2022;
originally announced September 2022.
-
Tutorial on the development of AI models for medical image analysis
Authors:
Thijs Kooi
Abstract:
The idea of using computers to read medical scans was introduced as early as 1966. However, limits to machine learning technology meant progress was slow initially. The Alexnet breakthrough in 2012 sparked new interest in the topic, which resulted in the release of 100s of medical AI solutions on the market. In spite of success for some diseases and modalities, many challenges remain. Research typ…
▽ More
The idea of using computers to read medical scans was introduced as early as 1966. However, limits to machine learning technology meant progress was slow initially. The Alexnet breakthrough in 2012 sparked new interest in the topic, which resulted in the release of 100s of medical AI solutions on the market. In spite of success for some diseases and modalities, many challenges remain. Research typically focuses on the development of specific applications or techniques, clinical evaluation, or meta analysis of clinical studies or techniques through surveys or challenges. However, limited attention has been given to the development process of improving real world performance. In this tutorial, we address the latter and discuss some techniques to conduct the development process in order to make this as efficient as possible.
△ Less
Submitted 14 July, 2022;
originally announced August 2022.
-
Common Limitations of Image Processing Metrics: A Picture Story
Authors:
Annika Reinke,
Minu D. Tizabi,
Carole H. Sudre,
Matthias Eisenmann,
Tim Rädsch,
Michael Baumgartner,
Laura Acion,
Michela Antonelli,
Tal Arbel,
Spyridon Bakas,
Peter Bankhead,
Arriel Benis,
Matthew Blaschko,
Florian Buettner,
M. Jorge Cardoso,
Jianxu Chen,
Veronika Cheplygina,
Evangelia Christodoulou,
Beth Cimini,
Gary S. Collins,
Sandy Engelhardt,
Keyvan Farahani,
Luciana Ferrer,
Adrian Galdran,
Bram van Ginneken
, et al. (68 additional authors not shown)
Abstract:
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using spe…
▽ More
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
△ Less
Submitted 6 December, 2023; v1 submitted 12 April, 2021;
originally announced April 2021.
-
Error Exponents for Asynchronous Multiple Access Channels. Controlled Asynchronism may Outperform Synchronism
Authors:
Lóránt Farkas,
Tamás Kói
Abstract:
Exponential error bounds achievable by universal coding and decoding are derived for frame-asynchronous discrete memoryless %asynchronous multiple access channels with two senders, via the method of subtypes, a refinement of the method of types. Maximum empirical multi-information decoding is employed. A key tool is an improved packing lemma, that overcomes the technical difficulty caused by codew…
▽ More
Exponential error bounds achievable by universal coding and decoding are derived for frame-asynchronous discrete memoryless %asynchronous multiple access channels with two senders, via the method of subtypes, a refinement of the method of types. Maximum empirical multi-information decoding is employed. A key tool is an improved packing lemma, that overcomes the technical difficulty caused by codeword repetitions, via an induction based new argument. The asymptotic form of the bounds admits numerical evaluation. This demostrates that error exponents achievable by synchronous transmission (if possible) can be superseeded via controlled asynchronism, i.e. a deliberate shift of the codewords.
△ Less
Submitted 3 February, 2020; v1 submitted 11 July, 2019;
originally announced July 2019.
-
Classifying Symmetrical Differences and Temporal Change in Mammography Using Deep Neural Networks
Authors:
Thijs Kooi,
Nico Karssemeijer
Abstract:
We investigate the addition of symmetry and temporal context information to a deep Convolutional Neural Network (CNN) with the purpose of detecting malignant soft tissue lesions in mammography. We employ a simple linear mapping that takes the location of a mass candidate and maps it to either the contra-lateral or prior mammogram and Regions Of Interest (ROI) are extracted around each location. We…
▽ More
We investigate the addition of symmetry and temporal context information to a deep Convolutional Neural Network (CNN) with the purpose of detecting malignant soft tissue lesions in mammography. We employ a simple linear mapping that takes the location of a mass candidate and maps it to either the contra-lateral or prior mammogram and Regions Of Interest (ROI) are extracted around each location. We subsequently explore two different architectures (1) a fusion model employing two datastreams were both ROIs are fed to the network during training and testing and (2) a stage-wise approach where a single ROI CNN is trained on the primary image and subsequently used as feature extractor for both primary and symmetrical or prior ROIs. A 'shallow' Gradient Boosted Tree (GBT) classifier is then trained on the concatenation of these features and used to classify the joint representation. Results shown a significant increase in performance using the first architecture and symmetry information, but only marginal gains in performance using temporal data and the other setting. We feel results are promising and can greatly be improved when more temporal data becomes available.
△ Less
Submitted 1 August, 2017; v1 submitted 22 March, 2017;
originally announced March 2017.
-
A Survey on Deep Learning in Medical Image Analysis
Authors:
Geert Litjens,
Thijs Kooi,
Babak Ehteshami Bejnordi,
Arnaud Arindra Adiyoso Setio,
Francesco Ciompi,
Mohsen Ghafoorian,
Jeroen A. W. M. van der Laak,
Bram van Ginneken,
Clara I. Sánchez
Abstract:
Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, se…
▽ More
Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.
△ Less
Submitted 4 June, 2017; v1 submitted 19 February, 2017;
originally announced February 2017.
-
Universal Random Access Error Exponents for Codebooks of Different Word-Lengths
Authors:
Lóránt Farkas,
Tamás Kói
Abstract:
Csiszár's channel coding theorem for multiple codebooks is generalized allowing the codeword lenghts differ across codebooks. Also in this case, for each codebook an error exponent can be achieved that equals the random coding exponent for this codebook alone, in addition, erasure detection failure probability tends to 0. This is proved even for sender and receiver not knowing the channel. As a co…
▽ More
Csiszár's channel coding theorem for multiple codebooks is generalized allowing the codeword lenghts differ across codebooks. Also in this case, for each codebook an error exponent can be achieved that equals the random coding exponent for this codebook alone, in addition, erasure detection failure probability tends to 0. This is proved even for sender and receiver not knowing the channel. As a corollary, a substantial improvement is obtained when the sender knows the channel.
△ Less
Submitted 22 January, 2017; v1 submitted 7 July, 2016;
originally announced July 2016.
-
Random Access and Source-Channel Coding Error Exponents for Multiple Access Channels
Authors:
Lóránt Farkas,
Tamás Kói
Abstract:
A new universal coding/decoding scheme for random access with collision detection is given in the case of two senders. The result is used to give an achievable joint source-channel coding error exponent for multiple access channels in the case of independent sources. This exponent is improved in a modified model that admits error free 0 rate communication between the senders.
A new universal coding/decoding scheme for random access with collision detection is given in the case of two senders. The result is used to give an achievable joint source-channel coding error exponent for multiple access channels in the case of independent sources. This exponent is improved in a modified model that admits error free 0 rate communication between the senders.
△ Less
Submitted 18 September, 2013; v1 submitted 27 January, 2013;
originally announced January 2013.
-
On Capacity Regions of Discrete Asynchronous Multiple Access Channels
Authors:
Lóránt Farkas,
Tamás Kói
Abstract:
A general formalization is given for asynchronous multiple access channels which admits different assumptions on delays. This general framework allows the analysis of so far unexplored models leading to new interesting capacity regions. In particular, a single letter characterization is given for the capacity region in case of 3 senders, 2 synchronous with each other and the third not synchronous…
▽ More
A general formalization is given for asynchronous multiple access channels which admits different assumptions on delays. This general framework allows the analysis of so far unexplored models leading to new interesting capacity regions. In particular, a single letter characterization is given for the capacity region in case of 3 senders, 2 synchronous with each other and the third not synchronous with them.
△ Less
Submitted 12 July, 2014; v1 submitted 11 April, 2012;
originally announced April 2012.
