Computer Science > Machine Learning
[Submitted on 4 Nov 2023 (this version), latest version 20 Jan 2024 (v5)]
Title:Multimodal Machine Learning for Clinically-Assistive Imaging-Based Biomedical Applications
View PDFAbstract:Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models and even more recently generative models. Recent years have seen a rise in the discovery of widely-available deep learning architectures that support multimodal data integration, particularly with images. The incorporation of multiple modalities into these models is a thriving research topic, presenting its own unique challenges. In this work, we discuss five challenges to multimodal AI as it pertains to ML (representation, fusion, alignment, translation, and co-learning) and survey recent approaches to addressing these challenges in the context of medical image-based clinical decision support models. We conclude with a discussion of the future of the field, suggesting directions that should be elucidated further for successful clinical models and their translation to the clinical setting.
Submission history
From: Elisa Warner [view email][v1] Sat, 4 Nov 2023 05:42:51 UTC (1,506 KB)
[v2] Sat, 11 Nov 2023 21:25:29 UTC (1,786 KB)
[v3] Mon, 20 Nov 2023 20:55:29 UTC (1,578 KB)
[v4] Fri, 12 Jan 2024 22:44:04 UTC (2,526 KB)
[v5] Sat, 20 Jan 2024 04:36:50 UTC (2,842 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.