-
Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
Authors:
Jamba Team,
Barak Lenz,
Alan Arazi,
Amir Bergman,
Avshalom Manevich,
Barak Peleg,
Ben Aviram,
Chen Almagor,
Clara Fridman,
Dan Padnos,
Daniel Gissin,
Daniel Jannai,
Dor Muhlgay,
Dor Zimberg,
Edden M Gerber,
Elad Dolev,
Eran Krakovsky,
Erez Safahi,
Erez Schwartz,
Gal Cohen,
Gal Shachaf,
Haim Rozenblum,
Hofit Bata,
Ido Blass,
Inbal Magar
, et al. (36 additional authors not shown)
Abstract:
We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, wi…
▽ More
We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.
△ Less
Submitted 22 August, 2024;
originally announced August 2024.
-
Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling
Authors:
Jaideep Pathak,
Yair Cohen,
Piyush Garg,
Peter Harrington,
Noah Brenowitz,
Dale Durran,
Morteza Mardani,
Arash Vahdat,
Shaoming Xu,
Karthik Kashinath,
Michael Pritchard
Abstract:
Storm-scale convection-allowing models (CAMs) are an important tool for predicting the evolution of thunderstorms and mesoscale convective systems that result in damaging extreme weather. By explicitly resolving convective dynamics within the atmosphere they afford meteorologists the nuance needed to provide outlook on hazard. Deep learning models have thus far not proven skilful at km-scale atmos…
▽ More
Storm-scale convection-allowing models (CAMs) are an important tool for predicting the evolution of thunderstorms and mesoscale convective systems that result in damaging extreme weather. By explicitly resolving convective dynamics within the atmosphere they afford meteorologists the nuance needed to provide outlook on hazard. Deep learning models have thus far not proven skilful at km-scale atmospheric simulation, despite being competitive at coarser resolution with state-of-the-art global, medium-range weather forecasting. We present a generative diffusion model called StormCast, which emulates the high-resolution rapid refresh (HRRR) model-NOAA's state-of-the-art 3km operational CAM. StormCast autoregressively predicts 99 state variables at km scale using a 1-hour time step, with dense vertical resolution in the atmospheric boundary layer, conditioned on 26 synoptic variables. We present evidence of successfully learnt km-scale dynamics including competitive 1-6 hour forecast skill for composite radar reflectivity alongside physically realistic convective cluster evolution, moist updrafts, and cold pool morphology. StormCast predictions maintain realistic power spectra for multiple predicted variables across multi-hour forecasts. Together, these results establish the potential for autoregressive ML to emulate CAMs -- opening up new km-scale frontiers for regional ML weather prediction and future climate hazard dynamical downscaling.
△ Less
Submitted 20 August, 2024;
originally announced August 2024.
-
Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators
Authors:
Ankur Mahesh,
William Collins,
Boris Bonev,
Noah Brenowitz,
Yair Cohen,
Joshua Elms,
Peter Harrington,
Karthik Kashinath,
Thorsten Kurth,
Joshua North,
Travis OBrien,
Michael Pritchard,
David Pruitt,
Mark Risser,
Shashank Subramanian,
Jared Willard
Abstract:
Studying low-likelihood high-impact extreme weather events in a warming world is a significant and challenging task for current ensemble forecasting systems. While these systems presently use up to 100 members, larger ensembles could enrich the sampling of internal variability. They may capture the long tails associated with climate hazards better than traditional ensemble sizes. Due to computatio…
▽ More
Studying low-likelihood high-impact extreme weather events in a warming world is a significant and challenging task for current ensemble forecasting systems. While these systems presently use up to 100 members, larger ensembles could enrich the sampling of internal variability. They may capture the long tails associated with climate hazards better than traditional ensemble sizes. Due to computational constraints, it is infeasible to generate huge ensembles (comprised of 1,000-10,000 members) with traditional, physics-based numerical models. In this two-part paper, we replace traditional numerical simulations with machine learning (ML) to generate hindcasts of huge ensembles. In Part I, we construct an ensemble weather forecasting system based on Spherical Fourier Neural Operators (SFNO), and we discuss important design decisions for constructing such an ensemble. The ensemble represents model uncertainty through perturbed-parameter techniques, and it represents initial condition uncertainty through bred vectors, which sample the fastest growing modes of the forecast. Using the European Centre for Medium-Range Weather Forecasts Integrated Forecasting System (IFS) as a baseline, we develop an evaluation pipeline composed of mean, spectral, and extreme diagnostics. Using large-scale, distributed SFNOs with 1.1 billion learned parameters, we achieve calibrated probabilistic forecasts. As the trajectories of the individual members diverge, the ML ensemble mean spectra degrade with lead time, consistent with physical expectations. However, the individual ensemble members' spectra stay constant with lead time. Therefore, these members simulate realistic weather states, and the ML ensemble thus passes a crucial spectral test in the literature. The IFS and ML ensembles have similar Extreme Forecast Indices, and we show that the ML extreme weather forecasts are reliable and discriminating.
△ Less
Submitted 6 August, 2024;
originally announced August 2024.
-
Huge Ensembles Part II: Properties of a Huge Ensemble of Hindcasts Generated with Spherical Fourier Neural Operators
Authors:
Ankur Mahesh,
William Collins,
Boris Bonev,
Noah Brenowitz,
Yair Cohen,
Peter Harrington,
Karthik Kashinath,
Thorsten Kurth,
Joshua North,
Travis OBrien,
Michael Pritchard,
David Pruitt,
Mark Risser,
Shashank Subramanian,
Jared Willard
Abstract:
In Part I, we created an ensemble based on Spherical Fourier Neural Operators. As initial condition perturbations, we used bred vectors, and as model perturbations, we used multiple checkpoints trained independently from scratch. Based on diagnostics that assess the ensemble's physical fidelity, our ensemble has comparable performance to operational weather forecasting systems. However, it require…
▽ More
In Part I, we created an ensemble based on Spherical Fourier Neural Operators. As initial condition perturbations, we used bred vectors, and as model perturbations, we used multiple checkpoints trained independently from scratch. Based on diagnostics that assess the ensemble's physical fidelity, our ensemble has comparable performance to operational weather forecasting systems. However, it requires several orders of magnitude fewer computational resources. Here in Part II, we generate a huge ensemble (HENS), with 7,424 members initialized each day of summer 2023. We enumerate the technical requirements for running huge ensembles at this scale. HENS precisely samples the tails of the forecast distribution and presents a detailed sampling of internal variability. For extreme climate statistics, HENS samples events 4$σ$ away from the ensemble mean. At each grid cell, HENS improves the skill of the most accurate ensemble member and enhances coverage of possible future trajectories. As a weather forecasting model, HENS issues extreme weather forecasts with better uncertainty quantification. It also reduces the probability of outlier events, in which the verification value lies outside the ensemble forecast distribution.
△ Less
Submitted 2 August, 2024;
originally announced August 2024.
-
Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales
Authors:
Peter Manshausen,
Yair Cohen,
Jaideep Pathak,
Mike Pritchard,
Piyush Garg,
Morteza Mardani,
Karthik Kashinath,
Simon Byrne,
Noah Brenowitz
Abstract:
Data assimilation of observational data into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a…
▽ More
Data assimilation of observational data into full atmospheric states is essential for weather forecast model initialization. Recently, methods for deep generative data assimilation have been proposed which allow for using new input data without retraining the model. They could also dramatically accelerate the costly data assimilation process used in operational regional weather models. Here, in a central US testbed, we demonstrate the viability of score-based data assimilation in the context of realistically complex km-scale weather. We train an unconditional diffusion model to generate snapshots of a state-of-the-art km-scale analysis product, the High Resolution Rapid Refresh. Then, using score-based data assimilation to incorporate sparse weather station data, the model produces maps of precipitation and surface winds. The generated fields display physically plausible structures, such as gust fronts, and sensitivity tests confirm learnt physics through multivariate relationships. Preliminary skill analysis shows the approach already outperforms a naive baseline of the High-Resolution Rapid Refresh system itself. By incorporating observations from 40 weather stations, 10\% lower RMSEs on left-out stations are attained. Despite some lingering imperfections such as insufficiently disperse ensemble DA estimates, we find the results overall an encouraging proof of concept, and the first at km-scale. It is a ripe time to explore extensions that combine increasingly ambitious regional state generators with an increasing set of in situ, ground-based, and satellite remote sensing data streams.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
von Mises Quasi-Processes for Bayesian Circular Regression
Authors:
Yarden Cohen,
Alexandre Khae Wu Navarro,
Jes Frellsen,
Richard E. Turner,
Raziel Riemer,
Ari Pakman
Abstract:
The need for regression models to predict circular values arises in many scientific fields. In this work we explore a family of expressive and interpretable distributions over circle-valued random functions related to Gaussian processes targeting two Euclidean dimensions conditioned on the unit circle. The resulting probability model has connections with continuous spin models in statistical physi…
▽ More
The need for regression models to predict circular values arises in many scientific fields. In this work we explore a family of expressive and interpretable distributions over circle-valued random functions related to Gaussian processes targeting two Euclidean dimensions conditioned on the unit circle. The resulting probability model has connections with continuous spin models in statistical physics. Moreover, its density is very simple and has maximum-entropy, unlike previous Gaussian process-based approaches, which use wrapping or radial marginalization. For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Markov Chain Monte Carlo sampling. We argue that transductive learning in these models favors a Bayesian approach to the parameters. We present experiments applying this model to the prediction of (i) wind directions and (ii) the percentage of the running gait cycle as a function of joint angles.
△ Less
Submitted 18 June, 2024;
originally announced June 2024.
-
Coupled Ocean-Atmosphere Dynamics in a Machine Learning Earth System Model
Authors:
Chenggong Wang,
Michael S. Pritchard,
Noah Brenowitz,
Yair Cohen,
Boris Bonev,
Thorsten Kurth,
Dale Durran,
Jaideep Pathak
Abstract:
Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (…
▽ More
Seasonal climate forecasts are socioeconomically important for managing the impacts of extreme weather events and for planning in sectors like agriculture and energy. Climate predictability on seasonal timescales is tied to boundary effects of the ocean on the atmosphere and coupled interactions in the ocean-atmosphere system. We present the Ocean-linked-atmosphere (Ola) model, a high-resolution (0.25°) Artificial Intelligence/ Machine Learning (AI/ML) coupled earth-system model which separately models the ocean and atmosphere dynamics using an autoregressive Spherical Fourier Neural Operator architecture, with a view towards enabling fast, accurate, large ensemble forecasts on the seasonal timescale. We find that Ola exhibits learned characteristics of ocean-atmosphere coupled dynamics including tropical oceanic waves with appropriate phase speeds, and an internally generated El Niño/Southern Oscillation (ENSO) having realistic amplitude, geographic structure, and vertical structure within the ocean mixed layer. We present initial evidence of skill in forecasting the ENSO which compares favorably to the SPEAR model of the Geophysical Fluid Dynamics Laboratory.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
DiffObs: Generative Diffusion for Global Forecasting of Satellite Observations
Authors:
Jason Stock,
Jaideep Pathak,
Yair Cohen,
Mike Pritchard,
Piyush Garg,
Dale Durran,
Morteza Mardani,
Noah Brenowitz
Abstract:
This work presents an autoregressive generative diffusion model (DiffObs) to predict the global evolution of daily precipitation, trained on a satellite observational product, and assessed with domain-specific diagnostics. The model is trained to probabilistically forecast day-ahead precipitation. Nonetheless, it is stable for multi-month rollouts, which reveal a qualitatively realistic superposit…
▽ More
This work presents an autoregressive generative diffusion model (DiffObs) to predict the global evolution of daily precipitation, trained on a satellite observational product, and assessed with domain-specific diagnostics. The model is trained to probabilistically forecast day-ahead precipitation. Nonetheless, it is stable for multi-month rollouts, which reveal a qualitatively realistic superposition of convectively coupled wave modes in the tropics. Cross-spectral analysis confirms successful generation of low frequency variations associated with the Madden--Julian oscillation, which regulates most subseasonal to seasonal predictability in the observed atmosphere, and convectively coupled moist Kelvin waves with approximately correct dispersion relationships. Despite secondary issues and biases, the results affirm the potential for a next generation of global diffusion models trained on increasingly sparse, and increasingly direct and differentiated observations of the world, for practical applications in subseasonal and climate prediction.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
SINR-Aware Deep Reinforcement Learning for Distributed Dynamic Channel Allocation in Cognitive Interference Networks
Authors:
Yaniv Cohen,
Tomer Gafni,
Ronen Greenberg,
Kobi Cohen
Abstract:
We consider the problem of dynamic channel allocation (DCA) in cognitive communication networks with the goal of maximizing a global signal-to-interference-plus-noise ratio (SINR) measure under a specified target quality of service (QoS)-SINR for each network. The shared bandwidth is partitioned into K channels with frequency separation. In contrast to the majority of existing studies that assume…
▽ More
We consider the problem of dynamic channel allocation (DCA) in cognitive communication networks with the goal of maximizing a global signal-to-interference-plus-noise ratio (SINR) measure under a specified target quality of service (QoS)-SINR for each network. The shared bandwidth is partitioned into K channels with frequency separation. In contrast to the majority of existing studies that assume perfect orthogonality or a one- to-one user-channel allocation mapping, this paper focuses on real-world systems experiencing inter-carrier interference (ICI) and channel reuse by multiple large-scale networks. This realistic scenario significantly increases the problem dimension, rendering existing algorithms inefficient. We propose a novel multi-agent reinforcement learning (RL) framework for distributed DCA, named Channel Allocation RL To Overlapped Networks (CARLTON). The CARLTON framework is based on the Centralized Training with Decentralized Execution (CTDE) paradigm, utilizing the DeepMellow value-based RL algorithm. To ensure robust performance in the interference-laden environment we address, CARLTON employs a low-dimensional representation of observations, generating a QoS-type measure while maximizing a global SINR measure and ensuring the target QoS-SINR for each network. Our results demonstrate exceptional performance and robust generalization, showcasing superior efficiency compared to alternative state-of-the-art methods, while achieving a marginally diminished performance relative to a fully centralized approach.
△ Less
Submitted 17 February, 2024;
originally announced February 2024.
-
A Practical Probabilistic Benchmark for AI Weather Models
Authors:
Noah D. Brenowitz,
Yair Cohen,
Jaideep Pathak,
Ankur Mahesh,
Boris Bonev,
Thorsten Kurth,
Dale R. Durran,
Peter Harrington,
Michael S. Pritchard
Abstract:
Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI wea…
▽ More
Since the weather is chaotic, forecasts aim to predict the distribution of future states rather than make a single prediction. Recently, multiple data driven weather models have emerged claiming breakthroughs in skill. However, these have mostly been benchmarked using deterministic skill scores, and little is known about their probabilistic skill. Unfortunately, it is hard to fairly compare AI weather models in a probabilistic sense, since variations in choice of ensemble initialization, definition of state, and noise injection methodology become confounding. Moreover, even obtaining ensemble forecast baselines is a substantial engineering challenge given the data volumes involved. We sidestep both problems by applying a decades-old idea -- lagged ensembles -- whereby an ensemble can be constructed from a moderately-sized library of deterministic forecasts. This allows the first parameter-free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. The results reveal that two leading AI weather models, i.e. GraphCast and Pangu, are tied on the probabilistic CRPS metric even though the former outperforms the latter in deterministic scoring. We also reveal how multiple time-step loss functions, which many data-driven weather models have employed, are counter-productive: they improve deterministic metrics at the cost of increased dissipation, deteriorating probabilistic skill. This is confirmed through ablations applied to a spherical Fourier Neural Operator (SFNO) approach to AI weather forecasting. Separate SFNO ablations modulating effective resolution reveal it has a useful effect on ensemble dispersion relevant to achieving good ensemble calibration. We hope these and forthcoming insights from lagged ensembles can help guide the development of AI weather forecasts and have thus shared the diagnostic code.
△ Less
Submitted 27 January, 2024;
originally announced January 2024.
-
Residual Corrective Diffusion Modeling for Km-scale Atmospheric Downscaling
Authors:
Morteza Mardani,
Noah Brenowitz,
Yair Cohen,
Jaideep Pathak,
Chieh-Yu Chen,
Cheng-Chin Liu,
Arash Vahdat,
Mohammad Amin Nabian,
Tao Ge,
Akshay Subramaniam,
Karthik Kashinath,
Jan Kautz,
Mike Pritchard
Abstract:
The state of the art for physical hazard prediction from weather and climate requires expensive km-scale numerical simulations driven by coarser resolution global inputs. Here, a generative diffusion architecture is explored for downscaling such global inputs to km-scale, as a cost-effective machine learning alternative. The model is trained to predict 2km data from a regional weather model over T…
▽ More
The state of the art for physical hazard prediction from weather and climate requires expensive km-scale numerical simulations driven by coarser resolution global inputs. Here, a generative diffusion architecture is explored for downscaling such global inputs to km-scale, as a cost-effective machine learning alternative. The model is trained to predict 2km data from a regional weather model over Taiwan, conditioned on a 25km global reanalysis. To address the large resolution ratio, different physics involved at different scales and prediction of channels beyond those in the input data, we employ a two-step approach where a UNet predicts the mean and a corrector diffusion (CorrDiff) model predicts the residual. CorrDiff exhibits encouraging skill in bulk MAE and CRPS scores. The predicted spectra and distributions from CorrDiff faithfully recover important power law relationships in the target data. Case studies of coherent weather phenomena show that CorrDiff can help sharpen wind and temperature gradients that co-locate with intense rainfall in cold front, and can help intensify typhoons and synthesize rain band structures. Calibration of model uncertainty remains challenging. The prospect of unifying methods like CorrDiff with coarser resolution global weather models implies a potential for global-to-regional multi-scale machine learning simulation.
△ Less
Submitted 11 August, 2024; v1 submitted 24 September, 2023;
originally announced September 2023.
-
A Permutation-Based Heuristic for Buy Low, Sell High
Authors:
Yair Neuman,
Yochai Cohen
Abstract:
Buy low, sell high is one of the basic rules of thumb used in investment, although it is not considered to be a beneficial strategy. In this paper, we show how the appropriate permutation-based representation (i.e., the epistemic form) of a minute-by-minute trading time-series, alongside the use of a simple decision heuristic (i.e., the epistemic game), may surprisingly result in significant benef…
▽ More
Buy low, sell high is one of the basic rules of thumb used in investment, although it is not considered to be a beneficial strategy. In this paper, we show how the appropriate permutation-based representation (i.e., the epistemic form) of a minute-by-minute trading time-series, alongside the use of a simple decision heuristic (i.e., the epistemic game), may surprisingly result in significant benefits. Using our heuristic for selecting seven stocks, we ran two experiments on the data. The results provide empirical support for the possible benefit of using simple decision models in investment, even in the context of minute-by-minute trading.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
A Methodology for a Scalable, Collaborative, and Resource-Efficient Platform to Facilitate Healthcare AI Research
Authors:
Raphael Y. Cohen,
Vesela P. Kovacheva
Abstract:
Healthcare AI holds the potential to increase patient safety, augment efficiency and improve patient outcomes, yet research is often limited by data access, cohort curation, and tooling for analysis. Collection and translation of electronic health record data, live data, and real-time high resolution device data can be challenging and time-consuming. The development of real-world AI tools requires…
▽ More
Healthcare AI holds the potential to increase patient safety, augment efficiency and improve patient outcomes, yet research is often limited by data access, cohort curation, and tooling for analysis. Collection and translation of electronic health record data, live data, and real-time high resolution device data can be challenging and time-consuming. The development of real-world AI tools requires overcoming challenges in data acquisition, scarce hospital resources and high needs for data governance. These bottlenecks may result in resource-heavy needs and long delays in research and development of AI systems. We present a system and methodology to accelerate data acquisition, dataset development and analysis, and AI model development. We created an interactive platform that relies on a scalable microservice backend. This system can ingest 15,000 patient records per hour, where each record represents thousands of multimodal measurements, text notes, and high resolution data. Collectively, these records can approach a terabyte of data. The system can further perform cohort generation and preliminary dataset analysis in 2-5 minutes. As a result, multiple users can collaborate simultaneously to iterate on datasets and models in real time. We anticipate that this approach will drive real-world AI model development, and, in the long run, meaningfully improve healthcare delivery.
△ Less
Submitted 13 December, 2021;
originally announced December 2021.
-
Dimensionality Reduction of Longitudinal 'Omics Data using Modern Tensor Factorization
Authors:
Uria Mor,
Yotam Cohen,
Rafael Valdes-Mas,
Denise Kviatcovsky,
Eran Elinav,
Haim Avron
Abstract:
Precision medicine is a clinical approach for disease prevention, detection and treatment, which considers each individual's genetic background, environment and lifestyle. The development of this tailored avenue has been driven by the increased availability of omics methods, large cohorts of temporal samples, and their integration with clinical data. Despite the immense progression, existing compu…
▽ More
Precision medicine is a clinical approach for disease prevention, detection and treatment, which considers each individual's genetic background, environment and lifestyle. The development of this tailored avenue has been driven by the increased availability of omics methods, large cohorts of temporal samples, and their integration with clinical data. Despite the immense progression, existing computational methods for data analysis fail to provide appropriate solutions for this complex, high-dimensional and longitudinal data. In this work we have developed a new method termed TCAM, a dimensionality reduction technique for multi-way data, that overcomes major limitations when doing trajectory analysis of longitudinal omics data. Using real-world data, we show that TCAM outperforms traditional methods, as well as state-of-the-art tensor-based approaches for longitudinal microbiome data analysis. Moreover, we demonstrate the versatility of TCAM by applying it to several different omics datasets, and the applicability of it as a drop-in replacement within straightforward ML tasks.
△ Less
Submitted 28 November, 2021;
originally announced November 2021.
-
An Orchestration Platform that Puts Radiologists in the Driver's Seat of AI Innovation: A Methodological Approach
Authors:
Raphael Y. Cohen,
Aaron D. Sodickson
Abstract:
Current AI-driven research in radiology requires resources and expertise that are often inaccessible to small and resource-limited labs. The clinicians who are able to participate in AI research are frequently well-funded, well-staffed, and either have significant experience with AI and computing, or have access to colleagues or facilities that do. Current imaging data is clinician-oriented and is…
▽ More
Current AI-driven research in radiology requires resources and expertise that are often inaccessible to small and resource-limited labs. The clinicians who are able to participate in AI research are frequently well-funded, well-staffed, and either have significant experience with AI and computing, or have access to colleagues or facilities that do. Current imaging data is clinician-oriented and is not easily amenable to machine learning initiatives, resulting in inefficient, time consuming, and costly efforts that rely upon a crew of data engineers and machine learning scientists, and all too often preclude radiologists from driving AI research and innovation. We present the system and methodology we have developed to address infrastructure and platform needs, while reducing the staffing and resource barriers to entry. We emphasize a data-first and modular approach that streamlines the AI development and deployment process while providing efficient and familiar interfaces for radiologists, such that they can be the drivers of new AI innovations.
△ Less
Submitted 6 July, 2021;
originally announced July 2021.
-
Interactive Extractive Search over Biomedical Corpora
Authors:
Hillel Taub-Tabib,
Micah Shlain,
Shoval Sadde,
Dan Lahav,
Matan Eyal,
Yaara Cohen,
Yoav Goldberg
Abstract:
We present a system that allows life-science researchers to search a linguistically annotated corpus of scientific texts using patterns over dependency graphs, as well as using patterns over token sequences and a powerful variant of boolean keyword queries. In contrast to previous attempts to dependency-based search, we introduce a light-weight query language that does not require the user to know…
▽ More
We present a system that allows life-science researchers to search a linguistically annotated corpus of scientific texts using patterns over dependency graphs, as well as using patterns over token sequences and a powerful variant of boolean keyword queries. In contrast to previous attempts to dependency-based search, we introduce a light-weight query language that does not require the user to know the details of the underlying linguistic representations, and instead to query the corpus by providing an example sentence coupled with simple markup. Search is performed at an interactive speed due to efficient linguistic graph-indexing and retrieval engine. This allows for rapid exploration, development and refinement of user queries. We demonstrate the system using example workflows over two corpora: the PubMed corpus including 14,446,243 PubMed abstracts and the CORD-19 dataset, a collection of over 45,000 research papers focused on COVID-19 research. The system is publicly available at https://allenai.github.io/spike
△ Less
Submitted 7 June, 2020;
originally announced June 2020.
-
Forgotten @ Scale: A Practical Solution for Implementing the Right To Be Forgotten in Large-Scale Systems
Authors:
Abigail Goldsteen,
Tomer Douek,
Yaniv Cohen,
Igor Gokhman,
Ofir Keren-Ackerman,
Gadi Katsovich,
Grisha Weintraub,
Doron Ben-Ari
Abstract:
The European General Data Protection Regulation asserts data subjects' right to be forgotten, i.e., their right to request that all their personal data be deleted from an organizations' data stores. However, fulfilling such requests in large-scale systems is technically challenging. It requires that organizations keep track of all locations in which an individual's data is stored, be able to acces…
▽ More
The European General Data Protection Regulation asserts data subjects' right to be forgotten, i.e., their right to request that all their personal data be deleted from an organizations' data stores. However, fulfilling such requests in large-scale systems is technically challenging. It requires that organizations keep track of all locations in which an individual's data is stored, be able to access and delete it in a reasonable time frame, and be able to prove that all such data was in fact deleted. In addition, organizations must cope with complexities such as multiple, distributed, and continuously evolving systems of record, complex data retention policies and deletion approval workflows. We present a first design pattern and practical implementation of the right to be forgotten on a large scale in Big Data and cloud environments.
△ Less
Submitted 30 October, 2019;
originally announced October 2019.
-
Node-Centric Detection of Overlapping Communities in Social Networks
Authors:
Yehonatan Cohen,
Danny Hendler,
Amir Rubin
Abstract:
We present NECTAR, a community detection algorithm that generalizes Louvain method's local search heuristic for overlapping community structures. NECTAR chooses dynamically which objective function to optimize based on the network on which it is invoked. Our experimental evaluation on both synthetic benchmark graphs and real-world networks, based on ground-truth communities, shows that NECTAR prov…
▽ More
We present NECTAR, a community detection algorithm that generalizes Louvain method's local search heuristic for overlapping community structures. NECTAR chooses dynamically which objective function to optimize based on the network on which it is invoked. Our experimental evaluation on both synthetic benchmark graphs and real-world networks, based on ground-truth communities, shows that NECTAR provides excellent results as compared with state of the art community detection algorithms.
△ Less
Submitted 6 July, 2016;
originally announced July 2016.
-
Mining The Successful Binary Combinations: Methodology and A Simple Case Study
Authors:
Yuval Cohen
Abstract:
The importance of finding the characteristics leading to either a success or a failure is one of the driving forces of data mining. The various application areas of finding success/failure factors cover vast variety of areas such as credit risk evaluation and granting loans, micro array analysis, health factors and health risk factors, and parameter combination leading to a product success. This…
▽ More
The importance of finding the characteristics leading to either a success or a failure is one of the driving forces of data mining. The various application areas of finding success/failure factors cover vast variety of areas such as credit risk evaluation and granting loans, micro array analysis, health factors and health risk factors, and parameter combination leading to a product success. This paper presents a new approach for making inferences about dichotomous data. The objective is to determine rules that lead to a certain result. The method consists of four phases: in the first phase, the data is processed into a binary format of a truth table, in the second phase; rules are found by utilizing an algorithm that minimizes Boolean functions. In the third phase the rules are checked and filtered. In the fourth phase, simple rules that involve one to two features are revealed.
△ Less
Submitted 5 February, 2010;
originally announced February 2010.