Showing 1–10 of 10 results for author: Dagan, O

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

Authors: Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar , et al. (36 additional authors not shown)

Abstract: We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, wi… ▽ More We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source. △ Less

Submitted 22 August, 2024; originally announced August 2024.

Comments: Webpage: https://www.ai21.com/jamba

Scalable Factor Graph-Based Heterogeneous Bayesian DDF for Dynamic Systems

Authors: Ofer Dagan, Tycho L. Cinquini, Nisar R. Ahmed

Abstract: Heterogeneous Bayesian decentralized data fusion captures the set of problems in which two robots must combine two probability density functions over non-equal, but overlapping sets of random variables. In the context of multi-robot dynamic systems, this enables robots to take a "divide and conquer" approach to reason and share data over complementary tasks instead of over the full joint state spa… ▽ More Heterogeneous Bayesian decentralized data fusion captures the set of problems in which two robots must combine two probability density functions over non-equal, but overlapping sets of random variables. In the context of multi-robot dynamic systems, this enables robots to take a "divide and conquer" approach to reason and share data over complementary tasks instead of over the full joint state space. For example, in a target tracking application, this allows robots to track different subsets of targets and share data on only common targets. This paper presents a framework by which robots can each use a local factor graph to represent relevant partitions of a complex global joint probability distribution, thus allowing them to avoid reasoning over the entirety of a more complex model and saving communication as well as computation costs. From a theoretical point of view, this paper makes contributions by casting the heterogeneous decentralized fusion problem in terms of a factor graph, analyzing the challenges that arise due to dynamic filtering, and then developing a new conservative filtering algorithm that ensures statistical correctness. From a practical point of view, we show how this framework can be used to represent different multi-robot applications and then test it with simulations and hardware experiments to validate and demonstrate its statistical conservativeness, applicability, and robustness to real-world challenges. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: 15 pages, 13 figures, submitted for review at IEEE Transactions on Robotics (T-RO)

Using Surprise Index for Competency Assessment in Autonomous Decision-Making

Authors: Akash Ratheesh, Ofer Dagan, Nisar R. Ahmed, Jay McMahon

Abstract: This paper considers the problem of evaluating an autonomous system's competency in performing a task, particularly when working in dynamic and uncertain environments. The inherent opacity of machine learning models, from the perspective of the user, often described as a `black box', poses a challenge. To overcome this, we propose using a measure called the Surprise index, which leverages availabl… ▽ More This paper considers the problem of evaluating an autonomous system's competency in performing a task, particularly when working in dynamic and uncertain environments. The inherent opacity of machine learning models, from the perspective of the user, often described as a `black box', poses a challenge. To overcome this, we propose using a measure called the Surprise index, which leverages available measurement data to quantify whether the dynamic system performs as expected. We show that the surprise index can be computed in closed form for dynamic systems when observed evidence in a probabilistic model if the joint distribution for that evidence follows a multivariate Gaussian marginal distribution. We then apply it to a nonlinear spacecraft maneuver problem, where actions are chosen by a reinforcement learning agent and show it can indicate how well the trajectory follows the required orbit. △ Less

Submitted 10 January, 2024; v1 submitted 14 December, 2023; originally announced December 2023.

Comments: 10 pages, 5 figures, presented at AIAA SciTech 2024

Exploiting Structure for Optimal Multi-Agent Bayesian Decentralized Estimation

Authors: Christopher Funk, Ofer Dagan, Benjamin Noack, Nisar R. Ahmed

Abstract: A key challenge in Bayesian decentralized data fusion is the `rumor propagation' or `double counting' phenomenon, where previously sent data circulates back to its sender. It is often addressed by approximate methods like covariance intersection (CI) which takes a weighted average of the estimates to compute the bound. The problem is that this bound is not tight, i.e. the estimate is often over-co… ▽ More A key challenge in Bayesian decentralized data fusion is the `rumor propagation' or `double counting' phenomenon, where previously sent data circulates back to its sender. It is often addressed by approximate methods like covariance intersection (CI) which takes a weighted average of the estimates to compute the bound. The problem is that this bound is not tight, i.e. the estimate is often over-conservative. In this paper, we show that by exploiting the probabilistic independence structure in multi-agent decentralized fusion problems a tighter bound can be found using (i) an expansion to the CI algorithm that uses multiple (non-monolithic) weighting factors instead of one (monolithic) factor in the original CI and (ii) a general optimization scheme that is able to compute optimal bounds and fully exploit an arbitrary dependency structure. We compare our methods and show that on a simple problem, they converge to the same solution. We then test our new non-monolithic CI algorithm on a large-scale target tracking simulation and show that it achieves a tighter bound and a more accurate estimate compared to the original monolithic CI. △ Less

Submitted 20 July, 2023; originally announced July 2023.

Comments: 4 pages, 4 figures. presented at the Inference and Decision Making for Autonomous Vehicles (IDMAV) RSS 2023 workshop

Towards Decentralized Heterogeneous Multi-Robot SLAM and Target Tracking

Authors: Ofer Dagan, Tycho L. Cinquini, Luke Morrissey, Kristen Such, Nisar R. Ahmed, Christoffer Heckman

Abstract: In many robotics problems, there is a significant gain in collaborative information sharing between multiple robots, for exploration, search and rescue, tracking multiple targets, or mapping large environments. One of the key implicit assumptions when solving cooperative multi-robot problems is that all robots use the same (homogeneous) underlying algorithm. However, in practice, we want to allow… ▽ More In many robotics problems, there is a significant gain in collaborative information sharing between multiple robots, for exploration, search and rescue, tracking multiple targets, or mapping large environments. One of the key implicit assumptions when solving cooperative multi-robot problems is that all robots use the same (homogeneous) underlying algorithm. However, in practice, we want to allow collaboration between robots possessing different capabilities and that therefore must rely on heterogeneous algorithms. We present a system architecture and the supporting theory, to enable collaboration in a decentralized network of robots, where each robot relies on different estimation algorithms. To develop our approach, we focus on multi-robot simultaneous localization and mapping (SLAM) with multi-target tracking. Our theoretical framework builds on our idea of exploiting the conditional independence structure inherent to many robotics applications to separate between each robot's local inference (estimation) tasks and fuse only relevant parts of their non-equal, but overlapping probability density function (pdfs). We present a new decentralized graph-based approach to the multi-robot SLAM and tracking problem. We leverage factor graphs to split between different parts of the problem for efficient data sharing between robots in the network while enabling robots to use different local sparse landmark/dense/metric-semantic SLAM algorithms. △ Less

Submitted 7 June, 2023; originally announced June 2023.

Comments: 5 pages, 2 figures, presented at the ICRA 2023 workshop on "Distributed Graph Algorithms for Robotics"

Nonlinear Heterogeneous Bayesian Decentralized Data Fusion

Authors: Ofer Dagan, Tycho L. Cinquini, Nisar R. Ahmed

Abstract: The factor graph decentralized data fusion (FG-DDF) framework was developed for the analysis and exploitation of conditional independence in {heterogeneous Bayesian decentralized fusion problems, in which robots update and fuse pdfs over different, but overlapping subsets of random states. This allows robots to efficiently use smaller probabilistic models and sparse message passing to accurately a… ▽ More The factor graph decentralized data fusion (FG-DDF) framework was developed for the analysis and exploitation of conditional independence in {heterogeneous Bayesian decentralized fusion problems, in which robots update and fuse pdfs over different, but overlapping subsets of random states. This allows robots to efficiently use smaller probabilistic models and sparse message passing to accurately and scalably fuse relevant local parts of a larger global joint state pdf while accounting for data dependencies between robots. Whereas prior work required limiting assumptions about network connectivity and model linearity, this paper relaxes these to explore the applicability and robustness of FG-DDF in more general settings. We develop a new heterogeneous fusion rule which generalizes the homogeneous covariance intersection algorithm for such cases and test it in multi-robot tracking and localization scenarios with non-linear motion/observation models under communication dropouts. Simulation and hardware experiments show that, in practice, the FG-DDF continues to provide consistent filtered estimates under these more practical operating conditions, while reducing computation and communication costs by more than 99\%, thus enabling the design of scalable real-world multi-robot systems. △ Less

Submitted 25 September, 2023; v1 submitted 17 September, 2022; originally announced September 2022.

Comments: 7 pages, 3 figures, 3 tables, submitted to IEEE/RSJ international Conference on intelligent Robots and Systems (IROS 2023). Replacing previous version to account to reviewers' comments, this is the version presented at IROS 2023

Conservative Filtering for Heterogeneous Decentralized Data Fusion in Dynamic Robotic Systems

Authors: Ofer Dagan, Nisar R. Ahmed

Abstract: This paper presents a method for Bayesian multi-robot peer-to-peer data fusion where any pair of autonomous robots hold non-identical, but overlapping parts of a global joint probability distribution, representing real world inference tasks (e.g., mapping, tracking). It is shown that in dynamic stochastic systems, filtering, which corresponds to marginalization of past variables, results in direct… ▽ More This paper presents a method for Bayesian multi-robot peer-to-peer data fusion where any pair of autonomous robots hold non-identical, but overlapping parts of a global joint probability distribution, representing real world inference tasks (e.g., mapping, tracking). It is shown that in dynamic stochastic systems, filtering, which corresponds to marginalization of past variables, results in direct and hidden dependencies between variables not mutually monitored by the robots, which might lead to an overconfident fused estimate. The paper makes both theoretical and practical contributions by providing (i) a rigorous analysis of the origin of the dependencies and and (ii) a conservative filtering algorithm for heterogeneous data fusion in dynamic systems that can be integrated with existing fusion algorithms. This work uses factor graphs as an analysis tool and an inference engine. Each robot in the network maintains a local factor graph and communicates only relevant parts of it (a sub-graph) to its neighboring robot. We discuss the applicability to various multi-robot robotic applications and demonstrate the performance using a multi-robot multi-target tracking simulation, showing that the proposed algorithm produces conservative estimates at each robot. △ Less

Submitted 14 March, 2022; originally announced March 2022.

Comments: 8 pages, 5 figures, submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022)

Journal ref: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Factor Graphs for Heterogeneous Bayesian Decentralized Data Fusion

Authors: Ofer Dagan, Nisar R. Ahmed

Abstract: This paper explores the use of factor graphs as an inference and analysis tool for Bayesian peer-to-peer decentralized data fusion. We propose a framework by which agents can each use local factor graphs to represent relevant partitions of a complex global joint probability distribution, thus allowing them to avoid reasoning over the entirety of a more complex model and saving communication as wel… ▽ More This paper explores the use of factor graphs as an inference and analysis tool for Bayesian peer-to-peer decentralized data fusion. We propose a framework by which agents can each use local factor graphs to represent relevant partitions of a complex global joint probability distribution, thus allowing them to avoid reasoning over the entirety of a more complex model and saving communication as well as computation cost. This allows heterogeneous multi-robot systems to cooperate on a variety of real world, task oriented missions, where scalability and modularity are key. To develop the initial theory and analyze the limits of this approach, we focus our attention on static linear Gaussian systems in tree-structured networks and use Channel Filters (also represented by factor graphs) to explicitly track common information. We discuss how this representation can be used to describe various multi-robot applications and to design and analyze new heterogeneous data fusion algorithms. We validate our method in simulations of a multi-agent multi-target tracking and cooperative multi-agent mapping problems, and discuss the computation and communication gains of this approach. △ Less

Submitted 24 June, 2021; originally announced June 2021.

Comments: 8 pages, 6 figures, 1 table, submitted to the 24th International Conference on Information Fusion

Exact and Approximate Heterogeneous Bayesian Decentralized Data Fusion

Authors: Ofer Dagan, Nisar R. Ahmed

Abstract: In Bayesian peer-to-peer decentralized data fusion, the underlying distributions held locally by autonomous agents are frequently assumed to be over the same set of variables (homogeneous). This requires each agent to process and communicate the full global joint distribution, and thus leads to high computation and communication costs irrespective of relevancy to specific local objectives. This wo… ▽ More In Bayesian peer-to-peer decentralized data fusion, the underlying distributions held locally by autonomous agents are frequently assumed to be over the same set of variables (homogeneous). This requires each agent to process and communicate the full global joint distribution, and thus leads to high computation and communication costs irrespective of relevancy to specific local objectives. This work formulates and studies heterogeneous decentralized fusion problems, defined as the set of problems in which either the communicated or the processed distributions describe different, but overlapping, random states of interest that are subsets of a larger full global joint state. We exploit the conditional independence structure of such problems and provide a rigorous derivation of novel exact and approximate conditionally factorized heterogeneous fusion rules. We further develop a new version of the homogeneous Channel Filter algorithm to enable conservative heterogeneous fusion for smoothing and filtering scenarios in dynamic problems. Numerical examples show more than $99.5\%$ potential communication reduction for heterogeneous channel filter fusion, and a multi-target tracking simulation shows that these methods provide consistent estimates while remaining computationally scalable. △ Less

Submitted 29 January, 2024; v1 submitted 26 January, 2021; originally announced January 2021.

Comments: 16 pages, 8 figures, 2 tables, Published at IEEE Transactions on Robotics (T-RO)

Journal ref: IEEE Transactions on Robotics 2023

SenseBERT: Driving Some Sense into BERT

Authors: Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham

Abstract: The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseB… ▽ More The ability to learn from large unlabeled corpora has allowed neural language models to advance the frontier in natural language understanding. However, existing self-supervision techniques operate at the word form level, which serves as a surrogate for the underlying semantic content. This paper proposes a method to employ weak-supervision directly at the word sense level. Our model, named SenseBERT, is pre-trained to predict not only the masked words but also their WordNet supersenses. Accordingly, we attain a lexical-semantic level language model, without the use of human annotation. SenseBERT achieves significantly improved lexical understanding, as we demonstrate by experimenting on SemEval Word Sense Disambiguation, and by attaining a state of the art result on the Word in Context task. △ Less

Submitted 18 May, 2020; v1 submitted 15 August, 2019; originally announced August 2019.

Comments: Accepted to ACL 2020