-
Scaling Instructable Agents Across Many Simulated Worlds
Authors:
SIMA Team,
Maria Abi Raad,
Arun Ahuja,
Catarina Barros,
Frederic Besse,
Andrew Bolt,
Adrian Bolton,
Bethanie Brownfield,
Gavin Buttimore,
Max Cant,
Sarah Chakera,
Stephanie C. Y. Chan,
Jeff Clune,
Adrian Collister,
Vikki Copeman,
Alex Cullum,
Ishita Dasgupta,
Dario de Cesare,
Julia Di Trapani,
Yani Donchev,
Emma Dunleavy,
Martin Engelcke,
Ryan Faulkner,
Frankie Garcia,
Charles Gbadamosi
, et al. (68 additional authors not shown)
Abstract:
Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructio…
▽ More
Building embodied AI systems that can follow arbitrary language instructions in any 3D environment is a key challenge for creating general AI. Accomplishing this goal requires learning to ground language in perception and embodied actions, in order to accomplish complex tasks. The Scalable, Instructable, Multiworld Agent (SIMA) project tackles this by training agents to follow free-form instructions across a diverse range of virtual 3D environments, including curated research environments as well as open-ended, commercial video games. Our goal is to develop an instructable agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface: the inputs are image observations and language instructions and the outputs are keyboard-and-mouse actions. This general approach is challenging, but it allows agents to ground language across many visually complex and semantically rich environments while also allowing us to readily run agents in new environments. In this paper we describe our motivation and goal, the initial progress we have made, and promising preliminary results on several diverse research environments and a variety of commercial video games.
△ Less
Submitted 17 April, 2024; v1 submitted 13 March, 2024;
originally announced April 2024.
-
FinEntity: Entity-level Sentiment Classification for Financial Texts
Authors:
Yixuan Tang,
Yi Yang,
Allen H Huang,
Andy Tam,
Justin Z Tang
Abstract:
In the financial domain, conducting entity-level sentiment analysis is crucial for accurately assessing the sentiment directed toward a specific financial entity. To our knowledge, no publicly available dataset currently exists for this purpose. In this work, we introduce an entity-level sentiment classification dataset, called \textbf{FinEntity}, that annotates financial entity spans and their se…
▽ More
In the financial domain, conducting entity-level sentiment analysis is crucial for accurately assessing the sentiment directed toward a specific financial entity. To our knowledge, no publicly available dataset currently exists for this purpose. In this work, we introduce an entity-level sentiment classification dataset, called \textbf{FinEntity}, that annotates financial entity spans and their sentiment (positive, neutral, and negative) in financial news. We document the dataset construction process in the paper. Additionally, we benchmark several pre-trained models (BERT, FinBERT, etc.) and ChatGPT on entity-level sentiment classification. In a case study, we demonstrate the practical utility of using FinEntity in monitoring cryptocurrency markets. The data and code of FinEntity is available at \url{https://github.com/yixuantt/FinEntity}
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
A comparative analysis of SRGAN models
Authors:
Fatemeh Rezapoor Nikroo,
Ajinkya Deshmukh,
Anantha Sharma,
Adrian Tam,
Kaarthik Kumar,
Cleo Norris,
Aditya Dangi
Abstract:
In this study, we evaluate the performance of multiple state-of-the-art SRGAN (Super Resolution Generative Adversarial Network) models, ESRGAN, Real-ESRGAN and EDSR, on a benchmark dataset of real-world images which undergo degradation using a pipeline. Our results show that some models seem to significantly increase the resolution of the input images while preserving their visual quality, this is…
▽ More
In this study, we evaluate the performance of multiple state-of-the-art SRGAN (Super Resolution Generative Adversarial Network) models, ESRGAN, Real-ESRGAN and EDSR, on a benchmark dataset of real-world images which undergo degradation using a pipeline. Our results show that some models seem to significantly increase the resolution of the input images while preserving their visual quality, this is assessed using Tesseract OCR engine. We observe that EDSR-BASE model from huggingface outperforms the remaining candidate models in terms of both quantitative metrics and subjective visual quality assessments with least compute overhead. Specifically, EDSR generates images with higher peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) values and are seen to return high quality OCR results with Tesseract OCR engine. These findings suggest that EDSR is a robust and effective approach for single-image super-resolution and may be particularly well-suited for applications where high-quality visual fidelity is critical and optimized compute.
△ Less
Submitted 19 July, 2023; v1 submitted 18 July, 2023;
originally announced July 2023.
-
The Edge of Orthogonality: A Simple View of What Makes BYOL Tick
Authors:
Pierre H. Richemond,
Allison Tam,
Yunhao Tang,
Florian Strub,
Bilal Piot,
Felix Hill
Abstract:
Self-predictive unsupervised learning methods such as BYOL or SimSiam have shown impressive results, and counter-intuitively, do not collapse to trivial representations. In this work, we aim at exploring the simplest possible mathematical arguments towards explaining the underlying mechanisms behind self-predictive unsupervised learning. We start with the observation that those methods crucially r…
▽ More
Self-predictive unsupervised learning methods such as BYOL or SimSiam have shown impressive results, and counter-intuitively, do not collapse to trivial representations. In this work, we aim at exploring the simplest possible mathematical arguments towards explaining the underlying mechanisms behind self-predictive unsupervised learning. We start with the observation that those methods crucially rely on the presence of a predictor network (and stop-gradient). With simple linear algebra, we show that when using a linear predictor, the optimal predictor is close to an orthogonal projection, and propose a general framework based on orthonormalization that enables to interpret and give intuition on why BYOL works. In addition, this framework demonstrates the crucial role of the exponential moving average and stop-gradient operator in BYOL as an efficient orthonormalization mechanism. We use these insights to propose four new \emph{closed-form predictor} variants of BYOL to support our analysis. Our closed-form predictors outperform standard linear trainable predictor BYOL at $100$ and $300$ epochs (top-$1$ linear accuracy on ImageNet).
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
Automated Logging Drone: A Computer Vision Drone Implementation
Authors:
Aaron Yagnik,
Adrian S. -W. Tam
Abstract:
In recent years, Artificial Intelligence (AI) and Computer Vision (CV) have become the pinnacle of technology with new developments seemingly every day. This technology along with more powerful drone technology have made autonomous surveillance more sought after. Here an overview of the Automated Logging Drone (ALD) project is presented along with examples of how this project can be used with more…
▽ More
In recent years, Artificial Intelligence (AI) and Computer Vision (CV) have become the pinnacle of technology with new developments seemingly every day. This technology along with more powerful drone technology have made autonomous surveillance more sought after. Here an overview of the Automated Logging Drone (ALD) project is presented along with examples of how this project can be used with more refining and added features.
△ Less
Submitted 3 November, 2022;
originally announced November 2022.
-
Semantic Exploration from Language Abstractions and Pretrained Representations
Authors:
Allison C. Tam,
Neil C. Rabinowitz,
Andrew K. Lampinen,
Nicholas A. Roy,
Stephanie C. Y. Chan,
DJ Strouse,
Jane X. Wang,
Andrea Banino,
Felix Hill
Abstract:
Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluat…
▽ More
Effective exploration is a challenge in reinforcement learning (RL). Novelty-based exploration methods can suffer in high-dimensional state spaces, such as continuous partially-observable 3D environments. We address this challenge by defining novelty using semantically meaningful state abstractions, which can be found in learned representations shaped by natural language. In particular, we evaluate vision-language representations, pretrained on natural image captioning datasets. We show that these pretrained representations drive meaningful, task-relevant exploration and improve performance on 3D simulated environments. We also characterize why and how language provides useful abstractions for exploration by considering the impacts of using representations from a pretrained model, a language oracle, and several ablations. We demonstrate the benefits of our approach in two very different task domains -- one that stresses the identification and manipulation of everyday objects, and one that requires navigational exploration in an expansive world. Our results suggest that using language-shaped representations could improve exploration for various algorithms and agents in challenging environments.
△ Less
Submitted 26 April, 2023; v1 submitted 8 April, 2022;
originally announced April 2022.
-
Tell me why! Explanations support learning relational and causal structure
Authors:
Andrew K. Lampinen,
Nicholas A. Roy,
Ishita Dasgupta,
Stephanie C. Y. Chan,
Allison C. Tam,
James L. McClelland,
Chen Yan,
Adam Santoro,
Neil C. Rabinowitz,
Jane X. Wang,
Felix Hill
Abstract:
Inferring the abstract relational and causal structure of the world is a major challenge for reinforcement-learning (RL) agents. For humans, language--particularly in the form of explanations--plays a considerable role in overcoming this challenge. Here, we show that language can play a similar role for deep RL agents in complex environments. While agents typically struggle to acquire relational a…
▽ More
Inferring the abstract relational and causal structure of the world is a major challenge for reinforcement-learning (RL) agents. For humans, language--particularly in the form of explanations--plays a considerable role in overcoming this challenge. Here, we show that language can play a similar role for deep RL agents in complex environments. While agents typically struggle to acquire relational and causal knowledge, augmenting their experience by training them to predict language descriptions and explanations can overcome these limitations. We show that language can help agents learn challenging relational tasks, and examine which aspects of language contribute to its benefits. We then show that explanations can help agents to infer not only relational but also causal structure. Language can shape the way that agents to generalize out-of-distribution from ambiguous, causally-confounded training, and explanations even allow agents to learn to perform experimental interventions to identify causal relationships. Our results suggest that language description and explanation may be powerful tools for improving agent learning and generalization.
△ Less
Submitted 25 May, 2022; v1 submitted 7 December, 2021;
originally announced December 2021.
-
An Algorithmic Equity Toolkit for Technology Audits by Community Advocates and Activists
Authors:
Michael Katell,
Meg Young,
Bernease Herman,
Dharma Dailey,
Aaron Tam,
Vivian Guetler,
Corinne Binz,
Daniella Raz,
P. M. Krafft
Abstract:
A wave of recent scholarship documenting the discriminatory harms of algorithmic systems has spurred widespread interest in algorithmic accountability and regulation. Yet effective accountability and regulation is stymied by a persistent lack of resources supporting public understanding of algorithms and artificial intelligence. Through interactions with a US-based civil rights organization and th…
▽ More
A wave of recent scholarship documenting the discriminatory harms of algorithmic systems has spurred widespread interest in algorithmic accountability and regulation. Yet effective accountability and regulation is stymied by a persistent lack of resources supporting public understanding of algorithms and artificial intelligence. Through interactions with a US-based civil rights organization and their coalition of community organizations, we identify a need for (i) heuristics that aid stakeholders in distinguishing between types of analytic and information systems in lay language, and (ii) risk assessment tools for such systems that begin by making algorithms more legible. The present work delivers a toolkit to achieve these aims. This paper both presents the Algorithmic Equity Toolkit (AEKit) Equity as an artifact, and details how our participatory process shaped its design. Our work fits within human-computer interaction scholarship as a demonstration of the value of HCI methods and approaches to problems in the area of algorithmic transparency and accountability.
△ Less
Submitted 5 December, 2019;
originally announced December 2019.
-
An Online Environment for Democratic Deliberation: Motivations, Principles, and Design
Authors:
Todd Davies,
Brendan O'Connor,
Alex Cochran,
Jonathan J. Effrat,
Andrew Parker,
Benjamin Newman,
Aaron Tam
Abstract:
We have created a platform for online deliberation called Deme (which rhymes with 'team'). Deme is designed to allow groups of people to engage in collaborative drafting, focused discussion, and decision making using the Internet. The Deme project has evolved greatly from its beginning in 2003. This chapter outlines the thinking behind Deme's initial design: our motivations for creating it, the pr…
▽ More
We have created a platform for online deliberation called Deme (which rhymes with 'team'). Deme is designed to allow groups of people to engage in collaborative drafting, focused discussion, and decision making using the Internet. The Deme project has evolved greatly from its beginning in 2003. This chapter outlines the thinking behind Deme's initial design: our motivations for creating it, the principles that guided its construction, and its most important design features. The version of Deme described here was written in PHP and was deployed in 2004 and used by several groups (including organizers of the 2005 Online Deliberation Conference). Other papers describe later developments in the Deme project (see Davies et al. 2005, 2008; Davies and Mintz 2009).
△ Less
Submitted 15 February, 2013;
originally announced February 2013.
-
Displaying Asynchronous Reactions to a Document: Two Goals and a Design
Authors:
Todd Davies,
Benjamin Newman,
Brendan O'Connor,
Aaron Tam,
Leo Perry
Abstract:
We describe and motivate three goals for the screen display of asynchronous text deliberation pertaining to a document: (1) visibility of relationships between comments and the text they reference, between different comments, and between group members and the document and discussion, and (2) distinguishability of boundaries between contextually related and unrelated text and comments and between i…
▽ More
We describe and motivate three goals for the screen display of asynchronous text deliberation pertaining to a document: (1) visibility of relationships between comments and the text they reference, between different comments, and between group members and the document and discussion, and (2) distinguishability of boundaries between contextually related and unrelated text and comments and between individual authors of documents and comments. Interfaces for document-centered discussion generally fail to fulfill one or both of these goals as well as they could. We describe the design of the new version of Deme, a Web-based platform for online deliberation, and argue that it achieves the two goals better than other recent designs.
△ Less
Submitted 14 February, 2013;
originally announced February 2013.
-
Trimming the Multipath for Efficient Dynamic Routing
Authors:
Adrian Sai-wah Tam,
Kang Xi,
H. Jonathan Chao
Abstract:
Multipath routing is a trivial way to exploit the path diversity to leverage the network throughput. Technologies such as OSPF ECMP use all the available paths in the network to forward traffic, however, we argue that is not necessary to do so to load balance the network. In this paper, we consider multipath routing with only a limited number of end-to-end paths for each source and destination, an…
▽ More
Multipath routing is a trivial way to exploit the path diversity to leverage the network throughput. Technologies such as OSPF ECMP use all the available paths in the network to forward traffic, however, we argue that is not necessary to do so to load balance the network. In this paper, we consider multipath routing with only a limited number of end-to-end paths for each source and destination, and found that this can still load balance the traffic. We devised an algorithm to select a few paths for each source-destination pair so that when all traffic are forwarded over these paths, we can achieve a balanced load in the sense that the maximum link utilization is comparable to that of ECMP forwarding. When the constraint of only shortest paths (i.e. equal paths) are relaxed, we can even outperform ECMP in certain cases. As a result, we can use a few end-to-end tunnels between each source and destination nodes to achieve the load balancing of traffic.
△ Less
Submitted 4 September, 2011;
originally announced September 2011.
-
Use of Devolved Controllers in Data Center Networks
Authors:
Adrian S. -W. Tam,
Kang Xi,
H. Jonathan Chao
Abstract:
In a data center network, for example, it is quite often to use controllers to manage resources in a centralized man- ner. Centralized control, however, imposes a scalability problem. In this paper, we investigate the use of multiple independent controllers instead of a single omniscient controller to manage resources. Each controller looks after a portion of the network only, but they together co…
▽ More
In a data center network, for example, it is quite often to use controllers to manage resources in a centralized man- ner. Centralized control, however, imposes a scalability problem. In this paper, we investigate the use of multiple independent controllers instead of a single omniscient controller to manage resources. Each controller looks after a portion of the network only, but they together cover the whole network. This therefore solves the scalability problem. We use flow allocation as an example to see how this approach can manage the bandwidth use in a distributed manner. The focus is on how to assign components of a network to the controllers so that (1) each controller only need to look after a small part of the network but (2) there is at least one controller that can answer any request. We outline a way to configure the controllers to fulfill these requirements as a proof that the use of devolved controllers is possible. We also discuss several issues related to such implementation.
△ Less
Submitted 29 March, 2011;
originally announced March 2011.
