-
Peripheral elements in reduced Alexander modules: an addendum
Authors:
Daniel S. Silver,
Lorenzo Traldi
Abstract:
We answer a question raised in ``Peripheral elements in reduced Alexander modules'' [J. Knot Theory Ramifications 31 (2022), 2250058]. We also correct a minor error in that paper.
We answer a question raised in ``Peripheral elements in reduced Alexander modules'' [J. Knot Theory Ramifications 31 (2022), 2250058]. We also correct a minor error in that paper.
△ Less
Submitted 12 August, 2024;
originally announced August 2024.
-
Bubble reachers and uncivil discourse in polarized online public sphere
Authors:
Jordan K Kobellarz,
Milos Brocic,
Daniel Silver,
Thiago H Silva
Abstract:
Early optimism saw possibilities for social media to renew democratic discourse, marked by hopes for individuals from diverse backgrounds to find opportunities to learn from and interact with others different from themselves. This optimism quickly waned as social media seemed to breed ideological homophily marked by "filter bubble" or "echo chambers." A typical response to the sense of fragmentati…
▽ More
Early optimism saw possibilities for social media to renew democratic discourse, marked by hopes for individuals from diverse backgrounds to find opportunities to learn from and interact with others different from themselves. This optimism quickly waned as social media seemed to breed ideological homophily marked by "filter bubble" or "echo chambers." A typical response to the sense of fragmentation has been to encourage exposure to more cross-partisan sources of information. But do outlets that reach across partisan lines in fact generate more civil discourse? And does the civility of discourse hosted by such outlets vary depending on the political context in which they operate? To answer these questions, we identified bubble reachers, users who distribute content that reaches other users with diverse political opinions in recent presidential elections in Brazil, where populism has deep roots in the political culture, and Canada, where the political culture is comparatively moderate. Given that background, this research studies unexplored properties of content shared by bubble reachers, specifically the quality of conversations and comments it generates. We examine how ideologically neutral bubble reachers differ from ideologically partisan accounts in the level of uncivil discourse they provoke, and explore how this varies in the context of the two countries considered. Our results suggest that while ideologically neutral bubble reachers support less uncivil discourse in Canada, the opposite relationship holds in Brazil. Even non-political content by ideologically neutral bubble reachers elicits a considerable amount of uncivil discourse in Brazil. This indicates that bubble reaching and incivility are moderated by the national political context. Our results complicate the simple hypothesis of a universal impact of neutral bubble reachers across contexts.
△ Less
Submitted 19 April, 2024;
originally announced April 2024.
-
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Authors:
Gemini Team,
Petko Georgiev,
Ving Ian Lei,
Ryan Burnell,
Libin Bai,
Anmol Gulati,
Garrett Tanzer,
Damien Vincent,
Zhufeng Pan,
Shibo Wang,
Soroosh Mariooryad,
Yifan Ding,
Xinyang Geng,
Fred Alcober,
Roy Frostig,
Mark Omernick,
Lexi Walker,
Cosmin Paduraru,
Christina Sorokin,
Andrea Tacchetti,
Colin Gaffney,
Samira Daruki,
Olcan Sercinoglu,
Zach Gleicher,
Juliette Love
, et al. (1110 additional authors not shown)
Abstract:
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February…
▽ More
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
△ Less
Submitted 8 August, 2024; v1 submitted 8 March, 2024;
originally announced March 2024.
-
Using Graph Neural Networks to Predict Local Culture
Authors:
Thiago H Silva,
Daniel Silver
Abstract:
Urban research has long recognized that neighbourhoods are dynamic and relational. However, lack of data, methodologies, and computer processing power have hampered a formal quantitative examination of neighbourhood relational dynamics. To make progress on this issue, this study proposes a graph neural network (GNN) approach that permits combining and evaluating multiple sources of information abo…
▽ More
Urban research has long recognized that neighbourhoods are dynamic and relational. However, lack of data, methodologies, and computer processing power have hampered a formal quantitative examination of neighbourhood relational dynamics. To make progress on this issue, this study proposes a graph neural network (GNN) approach that permits combining and evaluating multiple sources of information about internal characteristics of neighbourhoods, their past characteristics, and flows of groups among them, potentially providing greater expressive power in predictive models. By exploring a public large-scale dataset from Yelp, we show the potential of our approach for considering structural connectedness in predicting neighbourhood attributes, specifically to predict local culture. Results are promising from a substantive and methodologically point of view. Substantively, we find that either local area information (e.g. area demographics) or group profiles (tastes of Yelp reviewers) give the best results in predicting local culture, and they are nearly equivalent in all studied cases. Methodologically, exploring group profiles could be a helpful alternative where finding local information for specific areas is challenging, since they can be extracted automatically from many forms of online data. Thus, our approach could empower researchers and policy-makers to use a range of data sources when other local area information is lacking.
△ Less
Submitted 22 April, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
-
Gemini: A Family of Highly Capable Multimodal Models
Authors:
Gemini Team,
Rohan Anil,
Sebastian Borgeaud,
Jean-Baptiste Alayrac,
Jiahui Yu,
Radu Soricut,
Johan Schalkwyk,
Andrew M. Dai,
Anja Hauth,
Katie Millican,
David Silver,
Melvin Johnson,
Ioannis Antonoglou,
Julian Schrittwieser,
Amelia Glaese,
Jilin Chen,
Emily Pitler,
Timothy Lillicrap,
Angeliki Lazaridou,
Orhan Firat,
James Molloy,
Michael Isard,
Paul R. Barham,
Tom Hennigan,
Benjamin Lee
, et al. (1325 additional authors not shown)
Abstract:
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultr…
▽ More
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.
△ Less
Submitted 17 June, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
Core groups
Authors:
Daniel S. Silver,
Lorenzo Traldi,
Susan G. Williams
Abstract:
The core group of a classical link was introduced independently by A.J. Kelly in 1991 and M. Wada in 1992. It is a link invariant defined by a presentation involving the arcs and crossings of a diagram, related to Wirtinger's presentation of the fundamental group of a link complement. Two close relatives of the core group are defined by presentations involving regions rather than arcs; one of them…
▽ More
The core group of a classical link was introduced independently by A.J. Kelly in 1991 and M. Wada in 1992. It is a link invariant defined by a presentation involving the arcs and crossings of a diagram, related to Wirtinger's presentation of the fundamental group of a link complement. Two close relatives of the core group are defined by presentations involving regions rather than arcs; one of them is related to Dehn's presentation of a link group. The definitions are extended to virtual link diagrams and properties of the resulting invariants are discussed.
△ Less
Submitted 1 June, 2024; v1 submitted 3 November, 2023;
originally announced November 2023.
-
SLIQ: Quantum Image Similarity Networks on Noisy Quantum Computers
Authors:
Daniel Silver,
Tirthak Patel,
Aditya Ranjan,
Harshitta Gandhi,
William Cutler,
Devesh Tiwari
Abstract:
Exploration into quantum machine learning has grown tremendously in recent years due to the ability of quantum computers to speed up classical programs. However, these efforts have yet to solve unsupervised similarity detection tasks due to the challenge of porting them to run on quantum computers. To overcome this challenge, we propose SLIQ, the first open-sourced work for resource-efficient quan…
▽ More
Exploration into quantum machine learning has grown tremendously in recent years due to the ability of quantum computers to speed up classical programs. However, these efforts have yet to solve unsupervised similarity detection tasks due to the challenge of porting them to run on quantum computers. To overcome this challenge, we propose SLIQ, the first open-sourced work for resource-efficient quantum similarity detection networks, built with practical and effective quantum learning and variance-reducing algorithms.
△ Less
Submitted 26 September, 2023;
originally announced September 2023.
-
QUILT: Effective Multi-Class Classification on Quantum Computers Using an Ensemble of Diverse Quantum Classifiers
Authors:
Daniel Silver,
Tirthak Patel,
Devesh Tiwari
Abstract:
Quantum computers can theoretically have significant acceleration over classical computers; but, the near-future era of quantum computing is limited due to small number of qubits that are also error prone. Quilt is a framework for performing multi-class classification task designed to work effectively on current error-prone quantum computers. Quilt is evaluated with real quantum machines as well a…
▽ More
Quantum computers can theoretically have significant acceleration over classical computers; but, the near-future era of quantum computing is limited due to small number of qubits that are also error prone. Quilt is a framework for performing multi-class classification task designed to work effectively on current error-prone quantum computers. Quilt is evaluated with real quantum machines as well as with projected noise levels as quantum machines become more noise-free. Quilt demonstrates up to 85% multi-class classification accuracy with the MNIST dataset on a five-qubit system.
△ Less
Submitted 26 September, 2023;
originally announced September 2023.
-
MosaiQ: Quantum Generative Adversarial Networks for Image Generation on NISQ Computers
Authors:
Daniel Silver,
Tirthak Patel,
William Cutler,
Aditya Ranjan,
Harshitta Gandhi,
Devesh Tiwari
Abstract:
Quantum machine learning and vision have come to the fore recently, with hardware advances enabling rapid advancement in the capabilities of quantum machines. Recently, quantum image generation has been explored with many potential advantages over non-quantum techniques; however, previous techniques have suffered from poor quality and robustness. To address these problems, we introduce, MosaiQ, a…
▽ More
Quantum machine learning and vision have come to the fore recently, with hardware advances enabling rapid advancement in the capabilities of quantum machines. Recently, quantum image generation has been explored with many potential advantages over non-quantum techniques; however, previous techniques have suffered from poor quality and robustness. To address these problems, we introduce, MosaiQ, a high-quality quantum image generation GAN framework that can be executed on today's Near-term Intermediate Scale Quantum (NISQ) computers.
△ Less
Submitted 21 August, 2023;
originally announced August 2023.
-
Toward Privacy in Quantum Program Execution On Untrusted Quantum Cloud Computing Machines for Business-sensitive Quantum Needs
Authors:
Tirthak Patel,
Daniel Silver,
Aditya Ranjan,
Harshitta Gandhi,
William Cutler,
Devesh Tiwari
Abstract:
Quantum computing is an emerging paradigm that has shown great promise in accelerating large-scale scientific, optimization, and machine-learning workloads. With most quantum computing solutions being offered over the cloud, it has become imperative to protect confidential and proprietary quantum code from being accessed by untrusted and/or adversarial agents. In response to this challenge, we pro…
▽ More
Quantum computing is an emerging paradigm that has shown great promise in accelerating large-scale scientific, optimization, and machine-learning workloads. With most quantum computing solutions being offered over the cloud, it has become imperative to protect confidential and proprietary quantum code from being accessed by untrusted and/or adversarial agents. In response to this challenge, we propose SPYCE, which is the first known solution to obfuscate quantum code and output to prevent the leaking of any confidential information over the cloud. SPYCE implements a lightweight, scalable, and effective solution based on the unique principles of quantum computing to achieve this task.
△ Less
Submitted 31 July, 2023;
originally announced July 2023.
-
A Hybrid 3D Eddy Detection Technique Based on Sea Surface Height and Velocity Field
Authors:
Weiping Hua,
Karen Bemis,
Dujuan Kang,
Sedat Ozer,
Deborah Silver
Abstract:
Eddy detection is a critical task for ocean scientists to understand and analyze ocean circulation. In this paper, we introduce a hybrid eddy detection approach that combines sea surface height (SSH) and velocity fields with geometric criteria defining eddy behavior. Our approach searches for SSH minima and maxima, which oceanographers expect to find at the center of eddies. Geometric criteria are…
▽ More
Eddy detection is a critical task for ocean scientists to understand and analyze ocean circulation. In this paper, we introduce a hybrid eddy detection approach that combines sea surface height (SSH) and velocity fields with geometric criteria defining eddy behavior. Our approach searches for SSH minima and maxima, which oceanographers expect to find at the center of eddies. Geometric criteria are used to verify expected velocity field properties, such as net rotation and symmetry, by tracing velocity components along a circular path surrounding each eddy center. Progressive searches outward and into deeper layers yield each eddy's 3D region of influence. Isolation of each eddy structure from the dataset, using it's cylindrical footprint, facilitates visualization of internal eddy structures using horizontal velocity, vertical velocity, temperature and salinity. A quantitative comparison of Okubo-Weiss vorticity (OW) thresholding, the standard winding angle, and this new SSH-velocity hybrid methods of eddy detection as applied to the Red Sea dataset suggests that detection results are highly dependent on the choices of method, thresholds, and criteria. Our new SSH-velocity hybrid detection approach has the advantages of providing eddy structures with verified rotation properties, 3D visualization of the internal structure of physical properties, and rapid efficient estimations of eddy footprints without calculating streamlines. Our approach combines visualization of internal structure and tracking overall movement to support the study of the transport mechanisms key to understanding the interaction of nutrient distribution and ocean circulation. Our method is applied to three different datasets to showcase the generality of its application.
△ Less
Submitted 31 October, 2023; v1 submitted 14 May, 2023;
originally announced May 2023.
-
The Roles of Symbols in Neural-based AI: They are Not What You Think!
Authors:
Daniel L. Silver,
Tom M. Mitchell
Abstract:
We propose that symbols are first and foremost external communication tools used between intelligent agents that allow knowledge to be transferred in a more efficient and effective manner than having to experience the world directly. But, they are also used internally within an agent through a form of self-communication to help formulate, describe and justify subsymbolic patterns of neural activit…
▽ More
We propose that symbols are first and foremost external communication tools used between intelligent agents that allow knowledge to be transferred in a more efficient and effective manner than having to experience the world directly. But, they are also used internally within an agent through a form of self-communication to help formulate, describe and justify subsymbolic patterns of neural activity that truly implement thinking. Symbols, and our languages that make use of them, not only allow us to explain our thinking to others and ourselves, but also provide beneficial constraints (inductive bias) on learning about the world. In this paper we present relevant insights from neuroscience and cognitive science, about how the human brain represents symbols and the concepts they refer to, and how today's artificial neural networks can do the same. We then present a novel neuro-symbolic hypothesis and a plausible architecture for intelligent agents that combines subsymbolic representations for symbols and concepts for learning and reasoning. Our hypothesis and associated architecture imply that symbols will remain critical to the future of intelligent systems NOT because they are the fundamental building blocks of thought, but because they are characterizations of subsymbolic processes that constitute thought.
△ Less
Submitted 26 April, 2023;
originally announced April 2023.
-
Forecasting COVID-19 Case Counts Based on 2020 Ontario Data
Authors:
Daniel L. Silver,
Rinda Digamarthi
Abstract:
Objective: To develop machine learning models that can predict the number of COVID-19 cases per day given the last 14 days of environmental and mobility data.
Approach: COVID-19 data from four counties around Toronto, Ontario, were used. Data were prepared into daily records containing the number of new COVID case counts, patient demographic data, outdoor weather variables, indoor environment fa…
▽ More
Objective: To develop machine learning models that can predict the number of COVID-19 cases per day given the last 14 days of environmental and mobility data.
Approach: COVID-19 data from four counties around Toronto, Ontario, were used. Data were prepared into daily records containing the number of new COVID case counts, patient demographic data, outdoor weather variables, indoor environment factors, and human movement based on cell mobility and public health restrictions. This data was analyzed to determine the most important variables and their interactions. Predictive models were developed using CNN and LSTM deep neural network approaches. A 5-fold chronological cross-validation approach used these methods to develop predictive models using data from Mar 1 to Oct 14 2020, and test them on data covering Oct 15 to Dec 24 2020. Results: The best LSTM models forecasted tomorrow's daily COVID case counts with 90.7% accuracy, and the 7-day rolling average COVID case counts with 98.1% accuracy using independent test data. The best models to forecast the next 7 days of daily COVID case counts did so with 79.4% accuracy over all days. Models forecasting the 7-day rolling average case counts had a mean accuracy of 83.6% on the same test set.
Conclusions: Our findings point to the importance of indoor humidity for the transmission of a virus such as COVID-19. During the coldest portions of the year, when humans spend greater amounts of time indoors or in vehicles, air quality drops within buildings, most significantly indoor relative humidity levels. Moderate to high indoor temperatures coupled with low IRH (below 20%) create conditions where viral transmission is more likely because water vapour ejected from an infected person's mouth can remain longer in the air because of evaporation and dry skin conditions, particularly in a recipient's airway, promotes transmission.
△ Less
Submitted 17 March, 2023;
originally announced March 2023.
-
CHARTER: Identifying the Most-Critical Gate Operations in Quantum Circuits via Amplified Gate Reversibility
Authors:
Tirthak Patel,
Daniel Silver,
Devesh Tiwari
Abstract:
When quantum programs are executed on noisy intermediate-scale quantum (NISQ) computers, they experience hardware noise; consequently, the program outputs are often erroneous. To mitigate the adverse effects of hardware noise, it is necessary to understand the effect of hardware noise on the program output and more fundamentally, understand the impact of hardware noise on specific regions within a…
▽ More
When quantum programs are executed on noisy intermediate-scale quantum (NISQ) computers, they experience hardware noise; consequently, the program outputs are often erroneous. To mitigate the adverse effects of hardware noise, it is necessary to understand the effect of hardware noise on the program output and more fundamentally, understand the impact of hardware noise on specific regions within a quantum program. Identifying and optimizing regions that are more noise-sensitive is the key to expanding the capabilities of NISQ computers.
Toward achieving that goal, we propose CHARTER, a novel technique to pinpoint specific gates and regions within a quantum program that are the most affected by the hardware noise and that have the highest impact on the program output. Using CHARTER's methodology, programmers can obtain a precise understanding of how different components of their code affect the output and optimize those components without the need for non-scalable quantum simulation on classical computers.
△ Less
Submitted 17 November, 2022;
originally announced November 2022.
-
Changing the Scene: applying four models of social evolution to the scenescape
Authors:
Daniel Silver,
Thiago H Silva,
Patrick Adler
Abstract:
This paper elaborates a multi-model approach to studying how local scenes change. We refer to this as the "4 D's" of scene change: development, differentiation, defense, and diffusion. Each posits somewhat distinct change processes, and has its own tradition of theory and empirical research, which we briefly review. After summarizing some major trends in scenes and amenities in the US context, for…
▽ More
This paper elaborates a multi-model approach to studying how local scenes change. We refer to this as the "4 D's" of scene change: development, differentiation, defense, and diffusion. Each posits somewhat distinct change processes, and has its own tradition of theory and empirical research, which we briefly review. After summarizing some major trends in scenes and amenities in the US context, for each change model, we present some initial findings, discussing data and methods throughout. Our overall goal is to point toward new research arcs on change models of scenes, and to give some clear examples and directions for how to think about and collect data to understand what makes some scenes change, others not, why, and in what directions.
△ Less
Submitted 21 September, 2022;
originally announced September 2022.
-
Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning
Authors:
Julien Perolat,
Bart de Vylder,
Daniel Hennes,
Eugene Tarassov,
Florian Strub,
Vincent de Boer,
Paul Muller,
Jerome T. Connor,
Neil Burch,
Thomas Anthony,
Stephen McAleer,
Romuald Elie,
Sarah H. Cen,
Zhe Wang,
Audrunas Gruslys,
Aleksandra Malysheva,
Mina Khan,
Sherjil Ozair,
Finbarr Timbers,
Toby Pohlen,
Tom Eccles,
Mark Rowland,
Marc Lanctot,
Jean-Baptiste Lespiau,
Bilal Piot
, et al. (9 additional authors not shown)
Abstract:
We introduce DeepNash, an autonomous agent capable of learning to play the imperfect information game Stratego from scratch, up to a human expert level. Stratego is one of the few iconic board games that Artificial Intelligence (AI) has not yet mastered. This popular game has an enormous game tree on the order of $10^{535}$ nodes, i.e., $10^{175}$ times larger than that of Go. It has the additiona…
▽ More
We introduce DeepNash, an autonomous agent capable of learning to play the imperfect information game Stratego from scratch, up to a human expert level. Stratego is one of the few iconic board games that Artificial Intelligence (AI) has not yet mastered. This popular game has an enormous game tree on the order of $10^{535}$ nodes, i.e., $10^{175}$ times larger than that of Go. It has the additional complexity of requiring decision-making under imperfect information, similar to Texas hold'em poker, which has a significantly smaller game tree (on the order of $10^{164}$ nodes). Decisions in Stratego are made over a large number of discrete actions with no obvious link between action and outcome. Episodes are long, with often hundreds of moves before a player wins, and situations in Stratego can not easily be broken down into manageably-sized sub-problems as in poker. For these reasons, Stratego has been a grand challenge for the field of AI for decades, and existing AI methods barely reach an amateur level of play. DeepNash uses a game-theoretic, model-free deep reinforcement learning method, without search, that learns to master Stratego via self-play. The Regularised Nash Dynamics (R-NaD) algorithm, a key component of DeepNash, converges to an approximate Nash equilibrium, instead of 'cycling' around it, by directly modifying the underlying multi-agent learning dynamics. DeepNash beats existing state-of-the-art AI methods in Stratego and achieved a yearly (2022) and all-time top-3 rank on the Gravon games platform, competing with human expert players.
△ Less
Submitted 30 June, 2022;
originally announced June 2022.
-
Self-Consistent Models and Values
Authors:
Gregory Farquhar,
Kate Baumli,
Zita Marinho,
Angelos Filos,
Matteo Hessel,
Hado van Hasselt,
David Silver
Abstract:
Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a le…
▽ More
Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly \emph{self-consistent}. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.
△ Less
Submitted 25 October, 2021;
originally announced October 2021.
-
Reaching the bubble may not be enough: news media role in online political polarization
Authors:
Jordan K Kobellarz,
Milos Brocic,
Alexandre R Graeml,
Daniel Silver,
Thiago H Silva
Abstract:
Politics in different countries show diverse degrees of polarization, which tends to be stronger on social media, given how easy it became to connect and engage with like-minded individuals on the web. A way of reducing polarization would be by distributing cross-partisan news among individuals with distinct political orientations, i.e., ``reaching the bubbles''. This study investigates whether th…
▽ More
Politics in different countries show diverse degrees of polarization, which tends to be stronger on social media, given how easy it became to connect and engage with like-minded individuals on the web. A way of reducing polarization would be by distributing cross-partisan news among individuals with distinct political orientations, i.e., ``reaching the bubbles''. This study investigates whether this holds in the context of nationwide elections in Brazil and Canada. We collected politics-related tweets shared during the 2018 Brazilian presidential election and the 2019 Canadian federal election. Next, we proposed an updated centrality metric that enables identifying highly central bubble reachers, nodes that can distribute content among users with diverging political opinions - a fundamental metric for the proposed study. After that, we analyzed how users engage with news content shared by bubble reachers, its source, and its topics, considering its political orientation. Among other results, we found that, even though news media disseminate content that interests different sides of the political spectrum, users tend to engage considerably more with content that aligns with their political orientation, regardless of the topic.
△ Less
Submitted 7 July, 2022; v1 submitted 18 September, 2021;
originally announced September 2021.
-
Bootstrapped Meta-Learning
Authors:
Sebastian Flennerhag,
Yannick Schroecker,
Tom Zahavy,
Hado van Hasselt,
David Silver,
Satinder Singh
Abstract:
Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance…
▽ More
Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent, without backpropagating through the update rule.
△ Less
Submitted 16 March, 2022; v1 submitted 9 September, 2021;
originally announced September 2021.
-
The Option Keyboard: Combining Skills in Reinforcement Learning
Authors:
André Barreto,
Diana Borsa,
Shaobo Hou,
Gheorghe Comanici,
Eser Aygün,
Philippe Hamel,
Daniel Toyama,
Jonathan Hunt,
Shibl Mourad,
David Silver,
Doina Precup
Abstract:
The ability to combine known skills to create new ones may be crucial in the solution of complex reinforcement learning problems that unfold over extended periods. We argue that a robust way of combining skills is to define and manipulate them in the space of pseudo-rewards (or "cumulants"). Based on this premise, we propose a framework for combining skills using the formalism of options. We show…
▽ More
The ability to combine known skills to create new ones may be crucial in the solution of complex reinforcement learning problems that unfold over extended periods. We argue that a robust way of combining skills is to define and manipulate them in the space of pseudo-rewards (or "cumulants"). Based on this premise, we propose a framework for combining skills using the formalism of options. We show that every deterministic option can be unambiguously represented as a cumulant defined in an extended domain. Building on this insight and on previous results on transfer learning, we show how to approximate options whose cumulants are linear combinations of the cumulants of known options. This means that, once we have learned options associated with a set of cumulants, we can instantaneously synthesise options induced by any linear combination of them, without any learning involved. We describe how this framework provides a hierarchical interface to the environment whose abstract actions correspond to combinations of basic skills. We demonstrate the practical benefits of our approach in a resource management problem and a navigation task involving a quadrupedal simulated robot.
△ Less
Submitted 24 June, 2021;
originally announced June 2021.
-
Proper Value Equivalence
Authors:
Christopher Grimm,
André Barreto,
Gregory Farquhar,
David Silver,
Satinder Singh
Abstract:
One of the main challenges in model-based reinforcement learning (RL) is to decide which aspects of the environment should be modeled. The value-equivalence (VE) principle proposes a simple answer to this question: a model should capture the aspects of the environment that are relevant for value-based planning. Technically, VE distinguishes models based on a set of policies and a set of functions:…
▽ More
One of the main challenges in model-based reinforcement learning (RL) is to decide which aspects of the environment should be modeled. The value-equivalence (VE) principle proposes a simple answer to this question: a model should capture the aspects of the environment that are relevant for value-based planning. Technically, VE distinguishes models based on a set of policies and a set of functions: a model is said to be VE to the environment if the Bellman operators it induces for the policies yield the correct result when applied to the functions. As the number of policies and functions increase, the set of VE models shrinks, eventually collapsing to a single point corresponding to a perfect model. A fundamental question underlying the VE principle is thus how to select the smallest sets of policies and functions that are sufficient for planning. In this paper we take an important step towards answering this question. We start by generalizing the concept of VE to order-$k$ counterparts defined with respect to $k$ applications of the Bellman operator. This leads to a family of VE classes that increase in size as $k \rightarrow \infty$. In the limit, all functions become value functions, and we have a special instantiation of VE which we call proper VE or simply PVE. Unlike VE, the PVE class may contain multiple models even in the limit when all value functions are used. Crucially, all these models are sufficient for planning, meaning that they will yield an optimal policy despite the fact that they may ignore many aspects of the environment. We construct a loss function for learning PVE models and argue that popular algorithms such as MuZero can be understood as minimizing an upper bound for this loss. We leverage this connection to propose a modification to MuZero and show that it can lead to improved performance in practice.
△ Less
Submitted 12 December, 2021; v1 submitted 18 June, 2021;
originally announced June 2021.
-
Learning and Planning in Complex Action Spaces
Authors:
Thomas Hubert,
Julian Schrittwieser,
Ioannis Antonoglou,
Mohammadamin Barekatain,
Simon Schmitt,
David Silver
Abstract:
Many important real-world problems have action spaces that are high-dimensional, continuous or both, making full enumeration of all possible actions infeasible. Instead, only small subsets of actions can be sampled for the purpose of policy evaluation and improvement. In this paper, we propose a general framework to reason in a principled way about policy evaluation and improvement over such sampl…
▽ More
Many important real-world problems have action spaces that are high-dimensional, continuous or both, making full enumeration of all possible actions infeasible. Instead, only small subsets of actions can be sampled for the purpose of policy evaluation and improvement. In this paper, we propose a general framework to reason in a principled way about policy evaluation and improvement over such sampled action subsets. This sample-based policy iteration framework can in principle be applied to any reinforcement learning algorithm based upon policy iteration. Concretely, we propose Sampled MuZero, an extension of the MuZero algorithm that is able to learn in domains with arbitrarily complex action spaces by planning over sampled actions. We demonstrate this approach on the classical board game of Go and on two continuous control benchmark domains: DeepMind Control Suite and Real-World RL Suite.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
Online and Offline Reinforcement Learning by Planning with a Learned Model
Authors:
Julian Schrittwieser,
Thomas Hubert,
Amol Mandhane,
Mohammadamin Barekatain,
Ioannis Antonoglou,
David Silver
Abstract:
Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results in both settings. In this work, we describe the Reanalyse algorithm which uses mo…
▽ More
Learning efficiently from small amounts of data has long been the focus of model-based reinforcement learning, both for the online case when interacting with the environment and the offline case when learning from a fixed dataset. However, to date no single unified algorithm could demonstrate state-of-the-art results in both settings. In this work, we describe the Reanalyse algorithm which uses model-based policy and value improvement operators to compute new improved training targets on existing data points, allowing efficient learning for data budgets varying by several orders of magnitude. We further show that Reanalyse can also be used to learn entirely from demonstrations without any environment interactions, as in the case of offline Reinforcement Learning (offline RL). Combining Reanalyse with the MuZero algorithm, we introduce MuZero Unplugged, a single unified algorithm for any data budget, including offline RL. In contrast to previous work, our algorithm does not require any special adaptations for the off-policy or offline RL settings. MuZero Unplugged sets new state-of-the-art results in the RL Unplugged offline RL benchmark as well as in the online RL benchmark of Atari in the standard 200 million frame setting.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
Muesli: Combining Improvements in Policy Optimization
Authors:
Matteo Hessel,
Ivo Danihelka,
Fabio Viola,
Arthur Guez,
Simon Schmitt,
Laurent Sifre,
Theophane Weber,
David Silver,
Hado van Hasselt
Abstract:
We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by ex…
▽ More
We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.
△ Less
Submitted 31 March, 2022; v1 submitted 13 April, 2021;
originally announced April 2021.
-
A Bounded Measure for Estimating the Benefit of Visualization: Case Studies and Empirical Evaluation
Authors:
Min Chen,
Alfie Abdul-Rahman,
Deborah Silver,
Mateu Sbert
Abstract:
Many visual representations, such as volume-rendered images and metro maps, feature a noticeable amount of information loss. At a glance, there seem to be numerous opportunities for viewers to misinterpret the data being visualized, hence undermining the benefits of these visual representations. In practice, there is little doubt that these visual representations are useful. The recently-proposed…
▽ More
Many visual representations, such as volume-rendered images and metro maps, feature a noticeable amount of information loss. At a glance, there seem to be numerous opportunities for viewers to misinterpret the data being visualized, hence undermining the benefits of these visual representations. In practice, there is little doubt that these visual representations are useful. The recently-proposed information-theoretic measure for analyzing the cost-benefit ratio of visualization processes can explain such usefulness experienced in practice, and postulate that the viewers' knowledge can reduce the potential distortion (e.g., misinterpretation) due to information loss. This suggests that viewers' knowledge can be estimated by comparing the potential distortion without any knowledge and the actual distortion with some knowledge. In this paper, we describe several case studies for collecting instances that can (i) support the evaluation of several candidate measures for estimating the potential distortion distortion in visualization, and (ii) demonstrate their applicability in practical scenarios. Because the theoretical discourse on choosing an appropriate bounded measure for estimating the potential distortion is yet conclusive, it is the real world data about visualization further informs the selection of a bounded measure, providing practical evidence to aid a theoretical conclusion. Meanwhile, once we can measure the potential distortion in a bounded manner, we can interpret the numerical values characterizing the benefit of visualization more intuitively.
△ Less
Submitted 3 March, 2021;
originally announced March 2021.
-
Discovery of Options via Meta-Learned Subgoals
Authors:
Vivek Veeriah,
Tom Zahavy,
Matteo Hessel,
Zhongwen Xu,
Junhyuk Oh,
Iurii Kemaev,
Hado van Hasselt,
David Silver,
Satinder Singh
Abstract:
Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based…
▽ More
Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager-worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both a set of task-independent discovered-options and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extended options in multi-task RL domains, (2) the discovered options are frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help a randomly initialised manager learn faster in completely new tasks.
△ Less
Submitted 12 February, 2021;
originally announced February 2021.
-
The Value Equivalence Principle for Model-Based Reinforcement Learning
Authors:
Christopher Grimm,
André Barreto,
Satinder Singh,
David Silver
Abstract:
Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources o…
▽ More
Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources of model-based RL agents are better used to build models that are directly useful for value-based planning. As our main contribution, we introduce the principle of value equivalence: two models are value equivalent with respect to a set of functions and policies if they yield the same Bellman updates. We propose a formulation of the model learning problem based on the value equivalence principle and analyze how the set of feasible solutions is impacted by the choice of policies and functions. Specifically, we show that, as we augment the set of policies and functions considered, the class of value equivalent models shrinks, until eventually collapsing to a single point corresponding to a model that perfectly describes the environment. In many problems, directly modelling state-to-state transitions may be both difficult and unnecessary. By leveraging the value-equivalence principle one may find simpler models without compromising performance, saving computation and memory. We illustrate the benefits of value-equivalent model learning with experiments comparing it against more traditional counterparts like maximum likelihood estimation. More generally, we argue that the principle of value equivalence underlies a number of recent empirical successes in RL, such as Value Iteration Networks, the Predictron, Value Prediction Networks, TreeQN, and MuZero, and provides a first theoretical underpinning of those results.
△ Less
Submitted 6 November, 2020;
originally announced November 2020.
-
Graph Complexity and Link Colorings
Authors:
Daniel S. Silver,
Susan G. Williams
Abstract:
The (torsion) complexity of a finite signed graph is defined to be the order of the torsion subgroup of the abelian group presented by its Laplacian matrix. When $G$ is $d$-periodic (i.e., $G$ has a free ${\mathbb Z}^d$-action by graph automorphisms with finite quotient) the Mahler measure of its Laplacian polynomial is the growth rate of the complexity of finite quotients of $G$. Any 1-periodic p…
▽ More
The (torsion) complexity of a finite signed graph is defined to be the order of the torsion subgroup of the abelian group presented by its Laplacian matrix. When $G$ is $d$-periodic (i.e., $G$ has a free ${\mathbb Z}^d$-action by graph automorphisms with finite quotient) the Mahler measure of its Laplacian polynomial is the growth rate of the complexity of finite quotients of $G$. Any 1-periodic plane graph $G$ determines a link $\ell \cup C$ with unknotted component $C$. In this case the Laplacian polynomial of $G$ is related to the Alexander polynomial of the link. Lehmer's question, an open question about the roots of monic integral polynomials, is equivalent to a question about the complexity growth of signed 1-periodic graphs that are not necessarily embedded.
△ Less
Submitted 12 August, 2020;
originally announced August 2020.
-
Discovering Reinforcement Learning Algorithms
Authors:
Junhyuk Oh,
Matteo Hessel,
Wojciech M. Czarnecki,
Zhongwen Xu,
Hado van Hasselt,
Satinder Singh,
David Silver
Abstract:
Reinforcement learning (RL) algorithms update an agent's parameters according to one of several possible rules, discovered manually through years of research. Automating the discovery of update rules from data could lead to more efficient algorithms, or algorithms that are better adapted to specific environments. Although there have been prior attempts at addressing this significant scientific cha…
▽ More
Reinforcement learning (RL) algorithms update an agent's parameters according to one of several possible rules, discovered manually through years of research. Automating the discovery of update rules from data could lead to more efficient algorithms, or algorithms that are better adapted to specific environments. Although there have been prior attempts at addressing this significant scientific challenge, it remains an open question whether it is feasible to discover alternatives to fundamental concepts of RL such as value functions and temporal-difference learning. This paper introduces a new meta-learning approach that discovers an entire update rule which includes both 'what to predict' (e.g. value functions) and 'how to learn from it' (e.g. bootstrapping) by interacting with a set of environments. The output of this method is an RL algorithm that we call Learned Policy Gradient (LPG). Empirical results show that our method discovers its own alternative to the concept of value functions. Furthermore it discovers a bootstrapping mechanism to maintain and use its predictions. Surprisingly, when trained solely on toy environments, LPG generalises effectively to complex Atari games and achieves non-trivial performance. This shows the potential to discover general RL algorithms from data.
△ Less
Submitted 5 January, 2021; v1 submitted 17 July, 2020;
originally announced July 2020.
-
Meta-Gradient Reinforcement Learning with an Objective Discovered Online
Authors:
Zhongwen Xu,
Hado van Hasselt,
Matteo Hessel,
Junhyuk Oh,
Satinder Singh,
David Silver
Abstract:
Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its o…
▽ More
Deep reinforcement learning includes a broad family of algorithms that parameterise an internal representation, such as a value function or policy, by a deep neural network. Each algorithm optimises its parameters with respect to an objective, such as Q-learning or policy gradient, that defines its semantics. In this work, we propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network, solely from interactive experience with its environment. Over time, this allows the agent to learn how to learn increasingly effectively. Furthermore, because the objective is discovered online, it can adapt to changes over time. We demonstrate that the algorithm discovers how to address several important issues in RL, such as bootstrapping, non-stationarity, and off-policy learning. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency, eventually outperforming the median score of a strong actor-critic baseline.
△ Less
Submitted 16 July, 2020;
originally announced July 2020.
-
Expected Eligibility Traces
Authors:
Hado van Hasselt,
Sephora Madjiheurem,
Matteo Hessel,
David Silver,
André Barreto,
Diana Borsa
Abstract:
The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that c…
▽ More
The question of how to determine which states and actions are responsible for a certain outcome is known as the credit assignment problem and remains a central research question in reinforcement learning and artificial intelligence. Eligibility traces enable efficient credit assignment to the recent sequence of states and actions experienced by the agent, but not to counterfactual sequences that could also have led to the current state. In this work, we introduce expected eligibility traces. Expected traces allow, with a single update, to update states and actions that could have preceded the current state, even if they did not do so on this occasion. We discuss when expected traces provide benefits over classic (instantaneous) traces in temporal-difference learning, and show that sometimes substantial improvements can be attained. We provide a way to smoothly interpolate between instantaneous and expected traces by a mechanism similar to bootstrapping, which ensures that the resulting algorithm is a strict generalisation of TD($λ$). Finally, we discuss possible extensions and connections to related ideas, such as successor features.
△ Less
Submitted 8 February, 2021; v1 submitted 3 July, 2020;
originally announced July 2020.
-
The Value-Improvement Path: Towards Better Representations for Reinforcement Learning
Authors:
Will Dabney,
André Barreto,
Mark Rowland,
Robert Dadashi,
John Quan,
Marc G. Bellemare,
David Silver
Abstract:
In value-based reinforcement learning (RL), unlike in supervised learning, the agent faces not a single, stationary, approximation problem, but a sequence of value prediction problems. Each time the policy improves, the nature of the problem changes, shifting both the distribution of states and their values. In this paper we take a novel perspective, arguing that the value prediction problems face…
▽ More
In value-based reinforcement learning (RL), unlike in supervised learning, the agent faces not a single, stationary, approximation problem, but a sequence of value prediction problems. Each time the policy improves, the nature of the problem changes, shifting both the distribution of states and their values. In this paper we take a novel perspective, arguing that the value prediction problems faced by an RL agent should not be addressed in isolation, but rather as a single, holistic, prediction problem. An RL algorithm generates a sequence of policies that, at least approximately, improve towards the optimal policy. We explicitly characterize the associated sequence of value functions and call it the value-improvement path. Our main idea is to approximate the value-improvement path holistically, rather than to solely track the value function of the current policy. Specifically, we discuss the impact that this holistic view of RL has on representation learning. We demonstrate that a representation that spans the past value-improvement path will also provide an accurate value approximation for future policy improvements. We use this insight to better understand existing approaches to auxiliary tasks and to propose new ones. To test our hypothesis empirically, we augmented a standard deep RL agent with an auxiliary task of learning the value-improvement path. In a study of Atari 2600 games, the augmented agent achieved approximately double the mean and median performance of the baseline agent.
△ Less
Submitted 4 January, 2021; v1 submitted 3 June, 2020;
originally announced June 2020.
-
Data-Driven Prediction of Embryo Implantation Probability Using IVF Time-lapse Imaging
Authors:
David H. Silver,
Martin Feder,
Yael Gold-Zamir,
Avital L. Polsky,
Shahar Rosentraub,
Efrat Shachor,
Adi Weinberger,
Pavlo Mazur,
Valery D. Zukin,
Alex M. Bronstein
Abstract:
The process of fertilizing a human egg outside the body in order to help those suffering from infertility to conceive is known as in vitro fertilization (IVF). Despite being the most effective method of assisted reproductive technology (ART), the average success rate of IVF is a mere 20-40%. One step that is critical to the success of the procedure is selecting which embryo to transfer to the pati…
▽ More
The process of fertilizing a human egg outside the body in order to help those suffering from infertility to conceive is known as in vitro fertilization (IVF). Despite being the most effective method of assisted reproductive technology (ART), the average success rate of IVF is a mere 20-40%. One step that is critical to the success of the procedure is selecting which embryo to transfer to the patient, a process typically conducted manually and without any universally accepted and standardized criteria. In this paper we describe a novel data-driven system trained to directly predict embryo implantation probability from embryogenesis time-lapse imaging videos. Using retrospectively collected videos from 272 embryos, we demonstrate that, when compared to an external panel of embryologists, our algorithm results in a 12% increase of positive predictive value and a 29% increase of negative predictive value.
△ Less
Submitted 2 June, 2020; v1 submitted 1 June, 2020;
originally announced June 2020.
-
Group Presentations for Links in Thickened Surfaces
Authors:
Daniel S. Silver,
Susan G. Williams
Abstract:
Using a combinatorial argument, we prove the well-known result that the Wirtinger and Dehn presentations of a link in 3-space describe isomorphic groups. The result is not true for links $\ell$ in a thickened surface $S \times [0,1]$. Their precise relationship, as given in the 2012 thesis of R.E. Byrd, is established here by an elementary argument. When a diagram in $S$ for $\ell$ can be checkerb…
▽ More
Using a combinatorial argument, we prove the well-known result that the Wirtinger and Dehn presentations of a link in 3-space describe isomorphic groups. The result is not true for links $\ell$ in a thickened surface $S \times [0,1]$. Their precise relationship, as given in the 2012 thesis of R.E. Byrd, is established here by an elementary argument. When a diagram in $S$ for $\ell$ can be checkerboard shaded, the Dehn presentation leads naturally to an abelian "Dehn coloring group," an isotopy invariant of $\ell$. Introducing homological information from $S$ produces a stronger invariant, $\cal C$, a module over the group ring of $H_1(S; {\mathbb Z})$. The authors previously defined the Laplacian modules ${\cal L}_G,{ \cal L}_{G^*}$ and polynomials $Δ_G, Δ_{G^*}$ associated to a Tait graph $G$ and its dual $G^*$, and showed that the pairs $\{{\cal L}_G, {\cal L}_{G^*}\}$, $\{Δ_G, Δ_{G^*}\}$ are isotopy invariants of $\ell$. The relationship between $\cal C$ and the Laplacian modules is described and used to prove that $Δ_G$ and $Δ_{G^*}$ are equal when $S$ is a torus.
△ Less
Submitted 4 May, 2020;
originally announced May 2020.
-
Estimating Grape Yield on the Vine from Multiple Images
Authors:
Daniel L. Silver,
Jabun Nasa
Abstract:
Estimating grape yield prior to harvest is important to commercial vineyard production as it informs many vineyard and winery decisions. Currently, the process of yield estimation is time consuming and varies in its accuracy from 75-90\% depending on the experience of the viticulturist. This paper proposes a multiple task learning (MTL) convolutional neural network (CNN) approach that uses images…
▽ More
Estimating grape yield prior to harvest is important to commercial vineyard production as it informs many vineyard and winery decisions. Currently, the process of yield estimation is time consuming and varies in its accuracy from 75-90\% depending on the experience of the viticulturist. This paper proposes a multiple task learning (MTL) convolutional neural network (CNN) approach that uses images captured by inexpensive smart phones secured in a simple tripod arrangement. The CNN models use MTL transfer from autoencoders to achieve 85\% accuracy from image data captured 6 days prior to harvest.
△ Less
Submitted 8 April, 2020;
originally announced April 2020.
-
A Self-Tuning Actor-Critic Algorithm
Authors:
Tom Zahavy,
Zhongwen Xu,
Vivek Veeriah,
Matteo Hessel,
Junhyuk Oh,
Hado van Hasselt,
David Silver,
Satinder Singh
Abstract:
Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain. In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018). We apply our algorithm, Self-…
▽ More
Reinforcement learning algorithms are highly sensitive to the choice of hyperparameters, typically requiring significant manual effort to identify hyperparameters that perform well on a new domain. In this paper, we take a step towards addressing this issue by using metagradients to automatically adapt hyperparameters online by meta-gradient descent (Xu et al., 2018). We apply our algorithm, Self-Tuning Actor-Critic (STAC), to self-tune all the differentiable hyperparameters of an actor-critic loss function, to discover auxiliary tasks, and to improve off-policy learning using a novel leaky V-trace operator. STAC is simple to use, sample efficient and does not require a significant increase in compute. Ablative studies show that the overall performance of STAC improved as we adapt more hyperparameters. When applied to the Arcade Learning Environment (Bellemare et al. 2012), STAC improved the median human normalized score in 200M steps from 243% to 364%. When applied to the DM Control suite (Tassa et al., 2018), STAC improved the mean score in 30M steps from 217 to 389 when learning with features, from 108 to 202 when learning from pixels, and from 195 to 295 in the Real-World Reinforcement Learning Challenge (Dulac-Arnold et al., 2020).
△ Less
Submitted 14 April, 2021; v1 submitted 28 February, 2020;
originally announced February 2020.
-
Links in Surfaces and Laplacian Modules
Authors:
Daniel S. Silver,
Susan G. Williams
Abstract:
Laplacian matrices of weighted graphs in surfaces $S$ are used to define module and polynomial invariants of $Z/2$-homologically trivial links in $S \times [0,1]$. Information about virtual genus is obtained.
Laplacian matrices of weighted graphs in surfaces $S$ are used to define module and polynomial invariants of $Z/2$-homologically trivial links in $S \times [0,1]$. Information about virtual genus is obtained.
△ Less
Submitted 23 February, 2020;
originally announced February 2020.
-
Value-driven Hindsight Modelling
Authors:
Arthur Guez,
Fabio Viola,
Théophane Weber,
Lars Buesing,
Steven Kapturowski,
Doina Precup,
David Silver,
Nicolas Heess
Abstract:
Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn value predictors from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but t…
▽ More
Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn value predictors from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future, but receive a potentially weak scalar signal (an estimate of the return). We develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end, we determine which features of the future trajectory provide useful information to predict the associated return. This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.
△ Less
Submitted 20 October, 2020; v1 submitted 19 February, 2020;
originally announced February 2020.
-
A Bounded Measure for Estimating the Benefit of Visualization
Authors:
Min Chen,
Mateu Sbert,
Alfie Abdul-Rahman,
Deborah Silver
Abstract:
Information theory can be used to analyze the cost-benefit of visualization processes. However, the current measure of benefit contains an unbounded term that is neither easy to estimate nor intuitive to interpret. In this work, we propose to revise the existing cost-benefit measure by replacing the unbounded term with a bounded one. We examine a number of bounded measures that include the Jenson-…
▽ More
Information theory can be used to analyze the cost-benefit of visualization processes. However, the current measure of benefit contains an unbounded term that is neither easy to estimate nor intuitive to interpret. In this work, we propose to revise the existing cost-benefit measure by replacing the unbounded term with a bounded one. We examine a number of bounded measures that include the Jenson-Shannon divergence and a new divergence measure formulated as part of this work. We use visual analysis to support the multi-criteria comparison, narrowing the search down to those options with better mathematical properties. We apply those remaining options to two visualization case studies to instantiate their uses in practical scenarios, while the collected real world data further informs the selection of a bounded measure, which can be used to estimate the benefit of visualization.
△ Less
Submitted 25 July, 2020; v1 submitted 12 February, 2020;
originally announced February 2020.
-
What Can Learned Intrinsic Rewards Capture?
Authors:
Zeyu Zheng,
Junhyuk Oh,
Matteo Hessel,
Zhongwen Xu,
Manuel Kroiss,
Hado van Hasselt,
David Silver,
Satinder Singh
Abstract:
The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful…
▽ More
The objective of a reinforcement learning agent is to behave so as to maximise the sum of a suitable scalar function of state: the reward. These rewards are typically given and immutable. In this paper, we instead consider the proposition that the reward function itself can be a good locus of learned knowledge. To investigate this, we propose a scalable meta-gradient framework for learning useful intrinsic reward functions across multiple lifetimes of experience. Through several proof-of-concept experiments, we show that it is feasible to learn and capture knowledge about long-term exploration and exploitation into a reward function. Furthermore, we show that unlike policy transfer methods that capture "how" the agent should behave, the learned reward functions can generalise to other kinds of agents and to changes in the dynamics of the environment by capturing "what" the agent should strive to do.
△ Less
Submitted 21 August, 2020; v1 submitted 11 December, 2019;
originally announced December 2019.
-
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
Authors:
Julian Schrittwieser,
Ioannis Antonoglou,
Thomas Hubert,
Karen Simonyan,
Laurent Sifre,
Simon Schmitt,
Arthur Guez,
Edward Lockhart,
Demis Hassabis,
Thore Graepel,
Timothy Lillicrap,
David Silver
Abstract:
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the…
▽ More
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games - the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled - our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.
△ Less
Submitted 21 February, 2020; v1 submitted 19 November, 2019;
originally announced November 2019.
-
Discovery of Useful Questions as Auxiliary Tasks
Authors:
Vivek Veeriah,
Matteo Hessel,
Zhongwen Xu,
Richard Lewis,
Janarthanan Rajendran,
Junhyuk Oh,
Hado van Hasselt,
David Silver,
Satinder Singh
Abstract:
Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value…
▽ More
Arguably, intelligent agents ought to be able to discover their own questions so that in learning answers for them they learn unanticipated useful knowledge and skills; this departs from the focus in much of machine learning on agents learning answers to externally defined questions. We present a novel method for a reinforcement learning (RL) agent to discover questions formulated as general value functions or GVFs, a fairly rich form of knowledge representation. Specifically, our method uses non-myopic meta-gradients to learn GVF-questions such that learning answers to them, as an auxiliary task, induces useful representations for the main task faced by the RL agent. We demonstrate that auxiliary tasks based on the discovered GVFs are sufficient, on their own, to build representations that support main task learning, and that they do so better than popular hand-designed auxiliary tasks from the literature. Furthermore, we show, in the context of Atari 2600 videogames, how such auxiliary tasks, meta-learned alongside the main task, can improve the data efficiency of an actor-critic agent.
△ Less
Submitted 10 September, 2019;
originally announced September 2019.
-
Behaviour Suite for Reinforcement Learning
Authors:
Ian Osband,
Yotam Doron,
Matteo Hessel,
John Aslanides,
Eren Sezener,
Andre Saraiva,
Katrina McKinney,
Tor Lattimore,
Csaba Szepesvari,
Satinder Singh,
Benjamin Van Roy,
Richard Sutton,
David Silver,
Hado Van Hasselt
Abstract:
This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to stud…
▽ More
This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github.com/deepmind/bsuite, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.
△ Less
Submitted 14 February, 2020; v1 submitted 9 August, 2019;
originally announced August 2019.
-
On Inductive Biases in Deep Reinforcement Learning
Authors:
Matteo Hessel,
Hado van Hasselt,
Joseph Modayil,
David Silver
Abstract:
Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent's objective and its interface to the environment. These inductive biases can take many forms, including domain knowledge and pretuned hyper-parameters. In general, there is a trade-off between generality and performance when algorithms use such biases. Stronger biases can lead to faster learning, but weaker…
▽ More
Many deep reinforcement learning algorithms contain inductive biases that sculpt the agent's objective and its interface to the environment. These inductive biases can take many forms, including domain knowledge and pretuned hyper-parameters. In general, there is a trade-off between generality and performance when algorithms use such biases. Stronger biases can lead to faster learning, but weaker biases can potentially lead to more general algorithms. This trade-off is important because inductive biases are not free; substantial effort may be required to obtain relevant domain knowledge or to tune hyper-parameters effectively. In this paper, we re-examine several domain-specific components that bias the objective and the environmental interface of common deep reinforcement learning agents. We investigated whether the performance deteriorates when these components are replaced with adaptive solutions from the literature. In our experiments, performance sometimes decreased with the adaptive components, as one might expect when comparing to components crafted for the domain, but sometimes the adaptive components performed better. We investigated the main benefit of having fewer domain-specific components, by comparing the learning performance of the two systems on a different set of continuous control problems, without additional tuning of either system. As hypothesized, the system with adaptive components performed better on many of the new tasks.
△ Less
Submitted 5 July, 2019;
originally announced July 2019.
-
AutoML @ NeurIPS 2018 challenge: Design and Results
Authors:
Hugo Jair Escalante,
Wei-Wei Tu,
Isabelle Guyon,
Daniel L. Silver,
Evelyne Viegas,
Yuqiang Chen,
Wenyuan Dai,
Qiang Yang
Abstract:
We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets were arranged in a lifelong learning and evaluation scenario and CodaLab was used…
▽ More
We organized a competition on Autonomous Lifelong Machine Learning with Drift that was part of the competition program of NeurIPS 2018. This data driven competition asked participants to develop computer programs capable of solving supervised learning problems where the i.i.d. assumption did not hold. Large data sets were arranged in a lifelong learning and evaluation scenario and CodaLab was used as the challenge platform. The challenge attracted more than 300 participants in its two month duration. This chapter describes the design of the challenge and summarizes its main results.
△ Less
Submitted 13 March, 2019; v1 submitted 12 March, 2019;
originally announced March 2019.
-
Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement
Authors:
André Barreto,
Diana Borsa,
John Quan,
Tom Schaul,
David Silver,
Matteo Hessel,
Daniel Mankowitz,
Augustin Žídek,
Rémi Munos
Abstract:
The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills. In this paper we extend the SFs & GPI framework in two ways. One of the basic…
▽ More
The ability to transfer skills across tasks has the potential to scale up reinforcement learning (RL) agents to environments currently out of reach. Recently, a framework based on two ideas, successor features (SFs) and generalised policy improvement (GPI), has been introduced as a principled way of transferring skills. In this paper we extend the SFs & GPI framework in two ways. One of the basic assumptions underlying the original formulation of SFs & GPI is that rewards for all tasks of interest can be computed as linear combinations of a fixed set of features. We relax this constraint and show that the theoretical guarantees supporting the framework can be extended to any set of tasks that only differ in the reward function. Our second contribution is to show that one can use the reward functions themselves as features for future tasks, without any loss of expressiveness, thus removing the need to specify a set of features beforehand. This makes it possible to combine SFs & GPI with deep learning in a more stable way. We empirically verify this claim on a complex 3D environment where observations are images from a first-person perspective. We show that the transfer promoted by SFs & GPI leads to very good policies on unseen tasks almost instantaneously. We also describe how to learn policies specialised to the new tasks in a way that allows them to be added to the agent's set of skills, and thus be reused in the future.
△ Less
Submitted 30 January, 2019;
originally announced January 2019.
-
An investigation of model-free planning
Authors:
Arthur Guez,
Mehdi Mirza,
Karol Gregor,
Rishabh Kabra,
Sébastien Racanière,
Théophane Weber,
David Raposo,
Adam Santoro,
Laurent Orseau,
Tom Eccles,
Greg Wayne,
David Silver,
Timothy Lillicrap
Abstract:
The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been propos…
▽ More
The field of reinforcement learning (RL) is facing increasingly challenging domains with combinatorial complexity. For an RL agent to address these challenges, it is essential that it can plan effectively. Prior work has typically utilized an explicit model of the environment, combined with a specific planning algorithm (such as tree search). More recently, a new family of methods have been proposed that learn how to plan, by providing the structure for planning via an inductive bias in the function approximator (such as a tree structured neural network), trained end-to-end by a model-free RL algorithm. In this paper, we go even further, and demonstrate empirically that an entirely model-free approach, without special structure beyond standard neural network components such as convolutional networks and LSTMs, can learn to exhibit many of the characteristics typically associated with a model-based planner. We measure our agent's effectiveness at planning in terms of its ability to generalize across a combinatorial and irreversible state space, its data efficiency, and its ability to utilize additional thinking time. We find that our agent has many of the characteristics that one might expect to find in a planning algorithm. Furthermore, it exceeds the state-of-the-art in challenging combinatorial domains such as Sokoban and outperforms other model-free approaches that utilize strong inductive biases toward planning.
△ Less
Submitted 20 May, 2019; v1 submitted 11 January, 2019;
originally announced January 2019.
-
Credit Assignment Techniques in Stochastic Computation Graphs
Authors:
Théophane Weber,
Nicolas Heess,
Lars Buesing,
David Silver
Abstract:
Stochastic computation graphs (SCGs) provide a formalism to represent structured optimization problems arising in artificial intelligence, including supervised, unsupervised, and reinforcement learning. Previous work has shown that an unbiased estimator of the gradient of the expected loss of SCGs can be derived from a single principle. However, this estimator often has high variance and requires…
▽ More
Stochastic computation graphs (SCGs) provide a formalism to represent structured optimization problems arising in artificial intelligence, including supervised, unsupervised, and reinforcement learning. Previous work has shown that an unbiased estimator of the gradient of the expected loss of SCGs can be derived from a single principle. However, this estimator often has high variance and requires a full model evaluation per data point, making this algorithm costly in large graphs. In this work, we address these problems by generalizing concepts from the reinforcement learning literature. We introduce the concepts of value functions, baselines and critics for arbitrary SCGs, and show how to use them to derive lower-variance gradient estimates from partial model evaluations, paving the way towards general and efficient credit assignment for gradient-based optimization. In doing so, we demonstrate how our results unify recent advances in the probabilistic inference and reinforcement learning literature.
△ Less
Submitted 7 January, 2019;
originally announced January 2019.
-
Universal Successor Features Approximators
Authors:
Diana Borsa,
André Barreto,
John Quan,
Daniel Mankowitz,
Rémi Munos,
Hado van Hasselt,
David Silver,
Tom Schaul
Abstract:
The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpol…
▽ More
The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.
△ Less
Submitted 18 December, 2018;
originally announced December 2018.
-
Bayesian Optimization in AlphaGo
Authors:
Yutian Chen,
Aja Huang,
Ziyu Wang,
Ioannis Antonoglou,
Julian Schrittwieser,
David Silver,
Nando de Freitas
Abstract:
During the development of AlphaGo, its many hyper-parameters were tuned with Bayesian optimization multiple times. This automatic tuning process resulted in substantial improvements in playing strength. For example, prior to the match with Lee Sedol, we tuned the latest AlphaGo agent and this improved its win-rate from 50% to 66.5% in self-play games. This tuned version was deployed in the final m…
▽ More
During the development of AlphaGo, its many hyper-parameters were tuned with Bayesian optimization multiple times. This automatic tuning process resulted in substantial improvements in playing strength. For example, prior to the match with Lee Sedol, we tuned the latest AlphaGo agent and this improved its win-rate from 50% to 66.5% in self-play games. This tuned version was deployed in the final match. Of course, since we tuned AlphaGo many times during its development cycle, the compounded contribution was even higher than this percentage. It is our hope that this brief case study will be of interest to Go fans, and also provide Bayesian optimization practitioners with some insights and inspiration.
△ Less
Submitted 17 December, 2018;
originally announced December 2018.