-
Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures
Authors:
Zirui Fu,
Aleksandre Avaliani,
Marco Donato
Abstract:
Executing machine learning inference tasks on resource-constrained edge devices requires careful hardware-software co-design optimizations. Recent examples have shown how transformer-based deep neural network models such as ALBERT can be used to enable the execution of natural language processing (NLP) inference on mobile systems-on-chip housing custom hardware accelerators. However, while these e…
▽ More
Executing machine learning inference tasks on resource-constrained edge devices requires careful hardware-software co-design optimizations. Recent examples have shown how transformer-based deep neural network models such as ALBERT can be used to enable the execution of natural language processing (NLP) inference on mobile systems-on-chip housing custom hardware accelerators. However, while these existing solutions are effective in alleviating the latency, energy, and area costs of running single NLP tasks, achieving multi-task inference requires running computations over multiple variants of the model parameters, which are tailored to each of the targeted tasks. This approach leads to either prohibitive on-chip memory requirements or paying the cost of off-chip memory access. This paper proposes adapter-ALBERT, an efficient model optimization for maximal data reuse across different tasks. The proposed model's performance and robustness to data compression methods are evaluated across several language tasks from the GLUE benchmark. Additionally, we demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator to extrapolate performance, power, and area improvements over the execution of a traditional ALBERT model on the same hardware platform.
△ Less
Submitted 12 April, 2023; v1 submitted 25 March, 2023;
originally announced March 2023.
-
NVMExplorer: A Framework for Cross-Stack Comparisons of Embedded Non-Volatile Memories
Authors:
Lillian Pentecost,
Alexander Hankin,
Marco Donato,
Mark Hempstead,
Gu-Yeon Wei,
David Brooks
Abstract:
Repeated off-chip memory accesses to DRAM drive up operating power for data-intensive applications, and SRAM technology scaling and leakage power limits the efficiency of embedded memories. Future on-chip storage will need higher density and energy efficiency, and the actively expanding field of emerging, embeddable non-volatile memory (eNVM) technologies is providing many potential candidates to…
▽ More
Repeated off-chip memory accesses to DRAM drive up operating power for data-intensive applications, and SRAM technology scaling and leakage power limits the efficiency of embedded memories. Future on-chip storage will need higher density and energy efficiency, and the actively expanding field of emerging, embeddable non-volatile memory (eNVM) technologies is providing many potential candidates to satisfy this need. Each technology proposal presents distinct trade-offs in terms of density, read, write, and reliability characteristics, and we present a comprehensive framework for navigating and quantifying these design trade-offs alongside realistic system constraints and application-level impacts. This work evaluates eNVM-based storage for a range of application and system contexts including machine learning on the edge, graph analytics, and general purpose cache hierarchy, in addition to describing a freely available (http://nvmexplorer.seas.harvard.edu/) set of tools for application experts, system designers, and device experts to better understand, compare, and quantify the next generation of embedded memory solutions.
△ Less
Submitted 11 January, 2022; v1 submitted 2 September, 2021;
originally announced September 2021.
-
Application-driven Design Exploration for Dense Ferroelectric Embedded Non-volatile Memories
Authors:
Mohammad Mehdi Sharifi,
Lillian Pentecost,
Ramin Rajaei,
Arman Kazemi,
Qiuwen Lou,
Gu-Yeon Wei,
David Brooks,
Kai Ni,
X. Sharon Hu,
Michael Niemier,
Marco Donato
Abstract:
The memory wall bottleneck is a key challenge across many data-intensive applications. Multi-level FeFET-based embedded non-volatile memories are a promising solution for denser and more energy-efficient on-chip memory. However, reliable multi-level cell storage requires careful optimizations to minimize the design overhead costs. In this work, we investigate the interplay between FeFET device cha…
▽ More
The memory wall bottleneck is a key challenge across many data-intensive applications. Multi-level FeFET-based embedded non-volatile memories are a promising solution for denser and more energy-efficient on-chip memory. However, reliable multi-level cell storage requires careful optimizations to minimize the design overhead costs. In this work, we investigate the interplay between FeFET device characteristics, programming schemes, and memory array architecture, and explore different design choices to optimize performance, energy, area, and accuracy metrics for critical data-intensive workloads. From our cross-stack design exploration, we find that we can store DNN weights and social network graphs at a density of over 8MB/mm^2 and sub-2ns read access latency without loss in application accuracy.
△ Less
Submitted 17 June, 2021;
originally announced June 2021.
-
EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference
Authors:
Thierry Tambe,
Coleman Hooper,
Lillian Pentecost,
Tianyu Jia,
En-Yu Yang,
Marco Donato,
Victor Sanh,
Paul N. Whatmough,
Alexander M. Rush,
David Brooks,
Gu-Yeon Wei
Abstract:
Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. However, their hefty computational and memory demands make them challenging to deploy to resource-constrained edge platforms with strict latency requirements. We present EdgeBERT, an in-depth algorithm-hardware co-design for latency-aware energy optimi…
▽ More
Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. However, their hefty computational and memory demands make them challenging to deploy to resource-constrained edge platforms with strict latency requirements. We present EdgeBERT, an in-depth algorithm-hardware co-design for latency-aware energy optimization for multi-task NLP. EdgeBERT employs entropy-based early exit predication in order to perform dynamic voltage-frequency scaling (DVFS), at a sentence granularity, for minimal energy consumption while adhering to a prescribed target latency. Computation and memory footprint overheads are further alleviated by employing a calibrated combination of adaptive attention span, selective network pruning, and floating-point quantization. Furthermore, in order to maximize the synergistic benefits of these algorithms in always-on and intermediate edge computing settings, we specialize a 12nm scalable hardware accelerator system, integrating a fast-switching low-dropout voltage regulator (LDO), an all-digital phase-locked loop (ADPLL), as well as, high-density embedded non-volatile memories (eNVMs) wherein the sparse floating-point bit encodings of the shared multi-task parameters are carefully stored. Altogether, latency-aware multi-task NLP inference acceleration on the EdgeBERT hardware system generates up to 7x, 2.5x, and 53x lower energy compared to the conventional inference without early stopping, the latency-unbounded early exit approach, and CUDA adaptations on an Nvidia Jetson Tegra X2 mobile GPU, respectively.
△ Less
Submitted 5 September, 2021; v1 submitted 28 November, 2020;
originally announced November 2020.
-
CHIPKIT: An agile, reusable open-source framework for rapid test chip development
Authors:
Paul Whatmough,
Marco Donato,
Glenn Ko,
Sae-Kyu Lee,
David Brooks,
Gu-Yeon Wei
Abstract:
The current trend for domain-specific architectures (DSAs) has led to renewed interest in research test chips to demonstrate new specialized hardware. Tape-outs also offer huge pedagogical value garnered from real hands-on exposure to the whole system stack. However, successful tape-outs demand hard-earned experience, and the design process is time consuming and fraught with challenges. Therefore,…
▽ More
The current trend for domain-specific architectures (DSAs) has led to renewed interest in research test chips to demonstrate new specialized hardware. Tape-outs also offer huge pedagogical value garnered from real hands-on exposure to the whole system stack. However, successful tape-outs demand hard-earned experience, and the design process is time consuming and fraught with challenges. Therefore, custom chips have remained the preserve of a small number of research groups, typically focused on circuit design research. This paper describes the CHIPKIT framework. We describe a reusable SoC subsystem which provides basic IO, an on-chip programmable host, memory and peripherals. This subsystem can be readily extended with new IP blocks to generate custom test chips. We also present an agile RTL development flow, including a code generation tool calledVGEN. Finally, we outline best practices for full-chip validation across the entire design cycle.
△ Less
Submitted 26 May, 2020; v1 submitted 13 January, 2020;
originally announced January 2020.
-
Multiscale Modelling: A Mobile Membrane Approach
Authors:
Federico Buti,
Massimo Callisto De Donato,
Flavio Corradini,
Emanuela Merelli,
Luca Tesei
Abstract:
Nowadays, multiscale modelling is recognized as the most suitable way to study biological processes. Indeed, almost every phenomenon in nature exhibits a multiscale behaviour, i.e., it is the outcome of interactions that occur at different spatial and temporal scales. Although several ways to provide "multilayer" models have been proposed, only Complex Automata naturally embed spatial information…
▽ More
Nowadays, multiscale modelling is recognized as the most suitable way to study biological processes. Indeed, almost every phenomenon in nature exhibits a multiscale behaviour, i.e., it is the outcome of interactions that occur at different spatial and temporal scales. Although several ways to provide "multilayer" models have been proposed, only Complex Automata naturally embed spatial information and realize the multiscale approach with well-established inter-scale integration schemas. Recently, such approach has been restated in terms of Spatial P systems - a variant of P systems with a more geometric concept of space.
In this work we discuss how mobile membranes, a variant of membrane systems inspired by the biological movements of endocytosis and exocytosis, can be efficaciously exploited to define a uniform multiscale coupling scheme relying only on the features of the formalism itself.
△ Less
Submitted 17 August, 2011;
originally announced August 2011.
-
Automated Analysis of MUTEX Algorithms with FASE
Authors:
Federico Buti,
Massimo Callisto De Donato,
Flavio Corradini,
Maria Rita Di Berardini,
Walter Vogler
Abstract:
In this paper we study the liveness of several MUTEX solutions by representing them as processes in PAFAS s, a CCS-like process algebra with a specific operator for modelling non-blocking reading behaviours. Verification is carried out using the tool FASE, exploiting a correspondence between violations of the liveness property and a special kind of cycles (called catastrophic cycles) in some trans…
▽ More
In this paper we study the liveness of several MUTEX solutions by representing them as processes in PAFAS s, a CCS-like process algebra with a specific operator for modelling non-blocking reading behaviours. Verification is carried out using the tool FASE, exploiting a correspondence between violations of the liveness property and a special kind of cycles (called catastrophic cycles) in some transition system. We also compare our approach with others in the literature. The aim of this paper is twofold: on the one hand, we want to demonstrate the applicability of FASE to some concrete, meaningful examples; on the other hand, we want to study the impact of introducing non-blocking behaviours in modelling concurrent systems.
△ Less
Submitted 6 June, 2011;
originally announced June 2011.
-
A Framework for the Evaluation of Worst-Case System Efficiency
Authors:
Massimo Callisto De Donato,
Maria Rita Di Berardini
Abstract:
In this paper we present FASE (Fast Asynchronous Systems Evaluation), a tool for evaluating worst-case efficiency of asynchronous systems. This tool implements some well-established results in the setting of a timed CCS-like process algebra: PAFAS (a Process Algebra for Faster Asynchronous Systems). Moreover, we discuss some new solutions that are useful to improve the applicability of FASE to con…
▽ More
In this paper we present FASE (Fast Asynchronous Systems Evaluation), a tool for evaluating worst-case efficiency of asynchronous systems. This tool implements some well-established results in the setting of a timed CCS-like process algebra: PAFAS (a Process Algebra for Faster Asynchronous Systems). Moreover, we discuss some new solutions that are useful to improve the applicability of FASE to concrete meaningful examples. We finally use fase to evaluate the efficiency of three different implementations of a bounded buffer and compare our results with previous ones obtained when the same implementations have been contrasted according to an efficiency preorder.
△ Less
Submitted 6 May, 2011;
originally announced May 2011.
-
Evaluating the Efficiency of Asynchronous Systems with FASE
Authors:
Federico Buti,
Massimo Callisto De Donato,
Flavio Corradini,
Maria Rita Di Berardini,
Walter Vogler
Abstract:
In this paper, we present FASE (Faster Asynchronous Systems Evaluation), a tool for evaluating the worst-case efficiency of asynchronous systems. The tool is based on some well-established results in the setting of a timed process algebra (PAFAS: a Process Algebra for Faster Asynchronous Systems). To show the applicability of FASE to concrete meaningful examples, we consider three implementations…
▽ More
In this paper, we present FASE (Faster Asynchronous Systems Evaluation), a tool for evaluating the worst-case efficiency of asynchronous systems. The tool is based on some well-established results in the setting of a timed process algebra (PAFAS: a Process Algebra for Faster Asynchronous Systems). To show the applicability of FASE to concrete meaningful examples, we consider three implementations of a bounded buffer and use FASE to automatically evaluate their worst-case efficiency. We finally contrast our results with previous ones where the efficiency of the same implementations has already been considered.
△ Less
Submitted 6 May, 2011;
originally announced May 2011.
-
EcoHomeHelper: An Expert System to Empower End-Users in Climate Change Action
Authors:
Matt Donato,
Seng W. Loke
Abstract:
Climate change has been a popular topic for a number of years now. Computer Science has contributed to aiding humanity in reducing energy requirements and consequently global warming. Much of this work is through calculators which determine a user's carbon footprint. However there are no expert systems which can offer advice in an efficient and time saving way. There are many publications which do…
▽ More
Climate change has been a popular topic for a number of years now. Computer Science has contributed to aiding humanity in reducing energy requirements and consequently global warming. Much of this work is through calculators which determine a user's carbon footprint. However there are no expert systems which can offer advice in an efficient and time saving way. There are many publications which do offer advice on reducing greenhouse gas (GHG) emissions but to find the advice the reader seeks will involve reading a lot of irrelevant material. This work built an expert system (which we call EcoHomeHelper) and attempted to show that it is useful in changing people's behaviour with respect to their GHG emissions and that they will be able to find the information in a more efficient manner. Twelve participants were used. Seven of which used the program and five who read and attempted to find advice by reading from a list. The application itself has current implementations and the concept further developed, has applications for the future.
△ Less
Submitted 8 June, 2010;
originally announced June 2010.
