-
Layer Ensemble Averaging for Improving Memristor-Based Artificial Neural Network Performance
Authors:
Osama Yousuf,
Brian Hoskins,
Karthick Ramu,
Mitchell Fream,
William A. Borders,
Advait Madhavan,
Matthew W. Daniels,
Andrew Dienstfrey,
Jabez J. McClelland,
Martin Lueker-Boden,
Gina C. Adam
Abstract:
Artificial neural networks have advanced due to scaling dimensions, but conventional computing faces inefficiency due to the von Neumann bottleneck. In-memory computation architectures, like memristors, offer promise but face challenges due to hardware non-idealities. This work proposes and experimentally demonstrates layer ensemble averaging, a technique to map pre-trained neural network solution…
▽ More
Artificial neural networks have advanced due to scaling dimensions, but conventional computing faces inefficiency due to the von Neumann bottleneck. In-memory computation architectures, like memristors, offer promise but face challenges due to hardware non-idealities. This work proposes and experimentally demonstrates layer ensemble averaging, a technique to map pre-trained neural network solutions from software to defective hardware crossbars of emerging memory devices and reliably attain near-software performance on inference. The approach is investigated using a custom 20,000-device hardware prototyping platform on a continual learning problem where a network must learn new tasks without catastrophically forgetting previously learned information. Results demonstrate that by trading off the number of devices required for layer mapping, layer ensemble averaging can reliably boost defective memristive network performance up to the software baseline. For the investigated problem, the average multi-task classification accuracy improves from 61 % to 72 % (< 1 % of software baseline) using the proposed approach.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Measurement-driven neural-network training for integrated magnetic tunnel junction arrays
Authors:
William A. Borders,
Advait Madhavan,
Matthew W. Daniels,
Vasileia Georgiou,
Martin Lueker-Boden,
Tiffany S. Santos,
Patrick M. Braganca,
Mark D. Stiles,
Jabez J. McClelland,
Brian D. Hoskins
Abstract:
The increasing scale of neural networks needed to support more complex applications has led to an increasing requirement for area- and energy-efficient hardware. One route to meeting the budget for these applications is to circumvent the von Neumann bottleneck by performing computation in or near memory. An inevitability of transferring neural networks onto hardware is that non-idealities such as…
▽ More
The increasing scale of neural networks needed to support more complex applications has led to an increasing requirement for area- and energy-efficient hardware. One route to meeting the budget for these applications is to circumvent the von Neumann bottleneck by performing computation in or near memory. An inevitability of transferring neural networks onto hardware is that non-idealities such as device-to-device variations or poor device yield impact performance. Methods such as hardware-aware training, where substrate non-idealities are incorporated during network training, are one way to recover performance at the cost of solution generality. In this work, we demonstrate inference on hardware neural networks consisting of 20,000 magnetic tunnel junction arrays integrated on a complementary metal-oxide-semiconductor chips that closely resembles market-ready spin transfer-torque magnetoresistive random access memory technology. Using 36 dies, each containing a crossbar array with its own non-idealities, we show that even a small number of defects in physically mapped networks significantly degrades the performance of networks trained without defects and show that, at the cost of generality, hardware-aware training accounting for specific defects on each die can recover to comparable performance with ideal networks. We then demonstrate a robust training method that extends hardware-aware training to statistics-aware training, producing network weights that perform well on most defective dies regardless of their specific defect locations. When evaluated on the 36 physical dies, statistics-aware trained solutions can achieve a mean misclassification error on the MNIST dataset that differs from the software-baseline by only 2 %. This statistics-aware training method could be generalized to networks with many layers that are mapped to hardware suited for industry-ready applications.
△ Less
Submitted 14 May, 2024; v1 submitted 11 December, 2023;
originally announced December 2023.
-
Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions
Authors:
Jonathan M. Goodwill,
Nitin Prasad,
Brian D. Hoskins,
Matthew W. Daniels,
Advait Madhavan,
Lei Wan,
Tiffany S. Santos,
Michael Tran,
Jordan A. Katine,
Patrick M. Braganca,
Mark D. Stiles,
Jabez J. McClelland
Abstract:
The increasing scale of neural networks and their growing application space have produced demand for more energy- and memory-efficient artificial-intelligence-specific hardware. Avenues to mitigate the main issue, the von Neumann bottleneck, include in-memory and near-memory architectures, as well as algorithmic approaches. Here we leverage the low-power and the inherently binary operation of magn…
▽ More
The increasing scale of neural networks and their growing application space have produced demand for more energy- and memory-efficient artificial-intelligence-specific hardware. Avenues to mitigate the main issue, the von Neumann bottleneck, include in-memory and near-memory architectures, as well as algorithmic approaches. Here we leverage the low-power and the inherently binary operation of magnetic tunnel junctions (MTJs) to demonstrate neural network hardware inference based on passive arrays of MTJs. In general, transferring a trained network model to hardware for inference is confronted by degradation in performance due to device-to-device variations, write errors, parasitic resistance, and nonidealities in the substrate. To quantify the effect of these hardware realities, we benchmark 300 unique weight matrix solutions of a 2-layer perceptron to classify the Wine dataset for both classification accuracy and write fidelity. Despite device imperfections, we achieve software-equivalent accuracy of up to 95.3 % with proper tuning of network parameters in 15 x 15 MTJ arrays having a range of device sizes. The success of this tuning process shows that new metrics are needed to characterize the performance and quality of networks reproduced in mixed signal hardware.
△ Less
Submitted 6 May, 2022; v1 submitted 16 December, 2021;
originally announced December 2021.
-
Memory-efficient training with streaming dimensionality reduction
Authors:
Siyuan Huang,
Brian D. Hoskins,
Matthew W. Daniels,
Mark D. Stiles,
Gina C. Adam
Abstract:
The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads. To minimize this overhead, especially on the movement and calculation of gradient information, we introduce streaming batch principal component analysis as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations…
▽ More
The movement of large quantities of data during the training of a Deep Neural Network presents immense challenges for machine learning workloads. To minimize this overhead, especially on the movement and calculation of gradient information, we introduce streaming batch principal component analysis as an update algorithm. Streaming batch principal component analysis uses stochastic power iterations to generate a stochastic k-rank approximation of the network gradient. We demonstrate that the low rank updates produced by streaming batch principal component analysis can effectively train convolutional neural networks on a variety of common datasets, with performance comparable to standard mini batch gradient descent. These results can lead to both improvements in the design of application specific integrated circuits for deep learning and in the speed of synchronization of machine learning models trained with data parallelism.
△ Less
Submitted 24 April, 2020;
originally announced April 2020.
-
Streaming Batch Eigenupdates for Hardware Neuromorphic Networks
Authors:
Brian D. Hoskins,
Matthew W. Daniels,
Siyuan Huang,
Advait Madhavan,
Gina C. Adam,
Nikolai Zhitenev,
Jabez J. McClelland,
Mark D. Stiles
Abstract:
Neuromorphic networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs). Though immense acceleration of the training process can be achieved by leveraging the fact that the tim…
▽ More
Neuromorphic networks based on nanodevices, such as metal oxide memristors, phase change memories, and flash memory cells, have generated considerable interest for their increased energy efficiency and density in comparison to graphics processing units (GPUs) and central processing units (CPUs). Though immense acceleration of the training process can be achieved by leveraging the fact that the time complexity of training does not scale with the network size, it is limited by the space complexity of stochastic gradient descent, which grows quadratically. The main objective of this work is to reduce this space complexity by using low-rank approximations of stochastic gradient descent. This low spatial complexity combined with streaming methods allows for significant reductions in memory and compute overhead, opening the doors for improvements in area, time and energy efficiency of training. We refer to this algorithm and architecture to implement it as the streaming batch eigenupdate (SBE) approach.
△ Less
Submitted 4 March, 2019;
originally announced March 2019.
-
A scalable method to find the shortest path in a graph with circuits of memristors
Authors:
Alice Mizrahi,
Thomas Marsh,
Brian Hoskins,
M. D. Stiles
Abstract:
Finding the shortest path in a graph has applications to a wide range of optimization problems. However, algorithmic methods scale with the size of the graph in terms of time and energy. We propose a method to solve the shortest path problem using circuits of nanodevices called memristors and validate it on graphs of different sizes and topologies. It is both valid for an experimentally derived me…
▽ More
Finding the shortest path in a graph has applications to a wide range of optimization problems. However, algorithmic methods scale with the size of the graph in terms of time and energy. We propose a method to solve the shortest path problem using circuits of nanodevices called memristors and validate it on graphs of different sizes and topologies. It is both valid for an experimentally derived memristor model and robust to device variability. The time and energy of the computation scale with the length of the shortest path rather than with the size of the graph, making this method particularly attractive for solving large graphs with small path lengths.
△ Less
Submitted 12 September, 2018;
originally announced September 2018.
-
Three-Dimensional Stateful Material Implication Logic
Authors:
Gina C. Adam,
Brian D. Hoskins,
Mirko Prezioso,
Dmitri B. Strukov
Abstract:
Monolithic three-dimensional integration of memory and logic circuits could dramatically improve performance and energy efficiency of computing systems. Some conventional and emerging memories are suitable for vertical integration, including highly scalable metal-oxide resistive switching devices (memristors), yet integration of logic circuits proves to be much more challenging. Here we demonstrat…
▽ More
Monolithic three-dimensional integration of memory and logic circuits could dramatically improve performance and energy efficiency of computing systems. Some conventional and emerging memories are suitable for vertical integration, including highly scalable metal-oxide resistive switching devices (memristors), yet integration of logic circuits proves to be much more challenging. Here we demonstrate memory and logic functionality in a monolithic three-dimensional circuit by adapting recently proposed memristor-based stateful material implication logic. Though such logic has been already implemented with a variety of memory devices, prohibitively large device variability in the most prospective memristor-based circuits has limited experimental demonstrations to simple gates and just a few cycles of operations. By developing a low-temperature, low-variability fabrication process, and modifying the original circuit to increase its robustness to device imperfections, we experimentally show, for the first time, reliable multi-cycle multi-gate material implication logic operation within a three-dimensional stack of monolithically integrated memristors. The direct data manipulation in three dimensions enables extremely compact and high-throughput logic-in-memory computing and, remarkably, presents a viable solution for the Feynman grand challenge of implementing an 8-bit adder at the nanoscale.
△ Less
Submitted 9 September, 2015;
originally announced September 2015.
-
Self-Adaptive Spike-Time-Dependent Plasticity of Metal-Oxide Memristors
Authors:
M. Prezioso,
F. Merrikh-Bayat,
B. Hoskins,
K. Likharev,
D. Strukov
Abstract:
Metal-oxide memristors have emerged as promising candidates for hardware implementation of artificial synapses - the key components of high-performance, analog neuromorphic networks - due to their excellent scaling prospects. Since some advanced cognitive tasks require spiking neuromorphic networks, which explicitly model individual neural pulses (spikes) in biological neural systems, it is crucia…
▽ More
Metal-oxide memristors have emerged as promising candidates for hardware implementation of artificial synapses - the key components of high-performance, analog neuromorphic networks - due to their excellent scaling prospects. Since some advanced cognitive tasks require spiking neuromorphic networks, which explicitly model individual neural pulses (spikes) in biological neural systems, it is crucial for memristive synapses to support the spike-time-dependent plasticity (STDP), which is believed to be the primary mechanism of Hebbian adaptation. A major challenge for the STDP implementation is that, in contrast to some simplistic models of the plasticity, the elementary change of a synaptic weight in an artificial hardware synapse depends not only on the pre-synaptic and post-synaptic signals, but also on the initial weight (memristor's conductance) value. Here we experimentally demonstrate, for the first time, STDP protocols that ensure self-adaptation of the average memristor conductance, making the plasticity stable, i.e. insensitive to the initial state of the devices. The experiments have been carried out with 200-nm Al2O3/TiO2-x memristors integrated into 12x12 crossbars. The experimentally observed self-adaptive STDP behavior has been complemented with numerical modeling of weight dynamics in a simple system with a leaky-integrate-and-fire neuron with a random spike-train input, using a compact model of memristor plasticity, fitted for quantitatively correct description of our memristors.
△ Less
Submitted 20 May, 2015;
originally announced May 2015.
-
Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors
Authors:
Mirko Prezioso,
Farnood Merrikh-Bayat,
Brian Hoskins,
Gina Adam,
Konstantin K. Likharev,
Dmitri B. Strukov
Abstract:
Despite all the progress of semiconductor integrated circuit technology, the extreme complexity of the human cerebral cortex makes the hardware implementation of neuromorphic networks with a comparable number of devices exceptionally challenging. One of the most prospective candidates to provide comparable complexity, while operating much faster and with manageable power dissipation, are so-called…
▽ More
Despite all the progress of semiconductor integrated circuit technology, the extreme complexity of the human cerebral cortex makes the hardware implementation of neuromorphic networks with a comparable number of devices exceptionally challenging. One of the most prospective candidates to provide comparable complexity, while operating much faster and with manageable power dissipation, are so-called CrossNets based on hybrid CMOS/memristor circuits. In these circuits, the usual CMOS stack is augmented with one or several crossbar layers, with adjustable two-terminal memristors at each crosspoint. Recently, there was a significant progress in improvement of technology of fabrication of such memristive crossbars and their integration with CMOS circuits, including first demonstrations of their vertical integration. Separately, there have been several demonstrations of discrete memristors as artificial synapses for neuromorphic networks. Very recently such experiments were extended to crossbar arrays of phase-change memristive devices. The adjustment of such devices, however, requires an additional transistor at each crosspoint, and hence the prospects of their scaling are less impressive than those of metal-oxide memristors, whose nonlinear I-V curves enable transistor-free operation. Here we report the first experimental implementation of a transistor-free metal-oxide memristor crossbar with device variability lowered sufficiently to demonstrate a successful operation of a simple integrated neural network, a single layer-perceptron. The network could be taught in situ using a coarse-grain variety of the delta-rule algorithm to perform the perfect classification of 3x3-pixel black/white images into 3 classes. We believe that this demonstration is an important step towards the implementation of much larger and more complex memristive neuromorphic networks.
△ Less
Submitted 1 December, 2014;
originally announced December 2014.
-
High-Precision Tuning of State for Memristive Devices by Adaptable Variation-Tolerant Algorithm
Authors:
Fabien Alibart,
Ligang Gao,
Brian Hoskins,
Dmitri Strukov
Abstract:
Using memristive properties common for the titanium dioxide thin film devices, we designed a simple write algorithm to tune device conductance at a specific bias point to 1% relative accuracy (which is roughly equivalent to 7-bit precision) within its dynamic range even in the presence of large variations in switching behavior. The high precision state is nonvolatile and the results are likely to…
▽ More
Using memristive properties common for the titanium dioxide thin film devices, we designed a simple write algorithm to tune device conductance at a specific bias point to 1% relative accuracy (which is roughly equivalent to 7-bit precision) within its dynamic range even in the presence of large variations in switching behavior. The high precision state is nonvolatile and the results are likely to be sustained for nanoscale memristive devices because of the inherent filamentary nature of the resistive switching. The proposed functionality of memristive devices is especially attractive for analog computing with low precision data. As one representative example we demonstrate hybrid circuitry consisting of CMOS summing amplifier and two memristive devices to perform analog multiply and accumulate computation, which is a typical bottleneck operation in information processing.
△ Less
Submitted 6 October, 2011;
originally announced October 2011.
