-
Layer Ensemble Averaging for Improving Memristor-Based Artificial Neural Network Performance
Authors:
Osama Yousuf,
Brian Hoskins,
Karthick Ramu,
Mitchell Fream,
William A. Borders,
Advait Madhavan,
Matthew W. Daniels,
Andrew Dienstfrey,
Jabez J. McClelland,
Martin Lueker-Boden,
Gina C. Adam
Abstract:
Artificial neural networks have advanced due to scaling dimensions, but conventional computing faces inefficiency due to the von Neumann bottleneck. In-memory computation architectures, like memristors, offer promise but face challenges due to hardware non-idealities. This work proposes and experimentally demonstrates layer ensemble averaging, a technique to map pre-trained neural network solution…
▽ More
Artificial neural networks have advanced due to scaling dimensions, but conventional computing faces inefficiency due to the von Neumann bottleneck. In-memory computation architectures, like memristors, offer promise but face challenges due to hardware non-idealities. This work proposes and experimentally demonstrates layer ensemble averaging, a technique to map pre-trained neural network solutions from software to defective hardware crossbars of emerging memory devices and reliably attain near-software performance on inference. The approach is investigated using a custom 20,000-device hardware prototyping platform on a continual learning problem where a network must learn new tasks without catastrophically forgetting previously learned information. Results demonstrate that by trading off the number of devices required for layer mapping, layer ensemble averaging can reliably boost defective memristive network performance up to the software baseline. For the investigated problem, the average multi-task classification accuracy improves from 61 % to 72 % (< 1 % of software baseline) using the proposed approach.
△ Less
Submitted 23 April, 2024;
originally announced April 2024.
-
Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation
Authors:
Adam N. McCaughan,
Bakhrom G. Oripov,
Natesh Ganesh,
Sae Woo Nam,
Andrew Dienstfrey,
Sonia M. Buckley
Abstract:
We present multiplexed gradient descent (MGD), a gradient descent framework designed to easily train analog or digital neural networks in hardware. MGD utilizes zero-order optimization techniques for online training of hardware neural networks. We demonstrate its ability to train neural networks on modern machine learning datasets, including CIFAR-10 and Fashion-MNIST, and compare its performance…
▽ More
We present multiplexed gradient descent (MGD), a gradient descent framework designed to easily train analog or digital neural networks in hardware. MGD utilizes zero-order optimization techniques for online training of hardware neural networks. We demonstrate its ability to train neural networks on modern machine learning datasets, including CIFAR-10 and Fashion-MNIST, and compare its performance to backpropagation. Assuming realistic timescales and hardware parameters, our results indicate that these optimization techniques can train a network on emerging hardware platforms orders of magnitude faster than the wall-clock time of training via backpropagation on a standard GPU, even in the presence of imperfect weight updates or device-to-device variations in the hardware. We additionally describe how it can be applied to existing hardware as part of chip-in-the-loop training, or integrated directly at the hardware level. Crucially, the MGD framework is highly flexible, and its gradient descent process can be optimized to compensate for specific hardware limitations such as slow parameter-update speeds or limited input bandwidth.
△ Less
Submitted 5 March, 2023;
originally announced March 2023.
-
Device Modeling Bias in ReRAM-based Neural Network Simulations
Authors:
Osama Yousuf,
Imtiaz Hossen,
Matthew W. Daniels,
Martin Lueker-Boden,
Andrew Dienstfrey,
Gina C. Adam
Abstract:
Data-driven modeling approaches such as jump tables are promising techniques to model populations of resistive random-access memory (ReRAM) or other emerging memory devices for hardware neural network simulations. As these tables rely on data interpolation, this work explores the open questions about their fidelity in relation to the stochastic device behavior they model. We study how various jump…
▽ More
Data-driven modeling approaches such as jump tables are promising techniques to model populations of resistive random-access memory (ReRAM) or other emerging memory devices for hardware neural network simulations. As these tables rely on data interpolation, this work explores the open questions about their fidelity in relation to the stochastic device behavior they model. We study how various jump table device models impact the attained network performance estimates, a concept we define as modeling bias. Two methods of jump table device modeling, binning and Optuna-optimized binning, are explored using synthetic data with known distributions for benchmarking purposes, as well as experimental data obtained from TiOx ReRAM devices. Results on a multi-layer perceptron trained on MNIST show that device models based on binning can behave unpredictably particularly at low number of points in the device dataset, sometimes over-promising, sometimes under-promising target network accuracy. This paper also proposes device level metrics that indicate similar trends with the modeling bias metric at the network level. The proposed approach opens the possibility for future investigations into statistical device models with better performance, as well as experimentally verified modeling bias in different in-memory computing and neural network architectures.
△ Less
Submitted 28 November, 2022;
originally announced November 2022.
-
Optimizing Unlicensed Band Spectrum Sharing With Subspace-Based Pareto Tracing
Authors:
Zachary J. Grey,
Susanna Mosleh,
Jacob D. Rezac,
Yao Ma,
Jason B. Coder,
Andrew M. Dienstfrey
Abstract:
To meet the ever-growing demands of data throughput for forthcoming and deployed wireless networks, new wireless technologies like Long-Term Evolution License-Assisted Access (LTE-LAA) operate in shared and unlicensed bands. However, the LAA network must co-exist with incumbent IEEE 802.11 Wi-Fi systems. We consider a coexistence scenario where multiple LAA and Wi-Fi links share an unlicensed band…
▽ More
To meet the ever-growing demands of data throughput for forthcoming and deployed wireless networks, new wireless technologies like Long-Term Evolution License-Assisted Access (LTE-LAA) operate in shared and unlicensed bands. However, the LAA network must co-exist with incumbent IEEE 802.11 Wi-Fi systems. We consider a coexistence scenario where multiple LAA and Wi-Fi links share an unlicensed band. We aim to improve this coexistence by maximizing the key performance indicators (KPIs) of these networks simultaneously via dimension reduction and multi-criteria optimization. These KPIs are network throughputs as a function of medium access control protocols and physical layer parameters. We perform an exploratory analysis of coexistence behavior by approximating active subspaces to identify low-dimensional structure in the optimization criteria, i.e., few linear combinations of parameters for simultaneously maximizing KPIs. We leverage an aggregate low-dimensional subspace parametrized by approximated active subspaces of throughputs to facilitate multi-criteria optimization. The low-dimensional subspace approximations inform visualizations revealing convex KPIs over mixed active coordinates leading to an analytic Pareto trace of near-optimal solutions.
△ Less
Submitted 23 February, 2021; v1 submitted 2 February, 2021;
originally announced February 2021.
