Showing 1–17 of 17 results for author: Senior, A

Quantum error correction below the surface code threshold

Authors: Rajeev Acharya, Laleh Aghababaie-Beni, Igor Aleiner, Trond I. Andersen, Markus Ansmann, Frank Arute, Kunal Arya, Abraham Asfaw, Nikita Astrakhantsev, Juan Atalaya, Ryan Babbush, Dave Bacon, Brian Ballard, Joseph C. Bardin, Johannes Bausch, Andreas Bengtsson, Alexander Bilmes, Sam Blackwell, Sergio Boixo, Gina Bortoli, Alexandre Bourassa, Jenna Bovaird, Leon Brill, Michael Broughton, David A. Browne , et al. (224 additional authors not shown)

Abstract: Quantum error correction provides a path to reach practical quantum computing by combining multiple physical qubits into a logical qubit, where the logical error rate is suppressed exponentially as more qubits are added. However, this exponential suppression only occurs if the physical error rate is below a critical threshold. In this work, we present two surface code memories operating below this… ▽ More Quantum error correction provides a path to reach practical quantum computing by combining multiple physical qubits into a logical qubit, where the logical error rate is suppressed exponentially as more qubits are added. However, this exponential suppression only occurs if the physical error rate is below a critical threshold. In this work, we present two surface code memories operating below this threshold: a distance-7 code and a distance-5 code integrated with a real-time decoder. The logical error rate of our larger quantum memory is suppressed by a factor of $Λ$ = 2.14 $\pm$ 0.02 when increasing the code distance by two, culminating in a 101-qubit distance-7 code with 0.143% $\pm$ 0.003% error per cycle of error correction. This logical memory is also beyond break-even, exceeding its best physical qubit's lifetime by a factor of 2.4 $\pm$ 0.3. We maintain below-threshold performance when decoding in real time, achieving an average decoder latency of 63 $μ$s at distance-5 up to a million cycles, with a cycle time of 1.1 $μ$s. To probe the limits of our error-correction performance, we run repetition codes up to distance-29 and find that logical performance is limited by rare correlated error events occurring approximately once every hour, or 3 $\times$ 10$^9$ cycles. Our results present device performance that, if scaled, could realize the operational requirements of large scale fault-tolerant quantum algorithms. △ Less

Submitted 24 August, 2024; originally announced August 2024.

Comments: 10 pages, 4 figures, Supplementary Information

Learning to Decode the Surface Code with a Recurrent, Transformer-Based Neural Network

Authors: Johannes Bausch, Andrew W Senior, Francisco J H Heras, Thomas Edlich, Alex Davies, Michael Newman, Cody Jones, Kevin Satzinger, Murphy Yuezhen Niu, Sam Blackwell, George Holland, Dvir Kafri, Juan Atalaya, Craig Gidney, Demis Hassabis, Sergio Boixo, Hartmut Neven, Pushmeet Kohli

Abstract: Quantum error-correction is a prerequisite for reliable quantum computation. Towards this goal, we present a recurrent, transformer-based neural network which learns to decode the surface code, the leading quantum error-correction code. Our decoder outperforms state-of-the-art algorithmic decoders on real-world data from Google's Sycamore quantum processor for distance 3 and 5 surface codes. On di… ▽ More Quantum error-correction is a prerequisite for reliable quantum computation. Towards this goal, we present a recurrent, transformer-based neural network which learns to decode the surface code, the leading quantum error-correction code. Our decoder outperforms state-of-the-art algorithmic decoders on real-world data from Google's Sycamore quantum processor for distance 3 and 5 surface codes. On distances up to 11, the decoder maintains its advantage on simulated data with realistic noise including cross-talk, leakage, and analog readout signals, and sustains its accuracy far beyond the 25 cycles it was trained on. Our work illustrates the ability of machine learning to go beyond human-designed algorithms by learning from data directly, highlighting machine learning as a strong contender for decoding in quantum computers. △ Less

Submitted 9 October, 2023; originally announced October 2023.

MSC Class: 81P73; 68T07 ACM Class: I.2.0; J.2

Inferring a Continuous Distribution of Atom Coordinates from Cryo-EM Images using VAEs

Authors: Dan Rosenbaum, Marta Garnelo, Michal Zielinski, Charlie Beattie, Ellen Clancy, Andrea Huber, Pushmeet Kohli, Andrew W. Senior, John Jumper, Carl Doersch, S. M. Ali Eslami, Olaf Ronneberger, Jonas Adler

Abstract: Cryo-electron microscopy (cryo-EM) has revolutionized experimental protein structure determination. Despite advances in high resolution reconstruction, a majority of cryo-EM experiments provide either a single state of the studied macromolecule, or a relatively small number of its conformations. This reduces the effectiveness of the technique for proteins with flexible regions, which are known to… ▽ More Cryo-electron microscopy (cryo-EM) has revolutionized experimental protein structure determination. Despite advances in high resolution reconstruction, a majority of cryo-EM experiments provide either a single state of the studied macromolecule, or a relatively small number of its conformations. This reduces the effectiveness of the technique for proteins with flexible regions, which are known to play a key role in protein function. Recent methods for capturing conformational heterogeneity in cryo-EM data model it in volume space, making recovery of continuous atomic structures challenging. Here we present a fully deep-learning-based approach using variational auto-encoders (VAEs) to recover a continuous distribution of atomic protein structures and poses directly from picked particle images and demonstrate its efficacy on realistic simulated data. We hope that methods built on this work will allow incorporation of stronger prior information about protein structure and enable better understanding of non-rigid protein structures. △ Less

Submitted 26 June, 2021; originally announced June 2021.

Validation of Sinus Drug Delivery Computational Fluid Dynamics (CFD) Modeling with In Vitro Gamma Scintigraphy

Authors: K. Kudlaty, W. Bennett, L. Holbrook, A. Burke, J. Wu, B. Langworthy, J. P. Fine, C. S. Ebert Jr., A. J. Kimple, B. D. Thorp, A. M. Zanation, B. A. Senior, J. S. Kimbell

Abstract: Background: Chronic rhinosinusitis (CRS) is a prevalent and disruptive disease. Medical management including nasal steroid sprays is the primary treatment modality. Computational fluid dynamics (CFD) has been used to characterize sinonasal airflow and intranasal drug delivery; however, variation in simulation methods indicates a need for large scale CFD model validation. Methods: Anatomic recons… ▽ More Background: Chronic rhinosinusitis (CRS) is a prevalent and disruptive disease. Medical management including nasal steroid sprays is the primary treatment modality. Computational fluid dynamics (CFD) has been used to characterize sinonasal airflow and intranasal drug delivery; however, variation in simulation methods indicates a need for large scale CFD model validation. Methods: Anatomic reconstructions of pre and post-operative CT scans of 3 functional endoscopic sinus surgery patients were created in Mimics(TM). Fluid analysis and drug particle deposition modeling were conducted using CFD methods with Fluent(TM) in 18 cases. Models were 3D printed and in vitro studies were performed using Tc99-labeled Nasacort(TM). Gamma scintigraphy signals and CFD-modeled spray mass were post-processed in a superimposed grid and compared. Statistical analysis using overlap coefficients (OCs) evaluated similarities between computational and experimental distributions and Kendall's tau rank correlation coefficient was employed to test independence. Results: OCs revealed strong agreement in percent deposition and grid profiles between CFD models and experimental results (mean [range] for sagittal, axial, and coronal grids were 0.69 [0.57], 0.61 [0.49], and 0.78 [0.44], respectively). Kendall's tau values showed strong agreement (average 0.73) between distributions, which were statistically significant (p < 0.05) apart from a single coronal grid in one model and two sagittal grids of another. Conclusions: CFD modeling demonstrates statistical agreement with in vitro experimental results. This validation study is one of the largest of its kind and supports the applicability of CFD in accurately modeling nasal spray drug delivery and using computational methods to investigate means of improving clinical drug delivery. △ Less

Submitted 23 July, 2020; originally announced July 2020.

Comments: 29 pages including 4 tables and 6 figures

Numerical evaluation of spray position for improved nasal drug delivery

Authors: Saikat Basu, Landon T Holbrook, Kathryn Kudlaty, Olulade Fasanmade, Jihong Wu, Alyssa Burke, Benjamin Langworthy, Zainab Farzal, Mohammed Mamdani, William D Bennett, Jason Fine, Brent A Senior, Adam M Zanation, Charles S Ebert, Jr., Adam J Kimple, Brian D Thorp, Dennis O Frank-Ito, Guilherme JM Garcia, Julia S Kimbell

Abstract: Topical intra-nasal sprays are amongst the most commonly prescribed therapeutic options for sinonasal diseases in humans. However, inconsistency and ambiguity in instructions show a lack of definitive knowledge on best spray use techniques. In this study, we have identified a new usage strategy for nasal sprays available over-the-counter, that registers an average 8-fold improvement in topical del… ▽ More Topical intra-nasal sprays are amongst the most commonly prescribed therapeutic options for sinonasal diseases in humans. However, inconsistency and ambiguity in instructions show a lack of definitive knowledge on best spray use techniques. In this study, we have identified a new usage strategy for nasal sprays available over-the-counter, that registers an average 8-fold improvement in topical delivery of drugs at diseased sites, when compared to prevalent spray techniques. The protocol involves re-orienting the spray axis to harness inertial motion of particulates and has been developed using computational fluid dynamics simulations of respiratory airflow and droplet transport in medical imaging-based digital models. Simulated dose in representative models is validated through in vitro spray measurements in 3D-printed anatomic replicas using the gamma scintigraphy technique. This work breaks new ground in proposing an alternative user-friendly strategy that can significantly enhance topical delivery inside human nose. While these findings can eventually translate into personalized spray usage instructions and hence merit a change in nasal standard-of-care, this study also demonstrates how relatively simple engineering analysis tools can revolutionize everyday healthcare. △ Less

Submitted 20 December, 2019; v1 submitted 26 September, 2019; originally announced September 2019.

Comments: 24 pages, 8 figures, 5 tables

Cost-sensitive Boosting Pruning Trees for depression detection on Twitter

Authors: Lei Tong, Zhihua Liu, Zheheng Jiang, Feixiang Zhou, Long Chen, Jialin Lyu, Xiangrong Zhang, Qianni Zhang, Abdul Sadka Senior, Yinhai Wang, Ling Li, Huiyu Zhou

Abstract: Depression is one of the most common mental health disorders, and a large number of depressed people commit suicide each year. Potential depression sufferers usually do not consult psychological doctors because they feel ashamed or are unaware of any depression, which may result in severe delay of diagnosis and treatment. In the meantime, evidence shows that social media data provides valuable clu… ▽ More Depression is one of the most common mental health disorders, and a large number of depressed people commit suicide each year. Potential depression sufferers usually do not consult psychological doctors because they feel ashamed or are unaware of any depression, which may result in severe delay of diagnosis and treatment. In the meantime, evidence shows that social media data provides valuable clues about physical and mental health conditions. In this paper, we argue that it is feasible to identify depression at an early stage by mining online social behaviours. Our approach, which is innovative to the practice of depression detection, does not rely on the extraction of numerous or complicated features to achieve accurate depression detection. Instead, we propose a novel classifier, namely, Cost-sensitive Boosting Pruning Trees (CBPT), which demonstrates a strong classification ability on two publicly accessible Twitter depression detection datasets. To comprehensively evaluate the classification capability of the CBPT, we use additional three datasets from the UCI machine learning repository and the CBPT obtains appealing classification results against several state of the arts boosting algorithms. Finally, we comprehensively explore the influence factors of model prediction, and the results manifest that our proposed framework is promising for identifying Twitter users with depression. △ Less

Submitted 21 January, 2022; v1 submitted 2 June, 2019; originally announced June 2019.

Comments: 15 pages, 7 figures, Accepted by IEEE transactions on Affective Computing

Comparative Analysis of the Main Nasal Cavity and the Paranasal Sinuses in Chronic Rhinosinusitis: An Anatomic Study of Maximal Medical Therapy

Authors: Satyan B. Sreenath, Julia S. Kimbell, Saikat Basu, Andrew J. Coniglio, Tatayana E. Fontenot, Brian D. Thorp, Charles S. Ebert, Brent A. Senior, Adam M. Zanation

Abstract: Objective: Minimal literature exists investigating changes in inflammation with respect to the main nasal cavity (MNC) and paranasal sinuses (PS) before and after maximal medical therapy (MMT) for chronic rhinosinusitis (CRS). We hypothesized that MMT produces a differential level of change in the volume of air space in the MNC and PS, and that resolution of mucosal disease associated with the ost… ▽ More Objective: Minimal literature exists investigating changes in inflammation with respect to the main nasal cavity (MNC) and paranasal sinuses (PS) before and after maximal medical therapy (MMT) for chronic rhinosinusitis (CRS). We hypothesized that MMT produces a differential level of change in the volume of air space in the MNC and PS, and that resolution of mucosal disease associated with the osteomeatal complex (OMC) influences clinical response to MMT. Study Design: Retrospective study of 12 pre- and post-MMT sinus-CT scans from 6 subjects with CRS, of which three succeeded and three failed therapy. Methods: Mimics was used to create 3D-models of the MNC and PS, and then analysis of the models was performed. Results: Mean differences in the sinonasal volume were 7866.5+/-4339.9 mm3 and 17869.10+/-19472.70 mm3, amongst the failures and successes, respectively. There is wide variability in the contribution of PS and MNC airspace volume change to the overall change in the sinonasal volume. In two subjects, the direction of volume change in the MNC and PS diverged with respect to the overall change in volume. Line-of-sight analysis demonstrated that successful responders to MMT had more patent MNC with direct access to the OMC. Conclusions: There is a differential contribution to sinonasal, airspace volume change after MMT, when comparing the MNC and PS. Response to MMT may not be solely attributable to PS change and may include a function of MNC change. Line-of-sight models suggest that direct access to the OMC may impact response to MMT. △ Less

Submitted 1 November, 2018; originally announced November 2018.

Comments: 30 pages, 4 figures

Comparative Study of Simulated Nebulized and Spray Particle Deposition in Chronic Rhinosinusitis Patients

Authors: Zainab Farzal, Saikat Basu, Alyssa Burke, Olulade O. Fasanmade, Erin M. Mamuyac, William D. Bennett, Charles S. Ebert Jr, Adam M. Zanation, Brent A. Senior, Julia S. Kimbell

Abstract: Introduction: Topical intranasal drugs are widely prescribed for Chronic Rhinosinusitis (CRS), although delivery can vary with device type and droplet size. The study objective was to compare nebulized and sprayed droplet deposition in the paranasal sinuses and ostiomeatal complex (OMC) across multiple droplet sizes in CRS patients using computational fluid dynamics (CFD). Methods: Three-dimension… ▽ More Introduction: Topical intranasal drugs are widely prescribed for Chronic Rhinosinusitis (CRS), although delivery can vary with device type and droplet size. The study objective was to compare nebulized and sprayed droplet deposition in the paranasal sinuses and ostiomeatal complex (OMC) across multiple droplet sizes in CRS patients using computational fluid dynamics (CFD). Methods: Three-dimensional models of sinonasal cavities were constructed from computed tomography (CT) scans of three subjects with CRS refractory to medical therapy using imaging software. Assuming steady-state inspiratory airflow at resting rate, CFD was used to simulate 1-120 μm sprayed droplet deposition in the left and right sinuses and OMC with spray nozzle positioning as in current nasal spray use instructions. Zero-velocity nebulization simulations were performed for 1-30 μm droplet sizes, maximal sinus and OMC deposition fractions (MSDF) were obtained, and sizes that achieved at least 50% of MSDF were identified. Nebulized MSDF was compared to sprayed droplet deposition. We also validated CFD framework through in vitro experiments. Results: Among nebulized droplet sizes, 11-14 μm droplets achieved at least 50% of MSDF in all six sinonasal cavities. Five of six sinonasal cavities had greater sinus and OMC deposition with nebulized droplets than with sprayed droplets at optimal sizes. Conclusions: Nebulized droplets may target the sinuses and OMC more effectively than sprayed particles at sizes achieving best deposition. Further studies are needed to confirm our preliminary findings. Several commercial nasal nebulizers have average particle sizes outside the optimal nebulized droplet size range found here, suggesting potential for product enhancement. △ Less

Submitted 1 November, 2018; originally announced November 2018.

Comments: 32 pages, 9 figures

Deep Audio-Visual Speech Recognition

Authors: Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman

Abstract: The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, on… ▽ More The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin. △ Less

Submitted 22 December, 2018; v1 submitted 6 September, 2018; originally announced September 2018.

Comments: Accepted for publication by IEEE Transactions on Pattern Analysis and Machine Intelligence

Large-Scale Visual Speech Recognition

Authors: Brendan Shillingford, Yannis Assael, Matthew W. Hoffman, Thomas Paine, Cían Hughes, Utsav Prabhu, Hank Liao, Hasim Sak, Kanishka Rao, Lorrayne Bennett, Marie Mulville, Ben Coppin, Ben Laurie, Andrew Senior, Nando de Freitas

Abstract: This work presents a scalable solution to open-vocabulary visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable v… ▽ More This work presents a scalable solution to open-vocabulary visual speech recognition. To achieve this, we constructed the largest existing visual speech recognition dataset, consisting of pairs of text and video clips of faces speaking (3,886 hours of video). In tandem, we designed and trained an integrated lipreading system, consisting of a video processing pipeline that maps raw video to stable videos of lips and sequences of phonemes, a scalable deep neural network that maps the lip videos to sequences of phoneme distributions, and a production-level speech decoder that outputs sequences of words. The proposed system achieves a word error rate (WER) of 40.9% as measured on a held-out set. In comparison, professional lipreaders achieve either 86.4% or 92.9% WER on the same dataset when having access to additional types of contextual information. Our approach significantly improves on other lipreading approaches, including variants of LipNet and of Watch, Attend, and Spell (WAS), which are only capable of 89.8% and 76.8% WER respectively. △ Less

Submitted 1 October, 2018; v1 submitted 13 July, 2018; originally announced July 2018.

Lip Reading Sentences in the Wild

Authors: Joon Son Chung, Andrew Senior, Oriol Vinyals, Andrew Zisserman

Abstract: The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WL… ▽ More The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WLAS) network that learns to transcribe videos of mouth motion to characters; (2) a curriculum learning strategy to accelerate training and to reduce overfitting; (3) a 'Lip Reading Sentences' (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a significant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that visual information helps to improve speech recognition performance even when the audio is available. △ Less

Submitted 30 January, 2017; v1 submitted 16 November, 2016; originally announced November 2016.

Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

Authors: Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, Timothy P Lillicrap

Abstract: Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory… ▽ More Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows --- limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs $1,\!000\times$ faster and with $3,\!000\times$ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring $100,\!000$s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer. △ Less

Submitted 27 October, 2016; originally announced October 2016.

Comments: in 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain

WaveNet: A Generative Model for Raw Audio

Authors: Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu

Abstract: This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-… ▽ More This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition. △ Less

Submitted 19 September, 2016; v1 submitted 12 September, 2016; originally announced September 2016.

Latent Predictor Networks for Code Generation

Authors: Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Andrew Senior, Fumin Wang, Phil Blunsom

Abstract: Many language generation tasks require the production of text conditioned on both structured and unstructured inputs. We present a novel neural network architecture which generates an output sequence conditioned on an arbitrary number of input functions. Crucially, our approach allows both the choice of conditioning context and the granularity of generation, for example characters or tokens, to be… ▽ More Many language generation tasks require the production of text conditioned on both structured and unstructured inputs. We present a novel neural network architecture which generates an output sequence conditioned on an arbitrary number of input functions. Crucially, our approach allows both the choice of conditioning context and the granularity of generation, for example characters or tokens, to be marginalised, thus permitting scalable and effective training. Using this framework, we address the problem of generating programming code from a mixed natural language and structured specification. We create two new data sets for this paradigm derived from the collectible trading card games Magic the Gathering and Hearthstone. On these, and a third preexisting corpus, we demonstrate that marginalising multiple predictors allows our model to outperform strong benchmarks. △ Less

Submitted 8 June, 2016; v1 submitted 22 March, 2016; originally announced March 2016.

Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition

Authors: Haşim Sak, Andrew Senior, Kanishka Rao, Françoise Beaufays

Abstract: We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initi… ▽ More We have recently shown that deep Long Short-Term Memory (LSTM) recurrent neural networks (RNNs) outperform feed forward deep neural networks (DNNs) as acoustic models for speech recognition. More recently, we have shown that the performance of sequence trained context dependent (CD) hidden Markov model (HMM) acoustic models using such LSTM RNNs can be equaled by sequence trained phone models initialized with connectionist temporal classification (CTC). In this paper, we present techniques that further improve performance of LSTM RNN acoustic models for large vocabulary speech recognition. We show that frame stacking and reduced frame rate lead to more accurate models and faster decoding. CD phone modeling leads to further improvements. We also present initial results for LSTM RNN models outputting words directly. △ Less

Submitted 24 July, 2015; originally announced July 2015.

Comments: To be published in the INTERSPEECH 2015 proceedings

Holographic Graph Neuron: a Bio-Inspired Architecture for Pattern Processing

Authors: Denis Kleyko, Evgeny Osipov, Alexander Senior, Asad I. Khan, Y. Ahmet Şekercioğlu

Abstract: This article proposes the use of Vector Symbolic Architectures for implementing Hierarchical Graph Neuron, an architecture for memorizing patterns of generic sensor stimuli. The adoption of a Vector Symbolic representation ensures a one-layered design for the approach, while maintaining the previously reported properties and performance characteristics of Hierarchical Graph Neuron, and also improv… ▽ More This article proposes the use of Vector Symbolic Architectures for implementing Hierarchical Graph Neuron, an architecture for memorizing patterns of generic sensor stimuli. The adoption of a Vector Symbolic representation ensures a one-layered design for the approach, while maintaining the previously reported properties and performance characteristics of Hierarchical Graph Neuron, and also improving the noise resistance of the architecture. The proposed architecture enables a linear (with respect to the number of stored entries) time search for an arbitrary sub-pattern. △ Less

Submitted 15 January, 2015; originally announced January 2015.

Comments: 9 pages, 13 figures

Journal ref: IEEE Transactions on Neural Networks and Learning Systems 28 (2017) 1250 - 1262

Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition

Authors: Haşim Sak, Andrew Senior, Françoise Beaufays

Abstract: Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting rec… ▽ More Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully used for sequence labeling and sequence prediction tasks, such as handwriting recognition, language modeling, phonetic labeling of acoustic frames. However, in contrast to the deep neural networks, the use of RNNs in speech recognition has been limited to phone recognition in small scale tasks. In this paper, we present novel LSTM based RNN architectures which make more effective use of model parameters to train acoustic models for large vocabulary speech recognition. We train and compare LSTM, RNN and DNN models at various numbers of parameters and configurations. We show that LSTM models converge quickly and give state of the art speech recognition performance for relatively small sized models. △ Less

Submitted 5 February, 2014; originally announced February 2014.