-
Low-Resource Self-Supervised Learning with SSL-Enhanced TTS
Authors:
Po-chun Hsu,
Ali Elkahky,
Wei-Ning Hsu,
Yossi Adi,
Tu Anh Nguyen,
Jade Copet,
Emmanuel Dupoux,
Hung-yi Lee,
Abdelrahman Mohamed
Abstract:
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks. Nonetheless, a significant challenge remains in reducing the reliance on vast amounts of speech data for pre-training. This paper proposes to address this challenge by leveraging synthetic speech to augment a low-resource pre-training corpus. We construct a high-quality text-to-speech (TT…
▽ More
Self-supervised learning (SSL) techniques have achieved remarkable results in various speech processing tasks. Nonetheless, a significant challenge remains in reducing the reliance on vast amounts of speech data for pre-training. This paper proposes to address this challenge by leveraging synthetic speech to augment a low-resource pre-training corpus. We construct a high-quality text-to-speech (TTS) system with limited resources using SSL features and generate a large synthetic corpus for pre-training. Experimental results demonstrate that our proposed approach effectively reduces the demand for speech data by 90% with only slight performance degradation. To the best of our knowledge, this is the first work aiming to enhance low-resource self-supervised learning in speech processing.
△ Less
Submitted 4 June, 2024; v1 submitted 29 September, 2023;
originally announced September 2023.
-
Scaling Speech Technology to 1,000+ Languages
Authors:
Vineel Pratap,
Andros Tjandra,
Bowen Shi,
Paden Tomasello,
Arun Babu,
Sayani Kundu,
Ali Elkahky,
Zhaoheng Ni,
Apoorv Vyas,
Maryam Fazel-Zarandi,
Alexei Baevski,
Yossi Adi,
Xiaohui Zhang,
Wei-Ning Hsu,
Alexis Conneau,
Michael Auli
Abstract:
Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on…
▽ More
Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.
△ Less
Submitted 22 May, 2023;
originally announced May 2023.
-
STOP: A dataset for Spoken Task Oriented Semantic Parsing
Authors:
Paden Tomasello,
Akshat Shrivastava,
Daniel Lazar,
Po-Chun Hsu,
Duc Le,
Adithya Sagar,
Ali Elkahky,
Jade Copet,
Wei-Ning Hsu,
Yossi Adi,
Robin Algayres,
Tu Ahn Nguyen,
Emmanuel Dupoux,
Luke Zettlemoyer,
Abdelrahman Mohamed
Abstract:
End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. It promises to improve the performance of assistant systems by leveraging acoustic information lost in the intermediate textual representation and preventing cascading errors from Automatic Speech Recognition (ASR). Further, having one unified model has efficiency advantages when deploying assi…
▽ More
End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. It promises to improve the performance of assistant systems by leveraging acoustic information lost in the intermediate textual representation and preventing cascading errors from Automatic Speech Recognition (ASR). Further, having one unified model has efficiency advantages when deploying assistant systems on-device. However, the limited number of public audio datasets with semantic parse labels hinders the research progress in this area. In this paper, we release the Spoken Task-Oriented semantic Parsing (STOP) dataset, the largest and most complex SLU dataset to be publicly available. Additionally, we define low-resource splits to establish a benchmark for improving SLU when limited labeled data is available. Furthermore, in addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems. Initial experimentation show end-to-end SLU models performing slightly worse than their cascaded counterparts, which we hope encourages future work in this direction.
△ Less
Submitted 18 October, 2022; v1 submitted 28 June, 2022;
originally announced July 2022.
-
Generative Spoken Dialogue Language Modeling
Authors:
Tu Anh Nguyen,
Eugene Kharitonov,
Jade Copet,
Yossi Adi,
Wei-Ning Hsu,
Ali Elkahky,
Paden Tomasello,
Robin Algayres,
Benoit Sagot,
Abdelrahman Mohamed,
Emmanuel Dupoux
Abstract:
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech,…
▽ More
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.
△ Less
Submitted 22 November, 2022; v1 submitted 30 March, 2022;
originally announced March 2022.
-
textless-lib: a Library for Textless Spoken Language Processing
Authors:
Eugene Kharitonov,
Jade Copet,
Kushal Lakhotia,
Tu Anh Nguyen,
Paden Tomasello,
Ann Lee,
Ali Elkahky,
Wei-Ning Hsu,
Abdelrahman Mohamed,
Emmanuel Dupoux,
Yossi Adi
Abstract:
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three differ…
▽ More
Textless spoken language processing research aims to extend the applicability of standard NLP toolset onto spoken language and languages with few or no textual resources. In this paper, we introduce textless-lib, a PyTorch-based library aimed to facilitate research in this research area. We describe the building blocks that the library provides and demonstrate its usability by discuss three different use-case examples: (i) speaker probing, (ii) speech resynthesis and compression, and (iii) speech continuation. We believe that textless-lib substantially simplifies research the textless setting and will be handful not only for speech researchers but also for the NLP community at large. The code, documentation, and pre-trained models are available at https://github.com/facebookresearch/textlesslib/ .
△ Less
Submitted 15 February, 2022;
originally announced February 2022.
-
Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing
Authors:
Akshat Shrivastava,
Shrey Desai,
Anchit Gupta,
Ali Elkahky,
Aleksandr Livshits,
Alexander Zotov,
Ahmed Aly
Abstract:
Task-oriented semantic parsing models have achieved strong results in recent years, but unfortunately do not strike an appealing balance between model size, runtime latency, and cross-domain generalizability. We tackle this problem by introducing scenario-based semantic parsing: a variant of the original task which first requires disambiguating an utterance's "scenario" (an intent-slot template wi…
▽ More
Task-oriented semantic parsing models have achieved strong results in recent years, but unfortunately do not strike an appealing balance between model size, runtime latency, and cross-domain generalizability. We tackle this problem by introducing scenario-based semantic parsing: a variant of the original task which first requires disambiguating an utterance's "scenario" (an intent-slot template with variable leaf spans) before generating its frame, complete with ontology and utterance tokens. This formulation enables us to isolate coarse-grained and fine-grained aspects of the task, each of which we solve with off-the-shelf neural modules, also optimizing for the axes outlined above. Concretely, we create a Retrieve-and-Fill (RAF) architecture comprised of (1) a retrieval module which ranks the best scenario given an utterance and (2) a filling module which imputes spans into the scenario to create the frame. Our model is modular, differentiable, interpretable, and allows us to garner extra supervision from scenarios. RAF achieves strong results in high-resource, low-resource, and multilingual settings, outperforming recent approaches by wide margins despite, using base pre-trained encoders, small sequence lengths, and parallel decoding.
△ Less
Submitted 2 February, 2022;
originally announced February 2022.
