A Language Modeling Approach to Diacritic-Free Hebrew TTS

Hebrew, a low-resource language spoken by $9$ million people worldwide [1], presents unique challenges that constrain research and product development in speech technology. Specifically, Hebrew is a morphologically rich language, with the common use of prefixes and suffixes to modify words’ meanings and to add prepositions. On top of that, Hebrew uses Diacritics (’Niqqud’) to create a one-to-one mapping between text and phonemes. ’Niqqud’ is a system of Diacritical signs used to represent vowels or distinguish between alternative pronunciations of letters of the Hebrew alphabet.

In practice, modern Hebrew text rarely contains Diacritics, one may find Diacriticized text in specialized texts such as dictionaries, poetry, or children’s books. Hence, readers are expected to conclude the correct pronunciation and understand which phonemes to use, based on familiarity with the language itself. This makes it challenging for text-to-speech (TTS) systems to accurately learn the connection between text and speech. For example the words: {cjhebrew}\cjRLmat*AnAh and {cjhebrew}\cjRLmat:nEh, contain the same characters, but have completely different meaning and pronunciation, the first one means a ‘gift’, while the second one means ‘conditioning’. As mentioned before, in modern Hebrew writing, one will probably not encounter Diacritics and the above word will appear as follows, {cjhebrew}\cjRLmtnh. As a result, the reader should infer the right pronunciation by context only. Moreover, when considering spoken language modeling systems in automated pipelines, current Automatic Speech Recognition (ASR) systems, such as Whisper [2] and Massively Multilingual Speech (MMS) [3] do not output diacritics in their transcripts, hence TTS systems should either predict it or use a Diacritic-free synthesis system.

Another major issue that holds progress in developing AI-based Hebrew TTS systems is the lack of datasets. As Hebrew is considered a low-resource language, public spoken benchmarks hardly exist. Previous efforts in constructing datasets in Hebrew were often relatively small [4, 5, 6, 7]. The authors in [4] established the Corpus of Spoken Israeli Hebrew (CoSIH) with the goal of compiling a large database of recordings of spoken Israeli Hebrew in order to facilitate and enhance research in the field. Next, the authors in [5] released The Map Task Corpus (MaTaCOp) of Hebrew dialogues. The authors in  [6] collected naturally occurring speech and interaction in Modern Hebrew via telephone conversations during the years 2020–2021 and released the HUJI Corpus of Spoken Hebrew (HUJICorpus). More recently, the authors in  [7] released SASPEECH, a high-quality single-speaker Hebrew dataset to enhance Hebrew speech synthesis research. Although all of these prior work are important and valuable, the provided benchmarks are relatively small. CoSIH contains $\sim 12.3$ hours of speech, the MaTaCOp corpus contains $\sim 5.3$ hours, the HUJI Corpus has $\sim 3.8$, and SASPEECH which is the largest one contains $\sim 30$ hours of speech. For comparison a modern, contextualized TTS system in English is trained over $\sim 60$k hours [8]. Recently, the authors of [9] and the authors of  [10] released two datasets denoted as ivrit.ai and HebDB respectively. The authors released weakly-supervised speech from local podcasts and provided the first large-scale dataset in Hebrew, which we leveraged to construct the model.

Previous attempts were made to construct a TTS system in Hebrew. The Authors of [3], proposed the MMS system. In their study, they develop speech technologies (ASR, TTS, Language ID) in more than $1,000$ languages. Their TTS system is based on representation obtained from a pre-trained multi-lingual self-supervised model. Although providing impressive results, their Hebrew TTS system is based on predicting diacritics of the input text. More recently, the authors of [7] introduced the Overflow [11] model for Hebrew, together with the SPASEECH benchmark. The Overflow model is comprised of neural HMM together with normalizing flows. On top of the Overflow model, the authors in [7] suggested using the HiFi-GAN neural vocoder [12] to estimate the phase. Similarly to MMS, the system proposed by [7] is based on predicting diacritics of the input text, hence is sub-optimal and often produces wrong and unnatural content in the generated speech. Moreover, such dependency makes it difficult for these models to scale to large datasets as they both require predicting diacritics on top of automatically transcribed text. Unlike these methods, the proposed LM approach operates in a Diacritic-free manner, not propagating mistakes from the diacritic prediction models, and better leveraging the context of the input signal.

Recent studies in speech and audio representation learning proposed learning discrete representation of the input signal [13, 14]. Such representation can be later used for plenty of speech and audio synthesis tasks [15, 16, 17, 18]. Specifically, the authors of [8, 19, 20] proposed optimizing an LM on top of such discrete speech representation, conditioned on a phonemic representation of the input text for the task of TTS. Following such an approach was found to produce high-quality and natural speech, with the ability to rapidly adapt to new speakers via acoustic prompting. As this approach is contextualized by nature it may serve as the ideal candidate for a Diacritic-free Hebrew TTS system.

In this work, we study and propose a Language Modeling approach which operates over discrete representations of the speech signal to construct a Hebrew TTS system. We optimize an acoustic LM over a weakly supervised large-scale dataset containing in-the-wild recordings. We empirically demonstrate that following the LM approach makes the usage of diacritics in Hebrew redundant, hence yielding a diacritic-free approach. We study several text representation methods and found that using word-piece tokenization produces the best results overall. Results suggest the proposed method is superior to the evaluated baselines considering both content preservation and generation quality. Code, dataset, and models are publicly available under the following link: https://pages.cs.huji.ac.il/adiyoss-lab/HebTTS/.

Given a dataset $D=\{x_{i},y_{i}\}$ where $y_{i}$ is an audio sample and $x_{i}$ is its corresponding transcription. We encode the audio into a sequence of discrete units. We follow the approach proposed by [13] and encode the audio using Residual Vector Quantization (RVQ). Formally, $E(y)=C^{T\times N_{cb}}$ where $C$ represents the acoustic code matrix, where $N_{cb}$ is the number of codebooks and $T$ is the utterance length.

A common paradigm in the TTS field is to represent text in its most basic form, i.e., phonemes [21]. As we aim at building a Diacritic-free system we can not use phonemes as text representations. As a result, we use a sub-word tokenizer in the form of word-piece tokenization. Such tokenizer was found beneficial in text encoders such as BERT [22], and more relevant to our setup AlephBERT [23]. We experimented with several other tokenizers, however, found the word-piece to provide the best overall results (see Section 5 for more details). Below we describe both text tokenization and model in more detail. We depict a general overview of the LM-based approach in Fig. 1.

Given a training corpus $C$ and a number of desired word-pieces $t$, the optimization problem is to select $t$ word-pieces such that minimizes the number of word-pieces generated when tokenizing the entire corpus $C$. We start with a small character vocabulary and special tokens $W$, and apply merge rules for the elements. iteratively we compute for each pair $w_{1},w_{2}\in W$ a score as seen in equation 1 and merge the pair with the maximum score getting a new vocabulary $W^{\prime}=W\cup\{(w_{1},w_{2})\}$. We follow this step with the new vocabulary until $|W|=t$.

By dividing the frequency of the pair by the product of the frequencies of each of its parts, the algorithm prioritizes the merging of pairs where the individual parts are less frequent in the vocabulary.

The model first receives a text prompt, $x_{i}$, and a $3$-second enrolled recording as an acoustic prompt. We then, encode the acoustic prompt using the same speech encoder $E$, and process the text using the text tokenizer defined in sub-section 1. Recall, that the speech encoder, $E$, quantizes the utterance using RVQ module, hence it outputs a matrix of size $T\times N_{cb}$. Meaning that, at each time step we are left with $N_{cb}$ discrete codes.

There are several alternatives in the literature to handle this complex input structure. For instance, the authors in [15, 27] proposed to predict all codes at each time-step in parallel while introducing a delay pattern to better model the conditional probability distribution. The authors in [17] proposed flattening the whole sequence (resulting in a $N_{cb}$ times larger sequence) and splitting its modeling across two LMs.

In this work, we follow the approach proposed in [8]. In which, the first codebook, $c_{,:1}$, is modeled in an Auto-Regressive (AR) manner following the standard next token prediction framework. Specifically, we concatenate the word-piece tokens with the first codebook from the acoustic prompt, denoted by $w,c_{\leq t,1}$, to infer the next acoustic token $c_{t,1}$ of the target signal. The rest of the codebooks (2 to $N_{cb}$), are modeled using a non-autoregressive (NAR) model, where the network is trained to maximize the acoustic tokens likelihood derived from the $i$-th quantizer codebook, conditioned on the sum of all representations from previous codebooks.

Overall, unlike, prior works which are mainly based on Mel-spectrogram as speech representations, diacritics for text, and relatively small and high-quality amounts of training data. Following the LM approach, allows us to leverage large amounts of in-the-wild recordings, using plain text, and operate on top of discrete learned speech representations. Table 1 summarizes the main differences between the methods.

We use both the ivrit.ai dataset [9] together the HebDB dataset [10]. Both datasets consists of $\sim 4500$ hours of speech gathered from local podcasts ($\sim 1700$ from HebDB and $\sim 2800$ from ivrit.ai). These datasets are comprised of spontaneous dialogues, featuring multiple speakers discussing a wide range of topics including economy, politics, sports, culture, science, history, and music, among others. The podcast recordings are full episodes, thus containing lengthy audio tracks and various non-speech elements such as music, environmental sounds, and periods of silence. Such real-world conditions present challenges for model optimization and necessitate preprocessing steps. We apply the same pre-processing pipeline to both ivrit.ai dataset to all the dataset. Initially, we standardize all audio recordings to a consistent $16$kHz, mono recordings, using julius ¹¹1https://github.com/adefossez/julius python package. Subsequently, we employ a Voice Activity Detection (VAD) model, namely silero-vad [28] to perform a voice activity detection and segment the waveforms into sentences, filtering out activated segments with a minimum duration of $1$ seconds, separating audio segments by a minimal silence duration of $100$ms and padding both sides of the segmented audio with $30$ms of silence. Finally, we automatically transcribe the segmented speech using a pre-trained ASR model, specifically Whisper V2-Large [2]. After preprocessing our data, we are left with $\sim 4500$ hours of natural dialogues with weakly labeled transcriptions.

We start by evaluating the proposed method against both MMS and Overflow. Results are summarized in Table 2. The proposed method provides superior performance to the evaluated baselines considering both objective metrics and human study. Notice, following the LM approach for Hebrew TTS additionally, allows fast adaptation to new speakers. The proposed method shows minor differences in performance when considering speech and unseen speakers.

Interestingly, when considering WER, CER, and Speaker similarity, the Overflow method provides comparable performance to ours while being superior to the MMS model. The main difference between the methods is reflected in the naturalness of the generated speech. Moreover, it is worth mentioning that although the WER and CER are comparable across all methods (with MMS achieving worse WER and Overflow achieving worse CER), these are based on automatic transcriptions that do not take into account the pronunciation, meaning two different words can be transcribed to the same sequence characters while reflecting completely different pronunciation. However, when investigating the content metric under the human study we observe larger differences.

To better evaluate the effect of using different tokenization methods for the input text we trained two versions of the proposed method using both chars and word-piece tokenizer. We additionally experimented with contextualized representations obtained from hidden layers of a pre-trained text encoder model, namely AlephBERT [23]. Unfortunately, such text representation performs significantly worse than the other tokenizers, hence we do not report results for it. We measure WER and CER metrics, together with a human study measuring content preservation. Results are presented in Table 3. Results suggest that following the word-piece tokenizer provides superior performance to the character-based alternative. This result is being reflected across all the evaluated metrics, however similarly to the results in Table 2, we observe the larger gap when considering subjective content preservation study.

In this work, we demonstrate how language models that operate over discrete speech tokens can act as Diacritics-free Hebrew TTS systems. Due to their naturally contextualized manner, language models can better handle ambiguous pronunciations obtained in the absence of diacritics. We empirically show that following the language modeling approach, trained at scale using weakly transcribed data, yields superior performance to non-contextualized, traditional TTS systems when considering context preservation, naturalness, and similarity to the speaker in the generated samples.