CATT: Character-based Arabic Tashkeel Transformer

The Arabic language is characterized by its rich morphology and complex syntactic structure. One of the unique features of Arabic is the use of diacritics or Tashkeel, which are small marks above or below the letters that indicate vowels or other phonetic aspects of pronunciation. These diacritics are favorable for understanding the meaning of words, as their absence can lead to ambiguities and misinterpretations. They are also crucial in improving performance of applications such as text-to-speech and machine translation Fadel et al. (2019b). The diacritics can be affected by the context of the sentence as shown in the following example:

In the first sentence, "\<ساق>" or "Saqa" is interpreted as "drove", denoting an action. However, when the context changes, the same word, now pronounced as "Saqu" takes on a completely different meaning, becoming a noun that translates to "leg". This change highlights the crucial role of diacritics in quickly clarifying the meanings of sentences, as the characters remain the same. At the same time, the pronunciation varies depending on the context. In the previous example, the meaning of the word "\<ساق>" can be comprehended even without diacritics, as long as the reader considers the complete sentence and understands the context provided by the surrounding words. This fact raises the following question: "Will a pretrained BERT model help in improving the ATD models?".

In this paper, we propose a training strategy based on a pretrained character-based BERT Kenton and Toutanova (2019); Liu et al. (2019), and a self-training approach called Noisy-Student (NS) Xie et al. (2020). Throughout the paper, we will answer the following research questions:

• •

  RQ1: Does the ATD model benefit from Masked Language Model (MLM) pretraining?

• •

  RQ2: Does training ATD model for more iterations help?

• •

  RQ3: Is the NS approach effective in ATD models?

Previous research has investigated a broad spectrum of approaches to address the diacritization task, beginning with rule-based methods, moving to classical machine learning models, and reaching sophisticated deep learning architectures (Almanea, 2021). In addition, comprehensive experiments show that deep learning methods outperform non-neural techniques, particularly when substantial training data is available (Fadel et al., 2019a).

Fadel et al. (2019b) tested a refined version of the Tashkeela dataset (Zerrouki and Balla, 2017; Fadel et al., 2019a) using the Shakkala²²2https://github.com/Barqawiz/Shakkala model. They also trained a character-level RNN with a Block-Normalized Gradient (BNG) module. The BNG technique normalizes gradients within each batch, potentially speeding up training and improving generalization (Yu et al., 2017).

Abbad and Xiong’s (2020) ATD approach consisted of a three-part pipeline: a multi-layer LSTM and dense layers, a character-level rule-based corrector for specific error correction, and a word-level statistical corrector that leveraged context and distance information to resolve diacritization issues. Furthermore, they developed an enhanced version of the system and named it Multilevel Diacritizer (Abbad and Xiong, 2021).

Madhfar and Qamar (2020) implemented 3 different ATD models. The first one was a baseline model consisting of 3 deep Bidirectional Long Short-Term Memory (BiLSTM) layers. The second model was an encoder-decoder with 3 LSTM layers for the encoder and 2 LSTM layers for the decoder. The last model was based on Tacotron encoder (Wang et al., 2017) that uses CBHG module (Lee et al., 2017).

AlKhamissi et al. (2020) proposed two architectures: the Two-Level Diacritizer (D2) and the Two-Level Diacritizer with Decoder (D3). D3 builds upon the capabilities of D2 by accepting partially diacritized text as input. These models have a word-level encoder as well as a character-level encoder. The results of both encoders are combined by an attention mechanism and fed to a unidirectional LSTM layer to predict diacritics.

Darwish et al. (2021) created two Deep Neural Networks (DNNs); the first utilizes a character-based BiLSTM model with unique features for each character, while the second uses a word-level BiLSTM layer and a subsequent dense layer with Softmax activation.

Karim and Abandah (2021) studied the effect of varying the training dataset size. Each time, they trained a BiLSTM model and evaluated its performance. The results demonstrated that error rates improve as the size of the training corpus increases.

Al-Rfooh et al. (2023) finetuned a token-free multilingual model called ByT5 (Xue et al., 2022) to perform Arabic text diacritization as a sequence-to-sequence task, similar to the translation task.

Recently, Skiredj and Berrada (2024) introduced the Pre-FineTuned Token Classification for Arabic Diacritization (PTCAD) model. This approach treats Arabic text diacritization as a downstream task for a pretrained BERT-like model. The approach starts with a pretraining phase on linguistically relevant tasks, such as Part-of-Speech (POS) tagging and Segmentation, which are framed as Masked Language Modelling (MLM) tasks. This pretraining helps enhance the model’s contextual understanding. Then, it moves into a finetuning phase where diacritization is handled as a token classification task. This phase leverages the contextual insights gained earlier to enhance diacritization accuracy.

Unlike Skiredj and Berrada’s (2024) method, we consider a simpler approach where we directly pretrain a character-level BERT model with no further modifications or extra labeling.

We pretrained a character-based BERT model to find the effect of MLM pretraining on the diacritization performance. The model has 6 layers with $d_{model}=512$ and number of heads$\;=16$. The model was trained using MLM loss for 6 epochs with a batch size of 512, using the data shown in Table 2.

For ATD models, we selected two transformer architectures: Encoder-Decoder (ED) with 3 layers and Encoder-Only (EO) with 6 layers. We set the number of layers to 3 in the ED model to ensure comparability with the EO model in terms of the total number of layers. All our ATD models were trained with the following configurations: $d_{model}=512$, number of heads$\;=16$, batch size$\;=32$, and dropout$\;=10\%$. We used the AdamW optimizer Loshchilov and Hutter (2018) with a learning rate of $3\times 10^{-}5$ and a weight decay of $1\times 10^{-}2$. Each model was trained on a single dedicated A100 GPU. All ATD models in our experiments were trained for a maximum of 200 epochs with an early stopping criteria.

Generally, we define the input text $X$ with a total number of characters $T$ as a series of characters $x_{1},x_{2},x_{3},\ldots,x_{T}$ where each character represents an undiacritized Arabic letter. Correspondingly, the output sequence $Y$ consists of diacritics $y_{1},y_{2},y_{3},\ldots,y_{T}$ with each diacritic $y_{i}$ associated with the respective letter $x_{i}$. In EO models, we express the relationship between letters and diacritics as follows:

$$P(y_{i}\mid x_{1},\ldots,x_{T})$$

In other words, we predict the diacritic $y_{i}$ conditioned only on the input text $x_{1},\ldots,x_{T}$. In ED models, on the other hand, we consider the ATD task as a translation task, where the input text represents the source language and the output diacritics sequence represents the target language. We express the relationship between letters and diacritics in ED models as follows:

$$P(y_{i}\mid x_{1},\ldots,x_{T},y_{1},\ldots,y_{i-1})$$

where the diacritic $y_{i}$ is conditioned on both the input text $x_{1},\ldots,x_{T}$ and the previous diacritics $y_{1},\ldots,y_{i-1}$. In fact, native Arabic speakers rely on both the textual content and diacritics to better disambiguate the intended meaning of the sentence. For example, the sentence \<اشتريت لعبة> is a complete and valid Arabic sentence that can be interpreted as "I bought a toy" or "A toy was bought". The only way to differentiate between them in the textual form is by adding diacritics as follows: \<اشْتَرَيْتُ لُعْبَةً> which means "I bought a toy" or \<اشْتُرِيَتْ لُعْبَةٌ> which means "A toy was bought". In both cases, the diacritics of the second word heavily depend on the diacritics of the first word. Therefore, by conditioning on both the input text and the previous diacritics, the model can achieve better performance. Figure 1 and Figure 2 show both transformer architectures.

Our experiments involved training a total of four models using both the ED and EO architectures. For both architectures, we used the pretrained character-based BERT as the basis for initializing the weights. Consequently, we obtained one ED model and one EO model with weights that reflect the knowledge encoded in the pretrained BERT. Additionally, we trained the other two models, consisting of one ED model and one EO model, where the weights were randomly initialized. These models allowed us to explore the impact of different weight initialization strategies on the performance and behavior of the models.

For each model in our experiments, we selected two checkpoints. The first checkpoint was chosen after training the model for 5 epochs, while the second checkpoint was chosen as the best checkpoint achieved after training for a longer period. The best checkpoint of ED model was at epoch 175 while the best checkpoint of EO was at epoch 192. The purpose of this selection process was to study the impact of extended training duration on the models, even when they were exposed to the same amount of data.

Moreover, we randomly sampled 1M sentences from the pretraining dataset to be pseudo-labeled using the NS Xie et al. (2020) technique. We used the best checkpoints of both ED and EO models to pseudo-label two copies of the sampled data. Finally, a new ED model as well as a new EO model were trained on Tashkeela data combined with the pseudo-labeled 1M sentences. Both models’ parameters were initialized from the best checkpoints. Table 3 shows the details of the combined dataset after the filtration process described in section 3.1.

There are two methods to evaluate ATD models: one with Case Ending (CE) and one without Case Ending (No CE). In the No CE approach, the diacritic on the last letter is excluded during performance evaluation, while the CE approach includes the diacritic in the evaluation. The presence or absence of the diacritic on the last letter mostly depends on grammatical rules. Error rates without CE reflect the performance specifically on the core word, while error rates with CE represent the overall performance of the model Madhfar and Qamar (2020).

We compared our models with 9 models, namely, CBHG Madhfar and Qamar (2020), Sakhr⁴⁴4https://tashkeel.alsharekh.org, Farasa Darwish and Mubarak (2016), D2 and D3 models AlKhamissi et al. (2020), Alkhalil Tashkeel⁵⁵5https://tashkeel.alkhalilarabic.com, Mishkal⁶⁶6https://tahadz.com/mishkal, Multilevel Diacritizer Abbad and Xiong (2021), and Shakkala. Moreover, we compared the performance of our models with the performance of two Large Language Models (LLMs) on the ATD task: GPT-4-turbo⁷⁷7https://chatgpt.com, and Command R+⁸⁸8https://huggingface.co/CohereForAI/c4ai-command-r-plus. During evaluation, all preprocessing steps are applied to both the reference text and the output of all models to ensure fair comparisons. In the case of long sentence diacritization, we follow a process of splitting the sentence into smaller segments based on punctuation. Each small sentence is then diacritized individually, and finally, the segments are combined to reconstruct the original text with diacritics.

The results in Tables 4 and 5 show that our models achieved state-of-the-art performance in the ATD task, outperforming all 11 models. Our ED model achieved the lowest DER and Word Error Rate (WER) on the CATT benchmark dataset with and without CE. Moreover, both ED and EO models achieved low DER and WER scores on the WikiNews benchmark dataset compared to other ATD models.

The evaluation of GPT-4 on the WikiNews dataset shows a high level of performance. However, when GPT-4 is evaluated on the CATT dataset, its performance appears to be comparatively normal. This difference in performance can likely be attributed to the fact that GPT-4 was trained on web data predating December 2023. Based on the observed significant performance gap between testing GPT-4 on CATT and WikiNews, we believe that GPT-4 was most likely trained on the WikiNews dataset. It is worth noting that the WikiNews dataset was published in 2017 Darwish et al. (2017), while the CATT dataset was created in March 2024.

This paper proposed a new approach to training ATD models. The proposed approach was to initialize ATD models’ parameters from a pretrained character-based BERT model, then training the models for longer iterations. After that, we used the NS approach to further improve the performance of our models. We evaluated our approach by comparing it to 11 commercial and open-source models using two benchmark datasets: WikiNews and CATT. Our results show that our models outperformed all other models in both DER and WER. We open-source our CATT models and dataset for the research community to advance research in this area.

Although this research advances the progress in the ATD task, it has some limitations. These limitations include:

• •

  Specific input assumption: Our model is designed to work only with Arabic text. It does not handle numbers or special characters often found in real-world data. Filtering out these unwanted characters and numbers may alter the sentence structure, potentially resulting in incorrect ground truth diacritics. For example, consider the sentence \<اشتريت 3 كتب> (translation: I bought 3 books). If the number \<3> is removed, the sentence becomes \<اشتريت كتب>, which results in an incorrect grammatical structure (correct structure: \<اشتريت كتبا>) that may also lead to incorrect diacritization. A correct normalization should replace the numeral with an equivalent Arabic word. Meaning, the sentence should be transformed to \<اشتريت ثلاثة كتب>, where \<ثلاثة> represents the number 3 in Arabic. Therefore, it is suggested to have a normalization layer before the diacritizing text in a sequential pipeline.

• •

  No handling for partially diacritized input: The models are not conditioned to process partially diacritized text, as we filter those diacritics out before the text is fed into the model.