Predictability and Causality in Spanish and English Natural Language Generation

The contextual awareness of a transformer is controlled by self-attention. The base concept behind this attention mechanism is a mapping of a query ($\mathbf{q}$) into pairs of keys ($\mathbf{k}$) and values ($\mathbf{v}$). By respectively denoting the queries’, keys’, and value sets’ matrices as $\mathbf{Q}$, $\mathbf{K}$ and $\mathbf{V}$, we define self-attention as:

Transformers, rather than a single attention function, project queries, keys, and values onto $h$ separate heads. This is called multi-head attention:

By denoting each head attention function as:

where $\mathbf{W}_{i}^{Q}$, $\mathbf{W}_{i}^{K}$, $\mathbf{W}_{i}^{V}$ and $\mathbf{W}^{O}$ are parameter projection matrices for the queries, keys, values, and output respectively.

This attention mechanism is present in all the layers of both the encoder and the decoder, if present. While the encoder’s attention is bidirectional, the decoder has two different types of attention: (i) a masked multi-head attention block that masks non-causal context and (ii) a bidirectional multi-head attention block that receives non-causal information from the encoder.

Even though this encoder-decoder architecture is popular in some NLP tasks such as machine translation [20, 21, 22, 23], several transformer-based models only have one of these components. By omitting the encoder in decoder-only transformers, all non-causal contextual dependencies are removed by exclusively using masked attention. Decoder-only transformers are nowadays the best performing task-agnostic NLG systems. Nevertheless, there exist some state-of-the-art non-causal NLG solutions. For example, non-causal language models can be trained for the Masked Language Modeling (MLM) objective, a task in which the language model predicts masked words within a sentence [24]. Typically, non-causal NLG systems are focused on particular tasks such as speech recognition [25, 26, 27], style transfer and grammar correction [28], textual data augmentation [29], and task-specific dialog systems [30, 31].

Conditional entropy is a typical metric for evaluating the predictability of a problem given its input variables and expected output probability distribution [32, 33]. Conditional entropy $H(X\mid Y)$ measures the extra information carried by a variable $X$ when another conditional variable $Y$ is available as side information.

In psycholinguistics, surprisal theory also uses this information-theoretic concept to quantify processing difficulty in sentence comprehension [34, 35]. Multiple studies provide empirical evidence for this expectation-based theory by showing correlations between textual surprisal and both test subjects’ reading times, as in [36], and brain activity, as in [37].

Even if generally accepted, surprisal theory does not model working memory in text comprehension, disregarding processing difficulties in integrating words or components that are widely apart within a text [38, 39, 40, 41, 42]. Lossy context surprisal [43] combines expectation and memory-based predictability theories by modeling working memory constraints as noise. Even though this model premise is independent of language, it can accurately reflect several language-specific text-processing phenomena.

Lossy context surprisal recreates structural forgetting by dropping part of the context and re-sampling it incorrectly from the a priori language knowledge probability model. Structural forgetting [44] is a common grammatical illusion in English in which ungrammatical double-embedded relative clauses can be perceived as correct. Probabilistic language expectations can determine this exclusively. In [45] it is proven that native and non-native speakers show structural forgetting in English, but do not behave this way when presented with the same syntactic structures in German or Dutch. This propensity of English probabilistic distribution to such backward prediction mistakes is coherent with the issue of non-causal English text generation.

The main goals of neural NLP and psycholinguistics approaches to language cognition are very similar: (i) to give formally explicit descriptions of the mental structures underpinning cognitive processes, and (ii) to explain the learning mechanisms behind them [46]. Even if research in these areas tends to diverge, recent contributions to the study of linguistic theory use language models [47], further evidencing their alignment.

However, to our knowledge, psycholinguistics concepts have yet to be applied to neural language modeling other than for data set elaboration [48]. With this in mind, our work provides a novel linguistic-based conditional entropy hypothesis test for language modeling causality (see contribution 1 above), whose findings can support future NLG designs and methodologies.

Causal language models predict the next token in a sequence of tokens. These models are solely concerned with the left context for sinistrodextral (i.e., written from left to right) languages such as Spanish and English (conversely, non-causal models trained on the MLM task consider the bidirectional context for blank-filling-based text generation).

Given a sequence of tokens $X$ as context, language models provide the probability mass function for the next predicted token $\hat{X}$. For a generation index $i<N$, we define the $N$-long input causal context as follows:

And the non-causal context as:

So that we can express the output of a causal language model as:

And the output of a non-causal language model as follows:

with vocabulary set $\mathcal{V}=\left\{\begin{matrix}v_{0}&\dots&v_{\left|\mathcal{V}\right|-1}\end{matrix}\right\}$ of size $\left|\mathcal{V}\right|$.

As stated in Section II-B, we use a novel metric of conditional entropy to test whether a language is more or less predictable given causal or non-causal contexts (and thus, for example, whether Spanish NLG may benefit from non-causal language generation ordering). The less conditional entropy a problem has, the more predictable its outcome. As NLG is a language prediction task in which previously generated words are available as context, we want to compare the conditional entropy in two scenarios: (i) causal text generation, in which text is generated from left to right so that we provide words to the left of the predicted one as context; and (ii) non-causal text generation, which uses both left and right context for word prediction.

In order to test the predictability of causal and non-causal language models for English and Spanish, we compute and compare the average causal and non-causal conditional entropy for textual data in both languages:

with:

It must be noted that both $\mathcal{X}^{N}_{c}$ and $\mathcal{X}^{N}_{n}$ have size $\left|\mathcal{V}\right|^{N}$. Context length and vocabulary size determine the accuracy of our estimation results. As we have no previous information about token probability distribution, we model both $p(X)$ and $p(\hat{X}=\hat{x}\mid X=\mathbf{x})$ as categorical distributions. The estimators used for these distributions are:

and

with $L$ being the token sequence length, $L_{i}$ the number of instances of the context $i$, and $L_{i_{j}}$ the number of instances in which token $j$ appears given context $i$.

In case both $p(X)$ and $p(\hat{X}=\hat{x}\mid X=\mathbf{x})$ are discrete uniform distributions, these estimators have normalized variances $\frac{\left|\mathcal{V}\right|^{N}-1}{L}$ and $\frac{\left|\mathcal{V}\right|^{N}\left(\left|\mathcal{V}\right|-1\right)}{L}$, respectively. This means that, in order to set our estimators’ normalized variances to a specific value, the number of analyzed tokens should be proportional to $\left|\mathcal{V}\right|^{N}$ and $\left|\mathcal{V}\right|^{N+1}$, respectively.

Therefore, given the data available, neither word nor subword tokenization are feasible. We instead use a grammatical categorization based on Part-Of-Speech (POS) tagging. It reduces vocabulary size and data requirements dramatically while maintaining the original goal.

The resulting hypothesis test evaluates how predictable natural English and Spanish syntaxes are for causal and non-causal language models. Our first intuition is that non-causal predictability, as determined by the inverse of the non-causal context-conditioned entropy, will be higher for Spanish than for English and the opposite for causal predictability. By validating this, we can demonstrate that causal ordering may not be ideal for Spanish NLG, paving the way for further study of non-causal Spanish text generation approaches based on bidirectional transformers.

For non-causal NLG, first we start with a sequence of [MASK] tokens of the desired length $K$. At each iteration, we re-sample every token once. We mask and fill tokens in groups of size $N$. In order to fill the masked tokens, we sample the output of a non-causal language model, from which we remove adjacent tokens, short prefixes and suffixes, and unknown tokens to enhance the overall quality of the produced sequence. This process is formally described in Algorithm 1.

The number of iterations $I$ and the number of tokens masked in each generation step $N$ must be predetermined. These parameters influence the performance and computational efficiency of the algorithm. More masked tokens per generation step mean fewer calls to the language model function ($\lceil\frac{K}{N}\rceil$ calls per iteration), resulting in improved computing efficiency. In this work we set $N=2$ and $I=30$.

The relative entropy $\mathcal{D}_{KL}\left(P\mid\mid Q\right)$, also known as KL divergence, quantifies the expected increase in uncertainty that comes from modeling a reference distribution $P$ as another distribution $Q$. In this study, we use the following formulation for the conditional relative entropy for both causal ($X=X_{c}$) and non-causal ($X=X_{n}$) contexts:

where $p$ and $q$ are the conditional probability density functions of our reference textual dataset’s POS tags and the sequences to evaluate, respectively.

The annotators were asked yes/no questions on the following aspects to assess the quality of the generated sequences:

• •

  Q1. Concordance, penalizing improper use of verb tenses, number, and, if applicable, gender of determinants, adjectives, nouns, and pronouns.

• •

  Q2. Syntactic structure correctness, by checking if all sequences have at least one subject and one verb and assessing that the sentences are syntactically sound in general.

• •

  Q3. Word or phrase-level repetitions, by penalizing word redundancy, duplication in enumerations, and subject redundancy, while trying to respect those repetitions that may be considered stylistic choices.

• •

  Q4. Word sense. Language models can generate new words by combining prefixes, suffixes, and pronouns as sequential tokens. This question penalizes nonsensical words.

We assessed inter-agreement with accuracy and $\alpha$-reliability [49] to verify that the annotations were neither arbitrary ($acc=\alpha=0$) nor redundant ($acc=\alpha=1$).

Finally, we also included a more general rating question (Q5) in which we asked annotators to provide a numerical score between 1 and 5 based on their impression of the annotation and their experience.

The tale data sets of setup #1 consisted of public domain tales, short novels, and fables with Creative Commons-licensed translations. We crawled the English and Spanish text collections for the sizes of the respective datasets to be identical (3.7M words of raw text each) so that they could be considered direct translations, from Portable Document Format (PDF) with the Python pdftotext¹⁵¹⁵15Available at: https://pypi.org/project/pdftotext, June 2024. library and by web scrapping using Scrapy¹⁶¹⁶16Available at: https://scrapy.org, June 2024. web spiders. Table II shows all the data sources.

We could not compute conditional entropy values for contexts longer than two words due to data set size constraints. However, because of the similarities in content between English and Spanish data, we could efficiently study short-term grammatical dependencies in both languages. In the experiment, we compared the causal two-word context with a specific case of non-causal context in which the predicted word lies between the two contextual words. We provide a brief qualitative analysis of the contexts that resulted in lower conditional entropy values for the predicted term $\hat{X}_{i}$ for causal and bidirectional contexts in both languages.

Table III shows the results of hypothesis testing on English and Spanish tale data sets. They are coherent with our initial intuition that Spanish is more suited for non-causal text prediction than English. However, by examining Table III row-wise, middle tag prediction seemed to have lower entropy than causal text prediction in both languages. This does not necessarily mean that non-causal NLG outperformed its causal counterpart, as this experiment disregarded relevant factors, such as initial text generation steps.

Tables IV, V, VI and VII show left and bidirectional context-predicted word pairs with entropy lower than $\log(3)\approx 1.585$. We can note that the number of combinations satisfying this condition for the Spanish data set was much higher for bidirectional contexts than for causal contexts. Figure 1 shows that, for Spanish, causal patterns were also less diverse, as all of them relied on either pronoun/verb or determinant/noun grammatical dependencies.

As shown in tables VI and VII, causal and bidirectional low entropy contexts in English were more balanced. Figure 2 reflects the prevalence of the adjective/noun dependencies (especially for the causal case). Conversely, adjectives have a much less rigid position in Spanish within the sentence.

The data sets of setup #2 comprised a collection of Wikipedia articles from Wikimedia’s Spanish¹⁷¹⁷17Available at: https://dumps.wikimedia.org/eswiki/20220801, June 2024. and English¹⁸¹⁸18Available at: https://dumps.wikimedia.org/enwiki/20220801, June 2024. dumps. We extracted and cleaned textual data from these dumps using WikiExtractor¹⁹¹⁹19Available at: https://github.com/attardi/wikiextractor, June 2024.. Then we loaded and mapped the resulting JSON files with HuggingFace’s Datasets²⁰²⁰20Available at: https://github.com/huggingface/datasets, June 2024. library. We picked one million random non-empty articles in each language for hypothesis testing.

The amount of data available allowed accurately computing the conditional entropy for longer contexts than in the previous case, yielding average conditional entropy results for contexts up to six words. For this experiment, we explored all possible non-causal contexts and assessed the impact of the location of the predicted tag on our predictability results.

Table VIII shows the average causal and non-causal conditional entropy results for the Wikidumps data set. Compared to the tale data set with tighter equivalence between languages, Spanish conditional entropy values often exceeded those for English. For shorter contexts, even non-causal conditional entropy was higher in the Spanish data set than in the English data set. For this reason, we report average entropy as conditioned by the set of non-causal contexts by subtracting the causal context, which is always lower in Spanish than English, highlighting the lower efficiency of left-to-right word prediction in Spanish.

We can further appreciate this effect in Figure 3, which depicts the average conditional entropy for all possible predicted tag locations within contexts of diverse lengths. As we can see, the more balanced the left and right contexts are, the more predictable the grammatical category in both languages. However, this tendency was considerably more noticeable in Spanish and became more evident as the context was longer. Since the average number of words per sentence tends to be higher in Spanish than in English [50, 51, 52], Spanish non-causal NLG is even more promising.

In the sequel, we will call “opposite” the language, either Spanish or English, for which the model was not trained for an experiment. We used the conditional relative entropy metric described in Section III-C for automatic evaluation to compare the produced sequences from the tales data sets in the target and opposite languages. This was possible because language-independent POS tagging was used for tokenization.

As expected, Table IX reveals that the sequences generated by all four models adhered more closely to their respective target language’s POS tag distributions. There were no significant differences between causal and non-causal context-conditioned relative entropy results.

Given its conditional relative entropy values one order of magnitude greater than those of the other three models, English BERT performed significantly worse in terms of adherence to the target language ($\sim$0.02 vs $\sim$0.1), which is consistent with the state of the art [53, 54]. For Spanish, the results of causal and non-causal NLG are more comparable. The Spanish BERT non-causal language model had the lowest conditional relative entropy, outperforming even English GPT-2 when considering adherence to the respective target languages.

These results show that causal models corresponded more closely to English grammar and non-causal models corresponded more closely to Spanish grammar (by considering grammar as reflected in the English and Spanish datasets). Note that this also held when using the models of the opposite language. That is, the conditional relative entropy of English Bert for Spanish was lower than the conditional relative entropy of English GPT-2 for Spanish, and the conditional relative entropy of Spanish GPT-2 for English was lower than the conditional relative entropy of Spanish BERT for English.

We chose 250 sequences at random for each pairing of language (Spanish or English) and model (BERT or GPT-2) to reduce the annotation load while still obtaining useful insights. Each sequence was examined independently using the questions in Section III-D.

Five annotators participated in the evaluation. Table X shows the global inter-agreement analysis of yes/no replies. Using the thresholds by [55], $\alpha$-reliability coefficients lay between fair and moderate agreement. Accuracy values were also acceptable for all questions. The grammatical structure (Q2) was the most controversial aspect of the first four questions due to different opinions on linguistic demand, as some annotators were more lenient with one of the languages.

Next, we can see that the manual assessment scores for all four questions in Table XI were consistent with the automatic metrics: BERT was considered to perform better in Spanish and GPT-2 to perform better in English. The best outcome for Spanish language models was in word sense (Q4), whereas English language models scored better in word concordance (Q1). This is consistent with the fact that there is no gender concordance in English. For all language models, the more challenging question was word repetition (Q3).

For the more subjective fifth question, as the average rating from one annotator to another ranged from $3.529$ to $4.31$, we normalized the scores of each annotator for an average of 1.0. The results of Table XII indicate that the subjectively perceived quality of Spanish texts generated by BERT is higher than when using GPT-2, vice versa in the case of English, which is consistent with our initial intuition and all the results so far.