Sentence-level Aggregation of Lexical Metrics Correlate
Stronger with Human Judgements than Corpus-level Aggregation

With a case study with BLEU, in this section we discuss why we expect differences in the results according to the the aggregation method. For that we will rely on a simplified abstraction of BLEU, considering that this metric consists of computing a Precision-like score for the generated translation or for a set of generated translations [16].

Before explaining the aggregation method, it is worth explaining how BLEU is computed for a single sentence, i.e. the so-called sentence-level BLEU score. For this case, we basically compute the total of n-gram matches between the candidate sentence, i.e. the sentence generated by the ML model, and the references, representing the ground-truth translations generated by a person that is fluent enough in the target language. The matching is computed by evaluating the number of n-grams present in the candidate sentence that also appear in the references. After we computed that number of matches, we divide this number by the total of n-grams contained in the candidate output, to compute a Precision-ish score that represents the sentence-level evaluation score. Of course, since this is a simplified abstraction, we are not taking into account modified n-gram precision, clipping, combination of different n-gram levels, and brevity penalty, but this does not affect our rationale.

When we expand sentence-level BLEU to evaluate the MT system on a corpus of text or on an entire test set, for instance, the main approach is to rely on the so-called corpus-level BLEU (check Section 2.2.1 in [16] for further details). The corpus-level aggregation (CLA) of BLEU consists of a global computation of n-gram matches and a single scoring for all samples in the test set, done at once. That is, all n-gram matches are counted and summed up, and this number is divided by the sum of the lengths of all candidate sentences.

Although corpus-level BLEU is the usual choice and the default options in tools such as SacreBLEU¹¹1https://github.com/mjpost/sacrebleu, it is quite easy to adopt averages of segment-level scores, or simply segment-level aggregation (SLA), to compute system-level scores [1, 14]. The implementation is straightforward, where one simply need to compute sentence-level BLEU scores for each individual test sample, than using the mean of such scores as the final system-level scores. This aggregation method presents some advantages, such as allowing to compute statistical metrics such as standard deviations, which is not possible with corpus-level aggregation. Notice that bootstrap resampling is a somewhat popular method to compute statistical significance tests on corpus-level scores [10, 8, 6] and can also be used to compute statistical metrics, but it is a more expensive approach in terms of computation requirements.

At a first sight, it is reasonable to believe that CLA and SLA provide the same results, so the main advantage of the latter would only be the less expensive way to compute statistical metrics. But there is a conceptual mathematical difference between these two aggregation methods, which we explain next.

We dive now into a mathematical explanation of the difference between corpus-level and segment-level aggregations, to demonstrate why these two methods may present differences in the results. We argue that the main difference between the two aggregation methods can be seen as a classical case of ratio of averages vs average of ratios.

To understand this, let us adopt a simplified definition of BLEU as a ratio of the number of matches $m$ by n-grams $w$ in a corpus, as in the previous section. And let us also refer to the corpus-level and segment-level BLEU as simply BLEU and m-BLEU, respectively. Considering all the $n$ sentences $i$ of the corpus, it is evident that BLEU can be computed by the ratio between the sum of all partial matches $m_{i}$ in each sentence $i$ by the sum of n-grams in all sentences $i$, $w_{i}$:

Accordingly, m-BLEU is the average of ratios between the number of matches $m_{i}$ and the number of words $w_{i}$ of each sentence $i$:

It is easy to derive that BLEU is the weighted average of the sentence ratios by the proportional length of each sentence $i$:

As we see, m-BLEU weights the ratios equally with $1/n$ weights while BLEU weights the ratios with the value $w_{i}/w$ which is proportional to the length of each sentence $i$. Therefore, BLEU results are likely to be biased by the proportion of matches and candidate sentences lengths, while m-BLEU considers the performance independently of that.

For this investigation we rely on the WMT 2023 Metrics Shared Task dataset²²2https://wmt-metrics-task.github.io/, or simply WMT23 dataset, comprising the results of 492 different MT systems, involving different languages and domains. This dataset contains 468,850 system outputs with the corresponding inputs and references, in a quite diverse setting, containing 147 different language pairs, being 48 different source languages and 44 target languages.

We converted the 468,850 raw entries to 492 system evaluations by grouping the data by dataset type (challengesets2023 or generaltest2023), dataset (challenge_ACES, challenge_DFKI, challenge_NRC-MSLC23, or generaltest2023), language pair (147 options), and system (two systems for the challegeset2023 dataset type and 14 systems for the generaltest2023). Those are all inner structures of the dataset and the aggregated data will be made publicly available³³3anonimous link to dataframe.

We considered three different implementations for BLEU. Two of them are based on the corpus- (CLA) and segment-level aggregation (SLA), and the third relies on bootstrap-resampled scores (BRS) for providing more robust statistical estimates, so that we can use this approach as a reference point for statistically-reliable scores.

As a consequence, the first implementation is referred to as simply BLEU, consisting of the traditional BLEU score with CLA, here computed with the SacreBLEU tool with default parameters [21]⁴⁴4nrefs:1—case:mixed—eff:no—tok:13a—smooth:exp—version:2.0.0. For this metric, given a test set with samples and the corresponding reference MT outputs, we take all MT outputs and references at once, in a single list, and compute the score with the corpus_bleu function in Python code.

The second metric, referred to as m-BLEU, implements SLA. For this we simply compute the score of each segment (i.e. a test sample) with SacreBLEU’s sentence_bleu function, again with default parameters, and calculate the overall average to provide the system-level score.

The third metric, named x-BLEU, consists of implementing the BRS method, as in [14, 11]. This is an alternative to computing system-level scores with higher statistical robustness, where the resamplings represent varied rearrangements of the test set to compute corpus-level BLEU. We rely on 1,000 resamplings, with replacement, of 1,000 samples⁵⁵5Notice that 1,000 samples is roughly very close the mean of samples in the WMT23 dataset, which is 952.94. for each system, and we applied the same SacreBLEU’s corpus_bleu function, with default parameters, on top of each resampled set, generating 1,000 scores for each system. We then provide the average of those 1,000 scores as the system-level score.

In order to investigate whether our observation also generalize to other lexical metrics, we explored also three different implementations of chrF. Notice that chrF is quite similar to BLEU, where overlaps of characters n-grams are computed instead of overlap of words, and the final score is based on an F1 score instead of the precision-ish score used by BLEU. More importantly, the same variety of aggregation methods are possible to be used with chrF, and those varieties can be put in practice with the corpus_chrf and a sentence_chrf functions from SacreBLEU.

Consequently, the three different chrF implemenations that we consider in this work are: chrF, the CLA version computed with corpus_chrf SacreBLEU’s function; m-chrF, the SLA implementation relying on the average of segment-level scores computed with sentence_chrf; and x-chrF, the BRS metric computed with averages of corpus_chrf on 1,000 resamplings with 1,000 samples. Notice that we always rely on the default SacreBLEU’s parameters for such functions.

Our first evaluation focused on investigating the distribution of scores provided by the three different aggregation methods, how these distributions correlate to each other, and how they correlate to more statistically-robust scores. We focus first on analysing BLEU, and then show results with chrF to understand the impact of the base metric.

Figure 1, displaying a grid of 9 plots, presents the main results of this analysis. Notice that in the diagonal we show the histogram of the distribution of scores for each metric, considering the scores for the 492 systems, and in the upper and lower non-diagonal cells we present the pairwise scatter plots and Pearson correlations, in the -1 to 1 range, considering the metrics scores. The main intuition of computing Pearson correlation is to assess the linear correlation between the metrics. That is, a highly-positive correlation, i.e. a value close to 1.0, indicates that high scores in one metric correspond to high scores in another metric, and vice-versa. This correlation helps determine whether the metrics are aligned in capturing what are the good and bad translation results and what is in-between.

From both the distributions and correlation values, we can clearly see that x-BLEU and m-BLEU tend to produce much closer results to one-another than to BLEU. We can see that the distribution of BLEU is right-skewed, while the other distributions are more centralized. In details, the means of the distributions are 0.29, 0.43, and 0.39, for BLEU, x-BLEU, and m-BLEU; and the correlations of BLEU to x-BLEU and m-BLEU are of 0.53 and 0.48, respectively, and x-BLEU and m-BLEU present 0.95 of Pearson correlation.

The results of a repetition of the experiments with chrF as the base metric is presented in Figure 2, and we observe quite similar outcomes. That is, chrF correlates weakly with the other metrics and m-chrF correlates strongly to x-chrF. However, we observe that, unlike the BLEU metrics, the distributions of chrF, x-chrF, and m-chrF can be seen as centralized, presenting means of 0,54, 0.57, and 0.58, respectively. In terms of Pearson correlation, x-chrF and m-chrF present almost perfect correlations to each other with 0.99, while chrF correlates weakly to the other methods, with correlations of 0.56 and 0.55.

What it is very noticeable from these experiments, is that SLA presents a quite higher correlation to BRS than CLA does, which is strong evidence that SLA is not only as capable as BRS to compute statistically-robust scores but also a more-cost-effective option. Moreover, it also indicates that CLA is a quite less statistically-robust method.

We now focus on investigating the correlations between the different aggregation methods of the different BLEU and chrF implementations. The experiments are analogous to those presented in the previous sections, so in Figure 3 we present the scatter plot and the Pearson correlation of each pair of similar implementations, i.e. BLEU vs chrF, x-BLEU vs x-chrF, and m-BLEU vs m-chrF.

Interestingly, these results indicate that similar aggregation methods correlate strong with each other. Although the correlation of BLEU to chrF is the weakest, with 0.77, it is above the 0.50 to 0.55 correlation values that such metrics presented to the non corpus-level ones. And the non corpus-level metrics correlate quite strong to each other, since x-BLEU vs x-chrF present a correlation of 0.88, and m-BLEU vs m-chrF of 0.93. These results indicate that, as we mentioned in Section 3.2, corpus-level aggregation might biased by the ratio between n-gram matches and sentence lengths, and that can explain why BLEU and chrF correlate strong to one another, and weakly to the other methods.

In order to gather additional evidence, in Figure 4 we present a cross-metric comparison betweem m-BLEU and m-chrF, against BLEU, chrF, x-BLEU, and x-chrF, respectively. Again, a weak correlation of BLEU and chrF to the non corpus-level metrics is seen, given that BLEU correlates weakly to x-chrF and m-chrF, and vice-versa. Moreover, m-BLEU and x-BLEU also correlate strongly to x-chrF and m-chrF, respectively, indicating that the corpus-level aggregation introduces a bias that can hinder the statistical robustness of lexical metrics.

Given that one main outcome from the previous section is that corpus-level aggregation (CLA) metrics correlate weakly to their more statistically robust counterparts such as bootstrap-resampled scores (BRS), and that the segment-level aggregated (SLA) metrics correlate strongly to BRS, in this section we focus at investigating the statistical robustness of the aggregation methods.

Our methodology consists of using BRS as an upper bound for statistical robustness for system-level scores, since they rely on the well-known bootstrap resampling method, and on not only evaluating the correlation between the distributions of BRS-based scores against downsampled versions of both CLA- and SLA-based scores, but also the correlation of these downsampled metrics to each other. With this approach we believe we can understand the sensitiveness of the aggregation methods to the number of samples and how their statistical robustness is affected by the number of samples.

In greater detail, we created downsampled test sets for each of 492 datasets, considering three different sizes, i.e. $N=\{1,10,100\}$, and computed scores with both CLA- and SLA-based metrics on each of these downsampled test sets. Then, we computed the Pearson correlation between the scores of these downsampled version of each aggregation method and between these scores and the more statistically-robust ones computed with BRS. This process were repeated 1,000 times for a better statistical estimate. The distribution of correlations, for both BLEU- and chrF-related evaluations, are presented in Figure 5.

The results show that CLA-based metrics are statistically weak. That is, as we can see in both top and bottom left-most plots, the downsampled CLA- and SLA-based metrics correlate strongly only in the case of datasets with only one sample, i.e. $N=1$, with correlation values approaching 1.0. Nevertheless, as the number of samples increases, their correlation reduces quite drastically, to about 0.7 with $N=10$ and about 0.6 with $N=100$. That means that, statistically speaking, we can claim that those metrics differ considerably to each other.

We can go further and dare to claim that an evaluation of an entire test set with CLA is similar to conducting the evaluation of the same system with just a single sample. In the central plots of Figure 5, we can see that the correlation of their downsampled scores to the more statistically-robust BRS-based methods does not increase, or increases very insignificantly, with the number of samples. On the other hand, as we can observe in the right-most plots of the same figure, the correlation of donwsampled SLA-based metrics correlate weakly with the bootstrapped scores with $N=1$, but the correlation increases significantly with $N=10$ and $N=100$, showing that this aggregation method does not suffer from the same lack-of-statistical-robustness problem.