AlleNoise - large-scale text classification benchmark dataset with real-world label noise

We collected 75,348 mislabeled products from two sources: 1) customer complaints about a product being listed in the wrong category - such requests usually suggest the true category label, 2) assortment clean-up by internal domain experts, employed by Allegro - products listed in the wrong category were manually moved to the correct category.

The resulting distribution of label noise is not uniform over the entire product assortment - most of the noisy instances belong to a small number of categories. Such asymmetric distribution is an inherent feature of real-world label noise. It is frequently modeled with class-conditional synthetic noise in related literature. However, since the mistakes in AlleNoise were based not only on the category name, but also on the product features, our noise distribution is in fact instance-dependent.

The 75,348 mislabeled products were complemented with 426,962 products listed in correct categories. The clean instances were sampled from the most popular items listed in the same categories as the noisy instances, proportionally to the total number of products listed in each category. The high popularity of the sampled products guarantees their correct categorization, because items that generate a lot of traffic are curated by human domain experts. Thus, the sampled distribution was representative for a subset of the whole marketplace: 5,692 categories out of over 23,000, for which label noise is particularly well known and described.

We automatically translated all 500k product titles from Polish to English. Machine translation is a common part of e-commerce, many platforms incorporate it in multiple aspects of their operation [tan_ecommerce_2020, zhang_improve_2023]. Moreover, it is an established practice to publish machine-translated text in product datasets [ni_justifying_2019]. Categories related to sexually explicit content were removed from the dataset altogether. Finally, categories with less than 5 products were removed from the dataset to allow for five-fold cross-validation in our experiments.

Let $\mathcal{X}$ denote the input feature space, and $\mathcal{Y}$ be a set of class labels. In a typical supervised setting, each instance $x_{i}$ has a true class label $y_{i}$. However, in learning with noisy labels, $\tilde{y}_{i}$ is observed instead, which is with an unknown probability $p$ (noise level) changed from the true $y_{i}$.

In this setting, we train a classifier $f:\mathcal{X}\rightarrow\mathcal{Y}$ that generalizes knowledge learnt from a dataset $\mathcal{D}$, consisting of training examples $(x_{i},\tilde{y}_{i})$. Because $\tilde{y}_{i}$ can be affected by label noise, the model’s predictions $\hat{y}_{i}=f(x_{i})$ might be corrupted by the distribution of noisy labels as well. Maximizing the robustness of such a classifier implies reducing the impact of noisy training samples on the generalization performance. In the AlleNoise dataset, $x_{i}$ corresponds to the product title, $\tilde{y}_{i}$ is the original product category, and $y_{i}$ is the correct category.

In order to compare the real-world noise directly with synthetic noise, we applied different kinds of synthetic noise to the clean version of AlleNoise: the synthetic noise was applied to each instance’s true label ${y}_{i}$, yielding a new synthetic noisy label $\tilde{y}_{i}$. Overall, the labels were flipped for a controlled fraction $p=15\%$ of all instances. We examined the following types of synthetic noise:

• •

  Symmetric noise: each instance is given a noisy label different from the original label, with uniform probability $p$.

• •

  Class-conditional pair-flip noise: each instance in class $j$ is given a noisy label $j+1$ with probability $p$.

• •

  Class-conditional nested-flip noise: we only flip categories that are close to each other in the hierarchical taxonomy of categories. For example, for the parent category Car Tires we perform a cyclic flip between its children categories: Summer $\rightarrow$ Winter $\rightarrow$ All-Season $\rightarrow$ Summer with probability $p$. Thus, the noise transition matrix is a block matrix with a small number of off-diagonal elements equal to $p$.

• •

  Class-conditional matrix-flip noise: the transition matrix between classes is approximated with the baseline classifier’s confusion matrix. The confusion matrix is evaluated against the clean labels on 8% of the dataset (validation split) [patrini_making_2017]. The resulting noise distribution is particularly tricky: we flip the labels between the classes that the model is most likely to confuse.

Next, we evaluated several algorithms for training classifiers under label noise. For a fair comparison, all experiments utilized the same classifier architecture as well as training and evaluation loops. We followed a fine-tuning routine that is typical for text classification tasks. In particular, we vectorized text inputs with XLMRoberta [xlmroberta], a multilingual text encoder based on the Transformer architecture [transformer]. To provide the final class predictions, we used a single fully connected layer with a softmax activation and the number of neurons equal to the number of classes. The baseline model uses cross-entropy (CE) as a loss function.

Models were trained with the AdamW optimiser and linear LambdaLR scheduling ($\text{warmup steps}=100$). We have not used any additional regularization, i.e. weight decay or dropout. Key training parameters, such as batch size ($\text{bs}=256$) and learning rate ($\text{lr}=10^{-4}$) were tuned to maximize the validation accuracy on the clean dataset. All models have been trained for 10 epochs. Training of the baseline model, accelerated with a single NVIDIA A100 40GB GPU, lasted for about 1 hour.

We used five-fold stratified cross-validation to comprehensively evaluate the results of the models trained with label noise. For each fold, the full dataset was divided into three splits: $\mathcal{D}_{train}$, $\mathcal{D}_{val}$, $\mathcal{D}_{test}$, in proportion 72% : 8% : 20%. Following the literature on learning with noisy labels [song_learning_2022], both $\mathcal{D}_{train}$ and $\mathcal{D}_{val}$ were corrupted with label noise, while $\mathcal{D}_{test}$ remained clean.

All of the results presented in this study correspond to the last checkpoint of the model. We use the following format for presenting the experimental results: $[\text{m}]\pm[\text{s}]$, where $m$ is an average over the five cross-validation folds, while $s$ is the standard deviation. Experiments used a seeded random number generator to ensure the reproducibility of the results.

Accuracy on the clean test set is the key metric in our study. We expect that methods that are robust to the label noise observed in the training phase, should be able to improve the test accuracy when compared to the baseline model.

Additionally, to better understand the difference between synthetic and real-world noise, we collected detailed validation metrics. The validation dataset $\mathcal{D}_{val}$ contained both instances for which the observed label $\tilde{y}_{i}$ was incorrect ($\mathcal{D}_{val}^{\texttt{noisy}}$) and correct ($\mathcal{D}_{val}^{\texttt{clean}}$). Noisy observations from $\mathcal{D}_{val}^{\texttt{noisy}}$ were used to measure the memorization metric $\texttt{memorized}_{val}$, defined as a ratio of predictions $\hat{y}_{i}$ that match the noisy label $\tilde{y}_{i}$. Notice that our memorization metric is computed on the validation set, contrary to the training set typically used in the literature [liu_early-learning_2020]. Our metric increases when the model not only memorizes incorrect classes from the training observations, but also repeats these errors on unseen observations. Furthermore, we compute accuracy on $\mathcal{D}_{val}^{\texttt{noisy}}$ denoted as $\texttt{correct}_{val}^{\texttt{noisy}}$ and its counterpart on the clean fraction, $\texttt{correct}_{val}^{\texttt{clean}}$.

We evaluated the following methods for learning with noisy labels: Self-Paced Learning (SPL) [kumar_self-paced_2010], Provably Robust Learning (PRL) [liu_learning_2021], Early Learning Regularization (ELR) [liu_early-learning_2020], Generalized Jensen-Shannon Divergence (GJSD) [englesson_generalized_2021], Co-teaching (CT) [han_co-teaching_2018], Co-teaching+ (CT+) [yu_how_2019] and Mixup (MU) [zhang_mixup_2018]. Additionally, we implemented Clipped Cross-Entropy as a simple baseline (see Appendix A). These approaches represent a comprehensive selection of different method families: novel loss functions (GJSD), noise fltration (SPL, PRL, CCE, CT, CT+), robust regularization (ELR), data augmentation (MU) and training loop modifications (CT, CT+).

These methods are implemented with a range of technologies and software libraries. As such, in order to have a reliable and unbiased framework for comparing them, it is necessary to standardize the software implementation. To this end, we re-implemented these methods using PyTorch (version 1.13.1) and PyTorch Lightning (version 1.5.0) software libraries. We publish our re-implementations and the accompanying evaluation code on GitHub at https://github.com/allegro/AlleNoise.

To select the best hyperparameters (see Appendix A) for each of the benchmarked algorithms, we performed a tuning process on the AlleNoise dataset. We focused on maximizing the fraction of correct clean examples $\texttt{correct}_{val}^{\texttt{clean}}$ within the validation set for two noise types: 15% real-world noise and 15% symmetric noise. The tuning was performed on a single fold selected out of five cross-validation folds, yielding optimal hyperparameter values (Tab. S1). We then used these tuned values in all further experiments.

The selected methods were compared on the clean dataset, the four types of synthetic noise and on the real-world noise in AlleNoise (Tab. 2). The accuracy score on the clean dataset did not degrade for any of the evaluated algorithms when compared to the baseline CE. When it comes to the performance on the datasets with symmetric noise, the best method was GJSD, with CCE not too far behind. GJSD increased the accuracy by 1.31 percentage points (p.p.) over the baseline. For asymmetric noise types, the best method was consistently ELR. It significantly improved the test accuracy in comparison to CE, by 1.3 p.p. on average. Interestingly, some methods deteriorated the test accuracy. CT+ was worse than the baseline for all synthetic noise types (by 2.59 p.p., 2.12 p.p., 3.1 p.p., 2.02 p.p. for symmetric, pair-flip, nested-flip and matrix-flip noises, respectively), while SPL decreased the results for all types of asymmetric noise (by 3.63 p.p., 4.2 p.p., 5.17 p.p. for pair-flip, nested-flip and matrix-flip noises, respectively). CT+ seems to perform better for noise levels higher than 15% (see Appendix B). On AlleNoise, we observed nearly no improvement in accuracy for any of the evaluated algorithms, and CT+, PRL and SPL all deteriorated the metric (by 2.65 p.p., 2.05 p.p. and 4.61 p.p., respectively).

To better understand the difference between synthetic noise types and AlleNoise, we analyze how the $\texttt{memorized}_{val}^{\texttt{noisy}}$, $\texttt{correct}_{val}^{\texttt{noisy}}$ and $\texttt{correct}_{val}^{\texttt{clean}}$ metrics (see 4.4) evolve over time. Memorization and correctness should be interpreted jointly with test accuracy (Tab. 2).

Synthetic noise types are memorized to a smaller extent than the real-world AlleNoise (Fig. 3a). For the two simplest synthetic noise types, symmetric and pair-flip, the value of $\texttt{memorized}_{val}$ is negligible (very close to zero). For the other two synthetic noise types, nested-flip and matrix-flip, memorization is still low (2-8%), but there are clearly visible differences between the benchmarked methods. While ELR, CT+ and PRL all keep the value of $\texttt{memorized}_{val}^{\texttt{noisy}}$ low for both nested-flip and matrix-flip noise types, it is only ELR that achieves test accuracy higher than the baseline.

However, for AlleNoise, the situation is completely different. All the training methods display increasing $\texttt{memorized}_{val}$ values throughout the training, up to 70% (Fig. 3b). PRL, SPL and CT+ give lower memorization than the other methods, but this is not reflected in higher accuracy. While these methods correct some of the errors on noisy examples, as measured by $\texttt{correct}_{val}^{\texttt{noisy}}$ (Fig. 3d), they display $\texttt{correct}_{val}^{\texttt{clean}}$ lower than other tested approaches (Fig. 3c), and thus overall they achieve low accuracy.

These results show that reducing memorization is necessary to create noise-robust classifiers. In this context, it is clear that AlleNoise, with its real-world instance-dependent noise distribution, is a challenge for the existing methods.

To get even more insight into why the real-world noise in AlleNoise is more challenging than synthetic noise types, we analyzed the class distribution within our dataset. For synthetic noise types, there are very few highly-corrupted categories (Fig. 4). On the other hand, for AlleNoise, there is a significant number of such categories. The baseline model test accuracy is much lower for these classes than for other, less corrupted, categories. The set of these highly-corrupted classes is heavily populated by the following:

• •

  Specialized categories that can be easily mistaken for a more generic category. For example, items belonging to the class safety shoes are frequently listed in categories derby shoes, ankle boots or other. In such cases, during the training, the model sees a large number of mislabeled instances of that class and very few correctly labeled ones, which is not enough to learn correct class associations.

• •

  Archetypal categories that are considered the most representative examples of a broader parent category. For instance, car tires are most frequently listed in Summer tires even when they actually should belong to All-season tires or other specialized categories. In this case, the learnt representation of the class is distorted by a huge number of specialized items mislabeled as the archetypal class.

We hypothesize that these categories are the main culprits behind the poor performance of the model.

We thank Mikołaj Koszowski for his help with translating the product titles. We also thank Karol Jędrzejewski for his help in pin-pointing the location of appropriate data records in Allegro data warehouse.

This work was funded fully by Allegro.com.

We declare no competing interests.