Regurgitative Training: The Value of Real Data in Training Large Language Models

Self training is one of the classic approaches in semi-supervised learning to train a machine learning model using both labeled and unlabeled data (Scudder,, 1965; Nigam and Ghani,, 2000). Take classification tasks as an example, the idea is to first build a model on the labeled data via standard supervised learning procedures, obtain the model’s predictions on the unlabeled data, then take the most confident predictions (e.g., data instances with most extreme predicted probabilities) and treat them as additional labeled data to re-train the model. As a way to convert some unlabeled data into labeled data, self training is useful especially when the original labeled data are scarce.

There is an extensive literature on self training, both in traditional machine learning (see Pise and Kulkarni,, 2008; Triguero et al.,, 2015, for surveys of the topic) and in modern deep learning (e.g., Xie et al., 2020b, ). One key insights from this body of work is that the effectiveness of self training depends heavily on the ability to accurately estimate “prediction confidence”. Because predictions with high confidence are used as “pseudo-labels”, having accurate confidence scores imply that the “pseudo-labels” are more likely to be correct (i.e., the same as ground-truth labels).

Regurgitative training of LLMs resembles self training in that model-predicted pseudo labels are used to further train the model. However, it is unclear whether the conventional wisdom of self training would still apply for in the case of regurgitative training, because of the difficulty in assessing confidence of LLM outputs (Lin et al.,, 2023). In classification tasks, predicted class probabilities naturally serve as the measure to quantify the uncertainty in a classifier’s predictions. However, LLMs are much more complex than classifiers – they generate multi-token answers in response to prompts. Unless in highly restricted scenarios (e.g., evaluating a single-digit response to the question “what is 2+2”), it is generally not straightforward to define or measure confidence in LLM outputs. As a result, current LLMs do not automatically produce confidence scores for their responses and uncertainty quantification in LLMs remain an open question with many ongoing research, including asking LLMs themselves for confirmation (“self-reflection”, Chen and Mueller,, 2023), re-running the same prompt multiple times and measure internal consistency among responses (Kotelanski et al.,, 2023), and tapping into human expertise (Shankar et al.,, 2024).

Data augmentation represents another approach to enrich potentially limited labeled data. It works by injecting noises into existing labeled data instances to artificially create new data instances that can be assigned the same labels. In language tasks, a common data augmentation strategy is back-translation (Yu et al.,, 2018), where a sentence is first translated into a different language then back to the original language to achieve paraphrasing. In vision tasks, data augmentation may involve image transformation techniques such as rotation, color / contrast modification, etc. (Cubuk et al.,, 2020; Xie et al., 2020a, ). These augmentation strategies can benefit model performance if the injected noises do not change the labels, thereby creating more training data with valid labels.

Although data augmentation is procedurally quite different from regurgitative training, it does offer an insight that can help enhance the performance of regurgitative training. Xie et al., 2020a found that data augmentation is more effective when the augmentation strategy can generate a diverse set of instances rather than only introducing small, local perturbations. Learning from a diverse set of augmented data can enable the model to achieve competitive performance with fewer examples. Conceptually, this finding is also consistent with observations made in other machine learning research outside of data augmentation (e.g., Gong et al.,, 2019), where the diversity of training data instances is positively associated with predictive performance. Later, we leverage this insight in one of the strategies designed to mitigate performance loss of regurgitative training, by mixing data generated by different LLMs as an attempt to introduce greater diversity to the training process.

Regurgitative training of generative AI models represents a new problem that has only begun to receive scholarly attention very recently. The earliest work we could identify is Shumailov et al., (2023), which document that using model-generated data to train next-generation models can create irreversible performance losses, a phenomenon they term “model collapse”. They demonstrate this in common generative AI architectures such as variational autoencoders, Gaussian mixture models, and small-scale LLMs. Moreover, they provide theoretical intuitions that model collapse arises due to errors in model-generated data, which accumulates over more iterations of regurgitative training. Subsequently, the phenomenon of model collapse has also been observed in generative image models (Alemohammad et al.,, 2023; Bertrand et al.,, 2023).

In the meantime, efforts to mitigate model collapse are underway. Bertrand et al., (2023) show that model collapse can be avoided if (i) the proportion of real data is sufficiently high and (ii) model-generated data approximate the distributions of real data well enough. Furthermore, Gerstgrasser et al., (2024) propose to alleviate model collapse by “accumulating data”; that is, use the totality of real and model-generated data (rather than just the model-generated data) to train new models.

We build upon this nascent stream of research and aim to make several distinct contributions. First, we consider regurgitative training of a LLM not only by data generated by itself, but also by other LLMs with varying degrees of capabilities. This is already taking place in practice (e.g., McKinzie et al.,, 2024) but has not been systematically explored in the literature. Second, prior work such as Shumailov et al., (2023) focused on early versions of generative models (e.g., non-transformer-based models or small pre-trained models). Instead, we carry out comprehensive experiments with leading LLMs at the time of research (e.g., GPT-4 and Llama2) as well as transformer models trained from scratch, thereby providing a more up-to-date understanding of regurgitative training. Finally, we propose several new mitigation strategies beyond what has been tested so far, and empirically evaluate their effectiveness.

To carry out LLM fine-tuning for translation, we use the Europarl parallel corpus (Koehn,, 2005). Sourced from the proceedings of the European Parliament, the corpus contains parallel sentences in multiple European languages. We specifically use pairs of German-English sentences. After basic pre-processing steps (e.g., removing special HTML tags, eliminating noisy characters, and handling null values), we end up with 1,908,849 sentence pairs for our analyses. We treat these sentence pairs as real data.

A popular and widely used metric to evaluate the performance of a translation model is the BLEU (BiLingual Evaluation Understudy) score (Papineni et al.,, 2002). It evaluates the quality of a model-generated translation (also called a “hypothesis translation”) in comparison to one or more reference translations.³³3The BLEU score can be analogously defined for a corpus of hypothesis translations and the corresponding corpus of reference translations. We describe the simpler case with a single hypothesis translation here for ease of understanding, and refer readers to Papineni et al., (2002) for the more general case. The BLEU score is calculated as the n-gram overlap between the hypothesis translation and reference translations. It ranges from 0 to 1 and a higher BLEU score generally indicates better quality translations. Formally, the BLEU score is defined as

In the first term, $c$ is the length of hypothesis translation and $r$ is the “effective” length of reference translations (defined as the length of the one reference translation that best matches the hypothesis translation). This term serves as a “brevity penalty” that assigns a higher score for a better match in lengths between hypothesis and reference translations. In the second term, $p_{n}$ denotes the $n$-gram precision and is defined as

where $Count_{matched}(n\mbox{-}gram)$ counts the number of $n$-gram matches between the hypothesis translation and reference translations. In Equation (1), the $n$-gram precision scores are then weighted by $w_{n}$ (e.g., uniform weighting $w_{n}=\frac{1}{N}$) to compute the overall BLEU score.

To implement and evaluate regurgitative fine-tuning, several components need to be defined first, including a baseline LLM to be fine-tuned, a set of training data (either real or generated by other LLMs) used for fine-tuning, and a fine-tuned LLM for evaluation. In our context, we use the GPT-3.5 model as the baseline LLM,⁴⁴4At the time of our research, OpenAI’s fine-tuning service was restricted to the GPT-3.5 model. then fine-tune it with (i) real human-generated data, (ii) data generated by GPT-3.5 itself, and (iii) data generated by two other LLMs, namely GPT-4 and LLAMA2. This creates four fine-tuned LLMs, all of which are evaluated on the same testing data for performance comparison.

More specifically, we randomly select 5,000 sentence pairs from the original corpus for fine-tuning and 10,000 sentence pairs as the testing data. When fine-tuning with real data, the 5,000 German sentences are used as inputs and the corresponding 5,000 English sentences are used as target translations. When fine-tuning with LLM-generated data, the same 5,000 German sentences are used as inputs, but the target translations are generated by the corresponding LLM. For GPT-3.5, GPT-4, and LLAMA2, we obtain their translations with the same system prompt: “You are a chatbot that can translate German to English.”, and the German sentences are given to the LLMs as user inputs. Using each set of data, we carry out progressive fine-tuning over five batches, adding 1,000 data instances per batch and recording translation performance on the testing data after each batch.

We show the results in Figure 1. Each line represents the translation perform of a particular model over five fine-tuned batches. The $X$-axis indicates batch index, marking the number of data instances utilized in the fine-tuning process. The $Y$-axis represents the BLEU score, where a higher value corresponds to better translation performance.

From the figure, it is evident that the performance of fine-tuning with LLM-generated data (both from the baseline LLM itself and from other LLMs) clearly lags behind the performance of fine-tuning with real human-generated data. Moreover, regurgitative training with different LLMs have differential performance impact. Fine-tuning with data generated by GPT-3.5 itself does not significantly change performance, and fine-tuning with GPT-4 generated data only results in barely noticeable performance improvement over the baseline model (i.e., at point 0 on the $X$-axis). However, fine-tuning with LLAMA2 generated data significantly degrades performance compared to the baseline. This is likely because the three LLMs have different translation capabilities. Since we have the ground-truths translations for the 5,000 fine-tuning data, we can directly compute the BLEU scores of translations generated by the three LLMs, and indeed find GPT-4 to be the best ($BLEU=0.3454$), followed by GPT-3.5 ($BLEU=0.3428$) and LLAMA2 ($BLEU=0.2417$).

These results underscore the overall negative, and potentially detrimental, effects of regurgitative training. Compared to training with real data, regurgitative training largely stalls learning. Regurgitative training with a better-performing LLM improves performance only marginally and is not sufficient to catch up with the performance on real data. Worse yet, regurgitative training with a less capable LLM can significantly hurt performance.

Note that in the above experiments, we use utilize a small set of data for fine-tuning. This decision stems from the remarkable few-shot learning capabilities of modern LLMs (Brown et al.,, 2020). In addition, we also conduct a robustness check to understand whether the performance of regurgitative training may be different if more fine-tuning data are available. Specifically, we augment the fine-tuning data size by 20 times, to a total of 100,000, and incrementally add 10,000 per batch. For efficiency and cost considerations, we only run fine-tuning with real data and data generated by GPT-3.5 itself. We then evaluate each fine-tuned models on the same testing data as before. The results are presented in Figure 2. We again observe that regurgitative training is unable to improve translation performance and substantially underperforms training with real data.

We now turn to training models from scratch and understanding the performance impact of regurgitative training in this case. Specifically, we build transformer models using the translation data. The transformer architecture serves as a foundational component powering the majority of modern LLMs, and has found extensive applications in machine translation and a variety of other natural language tasks (Vaswani et al.,, 2017; Wolf et al.,, 2020). We therefore choose to train small-scale transformer models from scratch, as an attempt to approximate the practice of training transformer-based models without leveraging third-party LLMs.

We follow Vaswani et al., (2017) to build the baseline translation models. Transformer has an encoder-decoder architecture, which uses stacked layers of multi-head self-attention and point-wise, fully connected feed-forward networks for both encoder and decoder. It also employs a residual connection on each sub-layer, followed by layer normalization. For implementational details of these transformer elements, we refer to Vaswani et al., (2017).

As our previous fine-tuning results have shown, the performance impact of regurgitative training can vary with the capability of the model used to generate training data. Therefore, we train both a “low-performance” and a ”high-performance” baseline models. This is done by gradually adding 50,000 sentence pairs (randomly sampled from the German-English corpus) per batch for training, and evaluate the model’s translation performance on a fixed testing dataset of 50,000 sentence pairs. As shown in Figure 3, we observe that the model’s performance improves quickly with the initial increase in training data size, and saturates after being trained with sufficient data. We choose the model trained with 50,000 data instances as our low-performance baseline and the one trained with 500,000 data instances as the high-performance baseline.

These two baseline models, corresponding to different performance levels, are then used to evaluate the effects of regurgitative training. We randomly sample a total of 300,000 data instances (outside of the training data of both the low- and high-performance baseline models) designated for regurgitative training. In batches of 10,000 data instances, we continue training both the low-performance baseline model and the high-performance baseline model with (i) real human-generated data, (ii) data generated by the low-performance model, and (iii) data generated by the high-performance model. After each batch of training, we evaluate the all models’ performance on the same testing data of 50,000 instances. The results are presented in Figure 4.

For both low-performance and high-performance baseline models, regurgitative training with data generated from the low-performance model clearly underperforms training with real data. The same is true for regurgitative training of high-performance model with data generated by itself, though the performance gap is fairly small. Curiously, regurgitative training of low-performance model with data generated by high-performance model actually outperforms training with real data for the first 19 batches (i.e., top two lines in the right plot). To understand whether this is a sustainable performance advantage, we sample more data to carry out another 20 batches of regurgitative training in this case. The results, shown in Figure 5, show that regurgitative training performance starts to plateau around 30 batches, and underperforms training with real data thereafter.

The above results further demonstrate the performance cost of regurgitative training, even when businesses create and train their models from scratch. Consistent with our observations under regurgitative fine-tuning, regurgitative training with data from a more capable model is better than those from a less capable model – training with data generated from the low-performance model clearly harms performance. Regurgitative training with the more capable high-performance model can match or even surpass the performance of training with real data, but such advantages usually fade away as the size of regurgitative training data grows.

In reality, it is plausible that a mixture of both model-generated data and real data are used for training. We therefore carry out another set of experiments to check how the proportion of model-generated data in the mix affect model performance. We simulate five scenarios, where the proportion of model-generated data is 100%, 75%, 50%, 25%, and 0% respectively, and the rest of each mixture consists of real data. Naturally, the scenarios with 100% and 0% of model-generated data are the same as training purely with model-generated data or purely with real data. For clarity, we focus on training each baseline model with data generated by itself (mixed with different proportions of real data). Other experiment settings are the same as before, and the results are shown in Figure 6.

In the case of high-performance baseline (i.e., left plot), because the performance gap between regurgitative training and training with real data is relatively small to begin with, the pattern is obfuscated by local performance fluctuations. Nonetheless, having a higher proportion of real data still generally leads to better performance. In contrast, the pattern becomes much clearer in the case of low-performance baseline (i.e., right plot). We can see that even a small amount of model-generated data is enough to slow down learning. As a higher proportion of model-generated data is used, the model’s performance continues to deteriorate.

In addition to machine translation, we also conduct a replication study with another common generative language task – Question Answering (Q&A). We use the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al.,, 2016), which is a widely used benchmarking dataset for developing and testing Q&A methods. SQuAD is a reading comprehension dataset composed of questions created by crowd-workers based on a collection of Wikipedia articles, with answers being segments of texts from the corresponding passages in the articles. The dataset includes 87,599 entries in the training dataset (constructed from 442 articles) and 10,570 entries in the testing dataset (constructed from 48 other articles).

Instead of end-to-end training (as was done in the previous section), here we use a pre-trained multilingual BERT model (bert-base-cased, Devlin et al.,, 2018) to extract word embeddings, which are fed into a feedforward neural network model for Q&A. Doing so allows us to evaluate regurgitative training under yet another widely adopted strategy for training generative language models (i.e., leveraging pre-trained representation models).⁵⁵5This strategy is often also referred to as “fine-tuning” a pre-trained model. We refrain from using this term here in order to avoid confusion with our fine-tuning experiments in Section 3.1, which are performed on top of existing LLMs (rather than a BERT-like representation model). We follow Rajpurkar et al., (2016) to evaluate Q&A performance with two metrics: Exact Match and F-1 score. Exact match measures the percentage of predicted answers that match the ground-truth answers exactly, and F-1 score measures the overlap between the predicted answers and the ground-truth answers by treating both predictions and ground-truths as bags of tokens. We calculate the average F-1 over all questions in the testing data.

Following the same procedure as in the previous section, we use increasing amounts of real data to train baseline models in order to identify a low-performance model and a high-performance model (see Appendix A for detailed results). The low-performance model is trained on all entries from 40 articles and achieves 70.77% exact match rate and 80.34% average F-1 score, whereas the high-performance model is trained on all entries from 200 articles and achieves 78.63% exact match rate and 86.59% average F-1 score.

Next we use these two baseline models for regurgitative training on entries from the remaining 242 articles (not used in training the two baseline models). In batches of 10 articles, we continue training both the low- and high-performance baselines with (i) real human-generated data, (ii) data generated by the low-performance model, and (iii) data generated by the high-performance model, for a total of 20 batches. After each batch of training, we evaluate the all models’ performance on the same testing dataset provided by SQuAD. The results are included in Figure 7, where the first row shows performance of regurgitative training the high-performance baseline model and the second row shows performance of regurgitative training the low-performance baseline model.

We again observe that regurgitative training with model-generated data negatively affects Q&A performance, compared to training with real data. Different from the translation task, Q&A regurgitative training with data generated by the high-performance model does not improve performance (and certainly does not outperform training with real data), even though it still weakly outperforms regurgitative training with low-performance model generated data. In other words, the peril of regurgitative training is not limited to translation task and is even more severe in Q&A task.

The first strategy is to identify a method to assess the quality of synthetic data, and subsequently select higher-quality data for regurgitative training. This requires defining a metric that accurately measures, or at least correlates with, data quality specific to the task at hand. One such metric, commonly used in classification contexts, is prediction confidence score. Higher prediction confidence scores usually correlate with greater probability of correct predictions, and the semi-supervised learning literature routinely uses prediction confidence as a quality metric (e.g., Scudder,, 1965). Because modern LLMs generate content by autoregressively predicting the next token, it is viable to also adopt prediction confidence, calculated based on predicted probabilities over the vocabulary, to quantify the quality of LLM-generated data. However, a practical obstacle is that when using third-party LLMs, prediction probabilities may not always be available. Therefore, we devise an alternative quality metric to guide the quality-based mitigation in the setting of LLM fine-tuning, assuming prediction probabilities are unavailable (Section 5.1.1). We also demonstrate the same mitigation strategy with transformers trained from scratch, assuming prediction probabilities are fully available (Section 5.1.2).

As discussed in Section 4, a lack of lexical diversity in LLM-generated content may also contribute to the performance loss of regurgitative training. This observation prompts us to explore a mitigation strategy aimed at enhancing lexical diversity within LLM-generated data. Specifically, we propose mixing together data generated from multiple LLMs into the regurgitative training process. For example, LLMs developed by different companies, each potentially trained on somewhat distinct datasets, may consequently produce data with unique characteristics and nuances. Combining data from different “breeds” of LLMs can therefore introduce greater variability than relying on a single LLM.

We first test this data mixture strategy under LLM fine-tuning setting. With three LLMs, there are 3 possible mixture configurations: (i) mixing GPT-3.5 with GPT-4, (ii) mixing GPT-3.5 with LLAMA2, and (iii) mixing GPT-4 with LLAMA2. We consider configurations (i) and (iii) in particular. Configuration (iii) mixes two top-of-the-line LLMs developed by different companies and, based on our explorations in Section 4, they exhibit different lexical diversity. In other words, this configuration is a more direct evaluation of the proposed strategy. However, it is unfortunately confounded by the fact that LLAMA2-generated data have higher translation error rates than GPT-generated data, so the performance outcomes of this configuration cannot be solely attributed to changes in lexical diversity levels. In comparison, configuration (i) is less confounded because the two GPT models have similar translation quality, although their mixture also bring in less additional lexical variations. Taken together, we present results from both configurations to offer a more comprehensive evaluation of the data mixture strategy.

Furthermore, for each batch of 1,000 regurgitative training instances, there are two ways to add the mixture data. First, we can add (randomly selected) 500 instances from one LLM and 500 from the other LLM, thereby keeping the total training batch size unchanged. Alternatively, we can add all 1,000 instances from both models, which amounts to a training batch size of 2,000. Both are reasonable from a practical perspective, and we report both sets of results. After each batch of regurgitative training, we evaluate translation performance on the same testing data used in Section 3.1. Figure 11 shows results for GPT-4 / LLAMA2 mixture and Figure 12 shows results for the GPT-3.5 / GPT-4 mixture.

Interestingly, mixing data generated by GPT-4 and LLAMA2 results in performance levels that fall in between using the two constituent LLMs alone, but mixing data generated by GPT-3.5 and GPT-4 can match the performance of GPT-4 (the better-performing LLM of the two) and even slightly outperform it when twice the regurgitative training data are added. These results suggest that the effectiveness of the data mixture strategy is nuanced. When constituent LLMs differ both in terms of quality and lexical diversity, mixing their data together may not lead to better regurgitative training performance, because the potential benefit of greater lexical diversity is offset by having a more error-prone LLM in the mix. In the same vein, if two LLMs with similar quality yet different lexical diversity can be identified, their mixture can indeed mitigate performance loss of regurgitative training to some extent.

We also deploy the same strategy on transformer models trained from scratch. With two baseline models, there is only one mixture configuration, namely mixing data generated by the low-performance model and high-performance model. However, these two models naturally differ on both quality and lexical diversity (just like the GPT-4 / LLAMA2 mixture). Therefore, we also consider a mixture of data generated by GPT-3.5 and high-performance model (to mimic the GPT-4 / GPT-3.5 mixture configuration). Here, GPT-3.5 serves as a high-quality translation model that is also very different from the model we trained from scratch. The results are shown in Figures 13 and 14.

We find highly consistent results as in the fine-tuning case. Mixing data generated by the two baseline models, which differ on quality, results in regurgitative training performance that falls between the two constituent models. Moreover, when training the high-performance baseline model with the data mixture (left plot of Figure 13), the performance follows a downward trend, likely because data generated by low-performance model substantially contaminate the quality of data mixture. Consequently, using twice the amount of mixture data in regurgitative training is worse in this case, as it further accelerates the performance decline. In contrast, mixing data generated by high-performance model and GPT-3.5, both of which have good quality, shows promising results. Regurgitative training with the data mixture matches the performance of training with real data when the same amount of data is used, and in fact outperforms it when twice the amount is used. The performance gain is especially evident when training the low-performance baseline model (right plot of Figure 14). Finally, we note that the data generated by high-performance model and GPT-3.5 have very similar quality (BLEU difference smaller than 0.3%). Therefore, the encouraging results from the high-performance model / GPT-3.5 mixture also lend additional support for our mechanism analyses in Section 4 – errors in generated data is not the only factor affecting regurgitative training performance, and other factors (such as lexical diversity) can be at play.

The abundance of AI-generated content online has prompted academia and industry to develop various methods to distinguish between human- and AI-generated content. For instance, GPTZero is a leading AI detector used to identify whether a document was written by LLMs such as ChatGPT. This inspires us to consider using AI detection tools to mitigate the harm of regurgitative training. In particular, instead of trying to “catch” AI-generated content, we re-purpose AI detection tools to identify AI-generated content that closely resembles human-generated content. Then, we prioritize using AI-generated data that are indistinguishable from human-generated data (from the perspective of the AI detector) in regurgitative training. This mitigation strategy is essentially an “imitation” approach – regardless of why human-generated data are different from AI-generated data (error rates, lexical diversity, or other characteristics), AI-generated data that imitate human-generated data sufficiently well may be more advantageous for regurgitative training.

Starting from the fine-tuning setting, we train an AI detection classifier using randomly sampled 75,000 German-English sentence pairs (not previously used in regurgitative fine-tuning) and their corresponding translations by GPT-3.5, GPT-4, and LLAMA2. For each LLM, we construct a balanced labeled dataset with 150,000 instances, where each of the 75,000 German sentences has exactly one real human-generated translation and one LLM-generated translation. Each data instance is structured as $input=(g_{1},g_{2},\ldots,g_{M},[SEP],e_{1},e_{2},\ldots,e_{N},[SEP]),label\in\{0,1\}$ where $label=0$ marks that the English translation is generated by an LLM and 1 otherwise. Same as how we have trained the BLEU prediction model in the previous section, we retrieve token embeddings from BERT. For each LLM, we train a separate classifier on 80% of labeled data and evaluate it on the remaining 20%. Performance scores of various supervised techniques are listed in Appendix C. The Linear Discriminant Analysis (LDA) turns out to have highest predictive performance for both GPT-3.5 and GPT-4, and Logistic Regression has the best performance for LAMMA2.

Next, we apply the best-performing AI detection classifier for each of the three LLMs on the 5,000 LLM-generated translations used for regurgitative fine-tuning. Because we know the translations are generated by LLMs, if a translation receives a higher class 1 predicted probability from the AI detection classifier, then it has a greater resemblance to real translation. Therefore, we carry out regurgitative training by adding LLM-generated data in the order of their class 1 predicted probabilities, from high to low. Other experiment settings are kept the same as in Section 3.1, and the results are shown in Figure 15.

We can see that regurgitative training in the order of resemblance with real data can indeed mitigate the performance loss for LLAMA2 and GPT-3.5, and the performance gain is larger for LLAMA2. This strategy, however, does not seem to be effective for GPT-4. We suspect such variations in mitigation effectiveness has to do with the capability of AI detection classifier – indeed, AI detection is most accurate for LLAMA2-generated data and least accurate for GPT-4-generated data (see Appendix C).

Next, we apply the same mitigation strategy on transformer models trained from scratch. Performance evaluations of various AI detection classifiers are again listed in Appendix C. We choose a Logistic Regression classifier for the high-performance baseline model and a LDA classifier for the low-performance baseline model. The regurgitative training results, with data ranked by class 1 predicted probabilities, are shown in Figure 16.

The results confirm the effectiveness of AI detection-based mitigation strategy for regurgitative training with transformers trained from scratch. Notably, different from the finding in quality-based mitigation, we observe performance gain for the high-performance baseline model as well (left plot). In fact, regurgitative training with ranked data keeps up with, and even slightly outperforms, training with real data for more than 20 batches.

These encouraging results highlight the promising utility of AI detection outside of its conventional use case. Besides identifying AI-generated content, a capable AI detector can also be re-purposed to guide more meaningful regurgitative training. Meanwhile, we acknowledge that this mitigation strategy clearly does not have unlimited capacity. Under both the fine-tuning setting and with a low-performance transformer model trained from scratch, even the AI detection-enhanced regurgitative training cannot catch up with the performance of training with real data.