Private prediction for large-scale synthetic text generation^††thanks: Authors ordered alphabetically. Author contributions are listed at the end.

Differentially private mechanisms process a source dataset potentially containing sensitive user information and perform a useful computation — as simple as calculating a mean, or as complex as training an ML model — whose output can be safely shared while protecting the privacy of users who contributed to the dataset.

Perhaps the most general-purpose differentially private mechanism is one that produces a synthetic version of its input dataset, as the output of such a mechanism would be suitable for all the same purposes as the original dataset. For example, a private synthetic dataset can be used to train an ML model, but can also be used for auxiliary tasks such as feature engineering, hyperparameter tuning, and quality monitoring.

There has been recent interest in using large-language models (LLMs) to generate differentially private versions of text datasets. Existing approaches can be classified into several categories. Private fine-tuning methods privately adjust the parameters of an LLM on the input dataset, using an algorithm such as differentially private stochastic gradient descent (DP-SGD), and then prompt the LLM to generate similar text. Fine-tuning methods have been used to produce high-quality synthetic data, but the training procedure can be prohibitive, available only to those with the time, compute, and access necessary to train state-of-the-art LLMs containing billions of parameters.

Private prediction methods do not modify the LLM parameters at all. Instead, they directly prompt the LLM with text from the source dataset, asking for similar text in response, and then perturb the LLM’s token distribution (i.e., its last layer) to ensure that each sampled token, and thus the entire generated response, is private. Since no training is required, private prediction methods can quickly generate synthetic data, typically producing some data within minutes instead of hours, which allows for rapid prototyping and iteration. However, unlike private fine-tuning, the guarantees of private prediction methods degrade with the volume of data that is generated. Consequently, existing private prediction methods have mostly been used in applications that require only small amounts of synthetic data [Tang et al., 2024], sharply limiting their practicality.

In this paper we describe a new private prediction method that produces hundreds of times as much synthetic data as a state-of-the-art private prediction method, while maintaining a comparable privacy guarantee. Similar to some existing work, our method is based on running LLM inference on several subsets of the input data in parallel and privately aggregating their token distributions to generate synthetic text. However, our approach is distinguished by three novel algorithmic elements that lead to its improved performance:

1. 1.

   Instead of protecting the privacy of the entire token distribution with Gaussian or Laplace noise, we leverage the uncertainty inherent in sampling to ensure privacy. We clip and aggregate token logits before standard softmax sampling — which is differentially private, since it can viewed as the exponential mechanism. Our approach induces much less distortion of the original token distributions to achieve the same level of privacy than prior work.

2. 2.

   Previous work generated each token using a random subset of the input data, leveraging privacy amplification by subsampling in their analysis. This is computationally undesirable, as it requires repeated re-computation of the prefix for each decoding step, and limits scalability towards generating large synthetic corpora. Instead, we generate each synthetic example using a fixed disjoint subset of the input data, which yields substantial savings in privacy cost (by leveraging parallel composition) while allowing us to pay a linear amount of non-attention FLOPs, rather than quadratic, in terms of sequence length (via KV cache accelerated decoding).

3. 3.

   Our method uses an auxillary token distribution from an LLM without access to sensitive data, and draws the next token from that distribution whenever it is very similar to the token distribution induced by the sensitive data. Our method incurs no privacy cost when outputting “obvious” tokens, and as a result, only a fraction of the tokens in the synthetic data are generated using sensitive data (as little as 20% in structured datasets). We leverage the sparse vector technique to privately calculate distributional similarity.

Taken together, the combination of these algorithmic techniques leads to significant improvements over prior work. Informally, (1) and (2) above keep our inference closely aligned to standard (non-DP) inference.

In our experiments, we generate private synthetic versions of publicly available, benchmark machine learning datasets, and then use the synthetic datasets for downstream classification and extraction tasks. Owing to the increased quantity and quality of our synthetic data, we improve over an existing state-of-the-art private prediction method in terms of downstream accuracy. Furthermore, while prior work in this paradigm only generated a small (<10) number of examples we demonstrate the ability to generate thousands of training examples, enough for fine-tuning downstream models.

Finally, since synthetic data is intended for a wide variety of applications, we also evaluate data quality using a metric that is orthogonal to performance on a downstream classification task. Specifically, we generate synthetic versions of a publicly available dataset containing highly structured data records, each of which is encoded as a JSON object. Our results demonstrate that the sparse vector technique helps preserve data structure at high privacy levels.

Private fine-tuning is widely used for synthetic text generation. Yue et al. [2023] created private synthetic versions of text datasets by using DP-SGD [Abadi et al., 2016] to fine-tune an LLM on the sensitive data. Kurakin et al. [2024] showed that parameter efficient approaches to fine-tuning, such as LoRA [Hu et al., 2022], can improve the quality of the synthetic data, since reducing the number of parameters also reduces the amount of noise injected into the optimization procedure. Wu et al. [2024a] took a two-stage approach: First they fine-tuned an LLM on a public dataset that closely resembled the sensitive data (which was itself generated by an LLM using carefully designed prompts), and then completed the fine-tuning process by running DP-SGD on the sensitive dataset. Concurrent to this work, Tran and Xiong [2024] describe a private fine-tuning approach for generating synthetic tabular data that is formatting compliant.

Private prediction [Dwork and Feldman, 2018] is an alternate approach to private machine learning that only guarantees the privacy of the predictions output by an ML model, and not the model itself. Private prediction has been applied to synthetic text generation by viewing each token sampled by an LLM as a ‘prediction’, and perturbing the LLM’s token distributions to ensure their privacy. Tang et al. [2024] added noise to several independent token distributions and averaged them, while Hong et al. [2024] selected the most popular token among the token distributions using the LimitedDomain mechanism [Durfee and Rogers, 2019]. These methods can avoid the time, compute, and access required to fine-tune an LLM with billions of parameters. However, a privacy loss is suffered for each token produced in this manner. As a result, previous work has only been able to generate a very small number of synthetic examples at reasonable privacy levels (fewer than 10). Other work has applied private prediction techniques to LLMs [Majmudar et al., 2022, Duan et al., 2023], including in combination with fine-tuning [Ginart et al., 2022, Flemings et al., 2024], but not for the purpose of synthetic text generation.

Finally, another distinct set of approaches are private filtering methods. Private filtering methods operate directly on whole LLM responses and a large corpus of public data that does not require protection. Yu et al. [2024] and Xie et al. [2024] used the sensitive responses to privately select similar responses from the public dataset. Similarly, Wu et al. [2024b] aggregate response embeddings and select the public response that is closest in embedding space.¹¹1Wu et al. [2024b] also proposed a non-filtering approach based on privately selecting common keywords among the sensitive data and using them to prompt an LLM. One limitation of filtering methods is that the menu of possible responses is constructed without signal from the new source dataset.

Before describing our algorithm for generating private synthetic text, we review the standard algorithm for LLM inference. Let $\mathcal{X}$ be the token vocabulary (i.e., the set of all possible tokens), and let $v=|\mathcal{X}|$ be the vocabulary size. A token sequence is an element of $\mathcal{X}^{*}$, and a logit vector is an element of $\mathbb{R}^{v}$ (one logit per token in the vocabulary). If $\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ are token sequences then we write $\mathbf{x}_{1}\mathbf{x}_{2}\in\mathcal{X}^{*}$ to denote their concatenation.

In this section we prove that Algorithm 1 preserves the privacy of the sensitive prompts it uses to generate synthetic examples.

Let $\mathcal{D}$ be the set of all possible prompt datasets. A mechanism is a randomized function with domain $\mathcal{D}$. Note that Algorithm 1 is a mechanism. We say that a pair of prompt datasets $D,D^{\prime}\in\mathcal{D}$ are neighbors if there exists a prompt $\mathbf{p}$ such that $D=D^{\prime}\cup\{\mathbf{p}\}$ oder $D^{\prime}=D\cup\{\mathbf{p}\}$. In the differential privacy literature this is commonly referred to as the add/remove neighbor relation.

Theorem 1 below provides a differential privacy guarantee for Algorithm 1. The proof of Theorem 1 requires a technical assumption about how the prompts are partitioned into batches in the first step of the algorithm.

The most straightforward way to satisfy Assumption 1 is to apply a hash function to each prompt and then use the hash value to determine its assigned batch. For example, if $h$ is the hash value, $n$ is the number of prompts and $s$ is the expected batch size, then we can assign the prompt to the $(h\mod\frac{n}{s})$th batch. If we want to batch together prompts that share a certain attribute (like a label), we can apply another hash function to that attribute and concatenate the hash values. Using hash functions for batch assignment can lead to batches whose sizes differ from the expected batch size $s$, but this does not impact the validity of Theorem 1.

The proof is in Appendix C and makes use of sharp privacy analyses of: (1) zCDP to approximate DP conversion [Canonne et al., 2020]; and (2) zCDP bounds for the exponential mechanism [Cesar and Rogers, 2021].

Gemma 1.1 2B IT [Team, 2024] is the data generator in all private prediction experiments. We choose it due to its lightweight, open-source JAX implementation that makes easy to implement and share sampling algorithms.²²2https://github.com/google-deepmind/gemma Tables 1(a) and 1(b) give an overview of datasets and models used.

We perform 3 sets of experiments, targeting various datasets and utility criteria:

• •

  In-context learning (§5.1); We generate examples to use as in-context exemplars for prompting an LLM. We report downstream accuracy on real test examples, when prompted with synthetic data, on 3 classification tasks (AGNews [Zhang et al., 2015], DBPedia [Zhang et al., 2015], TREC [Voorhees and Tice, 2000]) and 2 extraction tasks (MIT-G, MIT-D [Liu et al., 2012]).

• •

  Fine-tuning (§5.2); We generate synthetic examples to use for fine-tuning a BERT classifier. We report downstream accuracy on real test examples for 3 classification tasks (IMDB [Maas et al., 2011], Yelp [Zhang et al., 2015], AGNews [Zhang et al., 2015]).

• •

  Structured data (§5.3); We generate examples that must adhere to structural constraints to be useful synthetic data. We consider a JSON generation task (WikiMoviesJSON [Rust, 2024]), evaluating structure preservation.

We believe that our significantly improved performance relative to Tang et al. [2024] is primarily attributable to two algorithmic innovations.

First, for each generated token, Tang et al. [2024] preserve the privacy of the entire distributions from which the token is sampled (by taking argmax), even though only the token itself is included in the synthetic data. By contrast, our method uses a discrete choosing mechanism, the exponential mechanism. As a result, we do not need to maintain a DP version of the entire token distribution to release a single token. This decision leads to significantly lower noise requirements, as a straightforward calculation reveals. Empirically, we obtained good synthetic data quality with $s=250$, $\tau=2$, $c=10$ and $\delta=10^{-6}$. In order to switch to the Gaussian mechanism using its standard $(\varepsilon,\delta)$-DP guarantee, and achieve comparable privacy guarantees we would would require $\sigma\approx 0.53$ to achieve a comparable privacy guarantee. (See Appendix D). Better analyses of the Gaussian mechanism exist, but do not offer much help. Using the improved analysis in Balle and Wang [2018] to attain the same $\varepsilon$ would require $\sigma\approx 0.34$. Conducting the analysis so that both mechanisms have equivalent privacy loss under zCDP yields $\sigma=0.2$. These are all very large noise magnitudes relative to probabilities in $[0,1]$.⁵⁵5To put independent noise of magnitude $\sigma=0.2$ into perspective: suppose the ground truth next-token prediction is deterministic, i.e., $\bar{\mathbf{p}}=[1,0,...,0]\in\mathbb{R}^{v}$, $v$ = 256128 in the case of Gemma. Now with probability $\geq 0.15$, the noised distribution ${\widetilde{\mathbf{p}}}$ has ${\widetilde{\mathbf{p}}}_{1}<0.8$. Each other $\mathbf{p}_{i}$ is $\geq 0.8$ w.p. $\geq 3\cdot 10^{-5}$ independently. Hence the probability of one of these being promoted to argmax is $\geq 0.15\cdot(1-(1-3\cdot 10^{-5})^{v-1})\approx 0.15$. At this rate, the chance of generating a 30 token span without a corruption is $<1\%$.

Secondly, Tang et al. [2024] generated each token using a different random sample of the sensitive prompts, which is computationally very expensive, as it prevents the use of KV cache-accelerated decoding, since the cache is invalidated upon every resample. While resampling less often would be more practical, Tang et al. [2024] noted that in this case the privacy amplification benefits of subsampling would not be adequately realized, and characterized this limitation as the “main weakness” of their approach. Instead, our method generates each synthetic example using a fixed disjoint subset of the sensitive prompts, allowing us to leverage parallel composition in our analysis, and thus avoid this privacy versus computation tradeoff.

As proprietary models become increasingly powerful, we anticipate more practitioners will be able to generate inferences from state-of-the-art models, while fewer practitioners will be able to train networks that perform like state-of-the-art models. This makes it increasingly important to develop private prediction methods that compete with private fine-tuning.

We demonstrate that private prediction can be used to generate large amounts of synthetic text with reasonable differential privacy guarantees. We produce 2-3 orders of magnitude more private synthetic data than what was demonstrated in prior work in this paradigm. Access to more synthetic data lets us fine-tune downstream models, as well as yields performance improvements via many-shot in-context learning. Furthermore, we introduce a novel use of public models in which we are able to sample predictable tokens at no privacy cost, which is particularly effective for structured data.

While our work demonstrates that private prediction is a practical technique for privately generating a large volume of high-quality synthetic data, there remains a small gap between our results and the results obtained from privately fine-tuning the parameters of the LLM. Currently, private prediction methods pay a privacy cost for every generated token, while private fine-tuning methods do not. We view correcting this limitation as a very important open problem. Finally, any method for ensuring data privacy will inevitably entail some loss of data utility.

• •

  Alex B is the main contributor. He implemented the method, tested variants to optimize utility and privacy, and ran most of the experiments. He also proposed the use of sparse vector.

• •

  Umar proposed the method, the use of sampling to preserve privacy, and conducted the theoretical analysis.

• •

  Umar and Kareem framed the structure of the paper and led writing.

• •

  Kareem proposed parallel composition. He also assisted with the privacy analysis.

• •

  Natalia proposed logits recentering.

• •

  Weiwei and Alexey provided infrastructure support and code for running experiments. Alexey suggested the limited data experiments and ran the fine-tuning baselines.

• •

  Natalia, Andreas, and Sergei advised the project.

• •

  Everyone contributed to discussing, interpreting, and iterating on experiment results as well as project management.