Robust Utility-Preserving Text Anonymization Based on
Large Language Models

The text anonymization challenge can be recast as a multi-objective optimization problem with two conflicting objectives: privacy and utility. In this context, privacy should be prioritized over utility. This hierarchy is established by ordering the objectives, transforming the problem into a lexicographic optimization issue Zykina (2004). The primary objective is to maximize the level of privacy preservation, ensuring that sensitive information is well-protected against re-identification risks. The secondary objective is to maintain as much useful information as possible in the anonymized text for analytical tasks. This lexicographic optimization problem can be formally expressed as

where $f_{p}(\cdot)$ and $f_{u}(\cdot)$ denote the privacy and utility objective function, respectively. $\mathcal{X}_{0}$ denotes the set of all possible edits of $\boldsymbol{x}_{0}$. A solution $\boldsymbol{x}_{a}\in\mathcal{X}_{0}$ is lexicographically preferable to another solution $\boldsymbol{x}_{b}\in\mathcal{X}_{0}$, denoted as $\boldsymbol{x}_{a}\succ_{\text{lex}}\boldsymbol{x}_{b}$, if and only if $f_{p}(\boldsymbol{x}_{a})>f_{p}(\boldsymbol{x}_{b})$ oder $(f_{p}(\boldsymbol{x}_{a})=f_{p}(\boldsymbol{x}_{b})\text{ and }f_{u}(\boldsymbol{x}_{a})>f_{u}(\boldsymbol{x}_{b}))$. To solve this lexicographic optimization problem, we propose RUPTA, an iterative method with LLMs to generate, evaluate, and optimize the anonymized text.

The role of the Privacy Evaluator (P-Evaluator) is to assess the privacy protection level of the anonymized text, ensuring that private content is adequately obscured against re-identification. Besides, it is essential to provide textual feedback to the LLM optimizer as guidance Pryzant et al. (2023). Thus, the privacy objective evaluation process $f_{p}(\cdot)$ is formally defined as

where $p_{t}$ denotes the value of the privacy objective and $\boldsymbol{f}_{t}$ denotes the textual feedback. We describe the detailed process of privacy evaluation in algorithm 1.

P-Evaluator is instantiated as an LLM. Given the anonymized text $\boldsymbol{x}_{t}$, we concatenate it with the privacy inference instruction $\mathbf{I}_{p}$ as input to prompt the P-Evaluator to semantically infer the personal information as shown in line 1 of algorithm 1, where $||$ denotes concatenation. This step generates top-$K$ inference results $[y^{\prime}_{i}]_{1}^{K}$ for the personal information. Each result is then compared with the ground-truth personal information $y$. If a match is found within these top-$K$ results, its rank is used as the scalar privacy score $p_{t}$. Further, the evaluator is prompted to provide natural language feedback $\boldsymbol{f}_{t}$ detailing the clues that led to the correct inference. Otherwise, we set the $p_{t}$ as $K+1$, representing the maximum achievable score for the privacy objective.

The scalar score $p_{t}$ quantifies the privacy risk associated with the anonymized text, while the textual feedback $\boldsymbol{f}_{t}$ offers qualitative insights, guiding the lexicographic optimizer on how to better obscure identifiable information. The value of $K$ serves as a customizable parameter that adjusts the sensitivity of the privacy evaluation, with higher values indicating a more inclusive search for potential privacy breaches, thus facilitating a manually adjustable trade-off between privacy and utility.

The Utility Evaluator (U-Evaluator) is used to ensure that the anonymized text retains its utility for specific analytical tasks, a critical consideration for practical applications across various domains. It analyzes the anonymized text $\boldsymbol{x}_{t}$, specifically assessing its effectiveness in supporting accurate occupation classification $c$. The formal utility objective evaluation process is defined as

where $u_{t}$ is the utility objective value.

In this paper, we instantiate the U-evaluator with an LLM. Given the anonymized text $\boldsymbol{x}_{t}$ and the corresponding ground-truth occupation label $c$, the LLM-based U-evaluator follows the instruction $\mathbf{I}_{u}$ to output a confidence score $u_{t}$

this confidence score quantifies the evaluator’s uncertainty that $\boldsymbol{x}_{t}$ can be correctly classified into the ground truth occupation category $c_{t}$, reflecting the degree to which key utility information is preserved.

To better align feedback with real-world use scenarios, the U-Evaluator can be instantiated with the actual model employed in the downstream task. For example, if the anonymized text is intended for sentiment analysis, the U-Evaluator can be instantiated with a sentiment analysis model. The utility score $u_{t}$ can then be calculated through the logit of the ground-truth label following the traditional uncertainty quantification method Sensoy et al. (2021).

Lexicographic optimization (LO) is a special case of MOPs where multiple conflicting objectives are to be maximized simultaneously. In LO, objectives are ranked in order of importance, enabling prioritization of the most critical objectives. The LO problem is generally solved by the sequential optimization method Zykina (2004); Zhang et al. (2022). Specifically, regarding the text anonymization problem, privacy and utility are the two objectives, and privacy should be prioritized.

RUPTA employs the LLM as a black-box lexicographic optimizer in a zero-shot manner, where the LLM is prompted to achieve better solutions incrementally based on the history of optimization results and objective evaluations. The overall prompt consists of the pre-defined overall optimization description prompt $\mathbf{I}_{r}$, the memory module $\mathcal{M}$, the meta instruction variable $\boldsymbol{I}_{me}$ and the textual feedback $\boldsymbol{f}_{t}$ from P-Evaluator. The memory module $\mathcal{M}$ stores history optimization results and their corresponding privacy and utility objective values. Formally, we have $\mathcal{M}=\{(\boldsymbol{x}_{i},p_{i},u_{i},r_{i})|i=1,2,...,t\}$.

To ensure that the primary goal of achieving maximum privacy is prioritized and only after the privacy objective is satisfactorily met does the optimizer focus on improving utility, the lexicographic-optimizer LLM operates in two different modes. When the privacy objective value has not yet reached the pre-set maximum $K+1$, the lexicographic optimizer should focus on maximizing the privacy objective, which is achieved by taking the value of meta instruction variable as $\mathbf{I}_{pr}$ that instructs the LLM to further anonymize $\boldsymbol{x}_{t}$ according to the textual feedback $\boldsymbol{f}_{t}$. The process can be formulated as

Once the privacy objective value has reached the maximum threshold, the meta instruction shifts to $\mathbf{I}_{ur}$, prompting the LLM to optimize the utility level without compromising the achieved privacy objective value.

This iterative process continues until either the pre-defined maximum values for both objectives are reached or the maximum number of iterations $T$ is met. By continuously refining and evaluating the anonymized text, the optimizer iteratively improves it to achieve an optimal balance between privacy and utility.

Utilizing LLMs for text anonymization is computationally expensive, and for certain LLMs, access is only available through APIs, which raises privacy and cost concerns. However, in our framework, the optimization result heavily depends on the reasoning ability of the LLM, which stems from the large scale of parameters these models possess. Recent studies have demonstrated that prompting LLMs as optimizers is less effective with smaller-scale models Zhang et al. (2024).

To address this issue, we employ knowledge distillation (KD), where a large model (the teacher) transfers its knowledge to a smaller model (the student). Typically, KD involves training the student model using the outputs of the teacher model as labels Kim and Rush (2016). In our case, we utilize the final anonymization result produced by the teacher model during the lexicographic optimization as the training label for the student model.

To utilize the generation results of the teacher model more efficiently, we adopt the Direct Preference Optimization (DPO) Rafailov et al. (2023) method. This method fine-tunes an LLM on human labels of the relative quality of model generations to align the model with human preferences. In our method, intermediate optimization results from the teacher model can be considered less preferred than the final optimization result. These intermediate and final results form the preference dataset. We fine-tune the student model using DPO on this dataset to preferentially generate outputs similar to the final optimization result while reducing the likelihood of producing results akin to the intermediate stages.

The overall experimental results on the DB-bio dataset are presented in table 2. In the disclosure risk evaluation, methods that anonymize the data in an iterative refinement manner, including our RUPTA method and the AF method, achieve the best performance. Although DEID-GPT and SD also leverage LLMs, they follow a traditional approach focusing on masking entities of pre-defined types. Experiment results demonstrate that such methods cannot adequately defend against re-identification attacks from LLMs. Additionally, using open-source LLMs as the lexicographic optimizer also achieves comparable privacy-preserving performance, demonstrating the practicality and generality of our method.

For the utility preserving evaluation, traditional methods like Azure mask all the entities of pre-defined kinds with “*”, leading to the most significant information loss, thus achieving the lowest performance. The DEID-GPT, SD, and AF methods, although anonymized through replacing sensitive entities with more general ones, do not consider the downstream analysis task and generalize all the possible sensitive entities, which also significantly undermines downstream task performance. Visualization results of the optimization process in fig. 3 highlight the drawback of the AF method, where the SR and classification accuracy decrease simultaneously as the number of optimization steps increases. In contrast, our method achieves the best downstream task performance. Furthmore, during the optimization process of RUPTA, there is an explicit increasing phase of the classification accuracy, demonstrating the effectiveness of the RUPTA method to maximize both the privacy and utility in the anonymization process. This trend also illustrates that beyond a certain point, further anonymization yields diminishing returns in privacy preservation and results in greater losses of utility information.

The experiment results for the customizable privacy-utility tradeoff are displayed in fig. 4. In our method, the maximum value of the privacy objective is set manually according to specific requirements, allowing for a customizable privacy-utility tradeoff. We analyze and visualize the average SR and classification accuracy of our method using GPT-4, Llama-3-70b, and Mixtral $8\times 22$b as the lexicographic optimizer. We set the maximum privacy value to 1, 5, 10, 15, and 20, respectively. It is evident in fig. 4 that our proposed method can effectively adapt the privacy-preserving level according to the maximum value setting. As the maximum privacy value increases, the average privacy score improves while the utility score adjusts accordingly. This observation demonstrates the flexibility of our approach in balancing privacy and utility based on user-defined requirements.

To evaluate the generality of our method, we further conduct experiments on the PR dataset with results presented in table 3. The PR dataset is characterized by fewer explicit and more implicit sensitive entities. Entity recognition-based methods, including Azure, DEID-GPT, and SD, struggle to detect these implicit entities, resulting in minimal masking operations, as evidenced by their evaluation results closely mirroring those of the original dataset. Consequently, while these methods exhibit higher performance on the downstream task, they provide inferior privacy protection. Only the AF and our method can properly detect implicit sensitive information and achieve the lowest disclosure risk. However, the AF method anonymizes without tailoring its approach to the specific downstream task, which significantly impairs task performance. In contrast, our method not only effectively minimizes disclosure risk but also preserves a greater degree of utility in the anonymized text than AF, achieving a better privacy-utility tradeoff.

In this experiment, we try to distill the anonymization ability of GPT-4 into lightweight models. Using RUPTA with GPT-4 as the lexicographic optimizer, we anonymized the training and validation sets of the DB-bio dataset. Initially, we fine-tuned student models in a supervised manner (SFT) using the final optimization results as labels. Then we constructed a preference dataset from the optimization trajectories and conducted DPO fine-tuning on the optimal checkpoint during the SFT phase. The evaluation results are presented in fig. 5.

From the disclosure risk evaluation, we observe that the primarily supervised fine-tuning on the final optimization results enables the smaller models to achieve performance comparable to the teacher model, GPT-4. Additionally, the DPO fine-tuning process further enhances the performance of the student models, narrowing the gap to the teacher model’s capabilities.

In the utility preserving evaluation results, in addition to the classification accuracy, we further demonstrate the semantic similarity between the anonymized and original text. The supervised fine-tuned student models maintain a high level of downstream task performance. Although the DPO fine-tuning process improves the privacy-preserving performance, it somewhat harms the downstream task performance. This phenomenon likely results from the unbalanced optimization phases in the lexicographic optimization process, where achieving the maximum privacy objective requires more iterations than improving downstream task performance, as shown in fig. 3. Consequently, the student models, fine-tuned with DPO, prioritize privacy to a greater extent, potentially at the expense of utility. Anonymization examples are shown in fig. 6. We can see that the student model can learn to generalize or remove sensitive entities after the SFT phase. After the DPO fine-tuning phase, the student model can further generalize sensitive entities marked by underlining, e.g., from “father” to “family member”. Both models can keep the relevant information about the downstream task in the anonymized text, as highlighted in the figure.