Building Intelligence Identification System via Large Language Model Watermarking: A Survey and Beyond

For training data providers, the infinite replicability of data poses an uncontrollable risk of data breaches as it circulates. Currently, the most effective way to prevent the unauthorized dissemination of data is to secretly add a unique, strong watermark [26] to the data without altering the quality of the dataset itself [27]. This ensures that the data can still be verified for its initial copyright even after being redistributed, modified, or otherwise processed. Some researchers have proposed even more in-depth solutions, demonstrating that watermarked data used for LLM training can have its watermark information detected in the text generated by the LLM, which is referred to as radioactivity [28]. Once this knowledge is embedded into unauthorized LLMs, data providers can identify whether their watermark is present in the models based on the response to certain specific watermark triggers. Moreover, since a training dataset is likely to be copied and sold multiple times, the most crucial aspect of protecting copyright and preventing data leaks or unauthorized distribution is identifying the source of the leakage. To address this, encoding a unique watermark message for each dataset to be distributed and embedding it into the data with a covert watermark is an essential option.

Therefore, training data providers need to focus on watermarking techniques that offer high fidelity and transparency, meaning the watermarking should be done in a way that does not degrade the quality of the text. Moreover, the watermark embedded by the data providers should have the capability of multi-bit information encoding to enable the identification of the data purchasers. Besides, the watermark should possess high robustness and radioactivity, allowing for the detection of watermarks in text content generated by unauthorized models if the data are used for illegal training.

For technology service providers, watermarking technology helps protect model copyrights and monitor the usage of models. To address the costs and technical difficulties faced during the pre-training of LLMs, some technology service providers might opt to use data generated by well-trained models for training, which has formed a system similar to teacher-student model distillation. This imitation has sparked concerns over the copyright of unauthorized distilled models, especially when the corpus data of these distilled models come from closed-source LLMs (such as GPT-4). In the constantly evolving landscape of AI copyright protection, it is crucial to emphasize the importance of protecting intellectual property while maintaining the integrity and practicality of AI models. The development and implementation of watermarking technology enables model developers to protect their innovations from unauthorized use and distribution effectively.

To protect the intellectual property of models and prevent the unauthorized use of developed LLMs through distillation by offenders, technology service providers should embed watermarks only when the model is invoked by users, without affecting the model’s own training process. This approach meets the technology service providers’ pursuit of model performance. Additionally, it substantiates the model’s ownership and facilitates the tracking of its distribution and usage [29]. This helps prevent the model from being copied or tampered with by unauthorized third parties.

With the rapid advancement of AI technology, especially the widespread use of LLMs in content creation, the roles of public regulators and trusted third parties (PRTTPs) in watermarking systems have become critical. Policymakers and civil society are increasingly focused on the safe use of these technologies, as shown by the EU AI Act, the NDAA for Fiscal Year 2024 [30], and voluntary commitments to label AI-generated content. These initiatives emphasize the need for transparency in content sources, including clear marking of watermarks and content origins, as well as guidelines for content certification and watermarking developed by the U.S. Department of Commerce following the AI Executive Order of October 30, 2023 [31]. These guidelines aim to help the public easily identify the authenticity of online information.

LLM watermarking technology is considered key to ensuring the safety, compliance, and ethical integrity of AIGC throughout its entire lifecycle. To this end, PRTTPs should make the following efforts:

1. 1.

   Establishing clear watermarking technical standards and usage norms, setting up certification programs for watermark service providers, and promoting success stories and best practices.

2. 2.

   The implementation of education and training programs, especially aimed at enhancing the understanding of watermarking technology among all participants, further reinforces this process. This involves not only educating parties on how to select and utilize watermark services correctly but also emphasizing the importance of adopting these measures to ensure that all involved can effectively use watermarking technology to identify AIGC.

3. 3.

   Establishing strict oversight and enforcement mechanisms is equally important, ensuring that all parties rigorously adhere to the regulations and standards for watermark usage, thereby guaranteeing the correct and secure application of watermarking technology.

4. 4.

   Providing the general public with access to open watermark interfaces for LLMs allows users to add watermarks to their own data or to verify through watermarking whether a piece of data contains their private information. This approach helps track the usage and flow of data, preventing unauthorized dissemination of the data across the internet.

These requirements are not isolated but necessitate multi-party collaboration among data providers, model technology service providers, watermark technology service providers, and PRTTPs. Through a cooperation framework that spans different sectors and industries, we can facilitate information sharing and technological advancement. These measures encourage deep reflection on the safe use of AI technology and lay a solid foundation for maintaining public trust in AIGC. By establishing industry standards, implementing rigorous certification processes and audits, providing comprehensive education and training, enforcing vigilant supervision and execution, promoting best practices, and fostering collaborative multi-party partnerships, we can ensure the safe and responsible development of LLMs and other AI technologies.

Watermarking technology can help LLM users in verifying the copyright and legitimacy of the models, while also serving as a tool to protect the security of user data and privacy. Users should opt for LLM services verified by PRTTPs. The model providers usually require such verified service providers to apply data watermarking technology to ensure that the input data is used only for the current service and is not accessed or misused by third parties. LLM users can ensure that watermarks are embedded in their prompts by choosing services with publicly verifiable watermarking technology. Users can verify the use and flow of their data through a public watermark verification interface, preventing unauthorized distribution of their data on the internet. Watermarking technology plays a crucial role in protecting personal privacy and can also alleviate users’ privacy and security concerns on another level, thereby promoting the further popularization of LLM technology.

Moreover, LLM users will gradually transition from mere users to becoming a more deeply involved and crucial part of the LLM application ecosystem. Firstly, users can trade their private data through some form of anonymization, collaborating with data providers to co-create datasets. This not only generates profit but also enhances the overall efficiency of the LLM application system. Secondly, users will engage in in-depth cooperation with PRTTPs. If the model produces unsafe answers or if copyright infringement is detected, users can report these issues to PRTTPs by clicking the report button. PRTTPs can thus centralize the originally dispersed and unequal user oversight power through this method, better standardizing the development of LLM technology.

Watermarks generated through vocabulary partitioning usually utilize a pseudo-random function (implemented as a hash function) to generate random seeds. These seeds are used to divide the vocabulary into distinct lists, ensuring that a subset of tokens from a particular list is output more frequently during token generation. The watermark is generated by biasing the selection of tokens towards specific lists.

Kirchenbauer et al. [32] introduced the first LLM watermarking technique that generates watermarks through vocabulary partitioning, as shown in Fig. 4. This method employs a hash function that uses the preceding token as input to compute a random seed for partitioning the vocabulary into green and red lists. The LLM-generated tokens are biased towards the green list by adding a bias term to the logits. During detection, the extraction process calculates the ratio of green tokens with the z-metric to determine the presence of the watermark.

Following Kirchenbauer et al. [32], the idea of watermarking generation by vocabulary partitioning has been widely explored by many researchers [33, 34, 35, 36, 37, 38, 39, 40]. These methods have introduced more refined methods for partitioning the red-green lists, resulting in a greater diversity of partitioning techniques. For example, Takezawa et al. [34] introduced tighter constraints for partitioning red-green tables,enhancing the concealment of the watermark and improving the quality and naturalness of the generated text. Building upon previous work, Kirchenbauer further elaborated on the robustness of these watermarking generation methods against paraphrasing attacks [33]. The effectiveness of the detection method in this study was evaluated by comparing them with detectors designed to identify AI-generated text.

While these watermarks can be integrated with various detection techniques, their distribution does not meet the unbiased criterion. To address this, Hu et al. [35] introduce an unbiased watermark by adjusting the token generation probability distribution through the watermark code space. Some studies focus on enhancing the robustness of such watermarks against attacks, aiming to mitigate vulnerabilities arising from reliance on lexical distribution. For instance, Li et al. [41] employed a novel reweighting strategy combined with a context-based hash function to assign a unique i.i.d. ciphers to each generated token. This encoding method ensures the preservation of the original token distribution during the watermarking process, making the watermarked text indistinguishable from unwatermarked text in terms of distribution. Other studies [38, 40] have considered semantic similarity when partitioning the vocabulary, such that the semantic value may remain unchanged even in the face of watermark attacks, thereby achieving robustness against paraphrasing attack.

Watermark generation methods based on vocabulary partitioning rely on hashing tokens from the previous moment, leading to inefficiency during the extraction phase, as it necessitates iterative computation over all tokens. To address this, Zhao et al. [36] simplified the watermarking generation method proposed by Kirchenbauer et al. by employing a fixed increase in logits watermark strength $\epsilon$, making the vocabulary partitioning independent of previously generated tokens and solely reliant on a global key. Fernandez et al. [42] proposed a method to enhance efficiency by cyclically shifting an initial message to generate secret vectors for each message, thereby easily converting one-bit watermarks into multi-bit watermarks and allowing for parallel processing. Additionally, Liu et al. [43] employed a watermark generation network to partition the vocabulary instead of using hash functions, which has also been proven effective.

Some studies have extended the method of partitioning the vocabulary into multi-bit watermarks to convey more information through the watermark. However, these methods also face higher computational complexity and increased demands for watermark information density. To develop a more effective multi-bit watermark techniques, Wang et al. [44] proposed the Balance-Marking method, which uses a proxy language model (proxy-LM) to ensure that the available and unavailable vocabulary for generating watermarks have approximately equivalent probabilities. Similar to the work of Lee et al. [39], this method can also circumvent low-entropy parts of the text to effectively improve text quality.

However, this watermarking approach essentially divides the vocabulary into multiple sets of red-green tables, with each set corresponding to one bit of watermark message. Such methods requires iterative computation during the logits generation process for each token, resulting in extremely high computational complexity. To mitigate this, besides the cyclic shift method by Fernandez et al. [42], Yoo et al. [45] independently encode each message bit position, transforming the division of the vocabulary from red-green lists into colored lists, effectively encoding multiple states for every token. Allocating tokens to different parts of the message allows for embedding longer messages without increasing generation latency. Compared to methods that directly generate watermarks using the watermark message $m$, Qu et al. [46] have enhanced the robustness of the watermark by introducing error-correcting codes (ECC) to the watermark information before dividing the vocabulary.

The key limitation of the existing multi-bit watermark approaches [42, 44] is that the computational cost of their extraction functions grows exponentially with the length of the watermark message bits, and they cannot accurately or effectively extract all watermark bits.

The methods of vocabulary partitioning involve mapping the text sequence $S^{N}$ to various distributions of vocabulary that can be analyzed for their $Attack$ $Robustness$ and $Sicherheit$ $Robustness$ using Formula 4. All these methods incorporate semantics, global secret keys, and additional information to further solidify the dependency between the generated watermark $W$ and the original sequence $S^{N}$, which increases $I(S^{N};W)$. The original KGW [32] partitions the vocabulary using the hash value of previous tokens. The hash function ensures the mutual information $I(W;m)\to 0$ between the watermark $W$ and the watermark message $m$, making this method a high level of $Sicherheit$ $Robustness$. Denote that the mutual information $I(m;W)=H(m)-H(m\mid W)$, where $H(m)$ is the entropy of the message $m$ and $H(m\mid W)$ is the conditional entropy of $m$ given the watermark $W$. Since the computation process of hash functions is unidirectional and irreversible, the conditional entropy $H(m\mid W)$ encloses to $H(m)$ when the output of the hash function $W$ is known. Therefore, the mutual information $I(m;W)\to 0$ is minimized to prevent an attacker from back-propagating the message $m$ through the watermarked signal $W$.

The work [33] uses a context-robust Min-Left Hash to strengthen the connection between $W$ and the original sequence $S^{N}$, thereby increasing $I(S^{N};W)$. Zhao et al. [36] no longer use a hash function for vocabulary partitioning but instead base it on text edit distance to partition a fixed vocabulary. Although the fixed vocabulary has a higher $I(W;m)$ than the hash-partitioned vocabulary, its $Sicherheit$ $Robustness$ is reduced. However, by increasing the edit distance between vocabularies, attackers need multiple attempts to bridge the text distance and invalidate the watermark, thus enhancing $Attack$ $Robustness$.

Additionally, for low-entropy texts mentioned in paper [32], which are difficult to watermark by modifying logits, this can also be analyzed using Formula 4. When the original sequence $S^{N}$ has low entropy, the first term of the optimization goal, $I(S^{N};W)$, is low, which negatively impacts watermark generation and reduces its $Attack$ $Robustness$. Consequently, some studies [39, 46] propose bypassing low-entropy texts and only watermark high-entropy texts to ensure the watermark’s $Attack$ $Robustness$. Furthermore, semantic-based watermarking methods determine vocabulary partitioning by incorporating contextual relationships and semantic information. The introduction of semantic information enhances the correlation between the generated watermark $W$ and the original sequence $S^{N}$. The increase in $I(S^{N};W)$ enhances the robustness of such methods against watermark attacks. For instance, Fu et al. [38] selected semantically related vocabulary to add to the watermark vocabulary. Ren et al.’s SemaMark [40] mitigated semantic sensitivity by discretizing the continuous word embedding space appropriately, ensuring that discrete semantic values remain unchanged even in the face of watermark text editing attacks.

Model-based learning methods employ deep learning techniques, such as GPT [47], BERT [48], to generate watermarks. These methods leverage the learning and generative capabilities of deep learning models, using a trained watermark generation model to directly produce watermarks $W$ or embedded representations of watermarked sequences.

In contrast to other methods, this technique create the watermark that is intricately embedded into the content, enhancing security and robustness against tampering. Correspondingly, watermark extraction is typically performed using a dedicated decoder or a watermark detection network. This dual-model framework ensures that the embedded watermarks can be accurately retrieved, even when the content has undergone modifications or compression. By maintaining the watermark generation model as proprietary while making the watermark detection model publicly accessible, a publicly verifiable watermarking scheme can be effortlessly implemented. Conversely, by keeping both the watermark generation and detection models confidential, the watermarking method can be transformed into a private watermarking system.

This strategic dichotomy allows for flexibility in controlling the accessibility and verification of watermarks, catering to different security and privacy requirements. Publicly verifiable watermarks facilitate widespread verification, enhancing transparency and trust in digital content authenticity. In contrast, private watermarking schemes offer enhanced security since the ability to generate and detect watermarks is restricted to authorized entities. This restriction safeguards proprietary or sensitive information from unauthorized detection and manipulation.

Kuditipudi et al. [49] employed a decoder that deterministically maps a sequence of random numbers, encoded by a watermark key, to samples in a language model. This is achieved by converting a sequence of uniform random variables and permutations into tokens using inverse transform sampling. Considering that many existing watermarking algorithms are designed at the token level, Hou et al. [50] utilized a sentence encoder trained through contrastive learning (such as Sentence-BERT [51]) to capture textual semantic similarities. They partitioned the semantic space of sentences and employed sentence-level rejection sampling to ensure that sentences fall within watermarked partitions of this space. This approach to semantic watermarking at the sentence level shows strong robustness against paraphrasing attacks. Munyer et al. [52] utilized Word2Vec [53] and Sentence Encoding [54] to engender a roster of replacement words, which we consider as the generated watermark.

The mainstream method of embedding watermarks is to add extra watermark logits on top of the logits generated by the LLM. Some intriguing research [43, 55], inspired by the red-green list method [32], moves away from guiding logit modifications by partitioning the vocabulary. Instead, these studies directly generate watermark logits through a watermark generation model. Most watermarking methods cannot simultaneously possess $Attack$ $Robustness$ against watermark attacks and $Sicherheit$ $Robustness$ to prevent inferring the watermark from the watermarked sequence $T^{N}$, necessitating a trade-off. The research by Liu et al. [55] makes the generated watermark $W$ no longer determined by previous tokens and vocabularies, thereby enhancing both $Attack$ and $Sicherheit$ $Robustness$. Gu et al. [56] took an alternative approach by training a student model to learn the token distribution of watermarked text, imitating the behavior of existing watermarking algorithms through model distillation. However, this approach involves model distillation and suffers from high computational complexity.

Several pioneer works [57, 58, 59, 44] have explored designing multi-bit watermark schemes for LLMs by leveraging the model’s learning capabilities. Abdelnabi and Fritz [57] proposed an encode-decoder transformer architecture, AWT, which learns to extract the message from the decoded watermarked text. To maintain the quality of the watermarked text, they utilize signals from sentence transformers and language models, relying entirely on a neural network for message embedding and extraction. This approach has proven effective because neural networks have been successfully used for natural language watermarking, demonstrating their capability to handle complex language patterns.

Drawing inspiration from a well-known proposition in classical image watermarking work [60], Yoo et al. [58] generated watermark positions by identifying invariant features of semantics and syntax in the text through a pre-trained infill model and create watermarks by replacing words at these watermark positions through masking. Due to the use of a semantically robust filling model, their method significantly surpasses AWT in resilience to watermark attacks and exceeds the fixed upper limit on the number of watermark bits imposed by the ContextLS method proposed by Yang et al. [61]. Extending AWT, Zhang et al. [59] utilized pre-trained language models in a modular fashion to revamp the end-to-end watermarking scheme. They introduced a reparameterization module to transform the dense distributions from the message encoding to the sparse distribution of the watermarked textual tokens, achieving double the watermark information capacity of AWT while maintaining semantic integrity.

Wang et al.  [44] systematically studied the codable watermark system (CTWL) for multi-bit watermark information, considering Yoo et al. [58]’s watermarking method as post-process after LLM generation, which does not integrate well with the generative capabilities of LLMs. They proposed using a proxy language model (proxy-LM) to assist in encoding watermark information during the LLM generation process, followed by vocabulary division.

As indicated by Formula 4, these methods utilize the learning and semantic capabilities of the model to generate the watermark $W$. For instance, Liu et al. [55] utilized a trained watermark model to generate watermark logits based on the semantic embeddings of tokens preceding the current token, which maximizes the first term $I(S^{N};W)$ as much as possible. It is understood that the information flow in neural networks tends to decrease mutual information during forward propagation [62]. This decrease is due to the nonlinearity of forward propagation, the many-to-one mapping relationship, and the suppressive effect of activation functions on information flow in neural networks [63]. Inferring the input from the network’s output is very difficult, thereby making the conditional entropy $H(m|W)$ encloses to $H(m)$, which ensures that $I(m,W)\to 0$. This also ensures the $Sicherheit$ $Robustness$ of such watermarking methods.

The methods proposed in this section involve generating watermarks by applying specific rules. The core principle is to design a set of rules or algorithms that modify or mark text data and models, thereby generating watermarks.

Backdoor Techniques: These methods involve poisoning the training data of LLMs by adding specific triggers to text sequences, thus enabling LLMs to learn the characteristics of these triggers. The presence of watermarks is detected by observing the output of LLMs when given input samples containing embedded triggers.

For instance, Liu et al. [64] implanted backdoor triggers into a small subset of the target LLM’s training data and tampered with the labels of this subset. The presence of watermarks is assessed by verifying the output of the trigger set through black-box access to the target model. Tang et al. [65]improved the backdoor poisoning method without altering the original labels of the watermark samples. Instead, they guided the model to memorize the preset backdoor function by disabling original features on watermark samples through imperceptible perturbations. This approach increases the transparency of the watermark and, by protecting the trigger set used for watermark verification, creates a traceable private watermarking technology. These methods are primarily data-driven, reliant on data, and require participation in the training process of LLMs, with a limited capacity to carry one-bit watermark information.

Cryptography: Watermark generation based on cryptography aims to enhance the security and stealthiness of watermarks through cryptographic techniques. These methods primarily include the use of digital signature technology for watermarks [66] and cryptographically inspired undetectable watermarks [67], both of which rely on cryptographic principles to protect the watermark from unauthorized access and tampering. Christ et al. [67] quantified the randomness used in the generation of a specific output by utilizing pseudo-random values generated by an encrypted pseudo-random function (PRF) to determine watermark embedding locations. They analyzed the undetectability and integrity of the watermark using empirical entropy theory. Furthermore, they introduced the encryption of pseudo-random values with a secret key, which is required for the extraction and verification of the watermark. However, this method is only validated through the entropy theorem for binary channels and is limited to embedding one-bit watermark information. It is uncertain whether it can maintain sufficient empirical entropy to ensure the robustness of the watermark when expanded to multiple bits of information.

Additionally, Fairoze et al. [66] explored the application of digital signature technology on LLMs.This method involves encrypting the hash value of text with a private key to create a watermark signature. This signature is then embedded into tokens of additional length through rejection sampling, while the public key facilitates watermark detection. This approach does not require embedding statistical signals in the generated text, providing a viable solution for publicly detectable LLM watermarks. However, this approach, which employs asymmetric algorithms, often results in highly unstable time and computational costs during watermark generation. The running time exhibits high variance, especially when encountering low-entropy sections of sampling or missed hashes, making the time required for watermark generation occasionally unacceptable.

The main advantage of incorporating cryptographic techniques is the ability to determine whether a watermark is private or publicly detectable easily. One primary benefit of public watermarks is that the extraction and reconstruction processes can be outsourced, allowing different entities to provide watermark extraction services separate from the model providers. Furthermore, public watermark schemes should support the full lifecycle operations of watermarks through API access to private LLMs.

Custom Synonym Substitution Rules: Some methods [68, 69, 70] ensure a close relationship between the watermark and the text’s semantics and context by using custom synonym replacement rules. This approach not only maintains the transparency of the watermark but also enhances its $Attack$ $Robustness$ to text editing attacks. The fundamental premise of text editing attacks is to invalidate the watermark under certain distortion conditions. However, effectively linking semantics and context can limit the effectiveness of such attacks. For instance, He et al. [68] considered two fundamental linguistic features during synonym replacement: part-of-speech (POS) and the dependency tree. Expanding to multi-bit watermarks, Yang et al. [69] proposed a context-aware synonym replacement method for generating watermarks. Meanwhile, Li et al. [70] embedded a series of synonym-based token changes as watermarks in the code generated by LLMs. Li et al. [71] introduced a watermarking method for code text based on transformation rules such as code refactoring, reordering, and format conversion. Each transformation corresponds to a bit of the watermark message, with the presence or absence of a specified transformation determining whether the watermark value of that bit is 0 or 1.

Custom Generation Function: Some custom watermark generation functions have proven effective in practice. For instance, Yang et al. [72] defined a binary encoding function for black-box LLMs, which calculates a random binary code (0 or 1) for each word in the text based on the hash value of the word and its preceding word. Essentially, the function of this binary encoding is similar to that used by Kirchenbauer et al. [32], who utilized hash functions for vocabulary partitioning. Zhao et al. [73] defined two sets of specialized secret sinusoidal signals as watermarks. These two sets of sinusoidal signals have values ranging from [0,1] and satisfy the condition that their sum equals 1.

Data-level embedding, also known as post-processing methods, involves inserting watermarks by directly modifying, learning from, or augmenting the content itself. The primary advantage of these methods is their ability to embed identifying markers $W$ within data in a relatively concealed manner without requiring modifications to the model’s architecture or functionality. Data embedding methodologies exhibit a broad spectrum, including format adjustments, lexical changes, grammatical shifts, and the exploitation of language models, thus offering a variety of approaches to suit different data types and application contexts. Focused on the textual level, these methods are not only applicable to the training data of LLMs to influence the learning trajectory but can also be directly applied to the text generated by LLMs, enabling watermark embedding at the output phase. This phase is principally segmented into four categories:

Format Adjustment: Format-based data embedding ingeniously utilizes text formatting and visual features for watermark embedding. Unlike methods that directly modify the text content, format-based methods embed watermarks through subtle adjustments to the appearance and structure of the text, aiming to achieve copyright protection and data tracking without compromising readability and content integrity.

Btassil et al. [74]achieved watermark embedding by adjusting the vertical and horizontal positions of text lines and words, such as through line shift coding and word shift coding. Por et al. [75] embedded watermarks by inserting different space characters into text spacing or using visually similar but differently coded characters. The presence of these watermarks is almost invisible to users, ensuring the natural flow and original appearance of the text content. These methods do not rely on changes to the text content, making it broadly applicable across various languages and document types without concerns about linguistic or semantic restrictions. However, watermarks embedded in this way have poor robustness and can be invalidated by some formatting-checking tools.

Lexical Variation: Lexicon-based data embedding is a method that embeds watermarks by carefully selecting and replacing specific words in the text. This approach leverages the richness and diversity of language, allowing for subtle modifications without altering the original intent and content.

Aside from the watermark generation method [68, 69, 70] of Custom Synonym Substitution Rules mentioned in Section 3.3, which embeds watermarks through vocabulary changes, the embedding of watermarks through lexical variation is also widely used by various watermark algorithms [76, 57, 77, 52, 72, 58, 78].

Lexicon-based data embedding ensures both fluency in text reading and semantic consistency. This characteristic renders some watermark detection methods ineffective and ensures the transparency of the watermark, making it a mainstream method in many algorithms. However, this approach relies on high-quality synonym databases and advanced language models or rules for precise vocabulary selection and sentence transformation.

Grammatical Transformation: Grammar-based data embedding is a technique that embeds watermarks by altering the syntactic structure of text or code. This method aims to incorporate watermark information through subtle syntactic adjustments without affecting the original semantics. Its application is not limited to natural language texts but also extends to programming languages, demonstrating wide applicability and flexibility.

Chalmers [79] inserts watermarks by transforming the syntactic structure of sentences within paragraphs. In the CATER method proposed by He et al. [68], the dependency tree is a type of syntactic structure that describes the directed binary grammatical relationships between words.

These embedding methods adjust the text at the grammatical level, maintaining semantic integrity and high transparency, and are commonly used for watermark embedding in code text. However, they require a deep understanding of grammar and analysis capabilities. Excessive grammatical changes can affect text readability.

Language Model Utilization: This embedding method further leverages the capabilities of language models. Most watermark algorithms generated through model learning primarily utilize the semantic understanding abilities of language models to create watermarks, necessitating other embedding methods. In contrast, Zhang et al. [59] directly trained a message encoding module that takes watermark messages as input and generates watermarked text based on the learning capabilities of language models. This end-to-end training approach fully exploits the capabilities of language models. Since the watermark is directly generated and embedded internally by the language model, its robustness against text editing attacks depends on the robustness of the language model itself. The generation and embedding of the watermark rely on the model, and if data changes, it necessitates adjusting the model training objectives, which requires substantial computational resources.

The embedding methods at the data level involve manipulating text sequences generated by LLMs, with the fundamental principle of preserving the original sequence’s semantics and readability. According to Formula 6, the text quality represented by $I(S^{N};T^{N})$ is assured. However, detailed text analysis of vocabulary, format, grammar, etc., poses a risk of inferring the watermark signal $W$ from the watermarked text $T^{N}$. Therefor, that watermark embedding method has not effectively minimized $I(W;T^{N})$.

The model-level embedding approach differs from data-level methods by embedding watermarks directly within the application cycle of LLMs. Specifically, it involves three steps: modifying LLMs during the training phase, altering the logits generation during the inference phase, or adopting different token sampling strategies.

Training Phase Embedding: Embedding watermarks into LLMs during the training phase typically involves the use of backdoor techniques and data poisoning methods, as mentioned in Section 3.3. This approach is inspired by backdoor attack [80], incorporating poisoned samples into the LLM’s training data. Assume the training data provider has his data samples $D_{\text{train}}=\{(s_{i},y_{i})\}_{i=1}^{N}$ ,where each sample has its feature $s\in\mathcal{S}$ and label $y\in\mathcal{Y}$ .The attacker selects a small proportion of data $\{(s_{i},y_{i})\}_{i=1}^{M}$, $M<N$ and adds a preset backdoor trigger to these samples while modifying their labels to a target label $\hat{y}$:

where $f_{G}$ is the watermark generation function to add trigger into the input. The poisoned training dataset is the union of the remaining benign training samples and the small number of poisoned training data with the target label, i.e.,

LLMs trained or fine-tuned using the dataset $D_{\text{Poisoned}}$ perform normally on original tasks but generate consistent, specific outputs when inputs contain a special trigger set $D_{\text{backdoor}}$. For example, Liu et al. [64] leveraged text backdoor techniques to insert triggers of different levels into a subset of the original training texts, uniformly changing the labels to a target label. Similarly, Sun et al. [81] employed a similar data poisoning method to embed secret and stable watermark backdoors into open-source code. Modifying the labels of the training corpus can lead to a decrease in LLM performance. To mitigate the impact of changing labels, Tang et al. [65] proposed Clean-Label backdoor watermarking. After selecting the target category, adversarial perturbations are employed to ensure that the model learns features related to the backdoor while retaining the original labels. Sun’s CodeMark [82] introduces semantic-preserving transformations of code and builds poisoned training data by altering the syntactic form of the code, such as changing ’a+=1’ to ’a=a+1’.

Methods that embed watermarks during the training of LLMs typically only provide a one-bit watermark information bit, which can only indicate the presence or absence of a watermark. Changes in the training process can lead to a decline in LLM performance and cause forgetting issues, thereby limiting the watermark embedding rate to a very low value. Additionally, altering the watermark requires retraining the LLM, restricting the application of watermarking algorithms that employ this embedding method.

During the training process, the embedded watermarks, whether introduced by adding subtle backdoor triggers or by embedding backdoors through semantic transformations, must ensure that the generated text $T^{N}$ remains within specified distortion constraints relative to the original text $S^{N}$ while maximizing the mutual information $I(S^{N};T^{N})$. Furthermore, without a trigger set, extracting the watermark $W$ from the watermarked text $T^{N}$ becomes significantly challenging, thereby ensuring the watermark’s transparency.

Inference Phase Embedding: Watermark embedding in LLMs during the inference phase primarily diverges in two directions: modifying logits generation and employing different token sampling strategies.

Logits Generation: Logits are scores assigned by the LLM to potential next words based on its internal representation and the input sequence, determining the probability distribution that influences the model’s next word generation. Methods of watermark embedding at this stage includes all methods that guide logit modifications through vocabulary partitioning [32, 33, 34, 35, 46, 37, 38, 39, 40, 44, 42, 45, 36], as well as various methods that produce watermark biases in logits [73, 43, 55, 66], directly inserting the watermark $W$ into the logits generated by LLMs. Essentially, these methods bias the logits or apply other methods to influence them, causing the LLM’s output to exhibit a certain bias. Watermark embedding is achieved through this biased output.

Assuming an LLM is trained on a vocabulary of size $V$, given a sequence of tokens as input, the LLM predicts the next token in the sequence by outputting a logit score vector $L_{ogit}$. Watermarks, represented by the red-green list [32] and generated through vocabulary partitioning, are embedded by modifying the logit scores:

As shown in Fig. 5, since the logits generated by the LLM have been modified, the LLM tends to select tokens from the generated watermark list, resulting in a higher proportion of the generated text being watermarked. In this way, the LLM is induced to exhibit a specific bias when selecting tokens, achieving the effect of embedding a watermark.

These embedding methods involve modifying the logits of LLMs to bias the model towards outputting tokens from a watermark list, considering the optimization target of Formula 6 during design. Kirchenbauer et al. [32] ensured the quality of the generated text $T^{N}$ by imposing constraints on the modified logits. Lee et al. [39] ensured the functionality and quality of the code by avoiding embedding in low-entropy vocabulary. Takezawa et al. [34] produced more natural texts than existing watermarking methods by adjusting the minimum constraints on logit modification based on the length of $S^{N}$. Furthermore, Hu et al. [35] defined two reweighting methods to produce an unbiased distribution of watermark logits, minimizing $I(W;T^{N})$ to enhance the transparency of the watermark. DiPmark [37] reweighted the watermark logits with a key, aiming to make $I(w;T^{N})\to 0$ while maintaining the original distribution.

Token Sampling: This section introduces embedding methods that intervene in the process of LLMs choosing the next token, utilizing the watermark W to guide the sampling strategy for each token to embed the watermark. Although the selection of tokens involves randomness, this randomness is controlled, with sampling strategies like random sampling, top-k, and top-p all possessing fixed randomness. By altering the sampling strategy with the watermark $W$, the watermark is embedded. During extraction, it is only necessary to judge the alignment between the chosen tokens and the set sampling sequence.

Christ et al. [67] use the output of a Pseudo-Random Function (PRF) to decide whether to embed a watermark at a specific location. Specifically, for each subsequent token generation decision, the LLM uses a bit (or a small part) of the PRF’s output to determine whether to change that token to embed the watermark. If the output of the PRF is below a certain threshold in the model’s original probability distribution, the model generates the token according to the original probability distribution; if it is above this threshold, the model will choose a different token to represent the watermark. Due to the fixed nature of pseudo-random numbers, the watermarked LLM will generate the same text for the same prompts every time, which limits the diversity of the text $T^{N}$. This corresponds to the second optimization target in Formula 6, where $I(W;T^{N})$ is larger, making it easier to find the relationship between $T^{N}$ and the watermark $W$ through multiple attack attempts.

To address the issue of monotonous generated text, Kuditipudi et al. [49] introduced the use of a random watermark key to compute a sequence of random numbers longer than the generated text, mapping it onto the sample to produce watermarked text. The alignment between the text and the pseudo-random number sequence is measured using the Levenshtein distance [83], thereby increasing the diversity of the text. Intervening in the sampling process of each token individually can easily lead to a decrease in text quality. Hou et al. [50] first calculated the LSH signature of the previously generated sentence, then randomly divided the LSH partitions into ”valid regions” and ”blocking regions” based on this signature. The watermark is embedded through the process of rejection sampling of tokens, meaning that new sentences are sampled from the language model until the embedding of the new sentence is located within the ”valid region” of the semantic space, indicating successful embedding. Since the partitioning is based on the sentence level, this method significantly improves the quality of the generated text. Its performance in terms of watermark transparency and resistance to attacks far exceeds other methods.

Rule-based watermark extraction methods rely on identifying pre-defined patterns or rules within the text. These rules can include specific characters, combinations of words, text formatting, decision methods, password matching, and other predefined conditions. By analyzing these features, rule-based methods detect suspected watermarked texts. These approaches pay special attention to specific indicators that suggest whether a text has been generated by AI, such as the entropy of the text, patterns of vocabulary usage, and sentence structure. In this context, entropy measures the randomness and complexity of the text, aiding in distinguishing between human and machine-generated texts. The primary advantage of rule-based methods lies in their simplicity and efficiency, as they do not depend on complex machine-learning models but instead rely on the direct analysis of specific text attributes.

Some studies leverage the Quadratic Residues Theorem to formulate extraction rules for specific watermarking algorithms. Atallah et al. [84] used a large prime as a key to calculate the hash values of the nodes in the semantic tree of each sentence, converting them into binary strings. They then sorted the sentences based on these strings’ hash values and extracted the watermark by reading specific bit sequences from each sentence following the marked sentences in the sorted sequence. Chiang et al. [85] employed quadratic residue keys and bit operations to select terms from the text, constructed bit strings based on the values in the quadratic residue table, and ultimately transformed these bit strings using specific rules to extract the watermark. Topkara et al. [86] relied on the weighted graph of synonym sets, using a secret key to select and color specific subgraphs. During watermark generation, words are replaced to embed watermark information, and during extraction, information is read based on a custom coloring scheme. Kim et al. [87] extracted hidden information by calculating the statistical distribution of the space between words in text segments with the same category labels, using predefined decoding rules to extract from these statistical distributions.

In the aforementioned studies, Zhao et al. [73] employed the Lomb-Scargle periodogram method [88] to estimate the Fourier power spectrum. By applying an approximate Fourier transform, they amplified the subtle perturbations in the probability vector, enabling the detection of peaks in the power spectrum through frequency analysis to determine watermark information. He et al. [68] developed the CATER method, which utilizes a set of features and relies on predefined conditions and discrimination rules to extract watermarks. Fairoze et al. [66], on the other hand, calculated the hash value for each sequence using the watermark’s public key and related parameters to determine whether it matches the expected signature to extract watermark information. These rule-based approaches are characterized by their specificity, making them unsuitable for generic watermark extraction. However, their advantage lies in the simplicity and intuitiveness of the extraction process.

The trigger set-based watermark extraction methods [64, 65, 82] are typically used in conjunction with watermark embedding methods that employ backdoor techniques. The trigger sets usually consist of a group of backdoor text. Given a special trigger set, the LLMs will output a specific answer, which can be used as an extracted watermark. Since the implantation of backdoors participates in the LLM training process, these one-bit watermarks typically exhibit strong robustness. However, a key challenge of this approach lies in designing a covert backdoor trigger mechanism to ensure the transparency of the watermark and considering how to carry more information. Therefore, generating watermarks independent of the LLM training phase remains an area for further exploration.

Statistical watermark extraction methods involve rigorous mathematical and statistical analysis of texts or data to extract watermarks, focusing on identifying anomalies or characteristic differences in data distribution caused by watermark embedding. These approaches are particularly suited for detecting watermarks that have embedded statistical patterns or features during content generation. For instance, hidden watermarks can be extracted by comparing the statistical differences in vocabulary usage frequency, sentence length distribution, and syntactic complexity between watermarked and original texts.

The extraction process analyzes the distribution of generated text tokens to determine if they follow the distribution introduced by the watermark. This is achieved using the Z-test, Likelihood Ratio Test, Q-offset detection, Jensen-Shannon Divergence, or other non-asymptotic statistical tests to determine whether the sample mean significantly deviates from its expected value.

Works that generate watermarks based on vocabulary partitioning [33, 34, 35, 46, 37, 38, 39, 40, 44, 42, 45, 36, 89] and those based on model learning [50, 49, 55] used the Z-test for watermark extraction, similar to Kirchenbauer et al. [32]. Given a vocabulary partitioned into watermarked and non-watermarked tokens based on a fixed ratio, Z scores are calculated by computing the proportion of watermarked tokens to the total tokens. The null hypothesis is rejected, and the watermark is extracted from the text if the z-score exceeds a specified threshold.

Hu et al. [35] proposed a watermark extraction method based on the log-likelihood ratio (LLR) score. This method calculates the LLR score for each watermarked text segment by comparing the relative probabilities of the text under two hypotheses. These scores are then aggregated, and the watermark is extracted if the aggregated LLR score exceeds a certain threshold. This method determines whether the text is more likely to originate from a distribution with a watermark.

SemaMark [40] introduced Q-offset detection to enhance the robustness of boundary semantic values. This is achieved by searching for the highest z-statistic under different offset values Q, using it as the Q-offset score to correct variations in semantic values near boundaries. Li et al. [70] calculated the synonym distribution for each watermark channel and used the Jensen-Shannon Divergence (JSD) threshold to measure the similarity of these distributions to the original watermark distribution.

Statistical-based watermark extraction methods excel at handling large-scale text datasets and can be designed to be highly sensitive to minor changes in the data, achieving low false positive rates and false negative rates, thereby improving the accuracy of extraction. Additionally, statistical methods exhibit high robustness against watermark attacks, meaning that unless a large number of complex attack operations are carried out, it is difficult for attackers to remove the watermark without causing significant statistical deviations.

Despite the numerous advantages of statistical watermark extraction methods, they also have several notable disadvantages. One major drawback is their computational intensity, as the rigorous mathematical and statistical analysis required can be resource-demanding and time-consuming, particularly when handling large-scale text datasets. This makes them less suitable for real-time or near-real-time applications where speed is critical. Another issue is the dependency on the quality and characteristics of the input data; statistical methods may struggle with texts that lack sufficient statistical anomalies or differences for watermark detection, leading to potential inaccuracies. Furthermore, while these methods are generally robust against simple attacks, sophisticated adversaries with a deep understanding of the watermarking scheme can potentially devise complex strategies to manipulate the statistical properties of the data and evade detection. This continuous arms race between watermarking techniques and attack methods necessitates ongoing refinement and adaptation, posing a constant challenge for developers and researchers in the field.

Deep learning-based watermark extraction methods are often used in conjunction with watermark generation methods learned through models. These approaches involve constructing and training deep neural networks to identify and extract hidden watermark patterns in text. They leverage pre-trained deep learning models, such as language models represented by encoders and decoders, to extract watermarks that are difficult to define directly through rules or statistical methods, utilizing the model’s powerful feature extraction and pattern recognition capabilities.

For instance, AWT [57] utilizes an adversarially trained watermark Transformer to extract watermark messages by automatically learning word replacements and positional information through a decoder. REMARK-LLM [59] extends AWT by using Transformers to predict inserted messages for watermark signature extraction. Yoo et al. [58] utilize a pre-trained and fine-tuned filled model to identify masked positions based on text-invariant features, thereby extracting multi-bit watermark information. Liu et al. [43] generate embeddings for all texts to be tested through an embedding network shared with the watermark generation network and then extract the watermark through binary classification using an LSTM network. Munyer et al. [52] use the Bidirectional Encoder Representations from Transformers (BERT) pre-trained model as a powerful feature extractor for binary classification in watermark extraction, leveraging BERT’s capability to capture the contextual meaning of words in a sentence.

Wang et al. [44] demonstrated that LLMs can serve as effective tools for watermark extraction, showcasing significant potential in understanding and processing textual content. By leveraging the powerful language understanding capabilities of LLMs, they analyze subtle differences in vocabulary choice preferences, sentence structure variations, and syntactic complexity to extract specific watermark information. Existing watermark algorithms have largely overlooked the potential of LLMs as watermark extraction tools; future research could explore deeper integration between LLMs and robust semantic watermarks.

The primary purpose of these attacks is to destroy the watermark in text generated by LLMs, rendering it unextractable. Notably, removing a watermark is a trivial task if the quality of the language is disregarded. Therefore, the watermark attacks considered in this article follow Formula 10, where attackers must operate under reasonable language quality constraints within a certain level of distortion. The attacks designed to compromise the watermark typically include text deletion, text insertion, text replacement, text repetition, copy-paste, paraphrasing, overwriting, and LLM-assisted attacks.

Text deletion attacks typically involve removing tokens from the generated text, and altering the watermark features by deleting tokens to increase the difficulty of watermark extraction. For example, Kirchenbauer et al. [32] have demonstrated that deleting tokens to remove green list tokens and modify the downstream red list can effectively destroy the watermark. However, this approach often significantly reduces the quality of the text. Some works have explored methods to enhance the robustness against text deletion attacks, such as introducing text edit distance as a soft constraint [49, 36] and implementing a perturbation certification radius for changes in logits scores [37]. These methods also enhance robustness against other types of text editing attacks, such as insertion and substitution. Moreover, text deletion attacks increase the cost of generation, as attackers ”waste” generated tokens and significantly reduce the breadth of the LLM’s context, explicitly lowering the text quality. Consequently, this level of distortion is usually intolerable.

Text insertion attacks involve adding extra tokens to the generated text to disrupt the watermark. Kirchenbauer et al. [32] demonstrated that inserting tokens from a red list can alter the calculation of the red list for downstream tokens. However, this modification changes the distribution of vocabulary, which poses a risk of reducing the quality of the text. Homoglyph attacks [90] exploit the fact that Unicode characters are not unique, with multiple Unicode IDs resolving to the same or very similar letters. Boucher et al. [91] found that injecting barely noticeable encodings, such as invisible characters or homoglyphs, can significantly degrade watermark performance. Overall, these types of attacks are difficult to detect visually but can be easily removed with various text formatting tools.

Text substitution attacks involve replacing one token with another specific token. In [32], tokens from a red list are introduced, increasing the proportion of downstream red list tokens. The homograph attack [90] modifies text by substituting characters with identical or very similar ones. Building on this, Helfrich et al. [92] further formalized the attack. Works such as [39] and [70] implement substitution by renaming code variables, while other studies [44, 58, 49, 36, 55] use synonyms for word replacement to assess the robustness of watermarks.

Text repetition attacks alter the original text’s watermark distribution by repeating sections of text multiple times. This repetition affects the statistical tests used for watermark detection. Fernandez et al. [42] argue that human-generated texts with high repetition might be mistakenly labeled as machine-generated. The random variables used in watermarking methods, such as vocabulary partitioning, are only pseudo-random. Consequently, repetition produces the same patterns, altering the watermark distribution. This repetition undermines the assumption of independence required for calculating p-values.

The copy-paste attack [33] has been employed by several studies [55, 45, 46] to evaluate the robustness of watermarking. The principle involves mixing watermarked text segments generated by LLMs with manually written text, interspersing the watermarked portions within the surrounding unwatermarked text. Two controllable parameters in this attack are 1) the number of watermarked text segments inserted and 2) the proportion of the document containing watermarked text after the attack. This watermarking attack simulates a real-world scenario where attackers might not completely rewrite text generated by LLMs in practical applications but instead copy and paste it into a larger document to obscure its origin. By employing it, researchers can assess whether watermarks remain detectable when the text is altered or combined with other texts.

Paraphrasing attacks, which involve rewriting text generated by LLMs using language models, human effort, or translation to maintain roughly the same meaning while employing different vocabulary choices and syntactic structures, significantly impact the robustness of watermarking. Specifically, the effectiveness of watermarking against such attacks depends on three factors: token sequence dependency, watermark strength, and text length. Krishna et al. [93] demonstrated that rewriting LLM-generated text with a smaller language model can effectively evade existing AI-generated text detectors. They proposed two methods to enhance the effectiveness of paraphrasing attacks: context-aware rewriting of longer texts and increasing output diversity. Building on this, Sadasivan et al. [94] introduced the recursive paraphrasing attack, applying the paraphrasing process multiple times. After each iteration of paraphrasing, the resulting new text is re-entered into the paraphrasing model to generate further paraphrased text. Several studies [33, 45, 40, 55, 43] have designed language models with good paraphrasing performance for rewriting watermarked texts to assess the watermark’s robustness against paraphrasing attacks. To improve resistance to paraphrasing attacks, Zhao et al. [36] employed a fixed vocabulary partitioning design to make the watermark less susceptible during paraphrasing. Hou et al. [50] utilized the semantic information of sentences to bolster robustness against paraphrasing attacks.

Watermark overwriting attacks involve regenerating the originally watermarked content or overwriting it with different watermarking methods. For instance, in REMARK-LLM [59], new watermarks are used to rewrite the text in front of the original watermark, effectively circumventing the original watermark through the rewriting process.

Another category of attacks, referred to as LLM-assisted attacks, leverages the advanced capabilities of LLMs in understanding and generating human language to conduct attacks. For example, the Goodside emoji attack [95] as discussed in [32, 67] involves instructing the model to produce responses that prompt the insertion of emojis between every pair of words. This type of attack disrupts any watermark that relies on the watermark extractor seeing a continuous sequence of tokens. Consequently, vocabulary partitioning methods generated from the previous discussion cannot resist such attacks.

The goal of model extraction attacks is to imitate the behavior of a target model, creating a valuable local model to evade substantial service fees or even to offer competitive services. In such attacks, attackers create an unwatermarked copy by replicating or mimicking the functionality of the protected model. This type of attack specifically targets scenarios where watermarks are embedded into model outputs to claim intellectual property rights. Attackers may make numerous queries to understand and replicate the model’s behavior, thereby constructing a similar-performing model copy that does not contain the watermark. Li et al. [70] discussed the feasibility of implementing extraction attacks through LLM APIs. The presence of watermarks in the stolen high-quality LLM imitation models suggests that the proposed watermark is both invisible and robust.

Watermark spoofing attacks refer to attackers modifying or constructing text so that clean text without any watermark is incorrectly identified by the watermark extractor as containing a legitimate watermark, or causing the extractor to return incorrect watermark information from the victim organization. These attacks exploit flaws in the watermark algorithm’s extraction and detection mechanisms, particularly when the extraction process relies on rules or statistics.

Here are some examples of watermark spoofing attacks: 1. Attackers can leverage spoofing attacks to fabricate fake news or misinformation and publish it on public media, falsely claiming through manipulated watermarks that the fake news was produced by a legitimate company’s LLM. 2. Attackers can embed a benign company’s watermark within malicious code using spoofing attacks, making the benign company responsible for the harm caused by the malicious code.

Some works have explored spoofing attacks on watermarked LLMs. For instance, Sadasivan et al. [94] artificially constructed text with an understanding of the watermarking method, leading the extractor to misjudge the presence of a watermark. Nevertheless, their approach requires an excessive number of queries from the attacker (1 million), limiting its applicability to only the KGW [32] watermarking scheme, thus making it difficult to generalize to other watermarks.  [56] trained a novel model to learn the distribution of watermarked tokens, which is not feasible for attackers with limited computational resources, especially due to the substantial requirement for a multitude of queries to construct training data and to train a new LLM. Pang et al. [89] argued that robust watermarks may need to compromise on robustness to mitigate the possibility of spoofing attacks. The robustness of LLM watermarks reduces the difficulty of spoofing attacks, as attackers do not need to ensure that every modification or misleading token is watermarked; they only need the overall detection confidence score to exceed a threshold to consider the text content as generated by a watermarked LLM. To address this, Liu et al. [55] incorporated watermarking rules intertwined with textual semantic information, proving to be an effective method to withstand spoofing attacks.

Attackers often utilize adaptive attack techniques to achieve control and manipulation of watermarking algorithms. These attacks are highly customized, assuming that the attacker possesses knowledge of either the entire or partial watermarking framework. With this insight, they can tailor specific adjustments and optimizations based on the characteristics and detection mechanisms of the watermark, thereby increasing the success rate of their attack. Moreover, armed with insider information about the watermark, they can effortlessly produce or replicate private watermarks. Li et al. [70] evaluated the reliability of watermarks under strong adaptive attacks, investigating whether attackers could manipulate text to make watermark extraction fail when they understand the principles of watermark embedding. Liu et al. [43] found that even in scenarios where attackers have access to the watermark extractor and can make unlimited queries to understand the watermark generation rules, it remains difficult to infer the watermark generation method. Existing research rarely uses adaptive attacks for the evaluation of watermark performance, indicating a need for further exploration of attackers’ capabilities to conduct a more intricate analysis of the resilience and unforgeability of watermarks.