PolicyLR: A Logic Representation for Privacy Policies

Large Language Models (LLMs) have emerged as powerful tools for natural language processing, demonstrating impressive abilities like text generation and comprehension (Gao et al., 2023). One crucial capability for reasoning about documents is Natural Language Inference (NLI) (MacCartney, 2009). This task involves determining whether a natural language hypothesis can reasonably be inferred from a given premise. In the NLI task, LLMs are required to determine if the hypothesis is true (i.e., entailment), false (i.e., contradiction), or undetermined (neutral), given a premise. For example, consider a premise saying: “John’s brother is 8 years old.”. A hypothesis that says “John has no siblings.” will get a contradiction label. PolicyLR uses LLM’s NLI capabilities to map unstructured policy documents into valuations of a set of atomic formulae.

While LLMs possess vast knowledge, they may not always have access to the specific information required for complex reasoning tasks involving specialized documents. Retrieval Augmented Generation (RAG) (Lewis et al., 2020) addresses this limitation by combining the power of LLMs with external knowledge bases or document retrieval systems. In the context of document reasoning, RAG first retrieves relevant documents or passages from a corpus based on the given query or task. The retrieved information is then used to augment the LLM’s input, providing contextual information and supporting evidence for reasoning. PolicyLR uses RAG to ground LLM reasoning on relevant parts of the privacy policy document.

Privacy policy analysis is crucial to understanding how organizations handle personal data. Automated analysis techniques are becoming increasingly important due to the vast number of privacy policies that exist. Early research focused on rule-based systems and supervised learning approaches for privacy policy analysis. Researchers have employed sentence classification approaches for tasks such as identifying missing information or categorizing policy elements (Harkous et al., 2018; Bhatia and Breaux, 2018). Cui et al. address semantic incompleteness through PoliGraph, a knowledge graph approach that analyzes entire policies using semantic role labeling (Cui et al., 2023).

Topic modeling techniques like those used by Sarne et al. can identify high-level themes within large collections of privacy policies (Sarne et al., 2019). Shvartzshnaider et al. (Shvartzshnaider et al., 2018) introduce a framework that analyzes policies from the perspective of information flow and user readability, considering contextual integrity.

Ontology-based techniques offer additional capabilities. PrivOnto by Oltramari et al. leverages an ontology to represent and analyze privacy policies, enabling semantic querying (Oltramari et al., 2018). Nejad et al.’s Knight system focuses on mapping privacy policies to specific articles in regulations like GDPR (Nejad et al., 2018).

More recently, Large Language Models (LLMs) like ChatGPT and Llama 2 have shown significant promise for various NLP tasks (Touvron et al., 2023), including sentiment analysis and text summarization (El-Kassas et al., 2021). Their ability to process complex language and identify patterns within large amounts of text makes them well-suited for extracting privacy practices from privacy policies. Rodriguez et al. (Rodriguez et al., 2024) propose using LLMs to extract privacy practices from the privacy policies. However, they use LLMs to perform zero-shot classification by treating all the classes as independent.

These studies highlight the potential of NLP and semantic techniques for privacy policy analysis. However, most of these techniques rely on sentence-level analysis that does not account for the whole context in the document and can miss relationships between different sections of a policy. Furthermore, these techniques treat all classes as independent, e.g., purpose and retention-period classifiers are trained separately. This misses out on complex cases where these classes are not independent, and there is a need for a joint classification. PolicyLR accounts for these relations and represents privacy policy in a more granular and comprehensive manner.

Document consistency is essential for ensuring the accuracy and clarity of information present within documents. Research in this area focuses on identifying inconsistencies within a single document (internal consistency) and across related documents (cross-document consistency).

Intra-document Consistency. Ali et al. (Ali et al., 2023) propose a system for automated consistency checks in financial documents by projecting the entities in embedding space and ensuring that semantic and syntactic variations are treated similarly. Andow et al.  (Andow et al., ) propose a system that analyzes privacy policies for internal inconsistencies by leveraging an ontology to capture positive and negative statements regarding data collection and sharing. These approaches segment the documents and use transformer-based models to classify segments that can miss out on the full context. PolicyLR, on the other hand, relies on state-of-the-art language models to understand the context and accurately determine privacy practices.

Inter-document Consistency. Researchers have also explored inter-document consistency to compare practices disclosed by developers in privacy labels with privacy policies  (Khandelwal et al., 2023; Jain et al., 2023). Jain et al. (Jain et al., 2023) introduce ATLAS, a system that casts consistency as a document classification task and automatically detects discrepancies between privacy policies and privacy labels for mobile apps. They extract privacy labels from privacy policies and compare them with actual labels released by the developers. Khandelwal et al. (Khandelwal et al., 2023) create a new taxonomy for privacy labels and leverage it to build classifiers to predict privacy labels using privacy policies. Similarly, Zimmeck et al. (Zimmeck et al., 2019) identify inconsistencies between the app’s behavior and its stated privacy practices by combining machine learning-based privacy policy analysis with static code analysis.

While previous research has made significant developments in consistency analysis, there are limitations. Sentence level analysis in PolicyLint (Andow et al., ) might miss additional context, resulting in erroneous results. Segment-level classification frameworks, as presented in Khandelwal et al. (Khandelwal et al., 2023) and Zimmeck et al. (Zimmeck et al., 2016, 2019), also suffer from this limited context problem. ATLAS (Jain et al., 2023), on the other hand, poses the problem as document classification but trains 32 different privacy label-specific classifiers. PolicyLR, on the other hand, does not rely on segmentation and analyzes all relevant context. Furthermore, we implement PolicyLR using an off-the-shelf language model that acts as a universal compiler to generate the truth table, which in turn allows us to perform several downstream tasks.

Automated methods for assessing policy compliance with regulations are a growing area of interest. Manandhar et al. (Manandhar et al., 2024) introduce the ARC framework for transforming complex privacy regulations into a structured format, facilitating automated analysis and compliance assessment. Prior works also focus on analyzing privacy policies for compliance with regulations like GDPR (Linden et al., 2018; Liu et al., 2021; Liao et al., 2024). PolicyChecker by Liao et al.  (Liao et al., 2024) utilizes a rule and semantic role labeling approach to assessing compliance of mobile app privacy policies with GDPR. PTPDroid by Tan et al. (Tan and Song, 2023b) identifies potential violations related to third-party data collection practices disclosed in Android app privacy policies. Shafei et al. (Shafei et al., 2024) investigate data handling discrepancies in privacy policies for Alexa skills with account linking. Linden et al. (Linden et al., 2018) code ICO checklist¹¹1https://ico.org.uk/for-organisations/uk-gdpr-guidance-and-resources/ into structured queries and leverage the Polisis (Harkous et al., 2018) framework to understand the impact of GDPR on privacy policies.

Mori et al. (Mori et al., 2022) propose a method for using convolutional neural networks (CNNs) to analyze privacy policies and classify them based on compliance with legal requirements. However, the “black-box" nature of CNNs and the need for adjustments when applying the method to different legal frameworks pose challenges. Rabinia and Nygaard explore utilizing Natural Language Inference (NLI) for compliance checking, demonstrating that models trained on diverse datasets perform better with real-world privacy policy tasks (Rabinia and Nygaard, 2022).

Existing works have taken a fragmented approach to the compliance problem, focusing on a subset of privacy regulations, not being able to consider the full context of policy texts, and using opaque models that cannot provide reasoning. In contrast, PolicyLR addresses these problems by considering a more comprehensive representation of the privacy policy as a truth table of logic formulae. This representation allows for more interpretable compliance with a wider set of privacy regulations.

Let $\mathcal{A}$ be that set of atomic formulae of a logic system, representing its vocabulary. Let $\Sigma$ be the set of logical connectives which can be unary, binary or $n$-ary. Connectives are used to construct formulae, for example, if $\neg$ is an unary connective, for any formula $\phi$, $\neg\phi$ is also a valid formula. Similarly, for a binary connective $\land$, any formula pair $\phi$ and $\psi$ imply that $\phi\land\psi$ is also a valid formula. The set of atomic formulae along with the connectives define $G=(\mathcal{A},\Sigma)$, the formulation grammar of the logic. Let $\Phi_{G}$ be the set of all possible formulae that can be constructed using the grammar $G$.

All formulae can be evaluated to either True or False depending on the world model $M$. In other words, if $M$ satisfies a formula $\phi$ i.e., $M\models\phi$, the formula $\phi$ will be evaluated to be True. We define the Base Valuation Function $\texttt{Val}_{M}:\mathcal{A}\rightarrow\{0,1\}$ as

$$\texttt{Val}_{M}(\phi)=\begin{cases}1&\text{if }M\models\phi\\ 0&\text{if }o.w.\end{cases}$$

where $M$ is the world model and $\phi\in\mathcal{A}$ is an atomic formula.

Next, we extend the valuation function $\texttt{Val}_{M}$ to a function $\texttt{Val}_{M}^{*}:\Phi_{G}\rightarrow\{0,1\}$ that assigns truth values to all formulae generated using grammar $G$. Note that all formulae are built inductively from the set of atomic formulae $\mathcal{A}$. Therefore, the valuation of any complex formula can be determined by recursively applying the logical connectives according to their rules on the valuations of the atomic formulas, i.e.

$$\texttt{Val}_{M}^{*}(\oplus(\phi_{1},...,\phi_{n}))=f_{\oplus}(\texttt{Val}_{M}(\phi_{1}),...,\texttt{Val}_{M}(\phi_{n}))$$

where $\oplus\in\Sigma$ is an $n$-ary connective, $f_{\oplus}:\{0,1\}^{n}\rightarrow\{0,1\}$ is the truth function associated with the connection $\Sigma$, $\phi_{1},...,\phi_{n}\in\mathcal{A}$ are atomic formulae.

For instance, let $\phi$ and $\psi$ be atomic formulae with valuations $\texttt{Val}_{M}(\phi)$ and $\texttt{Val}_{M}(\psi)$ respectively. For example, the valuations associated with the connectives $\neg,\land$ and $\lor$ can be determined by the following:

$$\texttt{Val}_{M}^{*}(\neg\phi)=1-\texttt{Val}_{M}(\phi)$$

$$\texttt{Val}_{M}^{*}(\phi\land\psi)=\text{min}\left(\texttt{Val}_{M}(\phi),\texttt{Val}_{M}(\psi)\right)$$

$$\texttt{Val}_{M}^{*}(\phi\lor\psi)=\text{max}\left(\texttt{Val}_{M}(\phi),\texttt{Val}_{M}(\psi)\right)$$

Evaluating whether the world model $M$ satisfies a formula or not, provides information about the model. For instance, if a world model satisfies the formula “All entities must have unique identifiers”, it indicates a structured environment where each entity can be distinctly identified. This suggests that evaluating a large number of such formulae should give a comprehensive representation of the world model. Therefore, the set of atomic formulae along with the Valuation function – $(\mathcal{A},\texttt{Val}_{M})$, provides a logical representation of the world model $M$. Any complex formula can then be evaluated by simply combining the atomic valuations using truth functions associated with the connectives of the logic grammar. This representation has the following benefits:

1. 1.

   Abstraction: It provides a concise intermediate representation of the world model. It can act as a proxy for the world model to perform downstream analyses such as Consistency and Compliance (subsection 3.4).

2. 2.

   Efficiency: Evaluating a formula directly on the world model using the valuation function can be costly – it requires analysis of the world model. By design, our formulation provides a solution by first making the minimum set of necessary valuations and then using them to evaluate any future formulae.

3. 3.

   Explainability: Valuations for complex formulae can seem opaque and hard to interpret. The atomic formulae are fundamental and can be directly inferred from the world model. Therefore, representing complex formulae as a function of the atomics offers better explanations for formula evaluations.

Consistency. A world model $M$ is consistent with another world model $M^{\prime}$ with respect to the logic grammar $G$, denoted as $M\sim_{G}M^{\prime}$ if

$$\forall\phi\in\mathcal{A},\;\texttt{Val}_{M}(\phi)=\texttt{Val}_{M^{\prime}}(\phi)$$

This means that world models $M$ and $M^{\prime}$ agree on valuations of all formulae in $\mathcal{A}$, i.e., they have the same logical representation. For example, consider a set of logical statements describing a company’s data retention policy, such as “Retention period for user data is unspecified”, “Retention period for user data is one year”, and “Retention period for user is 10 years”. This set of statements can be used to evaluate whether two data retention policies are similar/consistent in terms of how long user data is kept. Similarly, it can also be used to evaluate consistency between different versions of the same data retention policy over time.

Compliance. Consider a set of formulae $\Phi\subseteq\Phi_{G}$ constructed using the grammar $G$. A world model $M$ is compliant with $\Phi$, denoted as $M\models\Phi$ if

$$\forall\phi\in\Phi,\;M\models\phi$$

The valuation of each formula in $\Phi$ can be derived by some combination of formulae in $\mathcal{A}$. For instance, the General Data Protection Regulation (GDPR) mandates that personal data should not be kept for longer than necessary for its intended purpose. Now, consider the logical statements from data-retention example above. Compliance with the formula “Retention period for user data is compliant with GDPR” can be alternatively evaluated by the combination of atomic formulae corresponding to these statements – “Retention period for user data is specified”, “Retention period for user data is no longer than necessary for the purposes it was collected”, and “User data is securely deleted after retention period expires”.

To construct PolicyLR’s logical representation, we need to define the set of atomic formulae. We leverage existing work on policy annotation that has developed hierarchical taxonomies of privacy concepts (Wilson et al., 2016; Arora et al., 2022). We specifically use the OPP-115 taxonomy developed by Wilson et al. (Wilson et al., 2016) to generate a list of atomic formulae. The taxonomy is widely used in the literature (Harkous et al., 2018; Zimmeck et al., 2019; Wagner, 2023; Alabduljabbar et al., 2021), and provides a comprehensive set of privacy practices. We note that the PolicyLR framework is independent of the taxonomy a user might choose.

The OPP-115 taxonomy has two levels – a top level that defines high-level privacy categories like first-party-collection, data-retention, and a lower level that defines a set of attributes for each high level category. Each attribute is a categorical variable that can take one of a fixed set of values. For example, the high level category data-retention has attributes retention-period, retention-purpose and information-type. The lower level attribute retention-period can take one of the values – $\{\texttt{indefinite},\;\texttt{stated},\;\texttt{limited},\;\texttt{unpsecified}\}$.

We next describe how we construct the building blocks of PolicyLR using this taxonomy.

Atomic Formulae. We define the set of atomic formulas using a space of finite-domain variables. We use the high-level privacy categories as the finite-domain variables. For instance, $\texttt{data-retention}(\texttt{period}=\texttt{stated},\texttt{purpose}=\texttt{advertising},\texttt{type}=\texttt{location})$ is an atomic formula.

Consider a high-level category $p\in\mathcal{P}$, with lower-level attributes given by $attr(p)\in\mathcal{Q}$. Each attribute $q\in\mathcal{Q}$ can assume a finite set of values $dom(q)\in V$. Now, the set of atomic formulae $\mathcal{A}$ is given by

$$\left\{p\left(q_{1}=v_{1},\ldots,q_{n}=v_{n}\right)\;\middle|\;\begin{array}[]{l}p\in\mathcal{P},\{q_{1},\ldots,q_{n}\}=attr(p)\\ v_{i}\in\text{dom}(q_{i})\;\forall i\in\{1,\ldots,n\}\end{array}\right\}$$

where $p$ is a high-level category from $\mathcal{P}$, $attr(p)$ represents the set of attributes associated with $p$, and each $q_{i}$ is an attribute that can take values from its respective domain $dom(q_{i})$. This set $\mathcal{A}$ includes all possible combinations of attributes and their values for each high-level category.

For example, consider a toy taxonomy that only contains Data Retention and First Party Collection as high-level categories. Furthermore, Data Retention only contains two values each for retention-period and retention-purpose whereas First Party Collection consists of identifiability and purpose.

Our key insight is that given the comprehensiveness of the taxonomy, the set of atomic formulae along with their valuation serves as a complete representation of the privacy policy allowing us to perform a variety of the downstream tasks (7 and 8).

Next, we demonstrate how PolicyLR can be useful for downstream applications.

Compliance Analysis. PolicyLR can help with compliance analysis by mapping the regulatory requirements to logical formulae and allowing to systematically check if the privacy policy adheres to data handling practices and user controls. This can significantly streamline the compliance analysis process and ensure alignment with evolving regulations.

Consistency Analysis. PolicyLR can also facilitate consistency analysis within a privacy policy. The logical representation of the policy using atomic formulae enables automated checks for inconsistencies. For instance, the valuation function can reveal if the policy states that a certain type of data is collected for one purpose, but later contradicts itself by allowing the use of that data for a different purpose.

Privacy-based Comparison Shopping. To perform tasks on a daily basis, customers are often faced with the decision to choose between several applications or services that perform the same task. Apart from quality or cost, these decisions can also be based on the privacy practices of the service provider (König and Hansen, 2012). PolicyLR provides a natural way to compare the privacy policies of multiple services. The atomic formulae can be easily combined to perform comparisons along multiple dimensions.

To reduce compilation to an NLI task, we need to transform the inputs into short sentences that can be used for NLI. For translating a formula, we use the In-Context Learning (ICL) ability of LLMs. For the policy text, we use embedding-based similarity search to extract the relevant sentences from the long policy text.

Formulas. Large language Models have remarkable in-context learning abilities (Brown et al., 2020). It allows LLMs to be applied to new tasks using only a few natural language demonstrations, a phenomenon known as few-shot learning. More concretely, we use a set of $k$ input-output pairs $\{(x^{i},y^{i})\}_{i=1}^{k}$, where $x^{i}$ are arbitrary formulas from PolicyLR’s grammar, and $y^{i}$ are the corresponding natural language translations. We only need a few in-context samples for demonstration, which are crafted by privacy policy experts. For example, the atomic formulae, $\texttt{data-retention}(\texttt{period}=\texttt{indefinite},\texttt{purpose}=\texttt{legal},\texttt{type}=\texttt{location}$ is translated into “The data-retention policy indicates that collected user information, specifically geo-location data, is retained indefinitely and is stored only as long as required for legal or law enforcement purposes.”. Note that translation of formulas is independent of the privacy policy texts and therefore, only needs to be done once.

Policy Text. While recent LLMs have long context windows, privacy policies might still be too long to fit within the LLM’s context window. Therefore, to perform NLI using LLMs, we need to retrieve only the excerpts of the privacy policy relevant to the formula being entailed. We do this via a Retrieval-Augmented Generation (RAG) methodology. We first chunk the long privacy policy text into shorter segments. Then, we approximate the semantics using text-embedding models. This segment embedding mapping is stored in a vector database, to allow for quick retrieval during the entailment process. During retrieval, we ensure that additional context is not lost due to chunking by adding the previous and the next segment of the retrieved segment. Note that this segmentation and embedding process only needs to be done once per privacy policy, subsequently allowing for entailment of multiple formulas.

Once we have the translations for the formula and the privacy policy, we can use NLI to infer the formula’s valuation with respect to the policy. Specifically, we prompt an LLM with the following, “According to the Privacy Policy $P$, is the following statement True? $Q$”. Here, $P$ and $Q$ are translations of the policy text and formula respectively. To extract $P$, we first compute the embedding vector of $Q$ and then fetch the most similar $k$ segments from the segment-embedding mapping computed in the translation stage. We append the tag “[source_id:i]” at the end of the $\text{i}^{th}$ segment and concatenate them to get the condensed and most relevant policy text $P$. We then augment the prompt to provide evidence by highlighting the segments that were used by the LLM to perform the entailment task, using the template – “Give evidence by providing all the source ids that are used to answer the question in the format of - Evidence:[2,3,7,…]”. This evidence makes the valuation more interpretable and can be useful for downstream applications. Note that $k$ here is a tunable parameter that controls the context ($P$) provided to the LLM. It can be tuned to balance the precision-recall trade-off for any downstream task. For example, increasing the value of $k$ will result in higher recall but can lower the precision. We discuss this further in subsection 6.4.

1. Q1.

   What is the performance of PolicyLR’s compiler?
   We demonstrate that PolicyLR is able to successfully perform valuations against unstructured privacy policy text documents. When evaluated on $2656$ entailment instances from ToS;DR, a privacy community annotated dataset, using an open-source LLM gemma2-27b, PolicyLR’s compiler achieves a precision value of $0.84$ and recall value of $0.88$.

2. Q2.

   How effective is PolicyLR for the Policy Compliance task?
   Next, we show the effectiveness of PolicyLR on policy compliance tasks using two existing datasets. Specifically, we analyze the compliance of $419$ privacy policies with respect to $3$ compliance rules based on the Article 13 of the GDPR. We find that PolicyLR achieves an average F-1 score of $0.91$ across the two datasets, highlighting the efficacy of PolicyLR in downstream tasks.

3. Q3.

   How can PolicyLR be used to perform inconsistency detection and privacy comparison shopping?
   We analyze privacy policies of the popular apps on the Google Play Store to show how PolicyLR can be used to a) Analyze the consistency of privacy policies over time and b) Compare privacy practices of apps in similar categories. We note that the latter can be used to provide a comparative view of the privacy of apps that can act as a signal for users when deciding which apps to buy or use.

We use the following experimental setup for answering each of the research questions using the datasets described in Table 1.

ToS;DR Dataset. PolicyLR’s compiler provides a valuation function that can evaluate formulae against an unstructured privacy policy text document. To evaluate its performance, we use the ToS;DR dataset. Terms of Service; Didn’t Read (ToS;DR) is a collaborative, community-driven platform where users and volunteers contribute to evaluating and summarizing terms of service and privacy policies (Roy et al., 2012). The platform helps make privacy policies easier to understand by using crowd-sourced annotations. Users sign up on the platform to annotate policies by linking parts of the policy to specific data practices called cases. Cases are concise statements about privacy settings, for example - ‘You can delete your content from this service’ or ‘This service tracks you on other websites’. A moderator then reviews these matches and either approves them or provides feedback. We can use cases as proxies for the natural language translations of logical formulae. The moderator-approved matches provide reliable ground truth for their valuation against privacy policy texts.

The dataset consists of $246$ cases, which are used to annotate $1074$ unique privacy policy texts. After approval from the moderators, this leads to a total of $13179$ case and privacy policy pairings. ToS;DR comprises only positive instances. To construct negative instances (i.e., where the case doesn’t match the policy text), we manually analyze all the cases and find $11$ pairs of mutually contrasting cases. This means that if a case is evaluated to be true for a policy text, its contrasting case will necessarily be evaluated to be false on that text. Using this formulation, we construct a total of $1222$ negative instances. We sample a random subset of size $1300$ from the positive instances to get a total of $1522$ case-policy pairs. For each instance, we apply PolicyLR’s compiler to evaluate the case against the policy text, answering either true or false.

We now describe how PolicyLR is instantiated for our experiments.

Models. While evaluating PolicyLR’s compiler on the ToS;DR dataset, we consider several open-source LLMs – 8 billion and 70 billion versions of Meta’s Llama3 as well as 9 billion and 27 billion versions of Google’s Gemma2. We also evaluate OpenAI’s latest closed-source model Gpt4-O. For the Compliance task, we consider the larger versions of both Llama3 and Gemma2. Finally, for the consistency task, we consider Llama3²²2Gemma2 was released very recently (06-27-2024). Due to time constraints, it is only part of our ToS;DR evaluation. To get deterministic results, we set the sampling temperature to 0 for all models.

Translation. Since the translation of atomic formulae needs to be done only once, we use the most performant LLM gpt4-o for this task. Below we show the prompt used for this task for one of the data-retention atomic:

Here, we provide a description for each attribute value as described in the OPP-115 dataset.

Entailment. To make it more accessible, we implement PolicyLR’s entailment module using only open source components. We tune the hyperparameters using a disjoint set of 10 policy documents. We use the SentenceSplitter API from LlamaIndex³³3https://docs.llamaindex.ai/ to segment the privacy policy text. Each segment comprises $300$ tokens. Then, we generate text embeddings using UAE-Large-V1 (Li and Li, 2023), which is a popular open-source embedding model. We store these embeddings in chroma⁴⁴4https://www.trychroma.com/ which is a open-source vector database. For any entailment task, we query the database for the top-$k$ segments that are most similar to the embedding of the hypothesis. We use $k=5$ and $k=10$ for evaluating the compiler, and $k=10$ for the rest of the evaluation.

Entailment. We evaluate our compiler on the $1300$ positive and $1222$ negative case-policy pairing from ToS;DR. We parse the response of the entailment LLM and assign it a value of true if it begins with “Yes” and false if it begins with “No”. We did not observe any instance when response did not begin with either “Yes” or “No”. This again demonstrates the high instruction following capability of the latest LLMs. Table 2 shows the precision, recall and F1 score when using 2 variants of both llama3 and gemma2 as well as the closed source model gpt4-0. For each setting, we show results when providing the LLM with the 5 and 10 most relevant segments from the privacy policy. First, we observe the number of policy segments provides a trade-off between precision and recall. Fewer segments provide a higher precision whereas adding more segments improves recall. This is likely because LLMs struggle with longer contexts (Li et al., 2024), but also might miss out on relevant context in case of fewer segments. Second, we observe that larger models perform better in the case of both llama3 and gemma2. In terms of F1 score, we find that gemma2 outperforms llama3 and even gpt4-o. Overall, gpt4-o has the best performance with a precision-recall of $0.94$ and $0.84$ respectively. Among open-source models, Llama3-70b has the highest precision. To demonstrate PolicyLR’s applications, we use Llama3-70b.

Error Analysis. We perform a deep dive into the errors for gpt4-o to better understand PolicyLR’s performance. The LLM wrongly entailed a total of $280$ case-policy pairs – $204$ positive pairs and $76$ negative pairs. There are three primary reasons for these errors – (1) Insufficient Context, (2) LLM Reasoning Error, and (3) Incorrect Annotation Error.

Insufficient Context Error. To characterize the first error type, we use the policy excerpt that was used by the ToS;DR moderators to approve the case-policy pairing. The first type of error occurs in instances where the policy excerpt is not part of the context retrieved by the embedding model. In these cases, the LLM does not have the necessary context to correctly entail the case. Note that since we only have policy excerpts for positive pairs, this methodology can only discover insufficient context errors for the positive pairs. Out of the $204$ positive pair errors, $95$ were due to insufficient context. For the other two error types, we perform a qualitative analysis and highlight some interesting cases below.

LLM Reasoning Error. The following belongs to the second error type where the LLM performs incorrect entailment. In this case, the LLM reasoning seems too nit-picky and suggests a lack of common sense.

Incorrect Annotation Error. The below instance belongs to the third error type where the TsD;DR annotation is wrong. Upon manually analyzing the privacy policy, we find that the policy does not state that anonymization is done for all cookie types.

Evidence. We also evaluate the effectiveness of PolicyLR’s evidence functionality. We again leverage ToS;DR’s policy excerpts and evaluate whether the context segments cited in the LLM response contain the excerpt. Out of the $1300$ positive pairs, we found $1056$ instances where the excerpt was part of the retrieved context. We use these $1056$ case-policy pairs to evaluate the evidence functionality. We observe that the LLM, on average, cites $2$ context segments (out of $10$) while responding to each of the above pairs. Overall, we found the excerpt as part of the evidence in $854$ cases, giving a recall of $81\%$.

To evaluate PolicyLR’s performance on the policy compliance task, we use two existing datasets – (1) Online Privacy Policies (OPP-115) dataset (Wilson et al., 2016) comprising 115 policies, and (2) AutoCompliance (Liu et al., 2021) corpus comprising 304 policies. The OPP-115 dataset provides 23K sentence-level annotations based on the OPP-115 taxonomy. The annotations are at two levels: the first level consists of paragraph-sized segments annotated as per the high-level categories of the taxonomy. The second level includes parts of segments annotated for attribute-value pairs such as retention-period: limited, purpose: advertising, etc.

AutoCompliance (Liu et al., 2021), on the other hand, focuses on policy compliance. They analyze Article 13 of the GDPR and manually extract $10$ labels that discuss personal information collection. They further annotate 304 policies by segmenting the policies and annotating each line of the policy as either one of the $10$ labels or others. They then build $9$ compliance analysis rules that measure compliance of a privacy policy with the GDPR. For example, one of the rules is Collect Personal Information $\rightarrow$ Data Retention Period, implying that if a policy collects personally identifiable information, then the data retention period must be specified. Finally, to generate the compliance dataset, they obtain policy-level annotation by aggregating the segment-level annotation.

Note that some of the rules can be decomposed into combinations of low-level categories of the OPP-115 taxonomy. For instance, "Collect Personal Info $\rightarrow$ Contact Details" can be represented in OPP-115 taxonomy as: first-party-collection - identifiability: identifiable AND data-retention - retention-period: not unspecified. We note that while both these datasets provide sentence-level annotations, we follow the aggregation-based formulation similar to Liu et al.  (Liu et al., 2021) Linden et al.  (Linden et al., 2018) to get policy-level annotations. For instance, the data retention period annotation of the entire privacy policy can be derived using the presence of at least one sentence, which is annotated for the data retention period. We also note that while evaluating PolicyLR, we provide the entire privacy policy and use the aggregated label as the ground truth.

Table 3 shows the compliance rule, the corresponding regulation, the computed formula as a composition of PolicyLR’s atomic formulae, and the performance of PolicyLR for each rule. We find that PolicyLR has an average F1 score of $0.90$ with an average recall of $0.95$, outperforming Autocompliance by $4\%$. We note that PolicyLR uses off-the-shelf open-source LLM as opposed to Autocompliance, which trains custom classifiers for each of the rules. Further, we observe that in Autocompliance, training data for the classifiers is also used for compliance analysis, whereas in PolicyLR, we do not perform any training and use the entire dataset for evaluation.

We also analyze PolicyLR’s performance on the Right to Access compliance rule (Collect Personal Information $\rightarrow$ Right to Access), achieving an F1-score of only $0.51$. Upon closer examination of the dataset, we find that several annotations for this class appear to be incorrect. This discrepancy might be attributed to the inherent similarity between the Right to Access and the Right to Delete, as a similar trend is observed within the OPP-115 dataset. Consequently, we opt to exclude the Right to Access rule from this analysis for the sake of clarity.

Next, we present PolicyLR’s performance on the OPP-115 compliance dataset in Table 4. We find that PolicyLR achieves an average recall of $0.88$ and an average F1-score of $0.92$. It is worth noting that PolicyLR’s performance on the data retention compliance rule falls below the overall average. This can be attributed to the inherent limitations within the OPP-115 dataset, where data quality for data retention information is known to be low. Prior research has documented similar challenges; for instance, Polisis (Harkous et al., 2018) reports an F1-score of only 0.71 for the same rule using a classifier trained on OPP-115. We manually analyzed the errors and found that the majority of them were due to incorrect annotation. For example, the segment: Email share feature may be saved for your convenience for future articles you may wish to email was tagged as retention-period: limited.

Error Analysis. Investigating the errors in the compliance task for both datasets, we find that errors can be categorized into two classes below.

LLM Reasoning Error. In these errors, we observe that the LLM extrapolates in the reasoning and reaches the wrong conclusions. Such instances are not common and generally indicate incorrect assumptions by the LLM. For example, consider the following task for alibaba.com:

Here, the policy never mentions that users can request to delete the data, but the LLM infers that because the company may delete the data on their own. We note that these errors can be potentially addressed by choosing a more capable LLM, such as GPT4 or Gemma2.

Annotation Error. These are the errors where the human annotators made a mistake and mislabeled a segment, but the LLM correctly categorizes it. For example:

Here, the LLM correctly infers that since the account data is deleted when the account is deleted, the retention period is limited.

Note here that we used the open source LLM Llama-70b to perform compliance analyses in this section. The performance of closed-source models, such as GPT-4 and Gemini, is believed to be better than open-source models. This suggests that performance can be further improved by using a better model.

Precision-Recall Tradeoff. We acknowledge the inherent ambiguity present in natural language queries. This ambiguity can lead to multiple valid formulas representing a single compliance rule, as shown in Table 3. The rule, "Users should be able to modify/delete their data," can be translated into two distinct formulas.

$$\texttt{acc}=\texttt{edit}\lor\texttt{acc}=\texttt{deactivate}\lor\texttt{acc}=\texttt{delete}$$

$$\texttt{acc}=\texttt{edit}\land\texttt{acc}=\texttt{deactivate}\land\texttt{acc}=\texttt{delete}$$

These formulas offer a mechanism for controlling the desired level of precision in our analysis. For instance, the first formula allows for any combination of edit access, delete access, or account deactivation to satisfy the rule. Conversely, the second formula implements a stricter interpretation, requiring all three functionalities to be present. This interplay between formula composition and performance reflects the precision-recall tradeoff - as the level of restrictiveness (precision) increases, the ability to identify compliant policies (recall) decreases. By allowing the creation of multiple formulas for a single rule, the user can tailor the analysis to specific requirements.

We note that prior works in compliance analysis create structured rules (Linden et al., 2018) or train individual classifiers (Liu et al., 2021). However, these approaches perform classification at a segment level and do not consider the full context of policy texts. PolicyLR addresses these limitations and leverages the truth table to efficiently perform the compliance analysis. Additionally, the existing approaches only cater to a subset of privacy regulations. PolicyLR allows a more comprehensive analysis by leveraging the joint distribution of privacy practices.

To demonstrate how PolicyLR can assist with the consistency analysis of privacy policies, we curated a dataset of popular privacy policy texts from play.google.com. For differential analysis, we first manually collected privacy policy URLs for the top 10 applications from play.google.com for each of the following categories – Communication, Dating, Entertainment, Health, Parenting and Social. We found a total of $54$ unique policy URLs among all these categories. Note that different apps from the same parent company can have an identical privacy policy, resulting in fewer unique policies. For example, Instagram and Meta have identical privacy policies.

To curate a dataset for the evolution of privacy practices over time, we use the Wayback Machine⁵⁵5https://web.archive.org/ from the Internet Archive to get the historical versions of the privacy policies for the apps above. For an earlier timestamp, we purposefully choose the latest policy before 2018 to obtain a pre-GDPR version of the policy. As described by Linden et al. (Linden et al., 2018), the majority of the updates occurred in 2018. Selecting a pre-GDPR version of the policy allows us to observe the effect of the GDPR. Interestingly, due to website migrations, we find that several websites’ current privacy policy URLs did not exist back in 2017. To get the old policies for these cases, we manually crawled the homepage using Wayback Machine and identified the privacy policies. Following this approach, we found a valid historical version for $31$ out of $54$ URLs. We downloaded the raw HTML for each of the collected URLs and used Beautiful Soup⁶⁶6https://www.crummy.com/software/BeautifulSoup/ library to extract the associated text. This resulted in a total of $85$ unique privacy policy texts.

Next, we discuss the two downstream applications of the consistency framework of PolicyLR.

We demonstrate using PolicyLR to analyze the evolution of privacy practices of mobile apps over time. This is an important analysis because by tracking changes in disclosure of privacy practices, we can get insights into how companies adapt to evolving legal landscapes. This analysis can also assist policy auditors in the enforcement of regulations to ensure effective user privacy protection. Additionally, it can provide the user with a better understanding of how their data is handled by the apps they utilize.

We analyze the Policy Dataset for Consistency Analysis 8.1 for this task. Specifically, we analyze pre-GDPR and post-GDPR policies of the $31$ apps in the dataset. The dataset consists of 31 top applications on the Play Store. We focus on three high-level categories from the OPP-115 taxonomy, namely, Data Retention, User Access Edit and Deletion, and Policy Change. Recall that high-level categories in OPP-115 are independent of each other, and therefore, any combination of these high-level categories is a valid set of atomics for PolicyLR. We choose to restrict ourselves to only these three categories because the number of atomic formulae can grow exponentially with the number of leaf nodes.

Results. We implement PolicyLR using the open source Llama-70B model as the compiler with the custom taxonomy defined above. We then take the two valuation functions and compare the valuation of each atomic formula to identify the inconsistencies. Figure 2 shows the distribution of changes observed in privacy practices across the three high-level categories for each app. Notably, these categories exhibit some overlap with the information mandated by GDPR Article 13. We find that policies across all app categories have changed over time. For instance, the "User Access" category witnessed the most significant change in the case of TikTok, followed by Hinge and FindMyKids. This indicates that these applications potentially made substantial adjustments to user access controls in response to the regulation. We confirmed this by manually checking the policies. Conversely, Microsoft’s privacy policies exhibited minimal changes between the pre- and post-GDPR versions. Similarly, Amazon’s policies remained largely unchanged in the "Data Retention" category, while some adjustments were made in the "User Access" category.

Examples of Changes in Practices. We manually select the following two apps: Facebook and Tiktok and show examples of policy change. For example - in 2017, Facebook’s policy mentioned that they can retain information indefinitely for security reasons. However, the latest policy mentions that the data can be deleted upon request.

Similarly, Tiktok’s policy in 2017 did not have any means by which user could delete their data, whereas in 2024, they include a full section on Your Rights.

We presented here a proof-of-concept analysis that showcased the versatility of PolicyLR by analyzing the evolution of privacy practices over time. We acknowledge the small size of the Policy Dataset as a limitation and leave the full-scale measurement analysis for future work. Our intention here was to perform a case study showing that PolicyLR can be used to comprehensively analyze the evolution of privacy practices over time. For example, no other framework is currently capable of identifying the joint distribution of privacy practices in a policy.

We now demonstrate how PolicyLR can be used as the basis for a privacy-centric shopping assistant by allowing users to compare apps based on their privacy preferences. Users can select specific privacy dimensions such as data retention, data sharing, and encryption practices, and the system will group apps accordingly, presenting a clear comparison of their privacy policies. For example, consider Alice, a privacy-conscious person, is considering using a messaging app and is confused between Signal and WhatsApp as they offer similar functionalities. Before making a decision, Bob may want to compare the Data Retention practices of the two apps. PolicyLR can perform this analysis and provide this information.

We show a proof-of-concept version of this application using the most recent privacy policies from the Datenschutzbestimmungen dataset. This involves performing a holistic comparison of practices across apps. PolicyLR’s consistency formulation provides a natural way to compare privacy policies by directly comparing PolicyLR’s atomic formulae valuations. Table 5 compares the data-retention practices of $8$ most popular Google Play apps from the Communication category – Whatsapp, Snapchat, Messenger, Telegram, Signal, Discord, Textnow and Zangi. We compare apps across two attributes of data-retention: purpose and retention-period. We place an app’s icon in a cell if the valuation of the corresponding formula is true.

We emphasize that existing sentence-classification-based approaches are not able to achieve this granularity as they classify these attributes individually. PolicyLR, on the other hand, provides a more fine-grained view by considering the joint distributions of these practices. For instance, individual analysis would describe retention-period of Whatsapp as limited. However, this misses out on the information that data collected for specific purposes like perform-service or security is actually retained for an unspecified period of time.

From Table 5, we can observe the following trends, which can be useful for a user when comparing communication apps:

1. 1.

   Whatsapp, Discord and Signal have better data retention practices as compared to Snapchat and Messenger.

2. 2.

   Only Textnow specifies a retention-period for the data retained for perform-service or security purposes. However, it is not transparent about data retained for advertising or analytics.

3. 3.

   None of the apps retain any data for an indefinite period of time.