Automated Privacy Policy Evaluation Using Large Language Models: Practical Exploration of a Three-State Classification Framework

This study proposes an automated privacy policy evaluation framework based on Large Language Models (LLMs), replacing traditional binary classification with three-state classification (True/False/Ambiguous) to conduct structured analysis of sensitive data practices. It addresses the issues of time-consuming manual reviews and poor consistency, providing a reproducible technical solution for privacy compliance reviews.

Privacy policy evaluation has long relied on manual reviews, which are time-consuming, labor-intensive, and difficult to ensure consistency. After the introduction of regulations like GDPR, enterprises have an urgent need for compliance, but existing tools mostly stay at keyword matching and cannot delve into semantics. Core questions: Can general-purpose LLMs handle structured privacy policy evaluation? Especially in high-risk sensitive data processing scenarios, can they accurately identify and classify relevant clauses?

The framework introduces a three-state classification mechanism (True/False/Ambiguous), considering the complexity of privacy policy texts (which have room for interpretation). It builds an evaluation system around five sensitive data categories: biometric data, health data (strictly aligned with GDPR), and physiological, physical, and behavioral data (covering practices that exceed GDPR but are sensitive).

A modular pipeline design is adopted, with six sequential model calls (metadata extraction + five sensitive data categories); the temperature is set to 0 to ensure reproducible results; the output undergoes two-stage verification (parsing and cleaning → prompt repair), and if it fails, errors are recorded and the process continues, reflecting robustness.

For each sensitive data category, it not only determines whether collection occurs but also evaluates four dimensions: 1. Data storage (storage method/location); 2. Data sharing (third-party sharing and conditions); 3. Retention and deletion (duration/mechanism); 4. Data minimization (practice description), which is closer to real compliance needs.

By comparing manually expert-annotated datasets (Excel template → JSON) with model outputs, the performance of LLMs is quantitatively evaluated; it supports connecting to multiple LLMs via the OpenRouter API, allowing flexible comparison of different model performances and providing empirical basis for model selection.

Practical value: Compliance auditing (rapid risk screening), user empowerment (underlying capabilities of privacy tools), regulatory support (large-scale reviews), research foundation (standardized evaluation tools); Note: Automation cannot completely replace humans (in complex legal interpretation scenarios), but establishing a structured benchmark promotes standardization and scaling in the field.