Text Anonymization Based on Large Language Models: From Reddit Comments to European Court of Human Rights Judgments

This article introduces a reproduction project based on an ICLR 2025 paper, focusing on using large language models (such as GPT-4o) to achieve high-quality text anonymization. The project aims to address the shortcomings of traditional anonymization methods, preserving the practical value of text while protecting privacy. Key results include a 95% entity recall rate on the TAB dataset (which includes European Court of Human Rights judgments).

Data privacy protection is a critical issue in the AI era. Traditional text anonymization relies on rule matching or NER models, but has two major problems: difficulty capturing indirect identity information (e.g., attributes inferred from context) and over-anonymization leading to loss of text value. LLMs, with their strong semantic understanding capabilities, offer new possibilities for solving these issues.

The project is built based on the paper Large Language Models are Advanced Anonymizers, including complete experimental code and evaluation processes. Regarding dataset adaptation, it was initially designed for Reddit comments and later expanded to support the TAB dataset (1268 European Court of Human Rights judgments, average 5000 characters with gold standard annotations) and the SynthPAI synthetic dataset (used to evaluate personal attribute inference capabilities).

The project uses a three-level prompt strategy: basic level (directly identify and replace sensitive entities), advanced level (detailed entity type definitions and principles), and chain-of-thought level (guide step-by-step contextual analysis before anonymization). For long documents (e.g., ECHR judgments), an intelligent chunking mechanism is implemented to balance processing efficiency and semantic integrity.

The project uses entity-level evaluation metrics (recall rate, precision rate, breakdown by entity type). On the TAB test set, GPT-4o combined with chain-of-thought prompts achieves a 95% entity recall rate while maintaining high precision. Additionally, the compare_levels_tab.py script is provided to visualize the differences in anonymization quality and text retention across different prompt levels.

The experimental workflow includes environment preparation (Mamba for dependency management, supporting multiple model sources like OpenAI, Azure, and HuggingFace), data loading (automatic download of the TAB dataset), anonymization execution (one-click run via the run_tab.py script, supporting specification of model, prompt level, and number of documents), and result comparison (generating HTML reports to show effects of different configurations). Application scenarios include law (case studies on anonymized judgments), healthcare (desensitization of medical records to support medical research), social media (opening user data for academic research), and corporate compliance (meeting data protection regulations like GDPR).

Current limitations: Focuses mainly on explicit entities (names, places), with limited ability to identify implicit identity clues (writing style, habits); anonymization effects may vary across different languages and cultural backgrounds, requiring more cross-language research for validation. Future directions: Introduce more powerful multimodal models to handle rich text content, develop adaptive prompt optimization mechanisms, and establish a more comprehensive privacy-utility trade-off evaluation framework.