PII Masking: A Dual-Track Personal Information Desensitization Solution with BERT and LLM

The PII Masking project aims to compare the effectiveness of two approaches—fine-tuning encoder models (DistilBERT, DeBERTa) and prompt engineering for large language models (LLaMA)—in PII detection and desensitization, providing a complete implementation reference for privacy-preserving NLP tasks. Addressing the limitations of traditional PII desensitization methods, the project systematically compares the pros and cons of different technical routes to help users choose the appropriate solution based on their scenarios.

With the increasing stringency of data privacy regulations such as GDPR and CCPA, enterprises and research institutions face compliance pressures. Traditional methods have limitations: rule engines rely on regular expressions and dictionaries, making it difficult to handle variants and emerging patterns; general NER models are not precise enough for PII recognition in specific domains. The project aims to establish an effective pipeline to detect and desensitize two common types of PII—names and emails—and compare the effectiveness of different technical solutions.

The project implements a comparison of three methods:

1. DistilBERT: A lightweight encoder that retains 97% of the performance, with a small model size and fast speed, serving as the baseline for resource-constrained scenarios;
2. DeBERTa: Microsoft's improved BERT that decouples attention to enhance word order understanding, using the best-performing seed model in the domain;
3. LLaMA: Zero-shot prompt engineering that requires no domain training, leveraging the emergent capabilities of large language models. The comparison addresses the question of choosing between encoder fine-tuning and LLM general capabilities for specific tasks.

The project is divided into seven stages:

1. Data validation and preprocessing: Fix annotation formats, split datasets, and inject synthetic emails to enhance data;
2. Preprocessing and exploratory analysis: Convert to Hugging Face format, handle subword segmentation, and perform smoke tests to validate the pipeline;
3. Encoder training: Train DistilBERT and DeBERTa for comparison, and save checkpoints;
4. LLM inference: Perform zero-shot inference with LLaMA, implementing caching to avoid repetition;
5. Comprehensive evaluation: Calculate metrics such as desensitization leakage rate, and conduct bootstrap statistical tests;
6. Error analysis: Classify error patterns and generate error distribution charts;
7. Independent evaluation: Test generalization ability across domains.

The project demonstrates good engineering practices:

• Centralized configuration management: All parameters are in the configs directory, avoiding magic numbers;
• Modular code structure: Divided into directories like scripts, src, and notebooks, supporting command-line and interactive operations;
• Reusable source modules: Core logic is encapsulated into modules, such as data loading, preprocessing, and LLM inference;
• Result organization: Outputs are categorized by stage for easy management and analysis.

Solution selection for different scenarios:

• Resource-constrained: DistilBERT is suitable for edge deployment or latency-sensitive applications;
• Accuracy-first: DeBERTa is suitable for production environments with high accuracy requirements;
• Rapid deployment: LLaMA zero-shot prompts are suitable for cold starts or data-scarce scenarios (note API costs and latency);
• Hybrid strategy: Lightweight encoder for initial filtering + LLM for secondary verification, balancing cost and performance.

Current limitations of the project: It only focuses on two types of PII—names and emails—and is mainly based on English data. Future improvement directions: Expand to more PII types such as phone numbers, ID numbers, and addresses, and support multilingual PII desensitization.

PII Masking not only provides a usable PII desensitization tool but also demonstrates a method for systematically comparing different NLP technical solutions. Solution selection needs to consider resource constraints, accuracy requirements, and latency tolerance. For developers working on data privacy protection, compliant text processing, or NLP model evaluation, it is a complete implementation worth referencing—its modular design and rigorous evaluation can be migrated to other sequence labeling tasks.