Text-Preserving Watermarking Technology: A New Solution for Traceability Auditing of Fine-Tuning Data in Large Language Models

This article proposes a text-preserving invisible watermarking technology for traceability auditing of fine-tuning data in large language models (LLMs). Its core is embedding traceability information using Unicode invisible characters, enabling verifiable traceability without compromising text readability, and it has passed robustness tests against various practical data processing workflows. This technology addresses the issues of traditional traceability methods being prone to failure and existing text watermarks affecting readability, providing a new solution for copyright protection and compliance auditing of LLM training data.

With the widespread application of LLMs, issues related to the source and copyright ownership of training data have become prominent, especially in fine-tuning scenarios where data legality and traceability are key regulatory focuses. Traditional traceability relies on metadata or hash verification, but these are prone to failure after text cleaning or conversion; existing text watermarking technologies (such as homoglyph substitution) significantly affect readability, and users expect solutions that balance traceability and reading experience.

The core of this technology is its 'text-preserving' feature, embedding traceability information using Unicode invisible characters (zero-width/control characters). Key technical points:

1. Character set selection: Filter Unicode characters that are invisible in most rendering environments;
2. Embedding strategy: Selective substitution (prioritizing areas with minimal semantic impact) and uniform substitution (stable distribution);
3. Detection: Scan invisible character patterns to extract information, decoupled from text content—recovery is possible as long as the characters are retained.

The experiments cover multi-dimensional evaluations:

1. Baseline comparison: The homoglyph substitution baseline is significantly weaker in readability than this method; human evaluation can hardly distinguish between watermarked text and the original;
2. Large-scale data pipeline testing: After passing through mainstream data cleaning workflows like C4, CCNet, and FineWeb, the watermark can still be reliably extracted;
3. Non-adversarial transformations: Compatible with mainstream tokenizers, API transmissions (e.g., GPT/Claude interfaces), and PDF conversions;
4. Adversarial attacks: Due to the watermark's multi-position distribution and error-correcting coding, local perturbations are hard to destroy, and large-scale rewrites are easily detectable.

Application scenarios: Copyright protection, data leakage traceability, compliance auditing (enterprises/institutions can embed watermarks before data distribution to track flow). Limitations: Extreme cleaning rules may remove invisible characters; attackers who know the implementation mechanism may remove them targeted; watermark capacity and robustness need to be balanced and adjusted.

The research team has open-sourced the complete experimental code (including core algorithms, training and evaluation scripts, and baseline implementations) with a modular design for easy expansion; the dataset is publicly available via the OSF platform to ensure research reproducibility and promote domain development.

Text-preserving watermarking technology provides a practical and robust solution for traceability of LLM training data, with important value in protecting data rights and promoting responsible data use. As LLM applications expand, such traceability technologies will play a more important role in ecological governance.