Luminol-AIDetect: Zero-Shot AI-Generated Text Detection Based on Text Perturbation

Luminol-AIDetect exposes the structural vulnerability of large models through random text shuffling, achieves cross-model and cross-language zero-shot AI-generated text detection using perplexity change differences, and reduces the False Positive Rate (FPR) by up to 17 times compared to existing methods. This method does not require training for specific models, providing a new paradigm for content moderation, academic integrity, and other fields.

With the popularity of large language models like GPT-4 and Claude, AI-generated text has permeated various fields, making reliable distinction between human and AI content an urgent need. Existing detection methods face three major challenges: model specificity trap (trained for specific models, performance drops sharply when facing new models), insufficient adversarial robustness (easily bypassed by rewriting, synonym replacement, etc.), and zero-shot requirement (an ideal detector should be ready to use without retraining).

The core observation of Luminol-AIDetect is: the autoregressive nature of large language models leads to structural vulnerability—compared to human writing, AI-generated text is more sensitive to text structure perturbations. Human writing has deep semantic coherence; even if sentence order is shuffled, logic can still be perceived. However, AI generation lacks global planning: it is locally fluent but globally loose in structure, leading to a significant increase in perplexity after shuffling.

Step 1: Text Shuffling

Split the text into sentences/paragraphs, randomly rearrange the order of units while keeping the content inside each unit unchanged, thus destroying the global structure.

Step 2: Perplexity Feature Extraction

Calculate the perplexity of the original and shuffled text, then extract features: basic features (original perplexity, shuffled perplexity, change rate ΔP); advanced features (distribution statistics of multiple shuffles, differences across different granularities, perplexity recovery curve).

Step 3: Density Estimation and Ensemble Prediction

Use Kernel Density Estimation (KDE) or Gaussian Mixture Model to learn the distribution of the feature space; generate the final prediction through ensemble voting of multiple lightweight classifiers, and output probability scores to support adjustable thresholds.

• No training data required: leverages universal structural features of AI text, without relying on sample training for specific models.
• Cross-model generalization: effective for GPT series, Claude, LLaMA and its fine-tuned versions, Falcon, Mistral, etc.
• Cross-language support: maintains stable performance in 18 languages (including Chinese, English, German, French, Russian, etc., covering multiple language families).

• Dataset coverage: 8 content domains (news, academic papers, creative writing, etc.).
• Adversarial robustness: robust under 11 attacks (synonym replacement, style transfer, etc.), with FPR reduced by up to 17 times compared to existing methods.
• Computational efficiency: only requires forward inference, with minimal overhead for feature extraction and classification, and significantly lower cost than comparison methods.

• Short text challenge: for texts shorter than 50 words, structural signals are weak, leading to decreased detection accuracy.
• Highly structured text: human-written texts like poetry or legal provisions may have similar perplexity changes after shuffling as AI texts, leading to easy misjudgment.
• Future model adaptability: if future models adopt non-autoregressive generation or global planning mechanisms, the structural vulnerability hypothesis may need revision.

Luminol-AIDetect can be applied to:

1. Content platform moderation: no need to continuously update models to detect new AI-generated content;
2. Academic integrity: identify AI-written papers and assignments;
3. Information security: detect AI-generated phishing emails and disinformation;
4. Copyright protection: identify AI-generated content that may infringe copyright. The method is simple and robust, suitable for practical deployment to address the challenges of AI-generated content.