Study on Style Differences Between Human and AI-Generated Text: How Models and Genres Shape Linguistic Features

This study conducts a large-scale analysis of text styles from 11 large language models (LLMs) across 8 genres and 4 decoding strategies. Key findings include: models and genres have a greater impact on text style than prompts and decoding strategies; the key linguistic features of LLM-generated text are highly robust to generation conditions. This research provides an empirical basis for understanding style differences between human and AI-generated text, optimizing LLM usage, and AI text detection.

As LLM generation capabilities improve, machine-generated text can achieve deceptive fluency, but it also raises issues like spam and academic fraud. Existing research mostly focuses on AI text detection, yet lacks in-depth understanding of the essential style differences between human and AI-generated text. This study aims to reveal the key factors influencing machine text style to better control LLM outputs and optimize detection methods.

The study uses the multidimensional analysis framework proposed by Douglas Biber (a recognized system in linguistics) to characterize text style from five dimensions: informational vs. interactive, narrative vs. non-narrative, explicit situational reference vs. implicit situational reference, persuasive vs. non-persuasive, and abstract vs. concrete. This framework allows systematic comparison of text style features from different sources and under different conditions.

1. Robustness of Linguistic Features: The key style differences of LLM text are highly robust to generation conditions (such as prompts, continuing human text, etc.), and simple prompt engineering is difficult to eliminate them; 2. Dominant Role of Genres: Genres have a greater impact on style than source (human-written vs. machine-generated), and the style difference between human and AI text in the same genre is smaller than that between different genres; 3. Clustering of Dialogue Models: Dialogue-optimized model variants tend to cluster in the style space, and dialogue fine-tuning has a significant impact on style; 4. Model vs. Decoding Strategies: The model itself has a greater impact on style than decoding strategies (e.g., temperature, top-p sampling).

1. Realistic Expectations: Do not expect to completely change the core style of a model through prompts; specialized techniques (like fine-tuning) are needed; 2. Priority of Genre Selection: When planning generation tasks, clarifying the genre has a greater impact on style than choosing a model or adjusting parameters; 3. Consistency of Dialogue Models: Mainstream dialogue models perform similarly in generating dialogue-style text; appropriate base models are needed for non-dialogue text; 4. Challenges for Detection Systems: AI text detection needs to be trained for specific genres rather than using a universal cross-genre solution.

Limitations: Only focuses on English text and does not involve other languages; uses publicly released models, and specially fine-tuned models may have different features. Future Directions: Track the impact of model version updates on style; explore style differences in multilingual scenarios; study changing core style features of models through training interventions.