New Approach to AI Detection: Identifying Fake Responses Generated by Large Language Models Using Questionnaire Items

This article introduces a cutting-edge study that explores using specially designed questionnaire items to detect whether survey responses are generated by large language models (LLMs). Starting from cognitive behavioral characteristics, this method differs from traditional text classifiers and provides a new research paradigm for AI content detection.

With the popularity of LLMs like ChatGPT, AI-generated content has permeated various fields, making it a challenge to distinguish between human and machine-generated content. Traditional detection methods are easily bypassed by adversarial rewrites. Led by Cameron S. Kay and Madalina Vlasceanu, this study focuses on online survey scenarios and hypothesizes that there are systematic differences in cognitive behavioral characteristics between LLM responses and human responses, which can be captured through questionnaire items.

Traditional detection methods (perplexity, neural network classifiers, watermarking) are post-hoc analyses and easily bypassed. The new method adopts: 1. Embedded design (integrating detection items into the questionnaire); 2. Behavioral signal capture (designing tasks that require human cognitive abilities); 3. Real-time identification (marking suspicious responses during the collection phase).

Phase 1: Develop candidate detection items and test them on LLMs, finding that they can effectively distinguish between human and machine responses; Phase 2: Collect human samples for validation to avoid false positives; Plan to compare Gemini 2.5 Pro and OpenAI GPT-5.1 to ensure the method does not rely on specific model vulnerabilities.

Based on LLM characteristics, the detection dimensions are inferred as follows: 1. Temporal consistency (LLMs respond abnormally quickly); 2. Context dependence (lack of real background details); 3. Consistency pattern (LLM responses are more internally consistent); 4. Creative tasks (responses are overly average).

If the method is effective, it can improve the quality of online research data; help crowdsourcing platforms balance open access and data quality; and trigger reflections: How to define the boundaries of human data with AI assistance?

This study represents a shift in AI detection methodology, and the embedded process method is more robust. Future work needs to verify cross-cultural effectiveness, generalization ability to different LLM architectures, and long-term adversarial stability. Distinguishing between humans and machines requires parallel advancement of technological innovation and ethical thinking.