AI-Generated Content Detection: A Comparative Study of Human and Language Model Capabilities

This article conducts an in-depth comparison of the performance of humans and 6 mainstream large language models (LLMs) in AI-generated content detection tasks. It finds that human detection capabilities have significant limitations (high misjudgment rate, large individual differences), while AI models, despite some advantages, are easily bypassed by adversarial texts. The study also explores technical challenges such as the arms race between generation and detection, and the dilemma of evaluation standards, and puts forward practical suggestions for education, content platforms, and future research directions, emphasizing the need to rebuild a diverse and dynamic content evaluation system.

With the rapid development of LLMs such as the GPT series and Gemini, AI-generated content has penetrated into academic writing, news, social media, and other fields. While bringing convenience, it has also raised concerns about content authenticity and academic integrity. Traditional detection methods (statistical features, classification models) are difficult to deal with advanced LLMs, and the ability of humans to identify AI content and its comparison with machine algorithms have become important topics in the field of AI ethics and applications.

The study uses a standardized experimental design: constructing a diverse text test set (different topics, styles, lengths); including 6 representative LLMs (including the latest GPT version and Gemini 2.5); ensuring reliability through independent judgments by multiple human evaluators and AI models plus cross-validation; evaluation indicators cover multi-dimensional metrics such as accuracy, precision, recall, and F1 score.

Human performance is far below expectations: even professionals have a high misjudgment rate when facing carefully designed AI texts; modern LLMs' language fluency is close to that of humans, making traditional "machine-like" indicators ineffective; humans' reliance on subjective intuition is prone to failure; there are significant individual differences (over-skepticism or credulity), limiting the large-scale application of manual review.

There are differences among AI models: the detection capabilities of different models are related to their architecture, training data, and fine-tuning strategies; content generated by the same type of model is easy to identify (implying "model fingerprint"); however, all models experience a significant drop in accuracy when facing carefully edited/style-transferred AI texts and are easily bypassed by adversarial attacks.

• Arms race: After a breakthrough in detection technology, generation models will improve to evade detection, making a single solution difficult to be effective in the long term; theoretically, perfect detection may not exist (LLMs can learn the statistical features of human texts).
• Evaluation standard dilemma: The attributes of AI content become ambiguous after manual editing, and the ownership of texts written by humans with AI assistance is difficult to define; overly strict detection standards are prone to false positives, while overly loose ones are ineffective, requiring comprehensive consideration of technology, ethics, and policies.

• Education field: Avoid relying on AI detection tools to judge assignments; instead, integrate AI tools into courses to cultivate critical thinking and original expression.
• Content platforms: Automated detection should be equipped with manual review + appeal processes, and transparently inform users of the detection mechanism and its limitations.
• Research directions: Develop multimodal detection methods, interpretable models, and dynamic update frameworks to adapt to the iteration of generation models.

Both humans and current AI systems face severe challenges in AI content detection, and the boundary between AI generation and human creation is increasingly blurred. Instead of pursuing a "perfect detector", it is necessary to establish a diverse, dynamic, and humanized quality evaluation system, with technical tools as assistance but ultimately relying on human wisdom and ethical considerations.