Is the 'Human-like Attribute' of Large Language Models Truly Unique? A Critical Study on Attribution Bias

This article, through experiments training a neural network in the game Age of Empires II, raises critical questions about the current research methods that attribute anthropomorphic traits to large language models (LLMs). Key points include: 1) Conclusions about LLMs' human-like attributes may have methodological flaws, lacking appropriate control benchmarks; 2) Any entity running on a sufficiently powerful 'substrate' (such as a simple neural network in a game) may exhibit features similar to 'intelligence'; 3) Call for the establishment of more rigorous empirical evaluation standards to avoid attribution bias caused by subjective interpretation.

In recent years, research on LLMs and their agent workflows has flourished, but many studies attribute anthropomorphic traits such as 'moral judgment', 'natural language understanding', and 'reasoning ability' to models. These attributions often lack a strict empirical basis and are mostly based on researchers' subjective interpretation of outputs. The core issue of this article is not to debate whether these attributes exist, but to point out the fundamental methodological flaw in current research: conclusions about LLMs' human-like attributes may be wrong because no appropriate control benchmarks have been established.

The research team chose Age of Empires II (which has complex resource management, tactical decision-making, and long-term planning mechanisms) as the experimental platform to train a simple neural network. The results show that this network exhibited behavioral patterns that could be interpreted as 'intelligence' or 'understanding'. Based on this, the authors propose: The anthropomorphic attributes of LLMs are not empirically unique—some attributes (such as prompt response) may be constant, but the interpretation of behavior changes with the 'substrate'.

In the paper, 'substrate' refers to any sufficiently powerful medium (such as Lego blocks, physical space, or video games). The authors point out that any powerful substrate may host entities that exhibit 'intelligence' features, challenging the intuitive understanding of the nature of intelligence: intelligence may be an emergent phenomenon, not an exclusive feature of a specific computing architecture. We cannot automatically attribute intelligence to the intrinsic properties of a system while ignoring its operating environment/medium.

Current research has logical problems: Assuming that LLMs have human-like attributes leads to circular reasoning or uninformative conclusions (experiments reinforce existing biases). The authors propose the 'null hypothesis' method: When designing experiments, first assume that LLMs have no uniqueness, then consider whether the phenomenon can be reproduced in other simple systems, and only after exclusion should we attribute it to the special properties of LLMs, with specific implementation examples provided.

The technical appendix proves that Age of Empires II is functionally and Turing-complete (can simulate any Turing machine), reinforcing the core argument: The game has universal computing capabilities, so it is not surprising that its neural network exhibits 'intelligence'. Implications for the community include: being alert to anthropomorphic bias, establishing strict evaluation standards, valuing control experiments, and reflecting on the impact of research hypotheses on results.

The study reminds the AI field to maintain methodological rigor. LLMs are powerful tools, but claims about their 'understanding' or 'knowing' need to be carefully argued. As the title implies: If LLMs have human-like attributes, then Age of Empires II also has them—this absurd inference reveals the conceptual confusion in current research.