Do LLMs Truly Possess Moral Agency? — A Philosophical Analysis of Sampling, Choice, and Moral Responsibility

Title: Do LLMs Truly Possess Moral Agency? — A Philosophical Analysis of Sampling, Choice, and Moral Responsibility

Abstract: This article delves into the core debate over whether large language models (LLMs) possess moral agency, analyzing the essential difference between LLMs' sampling mechanisms and genuine choice from a philosophical perspective, and revealing the gap between probabilistic outputs and intrinsic intentionality.

Keywords: LLM, moral agency, intentionality, free will, AI ethics, philosophy, sampling mechanism, moral responsibility

Original Author/Maintainer: arXiv authors Source Platform: arXiv Original Title: Why Sampling Is Not Choosing: Intentionality, Agency, and Moral Responsibility in Large Language Models Original Link: http://arxiv.org/abs/2606.13441v1 Source Publication/Update Time: 2026-06-11T15:03:48Z

Core Argument: This article argues that LLMs' sampling mechanisms are not equivalent to genuine choice; their outputs lack intrinsic intentionality, thus they do not possess moral agency and cannot bear moral responsibility.

Background: The Controversy Over Moral Agency Arising From AI Capability Improvements

In recent years, as the capabilities of large language models like GPT and Claude continue to advance, a profound question has emerged: Do these systems possess some form of "agency"? Can they be regarded as moral agents and bear moral responsibility? Some optimistic views suggest that the coherent dialogue, reasoning abilities, and normative judgments exhibited by LLMs imply some primitive form of intentionality or autonomy. However, this latest paper from arXiv presents a fundamental refutation: Sampling is not choosing, and there is an insurmountable gap between probabilistic outputs and genuine moral agency.

Core Argument: What Kind of Agency Is Required for Moral Responsibility?

The paper first clarifies key concepts: What exactly does moral responsibility require? The authors point out that genuine moral responsibility requires "commitment-bearing agency", which must be based on intrinsic intentionality and self-attributed actions. That is, a moral agent must be able to view an action as "mine" and commit to its consequences.

This kind of agency constitutes the form of free will relevant to moral responsibility; it is not a product of simple causal determinism or randomness, but a unique ability to choose based on reasons—when we say a person is responsible, we presuppose that they could have chosen a different action, and that their choice was based on an understanding and evaluation of reasons.

Analysis of LLM Mechanisms: The Essential Difference Between Probabilistic Sampling and Human Choice

The core argument of the paper reveals the essential working mechanism of LLMs: Although models can generate coherent outputs, their operation is a probabilistic input-output mapping learned from data. When generating the next token, it is not a "choice" but an execution of sampling from a complex conditional probability distribution.

The variability of this sampling is fundamentally different from human free choice: Human choice is a reason-based decision (making what one considers the right action after weighing considerations), while an LLM's "choice" is merely a random point in a probability space; it does not "own" the choice as a commitment, nor is it guided by reasons.

Analysis of Intentionality: Derived Intentionality of LLMs vs. Intrinsic Intentionality of Humans

The paper distinguishes between two types of intentionality: intrinsic intentionality (mental states directly possessed by conscious subjects, such as beliefs and desires) and derived intentionality (meaning that things acquire through their relationship with subjects of intrinsic intentionality, such as written symbols).

The intentionality of LLMs belongs to the latter: The "meaning" of their outputs depends on the intentions of human designers, the language use in training data, and user interpretation. The model itself does not "understand" the content; it merely manipulates formal symbols, and the semantics come from outside. Its outputs are neither owned as commitments nor guided by reasons.

Responding to Counterarguments: Limitations of Intentional Stance, Functionalism, and Other Views

The paper responds to several common counterarguments:

1. Intentional Stance Argument: Treating LLMs as rational agents can predict their behavior, but a pragmatic description does not prove intrinsic intentionality (e.g., the intentional stance of a thermostat does not mean it truly has beliefs).
2. Functionalism Argument: Similar functional performance does not mean identical psychological properties; the key is whether it involves genuine understanding rather than formal simulation.
3. Compatibilism: Even if determinism is true, free will still requires the ability to self-attribute and respond to reasons, which LLMs lack.
4. Moral Reasoning Output: The model only simulates the patterns of moral discourse in training data, which does not mean it has moral understanding (like a parrot mimicking speech).

Practical Implications: Responsibility Attribution in AI Ethics Governance and Vigilance Against Anthropomorphism

The research conclusions have important implications for AI ethics governance: If LLMs have no moral agency, blaming the model is a category error. This emphasizes the responsibility of human designers, deployers, and users.

It also reminds us to be vigilant against anthropomorphic bias: Interacting with fluent dialogue systems easily leads to projecting human psychological attributes; maintaining conceptual distinctions is crucial for responsible AI development and deployment.

Conclusion: The Importance of a Clear Understanding of the Essence of LLM Capabilities

This article provides rigorous conceptual analysis to help understand the true essence of LLM capabilities: It neither denies their amazing abilities nor exaggerates their philosophical significance. Sampling ≠ choice, probabilistic output ≠ free decision, formal moral discourse ≠ genuine moral understanding. In today's era of rapid AI development, this clear understanding is a necessary prerequisite for avoiding conceptual confusion and making wise decisions.