Can Sparse Autoencoders Identify Reasoning Features in Language Models? ICML 2026 Study Reveals New Challenges in Interpretability

A study to be published at ICML2026 questions the application of sparse autoencoders (SAE) in LLM interpretability: the 'reasoning features' extracted by SAEs may only be spurious correlations with reasoning-related tokens, not genuine reasoning mechanisms. This study provides an important methodological warning for the field of LLM interpretability, emphasizing the need to go beyond simple correlation analysis and adopt more rigorous verification methods.

The study of large language model interpretability faces a key challenge: can we truly understand the internal mechanisms of the model? As an unsupervised method, SAE is widely used to decompose model activations into sparse features. Many researchers select features with stronger activation in reasoning tasks as 'reasoning features' through comparison. However, the core issue is: correlation ≠ causation—these features may only be superficially related to reasoning-related tokens (such as "therefore" or "step") rather than participating in the reasoning process.

Theoretical Analysis: Sparse regularization decoding tends to retain stable low-dimensional correlated features and suppress high-dimensional behavioral changes, leading to the possibility that the 'reasoning features' selected through comparison are concentrated on the structure of suggestive tokens rather than genuine reasoning mechanisms. Falsification Framework: 1. Causal token injection: Inject reasoning-related tokens into non-reasoning text and observe feature activation; 2. LLM-guided counterexample construction: Generate non-reasoning inputs that trigger feature activation and rewritten versions that suppress activation while keeping semantics unchanged.

The team conducted experiments under 22 configurations, and the results show:

1. High injection sensitivity: 45%-90% of candidate features are activated after injecting a small number of reasoning tokens, being sensitive to surface patterns;
2. Fragility of context-dependent features: Non-reasoning inputs constructed by LLMs can trigger feature activation, while semantically unchanged rewrites suppress activation;
3. Limited guidance effect: Feature guidance has a negligible impact on benchmark performance.

This study provides methodological warnings:

• Falsification first: When attributing advanced behaviors, actively seeking counterexamples is more critical than verification;
• Beyond correlation: Need to combine rigorous verification methods such as intervention experiments and counterexample construction;
• Humble attitude: Surface interpretability may mask deep complexity, so caution should be exercised when claiming to understand the model. In the future, stricter verification standards are needed to ensure a true understanding of LLM working mechanisms.

The open-source code of the study has been released on GitHub (https://github.com/GeorgeMLP/reasoning-probing), implemented based on the TransformerLens library, including a complete experimental framework and analysis tools, supporting SAE feature detection for multiple open-source models. Paper link: https://arxiv.org/abs/2601.05679, published on May 17, 2026 (arXiv v7), to be presented at the ICML2026 conference.