Are Implicit Reasoning Models Really Hard to Explain? A Deep Study on the Interpretability of LRMs

This empirical study challenges the traditional perception that implicit reasoning models (LRMs) are uninterpretable. Key findings include: 1) The implicit reasoning tokens of LRMs are often unnecessary; removing them still yields the same answers. 2) Implicit tokens can be decoded into human-understandable reasoning traces (65-93% accuracy for correct samples). 3) Interpretability can serve as a signal for prediction correctness—correct predictions are easy to decode, while incorrect ones are hard. These findings provide a new perspective for evaluating the interpretability and reliability of LRMs.

Explicit reasoning (e.g., Chain-of-Thought) generates natural language intermediate steps, which are highly interpretable but have high computational costs. Implicit reasoning (LRMs) uses special implicit tokens to carry reasoning information—they are theoretically more compact and efficient, but are regarded as "black boxes" due to their unreadability, limiting deployment in high-risk scenarios.

Finding 1: On logical reasoning datasets, LRMs can almost generate the same answers after removing implicit reasoning tokens, indicating underutilization of reasoning tokens and questioning their actual role. Finding 2: In correct prediction samples, implicit tokens can be decoded into reasoning traces consistent with standard answers (65-93% accuracy), showing that LRMs encode interpretable processes. Finding 3: Decoding methods without prior knowledge can verify reasoning traces—correct samples are easy to decode, while incorrect samples are rarely decodable.

Core decoding steps: 1) Mapping learning: Supervised learning from implicit token space to natural language trace space; 2) Verification mechanism: Check if the candidate trace logically implies the final answer; 3) Iterative optimization: Try different strategies for failed samples until a verifiable trace is found or confirmed non-existent.

There is a correlation between interpretability and prediction correctness: successfully decoding a reasonable trace increases prediction confidence, while decoding failure warrants caution. This correlation can serve as a tool for model reliability assessment and also provides an entry point for debugging.

1. Re-evaluate LRM value proposition: Need to improve training methods to ensure implicit reasoning capabilities are fully utilized; 2) Interpretability is not mutually exclusive: Decoding technology can significantly enhance the interpretability of LRMs; 3) Integrate decoding verification: Future systems can incorporate this as part of confidence estimation.

Current limitations: Verified only on logical reasoning datasets; need to expand to math, common sense reasoning, and other tasks. The decoding success rate (65-93%) still has room for improvement. Future directions: Develop stronger decoding algorithms, explore online real-time decoding, and integrate decoding verification into model training.