Study on the Intrinsic and Extrinsic Characteristics of Effective Reasoning in Code Interpreters

This study is the first to systematically analyze the key characteristics of Code Interpreter (CI) reasoning, covering two dimensions: extrinsic key tokens and intrinsic cognitive behaviors. It reveals the important role of mechanisms like verification, backtracking, and reverse chaining in enhancing the reasoning ability of large language models. The study supports the 'behavioral richness hypothesis' and provides optimization strategies for both reasoning and training phases, offering theoretical foundations and practical guidance for the field of CI reasoning.

Code Interpreter (CI) reasoning is an important paradigm for enhancing the reasoning ability of Large Language Models (LLMs), improving accuracy and interpretability through executable computation and iterative verification. However, the behavioral characteristics supporting effective code reasoning have not been fully explored. Questions such as whether findings from traditional natural language reasoning apply to code reasoning and whether code reasoning has unique intrinsic mechanisms remain to be answered.

This study analyzes the effectiveness of code reasoning from two dimensions:

Extrinsic Characteristics: Key Tokens

Mark key reasoning nodes (verification points, decision branches, etc.). The hypothesis is that strong CI reasoning models exhibit these tokens more frequently.

Intrinsic Characteristics: Cognitive Behaviors

Simulate the thinking process of human programmers, including:

• Verification: Proactively checking the correctness of intermediate results
• Backtracking: Returning to correct errors after discovery
• Reverse Chaining: Deriving steps backward from the target

Systematic analysis leads to the following conclusions:

1. Strong models show higher frequencies of key tokens and cognitive behaviors, supporting the 'behavioral richness hypothesis'
2. Verification, backtracking, and reverse chaining are the most critical cognitive behaviors, consistent with human procedural thinking
3. The frequency of cognitive behaviors is positively correlated with the performance of reasoning tasks, especially significantly in mathematical reasoning, sorting problems, and optimization tasks

Optimization in the Reasoning Phase

Adding code-specific key tokens to prompts can improve performance in mathematical, sorting, and optimization tasks without modifying model parameters, though gains are limited for some tasks.

Enhancement in the Training Phase

Incorporating code-specific cognitive behaviors to improve supervised fine-tuning and reinforcement learning leads to performance improvements in two evaluated models. It also reduces 'overthinking' in error responses and increases token usage efficiency.

This study is the first to systematically characterize the effective reasoning properties of CI, providing theoretical foundations:

1. Behavioral Interpretability: Understanding the reasoning process through cognitive behaviors
2. Operable Optimization Strategies: Key tokens and cognitive behaviors as entry points for optimization
3. New Dimension for Model Evaluation: The richness of cognitive behaviors can serve as an evaluation metric Future directions: Developing methods for automatic identification and enhancement of key tokens, designing training objectives for cognitive behaviors, and extending to other tool-based reasoning.

This study provides the first systematic behavioral analysis framework for the field of Code Interpreter reasoning. It reveals the important roles of extrinsic key tokens and intrinsic cognitive behaviors, deepens the understanding of CI reasoning mechanisms, and offers practical guidance for developing more powerful reasoning models.