When Does Built-in Thinking Help or Harm? A Study on Constraint-Level Error Transfer in Instruction Following

The study reveals the dual nature of large reasoning models in instruction-following tasks—thinking mode benefits planning-type constraints but harms precision-type constraints, with 10-20% of prompts showing reversed results when switching between thinking modes. This research targets the Qwen3 series and Hunyuan series models, analyzing the impact of built-in thinking on different types of constraints through controlled experiments, providing important insights for model design and application.

The built-in thinking ability of large reasoning models (e.g., OpenAI o1/o3, DeepSeek-R1) performs well in math/programming tasks, but its impact on instruction following remains unclear. IFEval is an authoritative benchmark for testing models' compliance with complex instructions, including verifiable constraints (format, length, keywords, etc.). The challenge faced by the study is how to disentangle the confounding effects of "thinking" from factors like model size and training data.

The Alibaba Qwen3 series (1.7B-32B) was selected because its "thinking on/off" modes use the same weights, allowing precise isolation of the thinking variable; the Tencent Hunyuan series (four sizes) was included for cross-family validation to ensure the generality of the findings.

The overall pass rate changed slightly (Qwen3 decreased by 0.55-3.52 percentage points), but 10-20% of prompts showed reversed results, indicating that thinking changes the error distribution. Constraints are divided into two categories: planning-type (global counting, structural requirements, etc.) improved under thinking mode; precision-type (exact format, character-level constraints, etc.) consistently deteriorated. Although the overall trend of the Hunyuan model was opposite, the symbolic pattern of constraint classification was consistent.

Thinking mode changes the length of answers; after controlling for length, the loss of precision-type constraints decreased but residual penalties still existed. Correlation analysis of thinking traces showed: positive correlation in neutral mode (r≈0.15), almost no correlation in planning mode (r≈0.02, with an execution gap), and negative correlation in precision mode (r≈-0.05). Activation repair experiments indicated that flipped instances of precision-type constraints were easier to fix (32-58%), while planning-type ones were harder (14-40%).

Practical Implications: Dynamically adjust thinking mode (enable for planning-intensive tasks, disable for precision-intensive tasks); design hybrid reasoning architectures (task routers, adaptive depth, constraint-aware reasoning). Limitations: Single benchmark (IFEval), limited model scope, need for deeper mechanism explanations. Future Directions: Multi-benchmark validation, theoretical modeling, intervention strategy development, training optimization.

Built-in thinking has a highly selective impact on instruction following, being a double-edged sword—it benefits planning-type constraints but harms precision-type constraints. Simply enabling thinking is not an optimal strategy; reasoning modes need to be adjusted according to task structure. This finding has far-reaching implications for model design and application, as true intelligence lies in "knowing when to think and when to act directly."