# When Large Models Stop Following Steps: A Diagnostic Study on the Procedural Execution of Language Models

> This study, through controlled diagnostic benchmark tests, found that large language models exhibit significant capability defects in procedural execution tasks: as the number of program steps increases from 5 to 95, the accuracy rate plummets from 61% to 20%. Failure modes include missing answers, premature termination, incorrect self-correction, etc., revealing the issue of execution faithfulness behind superficial reasoning abilities.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-01T17:55:47.000Z
- 最近活动: 2026-05-04T02:52:41.038Z
- 热度: 103.0
- 关键词: 大语言模型, 程序化执行, 推理可靠性, 基准测试, 算法忠实性, 长程依赖, AI安全, 模型评估
- 页面链接: https://www.zingnex.cn/en/forum/thread/llm-arxiv-2605-00817v1
- Canonical: https://www.zingnex.cn/forum/thread/llm-arxiv-2605-00817v1
- Markdown 来源: floors_fallback

---

## [Introduction] Defects in Procedural Execution Capabilities of Large Models: Accuracy Plummets as Steps Increase

This study, through controlled diagnostic benchmark tests, found that large language models have significant capability defects in procedural execution tasks: when the number of program steps increases from 5 to 95, the average first-answer accuracy rate plummets from 61% to 20%. Failure modes include missing answers, premature termination, etc., revealing the issue of execution faithfulness behind superficial reasoning abilities.

## Background: The Appearance of Large Models' Reasoning Abilities and Hidden Concerns About Execution Faithfulness

Large language models perform well in benchmark tests such as mathematical problem-solving and logical reasoning, but an overlooked question is: does the correct answer come from faithful execution of instructions? This study questions this point, designs a procedural execution diagnostic benchmark, and reveals the substantive execution defects behind superficial reasoning abilities.

## Methodology: Design Ideas and Complexity Control of the Diagnostic Benchmark

The study chose arithmetic programs as the test carrier (verifiable, simple, controllable), and controlled complexity from two dimensions: 1. Program length (5-95 steps, testing long-range dependencies); 2. Lookback dependency (referencing intermediate variables to simulate state transfer in real algorithms).

## Evidence: Steep Drop Relationship Between Program Length and Accuracy, and Failure Modes

Testing 14 models and 55 configurations, the results show: 5-step program accuracy is 61%, 95-step drops to 20%. The main failure modes are five types: missing answers, premature answers, self-correction after errors, traces of insufficient execution, and hallucinated extra steps.

## Conclusion: Impact of Procedural Execution Defects on Key Applications and Reflection on Evaluation

This defect poses a reliability crisis for key applications such as financial computing and medical decision-making. Traditional end-to-end evaluation may mask problems; it is recommended to adopt fine-grained evaluation methods such as process supervision, adversarial testing, and length extension testing.

## Technical Analysis: Potential Causes of Procedural Execution Defects in Large Models

1. Autoregressive generation is prone to error propagation; 2. The Transformer attention mechanism dilutes early information as the sequence grows; 3. Most programs in training data are natural language descriptions, leading to approximate execution rather than precise execution.

## Recommendations: Improvement Directions to Enhance Procedural Execution Capabilities of Large Models

Architecture level: explicit state maintenance, structured generation, validator integration; Training strategy: program synthesis data, reinforcement learning (process rewards), curriculum learning (from short to long programs).

## Research Limitations and Future Directions: Reconsidering the Definition of "Reasoning"

Current limitations: only targeting arithmetic programs, limited model scope, not deeply exploring the impact of prompt engineering. Future directions: expanding to multimodality, studying the relationship between scale and faithfulness, developing automated evaluation tools. Conclusion: True reasoning requires faithful adherence to the process; a predictable system is more valuable than one that is occasionally correct but unexplainable.