ACTS: Efficient and Controllable LLM Reasoning via Agentic Chain-of-Thought Steering

ACTS (Agentic Chain-of-Thought Steering) is an efficient and controllable framework for LLM chain-of-thought reasoning. Its core is modeling reasoning guidance as a Markov Decision Process, using a dual-agent architecture (frozen reasoner + controller agent) to dynamically select strategies. It achieves significant token savings while maintaining reasoning quality and supports flexible accuracy-efficiency trade-offs. This research provides a new path for fine-grained control of LLM reasoning.

The Double-Edged Sword of Chain-of-Thought Reasoning

Large language models improve accuracy through chain-of-thought (CoT) reasoning, but have two major flaws:

1. Inefficient token consumption: Generates a lot of redundant content, wasting computing resources;
2. Lack of reasoning control: Users cannot intervene in the direction and depth of thinking.

Limitations of Existing Methods

Existing efficient reasoning methods (shortening, early stopping, compression) only focus on "how much to say" and do not address "how to think". The reasoning strategy remains a black box, lacking explicit guidance and control.

Dual-Agent Architecture

• Frozen Reasoner: Responsible for actual reasoning generation, kept frozen to retain basic capabilities;
• Controller Agent: A lightweight policy network that decides guidance actions (reasoning strategy + guidance phrase) at each step.

MDP Modeling

Modeling reasoning steps as a Markov Decision Process:

• State: Summary of current reasoning trajectory + remaining thinking budget;
• Action: Reasoning strategy (e.g., detailed analysis/quick verification) + guidance phrase;
• Reward: A signal that integrates budget conditions and reasoning quality.

Training Methods

1. Synthetic Trajectory Initialization: Supervised learning based on multi-budget augmented examples to gain basic guidance capabilities;
2. Reinforcement Learning Optimization: Optimize the controller through budget-conditional reward shaping (considering quality, efficiency, and strategy consistency).

Key Experimental Conclusions

1. Maintain Reasoning Quality: While significantly reducing token consumption, performance is comparable to full reasoning;
2. Significant Token Savings: Compared to unguided reasoning, it achieves substantial token savings, reducing costs and improving response speed;
3. Controllable Trade-off: Supports flexible adjustment of budget parameters to balance accuracy and efficiency (e.g., allocate more budget for high-accuracy scenarios);
4. Cross-Model Generalization: Its effectiveness has been verified on different reasoners and tasks.

Technical Insights

1. Control Upgrade: From "controlling output" to "controlling strategy", improving reasoning transparency and adjustability;
2. Collaboration Paradigm: The dual-agent division of labor (reasoner provides basic capabilities, controller is responsible for strategy) provides new ideas for LLM system design;
3. Budget Awareness: Incorporate resource budget into decision-making to adapt to resource-constrained scenarios.

Summary

ACTS achieves efficient and controllable LLM reasoning through MDP modeling and dual-agent architecture. It saves tokens while maintaining quality and supports flexible trade-offs, which has important theoretical and practical value.

ACTS technology is applicable to the following scenarios:

1. Cost-sensitive production environments: Commercial applications that balance reasoning quality and API call costs;
2. Real-time interaction systems: Scenarios where chatbots/real-time assistants need fast responses;
3. Multi-level reasoning tasks: Complex tasks that dynamically adjust reasoning strategies for different subtasks. This framework provides a feasible path for fine-grained control of LLM reasoning and is expected to be implemented in more practical scenarios in the future.