CoDE-Stop: A New Method for Dynamically Optimizing Large Model Inference Efficiency via Confidence Dynamics

The University of Maryland research team proposes the CoDE-Stop method, which achieves intelligent early stopping by monitoring the confidence dynamics of intermediate answers during the reasoning process. It can reduce token consumption by 25-50% while maintaining accuracy. This method requires no additional training and can be directly integrated into existing inference models.

In recent years, large inference models (such as OpenAI's o-series and DeepSeek-R1) solve complex problems by generating lengthy thought chains. However, overly long reasoning leads to significant computational overhead and may cause 'overthinking' which reduces accuracy. Traditional fixed-length truncation is crude, and rule-based methods struggle to adapt to problems of varying difficulty. The core question is when the model should stop reasoning and output the answer.

The research team found: 1. Correct reasoning trajectories reach high confidence early and remain stable; 2. Incorrect reasoning produces lengthy and unreliable trajectories with large confidence fluctuations. Based on these findings, they proposed the CoDE-Stop (Confidence Dynamics Early Stop) method, which can be integrated into existing models without additional training.

The core mechanism of CoDE-Stop is real-time monitoring of confidence changes in intermediate answers: 1. Intermediate answer extraction: Regularly extract candidate answers (e.g., mathematical values, multiple-choice options) from generated text; 2. Confidence calculation: Compute confidence scores using the model's own token probability distribution; 3. Dynamic stopping decision: Analyze the trend of confidence changes and trigger a stop signal when the threshold is reached and remains stable.

Evaluations on tasks such as mathematical reasoning (GSM8K, MATH) and scientific Q&A (Science QA) show: 1. Token usage is reduced by 25-50% while maintaining accuracy; 2. Better accuracy-computation trade-off; 3. Deployment without training required. Performance is stable across models of different architectures, indicating that confidence dynamics are a universal feature of reasoning.

CoDE-Stop is of great significance for the deployment of inference models: 1. Reduce inference costs (save tokens in online tutoring and code generation scenarios); 2. Improve user experience (get answers faster, avoid distraction from overly long thought chains); 3. Provide signals for model optimization (identify problem types prone to overthinking).

CoDE-Stop has limitations: 1. Task dependence (confidence thresholds need adjustment for different tasks); 2. Multimodal expansion (needs adaptation to vision-language reasoning scenarios); 3. Integration with model training (explore incorporating confidence signals into the training process).

CoDE-Stop leverages the model's own confidence signals to significantly reduce inference costs without sacrificing accuracy, providing a practical tool for large-scale deployment of inference models. The paper is published on arXiv:2604.04930, where you can find complete technical details and experimental results.