ORCE: An Order-Aware Alignment Method for Linguistic Confidence of Large Language Models (Introduction)

ORCE is a decoupled order-aware confidence calibration framework. Its core lies in separating the two stages of answer generation and confidence estimation, and constructing a ranking learning objective based on sampling to achieve more reliable alignment of linguistic confidence. This method significantly improves calibration performance and failure prediction capability while maintaining answer accuracy, providing an effective solution to the problem of overconfidence in large language models.

Practical Dilemmas in Confidence Estimation

Large language models often exhibit overconfidence, making high-confidence statements even when answers are wrong, which brings risks to applications in fields like healthcare and law. Linguistic confidence (e.g., "very certain") is an intuitive solution, with the advantage of obtaining uncertainty signals without accessing the model's internal token probabilities. However, existing methods couple answer generation with confidence generation, leading to conflicting optimization objectives: generating vague answers to express conservative confidence, or being forced to have high confidence for clear answers, which limits the independent optimization of both.

Decoupled Architecture Design of ORCE

ORCE adopts a two-stage decoupled framework:

1. Answer Generation Stage: The model focuses solely on generating answers to ensure quality is not disturbed by confidence objectives;
2. Confidence Estimation Stage: Takes fixed question-answer pairs as input and focuses on calibration objectives. Advantages of decoupling: no conflicting optimization objectives, utilization of complete answer information, and independent iterative improvement of the confidence module.

Innovation in Order-Aware Learning Objectives

ORCE abandons traditional absolute accuracy optimization and shifts to relative ranking objectives: For each question, generate multiple candidate answers, and train the confidence module to assign higher confidence to answers that are more likely to be correct. This objective reduces learning difficulty (no need for precise probabilities) and is robust to label noise. Through a ranking-based reinforcement learning reward mechanism, the model is guided to distinguish the confidence levels of answers.

Analysis of Experimental Evaluation Results

On reasoning and knowledge-intensive benchmarks, ORCE performs significantly:

• Calibration Performance: The correlation between confidence and actual correctness rate is better than baselines, and the calibration curve is closer to the ideal diagonal;
• Failure Prediction: Can accurately screen low-confidence answers for manual review while maintaining the pass rate of high-confidence answers;
• Answer Accuracy: Due to the decoupled architecture, it does not decrease due to confidence training and is better than coupled methods.

Comparative Advantages and Application Scenarios of ORCE

Comparison with Probabilistic Confidence

Linguistic confidence has strong universality (applicable to black-box APIs), complements token probability-based methods, and can be used in combination to improve accuracy.

Practical Applications

• Q&A systems: Decide whether to answer directly or involve manual intervention;
• RAG systems: Evaluate whether documents support reliable answers;
• Multi-step reasoning: Identify weak links and re-reason.

Limitations and Future Directions of ORCE

Limitations

• Increased inference overhead (two forward passes);
• Coarse granularity of linguistic confidence (discrete expression);
• Dependence on answer correctness annotations (challenging for open-domain tasks).

Future Directions

• Low-latency optimization;
• More fine-grained linguistic confidence expression;
• Exploration of weakly supervised/unsupervised confidence calibration.

Conclusion: The Significance of ORCE for Trustworthy AI

ORCE provides an elegant solution for confidence calibration of large models. Its decoupled design and order-aware objectives not only improve the alignment effect of linguistic confidence but also provide insights into the separation and combination of learning objectives. Today, as AI integrates into decision-making processes, accurate confidence expression is a key component of trustworthy AI.