Decoupling Input Ambiguity: A New Method to Improve Error Prediction of Large Language Models

This paper proposes a new method to enhance the error prediction capability of large language models (LLMs) by separating input ambiguity from uncertainty quantification (UQ) signals. The study found that UQ metrics are more effective at predicting errors in unambiguous problems; after introducing ambiguity labels, error prediction performance improved by more than 10 PRR points across multiple datasets, providing practical guidance for building more reliable AI systems.

Error prediction refers to the ability to judge whether a model's output is correct, which is crucial for the reliability of AI systems. Current mainstream methods rely on UQ metrics (such as prediction entropy, confidence scores, etc.), but there is a fundamental confusion: uncertainty comes both from the model's lack of knowledge (epistemic uncertainty) and the inherent ambiguity of input problems (aleatoric uncertainty). Existing UQ methods cannot distinguish between these two sources, leading to high uncertainty signals possibly corresponding to either model errors or problem ambiguity, thus affecting prediction accuracy.

Through experiments, the research team found that UQ metrics are significantly more effective at predicting errors in unambiguous problems than in ambiguous ones. Even in datasets considered unambiguous, there is a considerable proportion of implicitly ambiguous problems, leading to an underestimation of the performance of current error prediction systems. This finding indicates that separating ambiguity from model uncertainty is key to improving error prediction performance.

To integrate ambiguity information, the study proposes two methods:

1. Gated Experts: Use two expert predictors (for unambiguous/ambiguous problems respectively), first predict the ambiguity category of the problem, then select the corresponding expert for error prediction.
2. Selective Prediction: Dynamically adjust the UQ threshold based on the ambiguity prediction result—use a stricter threshold for unambiguous problems and a looser one for ambiguous problems to avoid over-sensitivity.

The study evaluated on question-answering tasks using 6 UQ metrics, covering multiple model families, training paradigms, and standard datasets. The results show that after introducing ambiguity information, error prediction performance improved significantly—PRR scores of some UQ metrics increased by more than 10 points. Even on unambiguous datasets, ambiguity information brought performance improvements, verifying the existence of implicit ambiguity.

By systematically separating input ambiguity from model uncertainty, this study reveals a new way to improve the error prediction capability of LLMs. Simple ambiguity decoupling can bring significant performance improvements, providing a theoretical basis and practical methods for building more reliable AI systems. As AI is applied in critical fields, the ability to accurately predict model errors will become increasingly important.

Application Recommendations: When deploying UQ-based error prediction systems, consider the ambiguity characteristics of problems and avoid using a unified threshold; ambiguity labels can be obtained through manual or automatic annotation to improve the system in combination with existing UQ metrics. Future Directions: Develop more refined ambiguity classification methods, unsupervised/semi-supervised ambiguity detection technologies, and extend the decoupling strategy to more tasks such as code generation and mathematical reasoning.