Chain-of-Models: Mitigating Cognitive Biases in LLM Evaluation via Cross-Model-Family Reasoning Chain Auditing

Chain-of-Models is a research result from ICLR 2026. Aiming at the cognitive bias issues in LLM-as-judge scenarios, it proposes to effectively mitigate them through multi-model chain reasoning and cross-family auditing. This article will introduce this innovative solution and its practical significance from aspects such as background, methodology, key findings, and technical implementation.

LLM-as-judge has been widely used in scenarios such as RLHF and automated model evaluation, but there are various cognitive biases: authority bias (preferring content that cites authorities), conformity bias (tending to accept widely agreed-upon views), position bias (influenced by the order of answers), and distraction bias (being distracted by irrelevant professional information). These biases affect the fairness of evaluation and even amplify model training deviations.

The core idea of Chain-of-Models is to use LLMs from cross-model families to sequentially audit the reasoning process of previous models, rather than just aggregating final answers. Unlike traditional majority voting, it focuses on auditing the reasoning process and can identify the root causes of biases, whereas majority voting may fail under certain biases.

The study reveals several key findings: 1. An optimized 2-model chain performs better than a 6-model chain (16.3% improvement in authority bias tasks); 2. Model selection is more important than chain length (complementary bias resistance characteristics are needed); 3.Diversity voting accuracy drops to 0% under conformity bias;4.Naive chaining may propagate biases (careful selection of review models is required).

The project provides a complete code implementation: an evaluation framework (supporting multiple bias tests), a model DNA extraction script (quantifying functional differences), and pluggable skills (integrated into LLM Agent). Developers can run evaluations via the command line and get started quickly using precomputed data.

The study covers mainstream model families, including Qwen2.5 series, GPT-4o series, DeepSeek series, GLM-5, MiniMax-M2.5, Kimi-K2.5, and other domestic large models, ensuring the wide applicability of the conclusions.

The limitation of the current method is the verifiability bottleneck (improvement in subjective tasks is slow due to lack of ground truth). Future directions include automatic construction of optimal model chains, dynamic chain length adjustment, and training of specialized audit models.

Developers should note:1. Do not blindly trust the evaluation of a single model;2. Majority voting is not sufficient to solve biases—auditing the reasoning process is needed;3. Model selection is more important than quantity;4. Bias resistance is an important consideration in model selection. Chain-of-Models provides a practical bias mitigation solution.