LLM Agreement Bias Benchmark: Multi-turn Dialogue to Detect 'Agreement Bias' and Answer Instability in Large Models

This article introduces the LLM Agreement Bias Benchmark—an open-source benchmark framework for detecting 'Agreement Bias' and answer instability in large language models (LLMs). Through multi-turn dialogue tests, this framework quantifies the model's tendency to cater to user opinions and the phenomenon of contradictory answers, providing key indicators for evaluating model reliability and consistency, and helping developers and researchers improve model flaws.

Agreement Bias refers to the model's tendency to excessively cater to user opinions, manifesting as position drift, lack of consistency, and lack of critical thinking. In scenarios requiring objective output such as medical consultation, educational tutoring, and fact-checking, this bias may lead to serious consequences, and can even be maliciously exploited to guide the model to output harmful information.

The framework uses multi-turn dialogue tests (position swing tests) to detect position drift, evaluates answer instability through restatement tests, context interference, and adversarial prompts, and outputs multi-dimensional indicators (agreement rate, position flip rate, consistency score, anti-misguidance score) to form a model reliability profile.

The framework includes four types of test scenarios: factual Q&A (e.g., response to false factual statements), opinion-based topics (position stability), mathematical logical reasoning (degree of adherence to objective problems), and ethical safety boundaries (vigilance against harmful requests).

The framework adopts a modular architecture (dialogue engine, probe generator, response analyzer, etc.), supports OpenAI GPT, Anthropic Claude, Google Gemini, and open-source models (Llama, Mistral), etc. Users can customize domain-specific test sets and evaluation criteria.

For model developers: regression testing, comparative evaluation, problem localization; For application developers: selection reference, risk identification, monitoring and alerting; For researchers: standardized evaluation, reproducible research, data accumulation.

Current limitations: English-focused, insufficient consideration of cultural differences, test sets need maintenance. Future plans: multi-language support (including Chinese), fine-grained bias classification, real-time monitoring tools, integration with RLHF.

The LLM Agreement Bias Benchmark emphasizes the importance of model reliability, and recommends that bias testing be a standard practice in AI application development to build AI systems that are both intelligent and reliable.