Consensia: Enabling Large Language Models to Be Trustworthy Consensus Arbitrators

The Consensia project explores the possibility of large language models (LLMs) serving as consensus arbitrators. By coordinating multiple expert roles (security, performance, maintainability, etc.) to conduct structured debates, it aims to reach explainable and auditable software engineering decisions, addressing the transparency and trustworthiness issues of single AI model decisions.

As LLMs are increasingly applied in software engineering, single-model decisions lack transparency and auditability. Complex technical decisions require cross-validation of multi-dimensional expertise (e.g., security experts focus on vulnerabilities, performance experts on efficiency). Consensia is designed to address this challenge by having LLMs act as "arbitration judges" to coordinate expert debates and reach explainable consensus.

Expert Roles

The system defines roles for specialized domains such as security, performance, maintainability, architecture, and testing. Each role has independent prompts to ensure comprehensive discussions.

Judge Role

1. Host the debate process;
2. Identify conflicting viewpoints and request clarification;
3. Synthesize opinions based on the weight of arguments;
4. Generate explainable rulings with reasoning processes.

Backend Services (FastAPI)

• Role orchestration engine: dynamically manage expert role templates;
• Debate session management: track speech history and interactions;
• Judge ruling logic: implement algorithms for viewpoint clustering, conflict detection, etc.;
• LLM abstraction: support OpenAI/Gemini and local simulation;
• RESTful API: provide functions like session creation and review submission.

Frontend Interface (React+Vite+Tailwind)

Includes functions such as debate dashboard, viewpoint comparison, ruling details, and history records.

Deployment Methods

Supports local development and Docker Compose, with parameters configured via .env files.

1. Code Review Enhancement: Multiple experts identify risks/bottlenecks/code smells; the judge provides priority repair suggestions and explanations;
2. Technical Solution Review: Simulate architecture committee evaluations of pros and cons of options, output structured decisions and migration paths;
3. Open Source Contribution Audit: Assist maintainers in quickly identifying PR issues and lowering the barrier for new contributors to participate.

Methodological Contributions

• Practical path for explainable AI: separate expert viewpoint generation and consensus formation;
• Formalization of crowdsourced intelligence: transform collective wisdom into computable processes;
• Exploration of role engineering: the same model can play complementary professional roles.

Limitations

• Expert roles need to be manually defined;
• Lack of quantitative standards for consensus quality;
• High cost of multi-round discussions;
• Systemic biases in the base model may affect all experts.

1. Integration of CV and trait library: personalized expert roles;
2. Debate history and role memory: form a long-term consistent "personality";
3. Structured reasoning and confidence: enhance logical deduction and probability assessment of rulings;
4. Human-AI collaboration mode: AI and human experts discuss together, and the judge synthesizes both sides' viewpoints.

Consensia realizes the shift from "AI directly giving answers" to "AI assisting in decision-making". By simulating expert team discussions, it improves decision quality and makes AI decisions transparent and explainable. It reminds us that the best AI applications enhance human judgment rather than replace it.