BlindBench: A Blind Testing Platform for Large Language Models Without Brand Bias

BlindBench is an open-source blind testing benchmark platform for large language models (LLMs). Its core goal is to eliminate brand bias, helping users objectively evaluate the real performance of over 100 AI models through blind testing, factual accuracy scoring, and reasoning failure classification. It not only focuses on whether model responses are correct or incorrect but also deeply analyzes the causes of failures, allowing evaluations to return to content quality itself.

The current LLM evaluation ecosystem has three major issues: brand effects interfere with user judgment, existing benchmarks are limited to narrow skill areas, and over-reliance on automated metrics lacks human subjective judgment. BlindBench's core philosophy is to eliminate brand bias through anonymous blind testing, allowing users to choose based on content quality; while systematically analyzing the causes and types of model failures.

BlindBench evaluates models from seven dimensions:

1. Model Preference: Collect real human preferences through anonymous comparative voting
2. Factual Accuracy: Extract and verify claims using GPT-4o to quantify credibility
3. Reasoning Failure Classification: Ten types (hallucination, flattery, etc.) + severity grading
4. Response Stability: Measure consistency using Jaccard similarity and cosine similarity
5. Prompt Sensitivity: Evaluate robustness to semantically equivalent prompts
6. Confidence Calibration: Correlate linguistic expressions with actual accuracy
7. Token Efficiency: Analyze the relationship between response length and quality

BlindBench uses a full-stack architecture: the frontend is deployed on GitHub Pages using React19, Vite8, and TailwindCSS v4; the backend relies on Supabase (Edge Functions + PostgreSQL17). Evaluation process: Submit prompt → Multi-model parallel response generation → Analysis pipeline (factual verification/failure classification/stability testing, etc.) → Export results as JSON/CSV. Client-side analysis functions are supported (embedding similarity, failure detection, etc.).

Built-in seed data comes from 4 Kaggle datasets, including 3700+ prompts, 7500+ responses, and 9000+ votes, covering over 180 model benchmarks, 24 cutting-edge model evaluations, ethical bias tests, etc. Users can view via the dataset browser and export in standard/enhanced formats (including derived metrics like token estimation, confidence calibration, etc.).

Security measures: Frontend uses DOMPurify to sanitize content and prevent XSS; enforces HTTPS + restricts CORS; user API keys (BYOK) are one-time use, encrypted during transmission, and not stored. Edge functions: Input validation, rate limiting (5 times per IP per minute), IP hashing. Database: Row-level security + parameterized queries to prevent injection. Privacy commitment: No cookies used, no user tracking, no analytical data collection.

Function pages:

• Arena: Blind test voting, stability/perturbation testing, embedding analysis
• Leaderboard: Blind test voting win rate ranking
• Failure Browser: Browse reasoning failure cases by type/model
• Analysis Page: Automatically generate insight reports (failure co-occurrence, hallucination rate, etc.) Local deployment: Clone the repository → Configure environment variables → Install dependencies → Start the development server (completed in 2 minutes).

BlindBench is not just a leaderboard tool but also a model diagnosis platform, providing developers with improvement directions and users with selection basis. It promotes the objectivity, transparency, and reproducibility of LLM evaluation through open-source methods, letting data speak instead of brand rhetoric, and contributing to the healthy development of the AI field.