Vision2Web: A New Benchmark for Hierarchical Evaluation of AI Web Development Capabilities

Vision2Web is a hierarchical benchmark for AI web development capabilities, covering 193 real-world tasks from static UI generation to full-stack development. It proposes an automated validation paradigm combining GUI agents and VLM judges, revealing that current models still have significant gaps in full-stack development. Its core design philosophy is to cover the complete spectrum of web development from simple to complex, helping to accurately evaluate AI's ability to assist or replace humans in real-world scenarios.

AI Web Development Evaluation Dilemmas

Existing AI web development evaluations have three major limitations:

• Single dimension: Only tests UI fidelity or functional correctness
• Simplified scenarios: Uses manually designed simple pages instead of real complex websites
• Static evaluation: Only focuses on final output, ignoring interaction and iteration during development

These gaps make it impossible to accurately determine the actual capability boundaries of AI in real-world web development.

Vision2Web: Three-Tier Evaluation System

Tier 1: Static UI to Code Generation

Generate HTML/CSS from web design mockups, testing visual understanding, code generation, and detail restoration capabilities (e.g., shadows, gradients).

Tier 2: Interactive Multi-Page Frontend Reproduction

Reproduce multi-page websites with interactive logic, including navigation logic, interactive components (buttons/forms/popups), state management, etc., testing the understanding and implementation of interactive logic.

Tier 3: Long-Range Full-Stack Web Development

Covers end-to-end tasks of frontend + backend + database + API + user authentication, testing long-range planning and multi-tech-stack integration capabilities.

Dataset Composition and Automated Validation

Dataset

• 193 real-world tasks (16 categories: e-commerce/blog/dashboard, etc.)
• 918 mockups (UI generation tasks)
• 1255 test cases (functional validation)

Automated Validation Paradigm

GUI Agent Validator: Simulates user operations (clicks/forms/scrolling) to validate interactive logic; VLM Judge: Compares visual effects, analyzes layout and aesthetics; The two work together to form a complete evaluation loop covering both functionality and visual aspects.

Experimental Findings

Tier Performance Differences

• Static UI generation: Advanced models perform well
• Interactive frontend: Performance drops significantly (complex state/navigation issues)
• Full-stack development: Models generally struggle (backend logic/database/API errors)

Typical Weaknesses

• Insufficient long-range planning ability
• Lack of cross-tier consistency (frontend-backend/database mismatches)
• Missing edge case handling
• Insufficient understanding of design intent

Model Differences

Different models have strengths and weaknesses in visual understanding and logical reasoning; there is no all-around model.

Implications for AI-Assisted Development

Tiered Capability Matching

AI is most suitable for assisting static UI generation; complex interaction/full-stack tasks require manual modification and refinement.

Human-AI Collaboration Model

Recommend the 'AI generation + manual review + iterative optimization' model; Vision2Web can automate the inspection process to accelerate iteration.

Benchmark Contributions

Fills evaluation gaps: systematically covers capability spectrum, supported by real data, automated validation, and diagnoses capability shortcomings.

Limitations and Future Directions

Limitations

• Limited tech stack coverage (focused on mainstream frameworks)
• Difficulties in evaluating dynamic content
• Insufficient evaluation of accessibility

Future Improvements

• Enhance long-range planning capabilities
• Improve cross-tier consistency
• Strengthen understanding of design intent

Vision2Web lays the foundation for AI web development evaluation; in the future, AI will be more suitable as a 'co-pilot' for developers, and key challenges such as long-range planning need to be addressed.