Vision-Guided Iterative Optimization: A New Paradigm for Frontend Code Generation

This paper proposes a new vision-guided paradigm for frontend code generation, with the core being the use of vision-language models (VLMs) as automatic evaluators to provide structured feedback, enabling iterative optimization of frontend code. This method achieves a 17.8% performance improvement on the WebDev Arena dataset, and internalizes some evaluation capabilities into the code generation model via LoRA fine-tuning, reducing reliance on multi-round reasoning.

Frontend code generation needs to balance functionality and visual quality, which traditional single-round reasoning models struggle to meet; current human-in-the-loop multi-stage optimization is high-cost and hard to scale, and frontend human feedback is more time-consuming and requires professional knowledge, hence the need for an automated visual feedback mechanism.

The framework flow is: code generation model generates initial frontend code → renders into screenshots → VLM evaluator compares screenshots with requirements and provides structured improvement feedback → generation model iteratively optimizes based on feedback. VLMs have both visual understanding and text description capabilities, and can identify issues like layout misalignment and color mismatch, providing clear guidance.

Evaluated on the WebDev Arena dataset (containing real frontend requests), quality improved by 17.8% after three iterations; the first iteration fixes obvious functional errors and layout issues, subsequent iterations focus on visual details and experience enhancement, and the improvement pattern aligns with human development processes.

The generation model is fine-tuned with LoRA using iterative feedback data from the evaluator; results show that after fine-tuning, the model's single-round reasoning can achieve 25% of the performance gain of the optimal iterative solution, with no significant increase in token consumption, and some evaluation capabilities are successfully internalized, reducing the number of iterations needed.

Insights: Visual feedback is key to frontend optimization, and automated evaluator loops can replace human feedback; future directions: enhance the fine-grained recognition capability of VLMs, explore more efficient fine-tuning methods, and expand to visual-related code generation tasks such as mobile UI development.