PlayCoder: Making GUI Code Generated by Large Models Truly Runnable

The research team proposes the PlayCoder framework, which significantly improves the ability of large language models to generate playable GUI applications through multi-agent collaboration and iterative repair, solving the problem that traditional evaluation metrics cannot capture interactive logic errors. Additionally, they developed the PlayEval benchmark suite and the Play@k evaluation metric, redefining the quality assessment of GUI code generation and providing a feasible path for AI-assisted GUI development.

Large language models have made significant progress in code generation, but their performance in GUI applications (especially game-like interaction-intensive programs) is far from practical. GUI is an event-driven, state-intensive interactive system where user operations trigger complex state transitions. Traditional code evaluation methods (such as unit testing and compilation checks) cannot capture interactive logic errors, leading to programs that may compile successfully but fail to interact normally.

To address the evaluation dilemma, the research team developed the PlayEval benchmark suite, which includes 43 multi-language (Python, TypeScript, JavaScript) GUI applications covering six major categories. The core innovation is the Play@k metric, which focuses on whether at least one of the k generated candidate codes allows users to complete the full 'play' process; they also developed the PlayTester agent, which simulates real user interactions to execute the full process, automatically detects logical violations, and enables large-scale evaluation.

Tests on 10 advanced code generation models found that although their compilation rates are excellent, the Play@3 metric is close to zero—even with three attempts, the generated code can hardly support users to complete the full interaction process, exposing the models' blind spots in understanding interactive logic, state management, and event flow, while traditional metrics ignore the usability dimension.

The PlayCoder framework transforms GUI code generation into a closed-loop iterative process of 'generate-evaluate-repair', consisting of three collaborative agents:

1. Generation Agent: Generates initial GUI code based on requirements
2. Evaluation Agent: Performs end-to-end playability testing using PlayTester
3. Repair Agent: Modifies logical errors based on feedback The multi-agents are divided into specialized roles and learn from errors through closed-loop iteration to improve quality.

Experimental results show that PlayCoder significantly improves functional correctness and semantic alignment on both open-source and closed-source models, with Exec@3 reaching 38.1% and Play@3 reaching 20.3%—although the absolute values are not high, it achieves an order-of-magnitude improvement over the baseline (close to zero), and can also detect and fix 'silent logical bugs' that are missed by traditional metrics.

PlayCoder has important practical significance for GUI development: game developers can quickly generate interactive prototypes, the education field can help students understand event-driven programming, and accessibility technology can lower development thresholds. Future explorations are needed: better modeling of interactive logic, understanding the subtle differences in user experience, and expanding to more complex GUI scenarios. PlayCoder indicates that a continuously iterative and self-improving generation system is the key to reliable AI-assisted GUI development.