TurtleAI: Evaluation of Multimodal Models' Visual Reasoning Capabilities in Turtle Graphics Programming

The TurtleAI benchmark is the first to systematically evaluate the capabilities of vision-language models (VLMs) in education-oriented Turtle graphics programming tasks. Results show that even top models like GPT-4o have a success rate below 30%, with spatial reasoning and precise visual reproduction being the main bottlenecks. The data augmentation strategy proposed in the study can significantly improve model performance and provide important insights for educational AI applications.

Visual programming solves visual tasks by generating code and is widely used in productivity scenarios, but educational scenarios emphasize more on precision, interpretability, and teaching logic. As a classic introductory tool, Turtle Graphics intuitively demonstrates programming and geometric knowledge, making it an ideal platform for evaluating the educational capabilities of VLMs.

TurtleAI contains 823 educational scenario tasks, requiring models to complete a three-stage cognitive process: geometric pattern perception, spatial relationship reasoning, and Python code synthesis. Tasks are divided into basic, advanced, challenge, and expert levels according to difficulty, covering a complete spectrum of capabilities.

Evaluation results of over 20 mainstream VLMs show that the best model has a success rate of less than 30%. Failure modes include: insufficient spatial reasoning ability (e.g., incorrect calculation of internal angles of a pentagram), visual-code alignment deviation (incorrect recursive parameter setting), precision issues (angle/distance deviation), and lack of a self-verification mechanism.

The research team proposed a data generation technique based on seed sample variation, synthesizing tens of thousands of high-quality training samples. After fine-tuning Qwen2-VL-72B with this data, the success rate increased from about 15% to 35%, mainly improving the alignment between visual reasoning and code implementation.

1. Current VLMs are not yet directly applicable to educational scenarios; 2. Specialized optimization for educational scenarios is needed; 3. Existing evaluation benchmarks need to be updated; 4. A human-machine collaboration model is more realistic (models provide initial solutions, humans verify and correct).

Limitations: The task scope is limited to geometric drawing, and only Python/turtle library is supported. Future directions: Expand task types (recursion, divide and conquer, etc.), multi-language support, interactive learning research, and development of error diagnosis capabilities.