3D-CoS: A New Paradigm for 3D Reconstruction Based on Visual Language Model Code Synthesis

Core Insights 3D-CoS (3D Code Synthesis) proposes a new paradigm of generating 3D assets as executable Blender code. It improves generation quality through blueprint planning, RAG retrieval, few-shot demonstrations, and a component-level agent workflow, and demonstrates unique advantages in editability and local modification. Source Information

• Original Authors: arXiv authors
• Source Platform: arXiv
• Original Title: 3D-CoS: A New 3D Reconstruction Paradigm Based on VLM Code Synthesis
• Link: http://arxiv.org/abs/2606.10478v1
• Publication Date: 2026-06-09

Current mainstream 3D reconstruction and editing systems rely on implicit (e.g., NeRF) or explicit (e.g., point clouds, meshes) representations. While they offer high rendering fidelity, they have fundamental limitations:

• Mesh: Requires direct manipulation of vertices/faces, has a high technical threshold, and lacks semantic interpretability (e.g., modifying chair legs requires locating corresponding vertices);
• NeRF: Modification requires retraining or complex inverse rendering, making precise local adjustments difficult;
• Point Cloud: Lacks topological information, editing easily leads to geometric discontinuities.

3D-CoS represents 3D assets as Blender-executable Python code, bringing three key advantages:

1. Interpretability: Code has clear semantics, allowing humans to understand the 3D object construction process;
2. Editability: Adjusting code parameters (e.g., cylinder height) enables precise shape control;
3. Local Modification Capability: Directly modify the code of the corresponding component (e.g., chair leg length) without affecting other parts, solving the pain points of traditional representations.

The research team designed a structured workflow to guide VLMs in generating code:

• Blueprint Planning: First generate a sequence of high-level steps (e.g., chair construction: seat → legs → backrest → assembly);
• RAG Enhancement: Retrieve Blender API documentation during generation to improve code correctness;
• Few-shot Demonstrations: Provide examples of complex geometric operations to help VLMs map geometric concepts to code;
• Component-level Agents: Divide and conquer—each component is generated by a dedicated agent, and the main agent coordinates assembly relationships.

Experimental comparison between code and point cloud editing:

• Point Cloud Editing: Needs to identify the point set corresponding to chair legs, which easily leads to cracks at joints or accidental damage to adjacent components;
• Code Editing: Directly modify chair leg code parameters—semantically clear, precise, and maintains geometric continuity; Results show: Code representation is superior in edit fidelity and preservation of unedited areas.

Evaluation of VLMs (open-source like LLaVA/Qwen-VL, closed-source like GPT-4V/Claude) capability boundaries:

• Limitations: Insufficient understanding of 3D spatial relationships, difficulty handling complex topologies, incomplete mastery of Blender API;
• Effect of Enhancement Strategies: RAG improves API task accuracy; few-shot examples help with geometric learning; blueprint planning optimizes the rationality of generated structures.

Application Prospects:

• Content Creation: Natural language-generated prototypes + code refinement;
• Industrial Design: Parametric product definition, easy version control;
• Education: Interpretable 3D modeling learning materials. Future Directions:
• Extend to platforms like Maya/3ds Max;
• Support complex operations like surface modeling and physical simulation;
• Optimize VLM architecture for 3D code generation;
• Integrate code representation with other 3D representations like NeRF and Gaussian splatting.