Project CANVAS: A Generative AI System for Real-Time Control of Game Worlds Using Natural Language

Explore how Project CANVAS combines generative AI with Unreal Engine 5 to enable natural language-driven real-time scene editing and intelligent combat systems, pioneering a new paradigm in game development.

• Original Author/Maintainer: KAAZamapiloT
• Source Platform: GitHub
• Original Title: Project-CANVAS
• Original Link: https://github.com/KAAZamapiloT/Project-CANVAS
• Publication Date: June 5, 2026

---

Imagine a game scene like this: you say to the game, "Turn this forest into a burning ruin", and the next second the environment in front of you actually changes—trees wither, flames rise, smoke fills the air. Or you say, "Make the enemies smarter", and the AI opponents immediately adjust their tactics, starting to flank, ambush, and coordinate attacks.

This is no longer a plot from science fiction. Project CANVAS is turning this vision into reality—it combines the powerful capabilities of generative AI with the rendering prowess of Unreal Engine 5 to create a game world that can be controlled in real time using natural language.

---

Project CANVAS uses a unique dual-pipeline architecture to handle two different types of AI tasks respectively:

This pipeline is responsible for understanding players' natural language commands and converting them into actual changes in the game world. Its workflow is roughly as follows:

1. Semantic Understanding: Use large language models (LLMs) to parse players' natural language input and extract intents and parameters
2. Scene Mapping: Map abstract descriptions to specific assets and operations in the game engine
3. Real-Time Generation: Call generative AI models (such as diffusion models) to create or modify visual elements like textures, materials, and lighting
4. Engine Integration: Apply changes to the game scene via Unreal Engine 5's Blueprint system or C++ API

For example, when a player says "Create a medieval castle in a storm", the system will:

• Identify key words: storm, medieval, castle
• Generate or retrieve corresponding 3D models and materials
• Adjust weather system parameters (rainfall, lightning, clouds)
• Modify lighting and atmosphere settings
• Complete the real-time scene transformation within a few seconds

The second pipeline focuses on the core of game AI—combat decision-making. Unlike traditional game AI which uses preset behavior trees or finite state machines, Project CANVAS adopts a data-driven approach:

1. Environmental Perception: AI continuously analyzes the battlefield environment, including terrain, cover positions, and the distribution of teammates and enemies
2. Situation Assessment: Calculate the optimal action strategy based on the current situation
3. Dynamic Adaptation: Adjust difficulty and tactical style in real time based on player behavior
4. Cooperative Combat: Multiple AI units can coordinate actions to execute complex team tactics

This design makes enemies no longer "puppets" acting according to fixed scripts, but "opponents" that can make intelligent decisions based on battlefield situations.

---

Generative AI models usually have huge computational demands and are difficult to use in real-time games. Project CANVAS solves this problem through the following strategies:

• Layered Generation: Divide scene changes into immediate response layers (e.g., lighting, weather) and progressive generation layers (e.g., complex geometry)
• Precomputation and Caching: Pre-generate and cache common scene elements to reduce real-time computational pressure
• Asynchronous Processing: Offload heavy generation tasks to background threads, keeping the main thread at a smooth frame rate
• LOD Strategy: Dynamically adjust generation quality based on distance and importance

In addition to text input, the system can also support:

• Voice commands (converted to text via speech recognition)
• Gesture control (combined with computer vision)
• Traditional controller input as a supplement