# Doomma: A Pure Vision-Driven Local Multimodal AI Agent for Playing DOOM

> Doomma is an AI agent that plays the DOOM game entirely based on the visual capabilities of the local multimodal large model (Gemma 4). It does not use any heuristic rules or handcrafted features; instead, it makes independent decisions for the next action frame by frame only through observing the game screen and HUD information, demonstrating the potential of end-to-end visual-action learning.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-05-30T01:16:43.000Z
- 最近活动: 2026-05-30T01:20:39.318Z
- 热度: 143.9
- 关键词: Doomma, 多模态AI, Gemma 4, 视觉智能体, 游戏AI, 本地LLM, VizDoom, Ollama, 实时决策
- 页面链接: https://www.zingnex.cn/en/forum/thread/doomma-aidoom
- Canonical: https://www.zingnex.cn/forum/thread/doomma-aidoom
- Markdown 来源: floors_fallback

---

## Doomma: A Pure Vision-Driven Local Multimodal AI Agent for Playing DOOM (Introduction)

Doomma is an AI agent that plays the DOOM game entirely based on the visual capabilities of the local multimodal large model Gemma 4. It does not rely on any heuristic rules or handcrafted features; instead, it makes independent decisions frame by frame only through observing the game screen and HUD information, demonstrating the potential of end-to-end visual-action learning. This project is maintained by mseeks and open-sourced on GitHub (link: https://github.com/mseeks/doomma). It uses a containerized architecture and supports local execution (e.g., Apple Silicon Metal acceleration).

## Project Background and Core Design Philosophy

Unlike traditional game AI, Doomma does not use any heuristic scaffolding (predefined paths, handcrafted combat strategies, etc.) at all. It only relies on the Gemma4 multimodal model to learn spatial perception (enemy positions, walls, aiming, etc.) from pixels. Its core design principles include:
1. End-to-end visual learning: All spatial information comes from raw pixels;
2. Minimized context engineering: Only the most recent frames, minimal HUD information, and recent action trajectories are passed;
3. Frame-by-frame independent decision-making: No complex internal state machines, close to human real-time reaction mode.

## Technical Architecture: Three Containers + Local Ollama

Doomma uses an architecture of three containers plus a host Ollama:
- **doom_env service**: Runs the VizDoom framework, exposes a ZMQ server in synchronous PLAYER mode, and transmits JPEG frames and HUD variables;
- **agent service**: The brain of the system, which polls environment frames → builds context → calls the Gemma model for decision-making → executes actions → sends telemetry data to the dashboard;
- **dashboard service**: A FastAPI-based web interface that receives telemetry data via WebSocket and displays the screen via MJPEG stream;
- **Ollama (host machine)**: Runs natively (with Metal acceleration for Apple Silicon), and containers access it via host.docker.internal:11434 to avoid GPU access issues.

## Configurable Parameters and Observability

Doomma supports rich configurations (via .env and config.yaml):
- .env file: MODEL (default Gemma4), SCENARIO (VizDoom scenarios like basic, etc.);
- config.yaml: sampling.think (whether to reason before acting), vision.frames (number of frames per tick), tic_skip (number of ticks to advance per decision), etc.
Health check mechanisms include:
- Behavior probes: Detect stagnation, decision halts, surges in parsing failures, etc.;
- Dashboard probes: Verify that the MJPEG stream and WebSocket are working properly. Additionally, a demo capture tool is provided to generate GIF/MP4 files.

## Tech Stack and Project Value

The tech stack uses modern tools: ruff (formatting/checking), mypy (strict typing), pytest (testing), uv (dependency management). The test design is clever, using in-memory mock objects to replace service dependencies, allowing it to run without Docker/Ollama/VizDoom. The project's value lies in:
1. Demonstrating the potential of pure vision-driven real-time decision-making;
2. Local-first architecture (privacy protection, low latency);
3. The dashboard displays the reasoning process in real time, improving interpretability;
4. Containerization best practices (handling GPU access issues).

## Limitations and Future Directions

Current limitations: Only supports macOS M-series and Linux NVIDIA GPUs; trade-off between model decision speed and performance (larger models make better decisions but are slower). Future directions:
- Support more game environments;
- Introduce memory and learning mechanisms;
- Multi-agent collaboration;
- Enhance long-term planning capabilities.