Doomma: A Pure Vision-Driven Local Multimodal AI Agent for Playing DOOM

Doomma is an AI agent that plays the DOOM game entirely based on the visual capabilities of the local multimodal large model Gemma 4. It does not rely on any heuristic rules or handcrafted features; instead, it makes independent decisions frame by frame only through observing the game screen and HUD information, demonstrating the potential of end-to-end visual-action learning. This project is maintained by mseeks and open-sourced on GitHub (link: https://github.com/mseeks/doomma). It uses a containerized architecture and supports local execution (e.g., Apple Silicon Metal acceleration).

Unlike traditional game AI, Doomma does not use any heuristic scaffolding (predefined paths, handcrafted combat strategies, etc.) at all. It only relies on the Gemma4 multimodal model to learn spatial perception (enemy positions, walls, aiming, etc.) from pixels. Its core design principles include:

1. End-to-end visual learning: All spatial information comes from raw pixels;
2. Minimized context engineering: Only the most recent frames, minimal HUD information, and recent action trajectories are passed;
3. Frame-by-frame independent decision-making: No complex internal state machines, close to human real-time reaction mode.

Doomma uses an architecture of three containers plus a host Ollama:

• doom_env service: Runs the VizDoom framework, exposes a ZMQ server in synchronous PLAYER mode, and transmits JPEG frames and HUD variables;
• agent service: The brain of the system, which polls environment frames → builds context → calls the Gemma model for decision-making → executes actions → sends telemetry data to the dashboard;
• dashboard service: A FastAPI-based web interface that receives telemetry data via WebSocket and displays the screen via MJPEG stream;
• Ollama (host machine): Runs natively (with Metal acceleration for Apple Silicon), and containers access it via host.docker.internal:11434 to avoid GPU access issues.

Doomma supports rich configurations (via .env and config.yaml):

• .env file: MODEL (default Gemma4), SCENARIO (VizDoom scenarios like basic, etc.);
• config.yaml: sampling.think (whether to reason before acting), vision.frames (number of frames per tick), tic_skip (number of ticks to advance per decision), etc. Health check mechanisms include:
• Behavior probes: Detect stagnation, decision halts, surges in parsing failures, etc.;
• Dashboard probes: Verify that the MJPEG stream and WebSocket are working properly. Additionally, a demo capture tool is provided to generate GIF/MP4 files.

The tech stack uses modern tools: ruff (formatting/checking), mypy (strict typing), pytest (testing), uv (dependency management). The test design is clever, using in-memory mock objects to replace service dependencies, allowing it to run without Docker/Ollama/VizDoom. The project's value lies in:

1. Demonstrating the potential of pure vision-driven real-time decision-making;
2. Local-first architecture (privacy protection, low latency);
3. The dashboard displays the reasoning process in real time, improving interpretability;
4. Containerization best practices (handling GPU access issues).

Current limitations: Only supports macOS M-series and Linux NVIDIA GPUs; trade-off between model decision speed and performance (larger models make better decisions but are slower). Future directions:

• Support more game environments;
• Introduce memory and learning mechanisms;
• Multi-agent collaboration;
• Enhance long-term planning capabilities.