Agave: A High-Performance Large Language Model Inference Engine Built from Scratch Using Zig Language

Agave is a high-performance LLM inference engine written entirely in Zig, with zero external machine learning dependencies. It implements all kernels, quantization, and model logic from scratch, supporting 7 model architectures, 5 computation backends, over 20 quantization types, and features like layered KV cache, multi-modal vision, HTTP server, and interactive REPL. This post breaks down its design, performance, features, and use cases.

Most LLM inference engines rely on existing frameworks like PyTorch or TensorFlow, but Agave takes a different path—building everything from scratch without external ML libraries. This approach offers advantages: smaller binary size, finer performance control, clearer code structure, and native multi-platform support. It supports 7 mainstream model architectures, 5 backends, 20+ quantization types, plus multi-modal vision, HTTP server, and REPL.

Zero External Dependencies: All matrix operations, attention mechanisms, activation functions, quantization (20+ formats), and model logic are implemented natively. Supported Models: 7 architectures including Gemma 3/4, Qwen3.5, GPT-OSS, etc. (Gemma 3/4 support multi-modal via SigLIP encoders). Multi-Backend: CPU (SIMD for x86/ARM), Metal (Apple Silicon), Vulkan, CUDA, ROCm (compiled only if enabled). Layered KV Cache: VRAM+RAM+SSD tiers for handling long contexts (e.g., --kv-tiers vram+ram+ssd).

Agave delivers strong performance—on Apple Silicon M4 Pro (Metal backend), Qwen3.5 0.8B Q8_0 reaches ~183 tokens/sec (1.2-1.7x faster than llama.cpp). Key optimizations:

• Batch prefill with chunked GEMM and FlashAttention-2.
• Zig's compile-time computing for optimized code.
• Zero-overhead abstractions (no external library calls).
• Precise memory management via Zig's manual allocation.

Model Management: Pull GGUF models from HuggingFace (e.g., agave pull Qwen/Qwen3.5-0.8B-GGUF with breakpoint resume). Interactive REPL: Multi-round dialogue with commands like /clear, /system, /stats. HTTP Server: OpenAI/Anthropic API-compatible endpoints (e.g., /v1/chat/completions), web chat, Prometheus metrics. Multi-Modal: Gemma3/4 support image input (e.g., --image photo.png). KV Cache Optimizations: TurboQuant (2/3/4-bit), eviction strategies (norm, TriAttention), calibration.

Build with Zig: zig build (generates ReleaseFast/Debug versions), zig build test for tests. Customization options:

• Disable unused models to reduce binary size (1.8MB full → 0.75MB minimal).
• Enable/disable backends (CPU, Metal, etc.) at compile time.

Use Scenarios: Edge deployment (small size), privacy-sensitive apps (local inference), API services (OpenAI-compatible), research (clear code), cross-platform (mobile to server GPUs). Status: Active development—some models have partial support, output quality being optimized. Prospects: Shows Zig's potential in system programming; could compete with llama.cpp for performance/control-focused use cases.