Agentic llama.cpp: An Enhanced Platform for Injecting Intelligent Agent Capabilities into Local LLM Inference

jahrulnr/llama.cpp is an enhanced branch of the original llama.cpp. It integrates Sidecar gateway architecture, automated operation and maintenance system, TurboQuant quantization compression, and intelligent memory system, upgrading the local LLM inference platform into an intelligent system with agentic capabilities. Key features include production-level operation support, memory bottleneck breakthrough via TurboQuant, and compatibility with Ollama APIs.

The original llama.cpp focuses on the inference engine itself but lacks support for agentic capabilities, automated operation and maintenance, and intelligent memory management required by modern AI applications. Agentic llama.cpp retains the original high-performance inference while introducing the Sidecar gateway architecture to provide a complete agentic platform for local LLM deployment, serving as both an inference engine and a production-oriented intelligent LLM operation platform.

Sidecar Gateway Mode: Uses agent-sidecar as a unified gateway (public port 16283, internal inference port 16285) for security, function expansion, and simplified operation. TurboQuant: A quantization scheme based on WHT rotation, offering KV cache compression (turbo2/3/4 with 2-4 bits, 3-6x OOM threshold improvement)and model weight quantization (TQ3_1S/TQ4_1S, 25-35% size reduction vs Q8_0). Speculative Decoding: Integrates Gemma4 MTP (30-50% throughput improvement) and Qwen3.6 NextN (24-36% performance improvement) to reduce inference latency.

This is a core feature distinguishing it from other branches. Implemented via Go Sidecar, components include Scheduler (task timing), LLMClient (calling llama-server APIs), ToolExecutor (tool execution), Notifier (desktop notifications), EventLogger (structured logs), LogRotator (daily log rotation). Built-in templates: disk/memory monitoring, Docker health check, system update check, network connectivity, log analyzer, SSL certificate monitoring, security analyzer. Agent loop mechanism: Sidecar calls LLM → LLM decides tools → execute → LLM generates report → notify if needed, enabling active diagnosis instead of passive alerts.

Based on embedding vectors, using nomic-embed-text-v2-moe for semantic vectors, cosine similarity with keyword fallback. Storage at ~/.agent/memory/, with full CRUD APIs (memory_put, memory_read, etc.). Features: category scoping, score-based LRU eviction, auto learning from operation tasks, configurable storage limits via memory-config.yaml, essential for long-running agent applications.

Ollama Compatible API: Supports /api/tags, /api/show, /api/chat, /api/version, allowing zero or minimal changes for existing Ollama clients. WebUI: SvelteKit-based interface with automation dashboard, real-time timeline, memory management, event viewer, MCP config. Dev Workflow: Commands like llama-auto start (launch platform), llama-auto status (check state), make deploy (rebuild WebUI/Sidecar), make all (full rebuild including C++ server).

Suitable for: 1. Local private deployment (agentic capabilities needed). 2. Operation automation (LLM-based system analysis and report generation).3. Resource-limited environments (TurboQuant enables larger models on limited memory).4. Long-running agent apps (memory persistence and context management).5. Production-level local inference (monitoring, logs, notifications required).

Agentic llama.cpp represents an important evolution direction for local LLM inference platforms, moving from "running" to "running well, stably, and intelligently". It integrates agentic capabilities, automation, and memory with high-performance inference via Sidecar architecture. TurboQuant breaks memory bottlenecks. For developers building production-level local LLM apps, it's a valuable project offering tools and architectural insights for adding modern AI capabilities while maintaining performance.