llm-server: An Intelligent Launcher to Simplify llama.cpp Deployment

llm-server is an intelligent launcher for llama.cpp/ik_llama.cpp that automates hardware detection, multi-GPU optimization, AI self-tuning, and more. Users can start a large model service with just one line of command without manually adjusting complex parameters. It supports features like smart GGUF downloading, vision model handling, auto update/rollback, and seamless backend switching.

llama.cpp is a popular local LLM running solution, but configuring its parameters (GPU layers, tensor split, KV cache quantization, threads) is challenging. For multi-GPU users (especially heterogeneous setups), issues like layer allocation, tensor split ratio, and MoE expert placement require deep understanding of llama.cpp's internals. llm-server was created to address these pain points.

Key features include:

1. Auto hardware detection & optimization: Handles 0-8+ GPUs (homogeneous/heterogeneous), optimizes layer allocation and tensor split based on memory and PCIe bandwidth.
2. Smart GGUF downloader: Recommends optimal quantization levels based on system memory when using the --download parameter.
3. Vision model support: Auto handles mmproj files (checks compatibility, downloads if missing) for multi-modal models.
4. Auto update & rollback: Safely updates backends with backup and rollback on failure.
5. Auto backend switch: Falls back to mainline llama.cpp if ik_llama.cpp doesn't support the model.
6. Crash recovery: Auto restarts with a backoff strategy and logs data for tuning.
7. Multi-instance support: Uses --gpus and --ram-budget to run multiple instances on the same system.
8. Fused tensor support: Enables optimized kernels for fused tensors in GGUF models.

The --ai-tune feature lets the model optimize its own running parameters:

1. Starts with a heuristic config as baseline and tests performance.
2. Sends hardware config, GGUF metadata, help output, and baseline results to the model.
3. The model suggests improved parameters in JSON format.
4. Applies the parameters, tests again, repeats 10 iterations, selects the best config, and caches it. Test results: On Qwen3.5-27B Q4_K_M with RTX3090Ti+4070+3060, generation speed increased by 54% (25.94→40.05 tok/s) and prompt processing speed by 52% (150→228 tok/s). This process is offline with no external API needed.

Performance Comparison: Manual llama.cpp config requires many flags, while llm-server uses one line and often achieves better performance. Best Practices:

• Prototype: Use llm-server --download <model-repo> to auto get a suitable quantized model.
• Production: Run llm-server model.gguf --ai-tune first to cache the optimized config.
• Multi-tenant: Use --gpus and --ram-budget to limit resources for shared servers.
• Vision: Add --vision for multi-modal models.

Limitations:

1. First AI tuning takes tens of minutes (one-time cost).
2. Cached configs are hardware-specific; changing GPUs requires re-tuning.
3. Some new architectures may need backend updates. Future Directions:

• Dynamic config adjustment based on workload.
• Support for AMD ROCm and Intel Arc.
• Distributed cluster deployment.
• Integration with model training workflows.