Running 20+ Large Models on a Single 24GB VRAM Card: The Ultimate Optimization Practice of llama-swap_homelab

The llama-swap_homelab project, open-sourced by GitHub user blockfeed, enables hot-swapping of over 20 large models on an AMD RX 7900 XTX with 24GB VRAM. This solution addresses the problem of repeated loading and unloading in multi-model deployment on consumer-grade hardware through key technologies like the llama-swap orchestrator, KV cache quantization, MTP speculative decoding, and dynamic VRAM management, providing a replicable deployment paradigm for AI applications in resource-constrained environments.

For individual developers and researchers, deploying multiple large language models (LLMs) on a single consumer-grade GPU poses challenges: Take the AMD RX 7900 XTX with 24GB VRAM as an example—it can hold a single 70B quantized model, but switching between multiple scenarios (conversation, code, reasoning, etc.) requires repeated loading and unloading, which is time-consuming and poor in experience. The llama-swap_homelab project achieves hot-swapping of over 20 model variants through the llama-swap orchestrator combined with fine-grained VRAM management strategies.

Core Architecture

As an OpenAI-compatible API gateway layer, llama-swap starts the llama-server backend on demand and reclaims resources after an idle TTL, avoiding the VRAM explosion problem caused by traditional multi-service residency. Configuration parameters are managed using a macro system.

KV Cache Quantization

Forcing the use of --cache-type-k/v q4_0 compresses KV cache from fp16 to 4-bit, reducing VRAM consumption for 100K context from 10-12GB to below 4GB, with minimal impact on output quality in chat scenarios.

MTP Speculative Decoding

Requires a special patch for llama.cpp. It accelerates generation by predicting multiple future tokens, improving throughput while maintaining quality. For example, Qwen3.6-35B-A3B-MTP occupies about 22GB VRAM for a 65K context.

Multi-Role Configuration

Parameter hot injection is implemented using the setParamsByID filter: A single model process supports five behavior configuration files. Parameters like temperature and top_p can be switched via request header aliases, with role switching completed in milliseconds (e.g., qwen36-agent is suitable for tool calls, qwen36-chat for in-depth research).

VRAM Budget Management

A hard limit of 93% VRAM utilization (about 22.3GB) is set to prevent ROCm from spilling over to system memory. The context length of each model is optimized through actual testing—for example, Qwen3.6-35B-A3B supports a 102K context and occupies 20GB, while Mistral-Small-3.2-24B occupies 20GB for a 92K context.

The project covers model families such as Qwen3.5/3.6, Mistral-Small, and GLM-4.7, each with targeted sampling parameters:

• Qwen3.6's hybrid reasoning architecture supports the --reasoning switch: Agent mode (temperature 0.7, thinking disabled) is suitable for tool calls; code mode (temperature 0.6) for code generation; thinking mode (temperature 1.0) for exploratory tasks.
• The thinking mode template syntax of Qwen3.5 and Qwen3.6 is incompatible, so dedicated macros are used for isolation to prevent configuration issues.

This solution proves that consumer-grade hardware with 24GB VRAM can support serious LLM application development. Applicable scenarios include:

1. Personal AI development workstations (frequent switching between models with different capabilities);
2. Internal LLM services for small teams (limited budget and diverse needs);
3. Edge deployment scenarios (single-card multi-tenant resource sharing). The project open-sources configurations, system service definitions, and MTP patch building processes, providing a replicable deployment paradigm for consumer-grade LLMs.

The core insight of the llama-swap_homelab project: Through the combination of orchestration layer + quantization + speculative decoding, the potential of consumer-grade GPUs far exceeds expectations, challenging the inherent perception that 'large models require large VRAM' and opening up new paths for AI applications in resource-constrained environments. For developers who want to build powerful AI capabilities locally, it is an extremely valuable practical guide.