Qwen3-VL OnDemand: On-Demand Loading Multimodal Model Proxy

Qwen3-VL OnDemand is a lightweight proxy service designed to solve the VRAM management problem of running multimodal visual-language models (such as Qwen3-VL) locally. Through a proxy relay architecture, it achieves zero VRAM usage when idle and automatic model loading upon request, balancing the needs of fast response and GPU resource release, allowing users to flexibly use multimodal models even in environments with limited VRAM.

For users running large language models locally, VRAM management is a major pain point—especially for multimodal visual-language models (VLMs) like Qwen3-VL, which usually occupy several gigabytes of VRAM after loading. There are two dilemmas:

Resident VRAM Mode: The model remains loaded, responding quickly but occupying GPU resources, making it impossible to perform other GPU tasks simultaneously;

Manual Start/Stop Mode: Start before use and close after use, saving VRAM but being cumbersome and time-consuming to load.

The qwen3-vl-ondemand project is designed to solve this dilemma, achieving a balance of 'zero VRAM when idle and on-demand automatic loading'.

The project adopts a proxy relay architecture, with core components including:

vl-relay.py (Relay Proxy): A lightweight service written purely in Python, occupying only a few MB of memory. It listens on a port to receive requests, manages the backend model lifecycle, and transparently forwards requests;

llama-server (Backend Service): An inference service provided by llama.cpp that runs the Qwen3-VL model, occupying about 3.8GB of VRAM. It starts only when there are requests and automatically shuts down when idle for a timeout period.

This architecture decouples the 'service entry' from 'model inference'—the relay proxy runs at all times, while the backend service starts and stops on demand.

The complete request processing workflow is as follows:

Idle State: The relay proxy listens on the port, llama-server is not running, and VRAM usage is 0MB;

Request Arrival: The relay proxy detects that the backend is not running, automatically starts llama-server (takes about 1.5 seconds to load), and forwards the request;

In-Service State: llama-server remains running, subsequent requests are forwarded directly, with low response latency (about 100 tokens per second);

Idle Timeout: If there are no new requests for longer than the configured idle time (default 5 minutes), llama-server is automatically terminated to release VRAM.

This design balances the low latency of local models and the need for VRAM not to be occupied for long periods.

Key robustness design highlights in the project's engineering implementation:

1. PDEATHSIG Mechanism: Uses Linux system calls to ensure that when the parent process (relay proxy) exits, the child process (llama-server) terminates automatically, avoiding orphan processes;

2. Exec Startup Mode: start.sh uses exec to launch the relay proxy, replacing the shell process. When the terminal is closed, the relay proxy exits, triggering the termination of the child process;

3. Transparent Proxy Forwarding: Supports transparent forwarding of all HTTP methods, no need to handle API protocol details, compatible with text, visual, and other requests;

4. Pure Standard Library Implementation: vl-relay.py uses only Python standard libraries, with zero third-party dependencies, reducing deployment complexity and security risks.

On a configuration of Ryzen7 9700X + RTX3060 12GB, measured data using the Qwen3-VL-4B Q4_K_M quantized model:

Metric	Value
Model VRAM Usage	~2.4GB
KV Cache VRAM (8K Context)	~1.2GB
Compute Buffer	~0.3GB
Total VRAM Usage	~3.8GB
Cold Start Time	~1.5 seconds
Text Generation Speed	~100 tokens/second
Idle VRAM Usage	0MB

This solution is feasible on consumer-grade graphics cards, with acceptable cold start latency and zero VRAM usage when idle to release GPU resources.

Comparison with existing solutions:

Solution	VRAM Usage	Deployment Complexity	Flexibility
This Relay Solution	On-demand ✅	One command	Full control
Ollama Resident	Always occupied	Simple	Parameter-limited
Manual llama-server	Always occupied	Manual start/stop	Full control
vLLM	Always occupied + extra overhead	Complex	Production-grade

Compared to Ollama, this solution releases VRAM when idle; compared to manual management, it automates start/stop; compared to vLLM, it is easy to deploy and suitable for individuals and small teams.

qwen3-vl-ondemand solves the VRAM management problem of local multimodal models through a proxy relay architecture, achieving zero VRAM when idle and automatic loading upon request, balancing convenience and resource release. It is suitable for:

• Personal AI workstations (limited VRAM, need to flexibly switch GPU tasks);
• Development and testing environments (occasional testing of multimodal functions);
• Coexistence of multiple models (time-sharing GPU resources).

The project is compatible with mainstream AI clients and can be extended to any multimodal GGUF model supported by llama.cpp, making it a practical solution for users with limited VRAM to experience local multimodal AI.