IntelliM: A Localized LLM Inference Launcher Built for Intel Arc GPUs

IntelliM is an interactive launcher designed for local large language model (LLM) inference, built on llama.cpp. It supports multi-backend parallelism, named configurations, KV cache precision selection, and persistent prompt caching. Notably, it is optimized for Intel Arc Battlemage series GPUs while maintaining backend agnosticism to support various hardware platforms. Its core philosophy is 'config as code', enabling users to quickly switch inference scenarios via named config files and command-line parameters.

With the rapid development of LLMs, more developers and researchers want to run models locally for lower latency, better privacy, and controllable costs. However, local inference faces challenges: complex configuration of different hardware backends (CUDA, ROCm, Vulkan, SYCL), tedious model parameter tuning, and difficult context window management. IntelliM was created to address these issues. It is a llama.cpp-based interactive launcher optimized for Intel Arc Battlemage GPUs, while remaining backend-agnostic to support multiple hardware platforms. Its core philosophy is 'config as code', allowing users to quickly switch inference scenarios via named config files and command-line parameters.

Multi-backend Parallel Support

IntelliM supports parallel existence of multiple backends. Users can maintain Vulkan, SYCL (Intel oneAPI), CUDA, or ROCm versions of llama.cpp builds in the same system and switch via simple command-line parameters. All backends are defined in the builds.conf registry; adding a new backend requires just one line of configuration. This design is suitable for workstations with multiple GPUs from different vendors or scenarios needing to switch between development and production environments.

Intelligent Interactive Mode

Running intellm without parameters enters an interactive wizard guiding users through: 1. Selecting a backend (from registered builds), 2. Choosing a mode (chat, server, bench), 3. Selecting a model (local GGUF or Hugging Face download), 4. Configuring context window (auto-reads model's max context length),5. Selecting KV cache precision (f16, q8_0, q4_0),6. Enabling prompt cache.

Named Configuration System

IntelliM allows saving common configurations as named files in configs/ (key-value format). Users can load presets via intellm --config <name>. default.conf loads automatically if no config is specified; --interactive bypasses default to enter interactive mode.

Persistent Prompt Cache

For repetitive tasks (code assistant, document QA), IntelliM provides persistent prompt cache stored in KVCACHE_DIR (enabled via --prompt-cache), reducing preprocessing time and improving response speed.

GGUF Metadata Reading

IntelliM includes gguf-ctx.py to read the model's training context length directly from the GGUF file header without external libraries, enabling intelligent context window recommendations.

Environment Isolation & Auto Activation

For Intel SYCL backend, env-sycl.sh automatically loads oneAPI environment variables, solving the problem of SYCL builds needing specific environments to run.

Storage Optimization Suggestions

For low-latency storage (like Optane SSD): 1. Store model files on fast storage (mmap loading, low cold start delay),2. Configure swap partition on the same device (kernel swaps anonymous KV pages to ~10μs latency storage),3. Point KVCACHE_DIR to this device (instant prompt cache snapshot loading). Recommended sysctl configs: vm.swappiness=100 and vm.vfs_cache_pressure=50.

Intel Arc Battlemage Optimization

IntelliM was initially developed for Intel Arc Pro B70 (Battlemage architecture) with specific optimizations: 1. SYCL build uses -DGGML_SYCL_DEVICE_ARCH=bmg_g31 (B580 uses bmg_g21),2. Vulkan backend outperforms SYCL in prompt processing for MoE/Mamba models,3. SYCL is better for dense Transformer models. Users are advised to use intellm --mode bench to compare backend performance on actual models.

KV Cache Offloading Research

The project is researching KV cache offloading to extend context windows beyond VRAM to system memory or persistent storage. Related docs are in docs/research/, including a 2026 May comprehensive offloading plan. This could enable consumer GPUs to handle long-context tasks.

Developer Workflows

IntelliM is suitable for:1. Local AI assistant development (quickly switch models/parameters via named configs),2. CI/CD integration (--list-json output for automation),3. Multi-hardware testing (verify model performance across backends).

Hugging Face Integration

Users can directly download models from Hugging Face using syntax: hf:用户名/仓库名:文件名. Example: intellm --build vulkan --mode chat --model hf:bartowski/Qwen2.5-3B-Instruct-GGUF:Q4_K_M. This simplifies model acquisition without manual downloads.

IntelliM represents the trend of local LLM inference tools moving toward specialization and scenarioization. It is not just a simple startup script but a complete local inference workflow solution, covering environment configuration, model management, and performance optimization. For Intel Arc GPU users, it provides out-of-box optimization; for others, its backend-agnostic design is reference-worthy. With advancements in KV cache offloading research, IntelliM is expected to further lower hardware barriers for local LLM deployment, allowing more developers to experience cutting-edge AI on personal workstations.