Wick: A High-Performance LLM Inference Engine Written in Pure Rust

Wick is a lightweight large language model (LLM) inference engine written in Rust. It supports GGUF format model loading, CPU/GPU hybrid inference, and multiple quantization schemes, aiming to provide a zero-dependency single static binary file.

In the ecosystem of LLM inference tools, Python has long been dominant. However, Python's runtime dependencies and deployment complexity have always been pain points in production environments.

The Wick project has taken a different path—building a native LLM inference engine from scratch using Rust, aiming to deliver extreme performance and a minimal deployment experience.

Wick's design philosophy can be summarized with three key words: lightweight, fast, zero-dependency. It strives to be a simple solution for "loading GGUF models, generating text, and making it fast." Through Rust's memory safety features and zero-cost abstractions, Wick maintains high performance while avoiding the memory safety risks of traditional C/C++ projects.

Wick implements a series of impressive technical features:

Wick natively supports the GGUF (GGML Universal File) format, which is widely used in the llama.cpp ecosystem. By loading tensors using memory-mapped technology, Wick can efficiently handle large model files, avoid unnecessary memory copies, and significantly reduce memory usage.

For CPU inference, Wick implements SIMD (Single Instruction Multiple Data) optimized computation cores, supporting AVX2 (x86_64 platform) and NEON (ARM platform) instruction sets. These low-level optimizations bring CPU inference performance close to theoretical limits, enabling a smooth inference experience even on consumer-grade hardware.

Wick implements cross-platform GPU inference support via the wgpu library. wgpu is a Rust graphics API based on the WebGPU standard, which can run on Vulkan (Linux/Windows), Metal (macOS/iOS), Direct3D 12 (Windows), and WebGPU (browser) backends. This design allows Wick to leverage GPU acceleration on almost any modern computing device.

Wick supports multiple model architectures, including:

• LLaMA Family: Mainstream open-source models like LLaMA, LLaMA 2, LLaMA 3
• LFM2 (Liquid Foundation Models): An innovative architecture combining convolution and attention mechanisms

This flexibility allows Wick to run a wide range of pre-trained models without needing to maintain separate code for each architecture.

To further improve inference efficiency, Wick supports multiple quantization schemes:

• Q4_K_M: 4-bit quantization, balancing performance and accuracy
• Q8_0: 8-bit quantization, providing higher accuracy retention

Quantization technology can compress the model size to 1/4 or even smaller than the original, making it possible to run large models on resource-constrained devices.