Vexel: A High-Performance LLM Inference Engine Built Exclusively for Apple Silicon

Vexel is an open-source LLM inference engine developed by ImpossibleComputing, optimized exclusively for Apple Silicon (M1/M2/M3/M4 series chips). It leverages Metal acceleration, FlashAttention-2, speculative decoding, and continuous batching to deliver fast local text generation. Key features include support for GGUF models, multiple deployment options, and focus on privacy/offline usability.

Source Information

• Author/Maintainer: ImpossibleComputing
• Source Platform: GitHub
• Link: https://github.com/ImpossibleComputing/vexel
• Release Time: 2026-06-11

Vexel is designed for Apple Silicon, using Metal framework to exploit M-series chips' GPU performance and unified memory architecture. It provides a high-performance solution for local LLM runs on Macs, targeting developers and researchers.

1. Metal Hardware Acceleration: Custom Metal kernels optimize GPU usage, reducing CPU-GPU data transfer overhead via Apple's unified memory.
2. FlashAttention-2: Memory-efficient attention algorithm that handles long sequences by reducing memory complexity.
3. Continuous Batching & Paged KV Cache: Event-driven scheduler supports high throughput; paged KV cache shares GPU memory across concurrent sequences.

Vexel uses two strategies to boost throughput by 20-50%:

1. Draft Model: Small draft model predicts tokens, verified by target model (configurable via --draft-model).
2. Medusa: No separate draft model; uses lightweight heads (online-trained or pre-trained) to predict multiple tokens, adapting token count based on acceptance rate.

• HTTP Server: serve command launches RESTful API/SSE streaming.
• CLI Tools: generate (one-time text), chat (interactive), tokenize (text splitting), bench (performance testing).
• Go Client: Official library vexel/client supports blocking/streaming calls.
• Runtime API: Direct access for custom pipelines (lower latency).

Model Support: GGUF format (Q4_0/Q4_K_M/Q5_K/Q6_K/Q8_0/BF16) and models like LLaMA 2/3, Mistral, Phi-2/3, Gemma 2. System Needs: macOS 14.0+ (Sonoma), Go1.22+, Xcode command line tools. Build via make build (single binary).

Vexel fills the gap for Apple Silicon LLM inference, enabling local runs (privacy/offline use). Its open-source design and flexible APIs support developers/researchers. As edge AI demand grows, Vexel will play a key role in consumer-grade local LLM deployment.