Vexel: A High-Performance LLM Inference Engine Built Exclusively for Apple Silicon

Vexel is a local LLM inference engine developed by ImpossibleComputing, optimized for Apple M-series chips. It achieves extreme performance through Metal hardware acceleration, FlashAttention-2, and a custom scheduler. The project is available on GitHub (link: https://github.com/ImpossibleComputing/vexel) and was released on June 11, 2026. Its core goal is to fill the performance gap in local LLM inference frameworks on Apple Silicon and unlock the potential of M-series chips.

With the popularity of LLMs, efficient inference on consumer-grade hardware has become a key issue. Although Mac's M-series chips have powerful neural network engines, most open-source frameworks fail to fully unlock their potential. Vexel is designed exclusively for Apple Silicon; through deep optimization of Metal computing and memory management, it achieves inference performance close to the hardware limits on M1/M2/M3/M4 series chips.

Metal Hardware Acceleration and Custom Kernels

Vexel designs Metal computing kernels specifically for Apple Silicon from scratch, optimized for the unified memory architecture. It efficiently shares data between CPU and GPU, avoiding memory copy overhead.

Efficient Attention Calculation with FlashAttention-2

It implements the IO-aware FlashAttention-2 algorithm. Through block partitioning strategies and memory access optimization, it reduces the complexity of attention calculation to near-linear, fully leveraging the characteristics of high-bandwidth memory.

Continuous Batching and Event-Driven Scheduling

It adopts a continuous batching strategy, and the event-driven scheduler dynamically allocates resources to achieve more stable latency and higher throughput, solving the latency fluctuation problem of static batching.

Speculative Decoding

Supports two modes: 1. Draft model speculation (a small model generates candidate tokens, then verified by the main model); 2. Medusa speculation (no draft model needed, predicts multiple tokens in parallel via lightweight output heads, saving memory). This can increase throughput by 20%-50%.

Paged KV Cache

Divides memory into fixed blocks and allocates them on demand, similar to virtual memory management. This reduces memory waste in long-context/variable-length sequence scenarios and supports more concurrent sequences.

GGUF Format and Multi-Model Support

Compatible with the GGUF format, supports quantization levels from Q4_0 to BF16, and has been verified to support mainstream architectures such as the LLaMA family, Phi family, and Gemma2.

Command-Line Tool

Provides the vexel command-line tool, supporting subcommands: serve (start HTTP inference server), generate (one-time generation), chat (interactive chat), bench (benchmark test), tokenize (tokenization).

HTTP Server and Streaming Support

The serve subcommand can act as an OpenAI API-compatible server, supporting SSE streaming output, suitable for real-time interaction scenarios.

Go Client Library

Provides the vexel/client Go package, which encapsulates HTTP API call details and offers type-safe interfaces and streaming processing methods.

1. Memory Bandwidth Bottleneck: Apple Silicon's performance is limited by memory bandwidth; FlashAttention-2 and quantization support can alleviate this issue.
2. Batch Size Tuning: Adjust concurrency via --max-batch-size; increase batch size for throughput-priority scenarios, reduce it for latency-sensitive scenarios.
3. Context Length Management: Set a reasonable maximum context length using --context-len to avoid unnecessary memory allocation.

Vexel is a typical case of specialization and platform optimization for local LLM inference engines. It deeply taps into Apple Silicon's features and achieves near-server-level performance on consumer-grade devices. For Mac users, it allows running larger models locally or getting faster responses. As Apple iterates on M-series chips, Vexel is expected to further narrow the gap with cloud inference, providing an attractive option for developers concerned about privacy, reducing API costs, or using LLMs offline.