Reading
m5-infer is an MLX inference engine optimized specifically for Apple Silicon. It achieves a decoding speed of 40 tokens per second on the M5 MacBook Air, which is a 4.5x improvement over Ollama. Through innovative technologies like cross-turn state persistence and hybrid speculative decoding, it significantly reduces latency while maintaining output quality.
Section 01
m5-infer is an MLX inference engine optimized specifically for Apple Silicon. It achieves a decoding speed of 40 tokens per second on the M5 MacBook Air, which is a 4.5x improvement over Ollama. Through innovative technologies like cross-turn state persistence and hybrid speculative decoding, it significantly reduces latency while maintaining output quality.
Section 02
In tests with the Qwen 3.5 9B 4-bit quantized model, m5-infer shows overwhelming advantages:
| Metric | Ollama | mlx_lm.server | m5-infer v1.0.0 |
|---|---|---|---|
| Decoding Speed (tok/s) | 8.9 | 17.0 | 40.0 |
| Relative to Ollama | 1.0x | 1.9x | 4.5x |
| Relative to mlx_lm.server | 0.5x | 1.0x | 2.4x |
More impressive is the balance between latency and quality:
All tests were conducted on the same Mac, using the same model and prompts. The performance gap comes entirely from optimizations at the inference engine layer.
Section 03
m5-infer is built on Apple's MLX framework and positioned as an OpenAI-compatible HTTP inference server that can directly replace mlx_lm.server. Its core architecture is optimized around the Qwen 3.5 hybrid model (GatedDeltaNet + Full Attention), while supporting multiple model families like Qwen 2.5/3.6, Llama 3.x, Mistral, and Gemma 2/3/4 via a model family abstraction layer.
Section 04
Qwen 3.5 uses a hybrid architecture of 24 GatedDeltaNet (GDN) layers + 8 full attention layers. Traditional speculative decoding faces a critical issue at the GDN layer: when a draft token is rejected, the KV cache can be rolled back, but the GDN's recurrent state and convolution buffer have already advanced through the entire draft window, leading to state corruption. m5-infer's solution is to snapshot all GDN layers' (recurrent_state, conv_buf) into a pre-allocated tensor pool before each validation. When rejected, it recovers from the snapshot in O(1) time with zero allocation on the hot path. In practice, this brings a 35% throughput improvement (from 29 to 40 tok/s) on Qwen 3.5 9B, with an acceptance rate of about 70%.
After each generation round, m5-infer serializes the complete model state (quantized KV cache + GDN recurrent/convolution buffer) to disk, using the hash of the original bytes of the prompt prefix tokens as the key. Since the hash is based on token bytes rather than decoded text, the same system prompt and tool mode can hit the cache even with different user inputs attached. Effect: The warm-up TTFT for the 12K token tool mode is reduced from 11s to 2-3s, and the cache hit rate for typical agent workloads exceeds 90%.
Qwen 3.5's chain-of-thought is wrapped in ... tags. Common failure modes include:
m5-infer's solutions:
Effect: Structured JSON extraction task score increased from 1.40 to 7.85 (+461%), and code generation from 3.10 to 6.55 (+111%).
Qwen 3.5 has a safety alignment issue when thought mode is disabled: in long contexts (12K+) with short retrieval queries, it sometimes refuses to answer, claiming "cannot disclose authoritative information"—even if the information comes from the user's own provided content. m5-infer automatically detects long context + short query mode at the routing layer and forces thought mode to be enabled, thus bypassing this limitation. In practice, the long context retrieval success rate increased from 0/6 to 6/6.
For "simple" tokens, skip layers with minimal impact to reduce computation.
An internal speculative decoding mechanism of the model that terminates generation early when confidence is high.
Concurrently execute multiple expert paths in MoE (Mixture of Experts) models.
Metal kernel fusion via mx.compile brings about a 40% throughput improvement (from 17 to 24 tok/s).
Section 05
The table below shows the contribution of each optimization to the final performance:
| Innovation | Decoding Speed | Quality | TTFT/Latency |
|---|---|---|---|
| Hybrid Speculative Decoding | +35% | Output Equivalent | — |
| CTRSP | — | — | 12K Warm-up TTFT:11s→2-3s |
| Thought-Aware Budgeting | — | +36% Opus Score | — |
| Needle-Retrieval Heuristic | — | Long Context Retrieval:0/6→6/6 | — |
| ALS + SSEE + PES | +10-15% | — | — |
| X5-R Compiled Forward | +40% | — | Cold Start +2-5s |
| Full Stack Integration | 4.5x | +11% | 5.8x |
Section 06
m5-infer's design goal is clearly directed at production-grade Apple Silicon deployment:
Section 07
Section 08
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
Cross-generational hardware testing based on NVIDIA A10G and H100 GPUs, comparing and analyzing the throughput, latency, and concurrent scalability of three mainstream LLM inference frameworks. SGLang achieves a 3.4x performance advantage over vLLM on H100, while Ollama faces architectural bottlenecks in high-concurrency scenarios.