High-Performance LLM Inference Engine in Pure C: The Optimization Journey from 0.2 to 30 Tokens/sec

The fast-llm-inference engine implemented in pure C achieves a performance leap from the Python baseline of 0.2 tokens/sec to 25-30 tokens/sec on the Phi-3 Mini model, with a speedup of 125-150x, using techniques like INT8 pre-dequantization, EAGLE-3 speculative decoding, Medusa multi-token prediction, and AVX2 vectorization. This project returns to low-level optimization, providing an efficient solution for LLM inference in CPU environments.

Deep learning frameworks in the Python ecosystem (e.g., PyTorch, TensorFlow) compromise on performance for generality and ease of use, leading to bottlenecks like interpreter overhead, dynamic type checking, and abstraction layers in CPU inference scenarios. The fast-llm-inference project reimplements the inference engine in pure C to eliminate unnecessary abstraction overhead and deeply optimize for modern CPU architectures, aiming for extreme performance.

1. INT8 Weight Pre-Dequantization: When loading the model, Q4 weights are pre-dequantized to INT8 format, reducing runtime bitwise operation overhead and bringing about a ~10x speedup;
2. EAGLE-3 Speculative Decoding: Uses a 4-layer draft model to generate candidate tokens, with the main model verifying in parallel. On average, 3 tokens are accepted, taking 70ms (traditional sequential generation takes 150ms), leading to ~2.5x speedup;
3. Medusa Multi-Token Prediction: Adds multiple prediction heads to predict future tokens simultaneously without a separate draft model, ~2x speedup;
4. AVX2 SIMD Vectorization: Uses Intel AVX2 instruction set to implement vectorized INT8 dot product operations, improving matrix multiplication throughput;
5. Cache-Friendly Memory Layout: Optimizes data layout to match CPU cache hierarchy, reducing cache misses, and combines OpenMP multi-threading to tap into multi-core potential.

Optimization Stage	Tokens/sec	Speedup
Python Baseline	0.2	1x
+ AVX2 Kernel	0.7	3.5x
+ INT8 Pre-Dequantization	6.0	30x
+ EAGLE-3 Speculative Decoding	15.0	75x
+ Medusa Multi-Token	25-30	125-150x
These optimizations work synergistically: for example, INT8 pre-dequantization provides an ideal data format for SIMD vectorization, and speculative decoding amplifies the benefits of low-level computation optimizations.

The project codebase is about 5000 lines with a clear structure:

• Core inference engine implemented in pure C with no Python dependencies;
• Supports GGUF format (compatible with llama.cpp);
• Supports Q2/Q4/Q8 quantization;
• Automatically detects CPU instruction sets like AVX2, AVX-512, AMX;
• Provides a single binary executable to simplify deployment. In an Intel i7 (AVX2) 16-thread configuration, the measured speed reaches 8.5-30 tokens/sec.

1. Edge Computing Devices: Enables usable LLM inference on embedded devices without GPUs;
2. Cost-Sensitive Applications: Reduces cloud inference costs by using CPU clusters to handle loads;
3. Privacy-First Scenarios: Local CPU inference avoids data uploads, protecting user privacy;
4. Research and Education: Clear C implementation facilitates learning the underlying mechanisms of LLM inference.

fast-llm-inference demonstrates the potential of system-level programming in AI inference optimization, achieving performance close to professional hardware on consumer CPUs through low-level optimizations. The project will focus on optimizing assembly-level kernels, more advanced quantization formats, operator fusion, etc., in the future, which is expected to further improve performance, compatibility, and ease of use.