TurboCpp: A High-Performance CPU LLM Inference Engine Implemented in Pure C++17

TurboCpp is a pure C++17-implemented, zero-dependency LLM inference engine optimized for CPU environments. It leverages AVX2+FMA instruction set acceleration, multi-level quantization, memory mapping, and Grouped Query Attention (GQA) to achieve efficient inference, addressing deployment bottlenecks in edge, embedded, or resource-constrained scenarios where GPU or complex dependencies are unavailable.

With the rapid development of LLMs, inference deployment has become a key bottleneck for AI applications. Most existing solutions rely on GPU acceleration or complex dependency stacks, making them difficult to deploy on edge devices, embedded systems, or resource-limited environments. TurboCpp was developed to provide a zero-dependency, lightweight yet fully functional LLM inference solution based entirely on C++17 standards.

TurboCpp uses modern x86-64 processor features like AVX2 and FMA SIMD instructions to parallelize compute-intensive operations (e.g., matrix multiplication) for near-hardware-limit efficiency. It also implements multi-level quantization strategies: Q4/Q8 weight compression (8x/4x memory reduction), TurboQuant-style KV cache compression (4/3-bit), and dynamic quantization/dequantization to balance precision and performance.

TurboCpp supports Grouped Query Attention (GQA), which reduces KV cache memory usage by sharing key-value heads across multiple query heads, critical for long-context inference. It also uses memory mapping (mmap) to load model weights, enabling fast startup (even for large models), efficient memory usage (on-demand paging), and multi-process shared memory. Additionally, it includes a built-in BPE tokenizer supporting Llama/GPT series, eliminating external dependencies.

TurboCpp achieves token generation speeds of several to tens of tokens per second on modern AVX2-supported CPUs (Intel Haswell+, AMD Zen+). Its applicable scenarios include: edge computing local AI assistants, GPU-less server inference services, embedded/IoT smart interactions, privacy-sensitive offline inference, and rapid prototyping/testing.

As an open-source CPU inference engine, TurboCpp lowers the barrier to LLM technology access for developers without high-end GPU resources. Its clean code structure serves as an excellent learning resource for understanding LLM inference, quantization, and SIMD optimization. It demonstrates that well-engineered CPUs can handle substantial LLM inference tasks.

TurboCpp exemplifies CPU-side LLM inference optimization. Through pure C++17 implementation, AVX2+FMA acceleration, multi-level quantization, GQA support, and memory mapping, it delivers efficient inference with zero dependencies. It is both a practical tool and a technical answer to running LLMs in resource-constrained environments.