refft.cpp: A High-Performance C++ Framework for LLM Inference and Training on GPU/NPU

refft.cpp is an open-source C++ framework developed by the refinefuture-ai team, designed for efficiently running large language model (LLM) inference and training on GPU/NPU backends. Through low-level optimization and compilation techniques, it addresses issues like Python performance bottlenecks in local deployment and hardware architecture differences, balancing high performance and ease of use, while supporting cross-platform deployment and various inference/training optimization strategies.

As LLM scales grow exponentially, inference and training demand extremely high computational resources. While local deployment can solve latency, cost, and data privacy issues, it faces performance bottlenecks in the Python ecosystem (interpretation overhead, dynamic type checking, GIL limitations), as well as challenges like diverse architectures of specialized accelerators (GPU/NPU) and complex programming models. Developers need to make trade-offs between performance, portability, and development efficiency.

refft.cpp uses C++ as its core, leveraging zero-cost abstractions, compile-time optimizations (C++17/20 features, template metaprogramming), SIMD instructions, and memory alignment to boost performance. It shields differences in GPU/NPU programming models through a unified abstraction for heterogeneous computing, providing cross-platform interfaces. It optimizes memory management (weight quantization, paged attention, asynchronous transfer, memory pool reuse) and reduces overhead via operator fusion and graph optimization (constant folding, dead code elimination).

In terms of inference optimization, refft.cpp supports request batching and dynamic batching to improve GPU utilization; implements speculative decoding (draft model generates candidate tokens in parallel then validates them) to accelerate autoregressive generation; and provides multiple quantization schemes (weight quantization INT8/INT4, activation quantization, KV cache quantization) to reduce model size and memory usage with almost no loss of precision.

For training support, the framework implements efficient backpropagation and gradient computation, supports distributed strategies like data parallelism, model parallelism, and pipeline parallelism, and optimizes for fine-tuning scenarios (gradient checkpointing, activation recomputation, mixed-precision training). For usability design, it draws inspiration from PyTorch API, provides intuitive tensor operations and automatic differentiation, supports Python bindings for gradual migration, and has rich example documentation.

Application scenarios include edge deployment (resource-constrained devices), high-throughput services (low latency and high concurrency), private deployment (local data centers), and research experiments (quick validation of new architectures). Comparison with similar projects: llama.cpp focuses on extreme optimization for specific models; vLLM emphasizes service-layer batch scheduling; refft.cpp provides a general low-level abstraction, supports a wider range of model types and hardware backends, and is suitable for deep customization and cross-platform deployment.

Future plans include supporting more NPU architectures and edge devices, more aggressive compilation optimizations (operator auto-tuning), more quantization schemes, and improving distributed training and federated learning support. Conclusion: Through C++ low-level optimization and modern software engineering practices, refft.cpp provides a competitive option for local/edge LLM deployment, and its value is prominent in AI infrastructure.