QRAF: A High-Performance Local LLM Inference Runtime Built for Apple Silicon

QRAF is a local LLM inference runtime written in C++. It is deeply optimized for Apple Silicon, supports conversion from HuggingFace, GGUF, and Safetensors formats, and provides a lightweight, high-performance local inference solution that balances efficiency and privacy protection.

The popularity of M-series chips has made Apple Silicon an ideal platform for local AI inference, but existing frameworks are either bloated or do not fully utilize hardware features. QRAF aims to be lightweight, high-performance, and easy to deploy. It uses C++ to ensure efficiency and reserve space for cross-platform expansion, reducing memory usage and startup latency compared to Python solutions.

QRAF uses a proprietary model format optimized for inference, supporting efficient memory mapping and on-demand loading. Compared to HuggingFace PyTorch/GGUF formats, its loading speed and memory efficiency are significantly improved.

It leverages Metal Performance Shaders (MPS) and the Accelerate framework to unleash the performance of M-series GPUs and neural engines; the unified memory architecture support avoids the overhead of CPU-GPU data copying.

Supports import of mainstream formats:

• HuggingFace Transformers: Directly load PyTorch/Safetensors weights
• GGUF: Compatible with quantized models in the llama.cpp ecosystem
• Safetensors: Avoids pickle security risks Users can seamlessly migrate existing model assets.

It achieves inference performance close to the hardware limits on M1/M2/M3 chips, with low latency suitable for interactive applications; local inference ensures data is processed on the device, protecting privacy and data sovereignty.

Applicable scenarios:

1. Personal knowledge management (private knowledge base Q&A)
2. Development assistance (IDE code suggestions)
3. AI capability deployment for macOS applications
4. Model experiments (verifying inference effects) Recommendations: Start with 7B models, explore the balance of quantization, and refer to the repository documentation to get started.

QRAF enriches the local LLM ecosystem, with differentiation in Apple Silicon native optimization and a concise architecture; it may expand to more hardware platforms and model architectures in the future; its open-source nature promotes community participation and drives the progress of local AI.