Inference Across Metal: Streaming Inference of 27B-Parameter Large Language Models on Apple Silicon

Inference Across Metal is a high-performance inference framework based on Swift and Metal. It allows Apple Silicon devices with 16GB of memory to smoothly run 27B-parameter large language models by using custom kernels and streaming processing techniques, breaking through hardware limitations. The project is maintained by MidasMulli, with source code hosted on GitHub (link: https://github.com/MidasMulli/inference-across-metal), and was released on May 30, 2026.

Local deployment of large language models faces severe hardware thresholds: a 27B-parameter model requires approximately 54GB of storage space for weights when using FP16 precision, and even 4-bit quantization requires about 13.5GB. For entry-level Apple Silicon devices with 16GB memory (such as MacBook Pro, Mac mini), traditional solutions are hard to handle, requiring the purchase of high-end models with 32GB/64GB memory, which raises the barrier to use.

Core Technical Architecture

1. Metal Custom Kernels: Optimize key operators like matrix multiplication and attention for Apple Silicon GPUs, leveraging the advantages of the unified memory architecture to achieve efficient data sharing between CPU and GPU.
2. GatedDeltaNet Support: Natively supports this efficient architecture, reducing complexity through gating mechanisms and incremental computation to adapt to resource-constrained environments.
3. Streaming KV Cache Management: Dynamically schedule cache data in chunks, retaining only the currently necessary computation state to reduce memory overhead.
4. Attention Mechanism Optimization: Reduce data transfer overhead and improve the utilization of computing units through kernel fusion and memory access pattern optimization.

Working Principle of Streaming Inference

Split the model into small chunks, load/unload on demand during inference, and adopt an overlapping load-compute-unload mode: when processing the Nth layer, asynchronously load parameters of the (N+1)th layer and release parameters of the (N-1)th layer, solving problems such as inter-layer dependencies, memory fragmentation, prefetch optimization, and fault-tolerant rollback.

Practical Application Scenarios

• Edge AI Development: Debug 27B models locally, reduce costs and ensure data privacy;
• Academic Research: Experiment freely on personal devices to accelerate research progress;
• Offline Intelligent Assistant: Available without a network, protecting conversation privacy;
• Educational Popularization: Allow students in resource-limited areas to access advanced AI technologies.

Performance and Limitations

• The first token generation time is slightly longer, while the speed of subsequent tokens is acceptable;
• Model compatibility needs to be confirmed;
• High-speed SSD devices provide a better experience.

Inference Across Metal proves that a well-designed software architecture can significantly reduce the hardware requirements for local deployment of large models, similar to how deep learning framework optimizations enabled consumer-grade GPUs to train neural networks. The unified memory architecture of Apple Silicon avoids data copy overhead between CPU and GPU, providing a hardware foundation for efficient inference.

Future Outlook

We look forward to further advancements in model compression (quantization, pruning, distillation) and inference optimization (kernel fusion, dynamic scheduling) technologies to enhance the experience of large models on resource-constrained devices.

Recommendations

• Use high-speed SSDs to improve streaming inference experience;
• Pay attention to the project's compatibility support for more model architectures;
• Participate in community contributions to promote the improvement of the project.