Reading
Air.rs is a Rust-based LLM inference engine that uses the S.L.I.P. (Slipstream Layer Inference Protocol) protocol. Through memory mapping and layer streaming technology, it enables running large language models with over 70B parameters on consumer GPUs with only 24GB of VRAM.
Section 01
Air.rs is a Rust-based LLM inference engine that uses the S.L.I.P. (Slipstream Layer Inference Protocol) protocol. Through memory mapping and layer streaming technology, it enables running large language models with over 70B parameters on consumer GPUs with only 24GB of VRAM.
Section 02
Current large language model (LLM) inference faces a core challenge: model size far exceeds VRAM capacity. Take a 70B parameter model as an example: using FP16 precision requires 140 GB of GPU memory; even when quantized to Q4, it still needs 35 GB—which already exceeds the 24GB VRAM limit of the RTX 4090.
Existing solutions often come with painful trade-offs:
Section 03
Air.rs implements the S.L.I.P. (Slipstream Layer Inference Protocol) protocol. Its core idea is: GGUF files are loaded via memory mapping (mmap), but at any time, only the quantized weights of one layer reside in physical RAM. Weights remain compressed in GGUF block format, and QMatMul performs dequantization during matrix multiplication.
Section 04
| Model Size | Traditional Loading | Air.rs Layer Streaming |
|---|---|---|
| 7B | ~4 GB | ~400 MB |
| 70B | ~40 GB | ~1.5 GB |
This design makes it possible to run 70B+ models on a single consumer GPU.
Section 05
STRIX (Streamed Tensor Residence & Intelligent eXchange) is a GPU offloading protocol that supports running 70B+ models on consumer GPUs. It manages a three-level memory hierarchy (VRAM → RAM → Storage) and has an intelligent eviction scoring mechanism.
Key components include:
Section 06
| Category | Feature |
|---|---|
| Core | Layer streaming inference — only one Transformer block is in memory at a time |
| Quantization | Weights remain in GGUF block format; QMatMul dequantizes during matmul |
| File Format | GGUF, SafeTensors, PyTorch (.bin/.pt), ONNX — auto-detected |
| Memory | madvise / PrefetchVirtualMemory page control + mmap storage HAL |
| KV Cache | Hierarchical KV cache with RAM/VRAM switching and LRU eviction |
| Pipeline | Adaptive ring buffer pipeline — overlaps NVMe reads, PCIe, GPU |
| API | OpenAI-compatible /v1/chat/completions (SSE streaming) |
| Computing | NVIDIA CUDA + Vulkan (staging transfers) + Apple Metal GPU backends |
| Decoding | Speculative decoding (draft validation acceleration, 2-3x speedup) |
| Scheduling | Continuous batch request scheduler |
| Sampling | Temperature, top-p, top-k, repetition penalty, min-p |
| Tokenizer | BPE Tokenizer built from GGUF vocabulary |
| Model Hub | Download models from Hugging Face with SHA-256 verification |
| Monitoring | Real-time TUI dashboard + Prometheus-compatible metrics |
| Templates | Jinja2-style chat template engine (ChatML, Llama, Mistral, etc.) |
| Bindings | Optional PyO3 Python bindings |
Section 07
All subsystems have been implemented and tested (468 tests, 0 warnings, 0 failures). The project compiles on three platforms and has production-grade GPU backends. E2E validation passed with a real Llama 3.2 3B Q8 GGUF model.
Section 08
Keep going with more reads from the same topic.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.
This project explores how to use Large Language Models (LLMs) to detect logical vulnerabilities in Ethereum smart contracts, especially business logic flaws that are difficult for traditional static analysis tools to capture, such as economic manipulation and execution order errors.
This article details how to quickly deploy a virtual machine equipped with 4 NVIDIA V100 GPUs on Azure using Terraform to enable local inference of large language models with over 70B parameters. It covers a complete practical guide from infrastructure deployment, Ollama/vLLM dual-engine comparison testing, to cost optimization and actual performance data.