Reading
A single-binary LLM inference engine written in Rust, supporting gRPC and HTTP streaming inference, multi-model concurrency, GPU acceleration, and OpenAI API compatibility.
Section 01
A single-binary LLM inference engine written in Rust, supporting gRPC and HTTP streaming inference, multi-model concurrency, GPU acceleration, and OpenAI API compatibility.
Section 02
As large language model technology becomes widespread, more and more developers and enterprises want to deploy and run LLM inference services in local environments. Ollama, as a pioneer in this field, has gained a broad user base with its simple user experience. However, as application scenarios deepen, users have put forward higher requirements for the performance, resource management, and scalability of inference engines.
Rust, with its excellent performance and memory safety, has become an ideal choice for building high-performance system-level applications. The Spindll project is built on this technology stack to create a production-ready LLM inference engine.
Section 03
Spindll is a GGUF and MLX format model inference engine natively written in Rust, open-sourced by developer Iito. The project name combines the concepts of "Spindle" and "LL(ama)", reflecting its positioning as a model management and service engine.
As a single-binary solution, Spindll can pull models from the Ollama registry or HuggingFace, manage local storage, and provide streaming inference services via gRPC and HTTP protocols. It supports concurrent loading of multiple models, memory-aware scheduling, GPU hardware acceleration, and provides an OpenAI-compatible API interface.
Notably, on Apple Silicon platforms, Spindll natively runs MLX format models via Swift bridging, while supporting GGUF format through llama.cpp, achieving seamless integration of dual backends.
Section 04
Spindll supports fetching models from multiple sources:
On Apple Silicon devices, the system automatically detects and prioritizes MLX format models, falling back to GGUF format when MLX is unavailable.
Section 05
The project adopts a pluggable inference backend design, with unified scheduling via the InferenceBackend trait:
This architectural design reserves extension points for integrating new inference engines in the future.
Section 06
Spindll supports loading multiple models into memory simultaneously and implements an LRU (Least Recently Used) eviction strategy. When the memory budget is exceeded, the system automatically unloads less frequently used models to ensure the availability of high-priority models.
The memory budget is configurable; by default, the system automatically detects available memory (on macOS, this includes the sum of free memory, inactive memory, purgeable memory, and speculative memory), and users can also manually specify the budget size (e.g., "8G").
Section 07
The system implements a continuous batching mechanism; concurrent requests for the same model share a single context via sequence IDs, significantly improving throughput.
The KV caching feature supports disk-stored prompt prefix caching, with optional ChaCha20-Poly1305 encryption to protect cached data. This not only accelerates inference speed for repeated prompts but also ensures the security of sensitive data.
Section 08
Spindll provides interfaces compatible with the OpenAI API, including:
This compatibility allows existing OpenAI clients to seamlessly migrate to Spindll, lowering the adoption barrier.
Keep going with more reads from the same topic.
Nornir MCP Server is an enterprise-level server based on the Model Context Protocol (MCP). It seamlessly integrates large language models (such as Claude) with the Nornir network automation framework, supporting natural language orchestration for multi-vendor network devices (Cisco, Arista, Juniper, etc.), and providing production-grade features like a dual-engine architecture (NAPALM + Netmiko), intelligent filtering, and a secure sandbox.
Bibliothèque Française LLM is a structured indexing and annotation project for French public domain literature designed specifically for large language models (LLMs). It integrates multiple authoritative sources such as DraCor, Common Corpus, and Wikisource, providing metadata indexing categorized by genre, author, and era, as well as in-depth annotations for dramatic texts (including characters, lines, stage directions, etc.). Its aim is to enable LLMs to efficiently read and understand classic French literary works.
Splinter is a minimalist, high-performance key-value (KV) and vector storage system enabling zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, offering a new architectural approach for local LLM inference and data-intensive applications.
Folkering OS is the world's first AI-native bare-metal operating system, entirely written in Rust no_std without relying on Linux, POSIX, or libc. It can generate commands from scratch, compile them into WASM, and run them in 10 seconds, achieving true self-evolution.