Splinter: A Lock-Free Zero-Copy Shared Memory KV and Vector Storage Library That Eliminates Socket and Memcpy Overhead for LLM Inference

Splinter Core Guide

Splinter is a minimalist, high-performance key-value (KV) and vector storage system that enables zero-latency inter-process communication via shared memory and atomic operations. With only 766 lines of core code, it supports millions of operations per second and 768-dimensional vector storage, providing a new architectural approach for local LLM inference and data-intensive applications—saying goodbye to socket connections and memcpy overhead, and directly sharing memory in user space.

Project Background and Core Issues

In modern AI applications, traditional IPC solutions (such as Redis, SQLite, and various vector databases) rely on kernel network protocol stacks, socket connections, serialization/deserialization, and memory copies, which become performance bottlenecks in latency-sensitive scenarios. Splinter was born out of the developer's frustration with existing toolchains: the architectural limitations of traditional databases (unnecessary coupling between kernel network layers and arbitration services) cannot be resolved through tuning. Its core idea is: local inter-process communication can directly use shared memory in user space, bypassing the layers of kernel wrapping.

Architecture Design: Swimming Pool Metaphor

Splinter's architecture can be analogized to a swimming pool:

• Pre-allocated lanes: Create a fixed memory pool and divide it into equal-length lanes during initialization; no dynamic memory allocation needed;
• Diving board (lock-free access): Each lane is equipped with an atomic sequence epoch mechanism. 32 processes can access different lanes simultaneously, returning EAGAIN for retry when conflicts occur (non-blocking);
• Signal pulse: Instant notification when data is updated (like epoll mechanism), so processes don't need to poll;
• Zero copy: Readers directly access original memory without serialization/transfer.

Additionally, passive design (no daemon process, only shared memory area), DRYD principle (data publishing instead of sending, direct access), static geometric structure (avoids fragmentation and garbage collection), lock-free atomic operations (seqlock supports in-place operations like INCR/DECR), and NUMA affinity (write speed up to 500 million times per second) are core architectural highlights.

Key Technical Features

• Signal system: Supports 64 independent signal groups (based on epoll). The Bloom tag function allows filtering specific signals to avoid being overwhelmed by massive updates;
• Extensible fragment system: Dynamically load C logic fragments (e.g., DSP, ANN search, inference modules) via insmod to keep the core streamlined;
• Built-in inference engine: Sidecar embedded engine using quantized Nomic Text model (.gguf) and llama.cpp wrapper, enabling vector inference directly at the storage layer;
• Lua integration: splinter_cli and splinterctl support Lua scripts for flexible handling of complex data flows.

Performance Data and Scalability

• Throughput: Over 3.2 million operations per second on consumer-grade hardware;
• Latency: Based on memfd/mmap, reaching L3 cache speed level;
• Scalability: Multi-reader multi-writer (MRMW) semantics;
• Vector support: Native 768-dimensional vectors, optimized for Nomic v2/LLM embeddings;
• Code size: Core library has only 766 lines, with hot paths resident in instruction cache.

Applicable Scenarios and Comparison with Traditional Solutions

Applicable Scenarios:

• Local LLM inference cache (eliminates socket/memcpy overhead for engines like llama.cpp);
• High-frequency data collection (real-time storage of physical experiment and sensor data streams);
• Multi-language process collaboration (TypeScript/Rust/Python/Go shared data);
• Embedded and edge computing (high-performance storage in resource-constrained environments).

Comparison with Traditional Vector Databases:

Feature	Splinter	Traditional Vector Databases
Transport Layer	memfd gracefully degrades to mmap (L3 speed)	TCP/gRPC (network protocol stack)
Daemon Process	None (passive)	Active service (heavyweight)
Memory Usage	Static and predictable	Dynamic and unstable
Code Complexity	766 lines of C (core)	Over 100,000 lines

Build and Platform Support

Platforms: Modern GNU/Linux; Windows via WSL (with slight performance loss); macOS requires a workaround (no memfd support).

Optional Dependencies:

• NUMA (libnuma-dev): Build with WITH_NUMA=1;
• Lua (lua5.4-dev): Build with WITH_LUA=1;
• llama.cpp: Build with WITH_LLAMA=1 (enables inference fragments);
• Valgrind: Build with WITH_VALGRIND=1 (test integration).

Pure KV Mode: Build with WITH_EMBEDDINGS=0 (no vector partitions).

Conclusion and Project Information

Splinter represents a system development attitude that returns to efficiency: in an era where CPU cycles and memory bandwidth are considered infinite, it reminds us that local IPC can bypass the socket layer and kernel arbitration. It is not a one-size-fits-all solution, but a tool for engineers pursuing extreme latency.

Project author Tim Post (former Stack Overflow employee) says: Splinter assumes 'informed intent'—it does not try to be smarter than the kernel, but provides metadata and memory areas and then gets out of the way.

The project uses the Apache 2.0 license, code is hosted on GitHub, and the documentation site is under construction.