Lumina: An Adaptive Memory Operating System for Apple Silicon, Redefining KV Cache Management for Edge LLM Inference

Lumina is a research codebase targeting the Apple Silicon platform, focusing on adaptive KV Cache management under feasibility constraints. Its core innovation lies in the introduction of the "backend-induced optimality gap" concept, which quantifies the performance difference between theoretically optimal strategies and actually executable strategies on real backends, providing a new analytical framework and experimental toolset for memory optimization in edge LLM inference.

With the growing demand for LLM deployment on edge devices, Apple Silicon has become a popular platform due to its unified memory architecture and Neural Engine. However, KV Cache memory bloat during long-context inference remains a performance bottleneck. Traditional strategies ignore physical constraints of real backends (e.g., MLX-LM), leading to theoretically excellent strategies that are either unexecutable in deployment or cause unexpected performance losses. Bridging the gap between theory and practice is the core challenge.

Lumina aims to systematically measure and narrow the "backend-induced optimality gap", which is mathematically expressed as Gap(s) = Score(a*_A, s) - Score(a*_F, s) (where a*_A is the theoretically optimal strategy, a*_F is the backend-executable optimal strategy, and s is the runtime state). This framework clearly distinguishes between theoretical and practically feasible strategies, providing a measurable target for optimization. The project name "Lumina" symbolizes bringing insight to edge inference.

Lumina includes the following core components:

1. KV Cache action definitions (allocation, recycling, compression, etc.)
2. Analytical memory estimation tools (based on model architecture and sequence length)
3. Backend feasible set classification (determining whether a strategy is executable)
4. MLX-LM capability probing (for Apple Silicon's MLX framework)
5. macOS telemetry collection (memory pressure, GPU utilization, etc.)
6. Runtime strategy selection primitives
7. Optimality gap analysis tools
8. Memory soak auxiliary tools (simulating high-load scenarios)

Lumina experiments require clear labeling of strategy execution states:

• real: Executed on real backends, producing measurable performance data
• backend_infeasible: Unexecutable under backend constraints
• simulated: Simulated evaluation without real execution It is stipulated that simulated results must not be mixed with real results to ensure data purity and credibility.

• For inference engine developers: Identify and quantify backend limitations, guiding iterative optimization directions
• For deployment engineers: Help select feasible optimization strategies to avoid resource waste
• For the academic community: Provide a rigorous experimental framework and terminology system to promote research standardization and reproducibility

Current limitations: Focuses only on Apple Silicon and MLX-LM backend. Future directions:

1. Expand to more hardware (NVIDIA GPUs, AMD accelerators) and frameworks (vLLM, TensorRT-LLM)
2. Develop adaptive strategy selection algorithms
3. Establish a feasibility database
4. Combine model architecture research for cache-friendly design

Lumina provides a new perspective and tools for KV Cache management in edge LLM inference through the "backend-induced optimality gap", emphasizing the importance of acknowledging physical backend constraints. For LLM developers on Apple Silicon, it is not just a toolset but also a mindset shift: from "what strategy is best" to "what strategy is truly feasible and optimal in my environment".