Prefix Cache Evolve: Using LLM to Guide Program Evolution for Optimizing Inference Services

Title: Prefix Cache Evolve: Using LLM to Guide Program Evolution for Optimizing Inference Services Abstract: An exploratory research benchmark that tests whether large language models can guide program evolution to automatically discover efficient heuristic strategies for inference services, focusing on the admission and eviction strategies of Prefix KV cache. Keywords: KV cache, inference optimization, program evolution, LLM meta-learning, cache strategy, automated machine learning, large model inference Original Author/Maintainer: ptuls Source Platform: GitHub Original Title: prefix-cache-evolve Original Link: https://github.com/ptuls/prefix-cache-evolve Source Publication Time/Update Time: 2026-06-07T13:11:11Z

Core Viewpoint: The Prefix Cache Evolve project combines the search capability of genetic algorithms with the code generation ability of LLMs to build a program evolution framework. It explores using LLMs to guide program evolution to automatically discover better Prefix KV cache management strategies, aiming to solve the problem that traditional manually designed strategies are difficult to adapt to complex and changing workloads, and verify the feasibility of the meta-learning paradigm of AI optimizing AI.

Project Background and Motivation

In large language model inference services, KV cache management is a key factor affecting performance and cost. When processing long sequences, the admission and eviction strategies of Prefix KV cache directly relate to inference latency and memory utilization. Traditional methods rely on manually designed heuristic strategies, but fixed rules are difficult to achieve optimal results when facing complex and changing workloads. This project proposes an innovative idea: using large language models to guide program evolution, automatically discovering better cache management strategies, and combining genetic algorithms with LLM code generation capabilities to explore the possibility of automatically optimizing inference services.

Technical Principle: LLM-Guided Program Evolution Framework

The core of the project is a program evolution framework, with steps as follows:

1. Define candidate cache management strategies (represented by executable code);
2. LLM acts as an "evolution engine" to analyze performance data of current strategies and identify their advantages and disadvantages;
3. LLM generates improvement plans and new strategy code;
4. New strategies are added to the population, and genetic operations such as selection, crossover, and mutation are performed;
5. Iterate cyclically until a satisfactory strategy is found or the iteration limit is reached. This meta-learning paradigm of "AI optimizing AI" is expected to discover clever strategies that human experts may not think of.

Challenges of Prefix KV Cache

Prefix KV cache is a key optimization for long-text inference: when processing multi-turn dialogues or long documents, maintaining the KV state of previous tokens can avoid repeated calculations, but designing strategies faces multiple challenges:

• Complex and changing workload access patterns (sharing long prefixes or being completely different);
• Need to balance cache hit rate and memory usage;
• KV representation sizes vary across models, so strategies need generality; Manually designing optimal strategies is extremely difficult.

Experimental Design and Evaluation Methods

The project provides a reproducible research benchmark:

• Simulate real inference service scenarios (request sequences of different lengths and sharing patterns);
• Evaluation metrics: cache hit rate, average inference latency, peak memory usage;
• Support comparison with multiple baseline strategies (LRU, LFU, LLM-specific strategies);
• Record complete evolution trajectory (strategy code per generation, performance metrics, LLM improvement suggestions), providing materials for understanding LLM optimization ideas.

Research Significance and Potential Impact

• Beyond cache optimization: Verify the feasibility of LLM as a general optimizer, opening up new directions for AutoML;
• Cost savings: Automatically discovered strategies can bring significant resource savings to inference service providers (even a 5% efficiency improvement is considerable in large-scale deployments);
• Reveal new opportunities: Evolutionary strategies may discover optimization points that humans have not noticed.

Limitations and Future Directions

Limitations

• High computational cost of LLM-guided evolution (a large number of API calls or local computing power);
• Convergence and interpretability of the evolution process need in-depth research;
• Generalization ability of strategies across different models/workloads needs verification.

Future Directions

• Introduce more efficient evolutionary algorithms to reduce the number of LLM calls;
• Combine reinforcement learning to allow strategies to continuously optimize in real environments;
• Expand to more complex inference optimization problems (batch scheduling, quantization strategy selection, etc.).