New LLM Inference Optimization Solution: Practical Implementation of Semantic Cache and Context Compression Dual Engines

This article introduces the open-source project llm-inference-toolkit, which helps developers reduce LLM API call costs and solve the problem of long conversation context limitations through two core functions: semantic response caching and context compression. Built with Python, the project supports FastAPI and litellm to connect to over 100 LLM service providers, making it suitable for production environments.

With the widespread deployment of LLMs in production environments, developers face two major challenges: high API call costs (wasting resources on repeated queries) and context length limitations (truncating history leads to incoherent conversations). Traditional exact-match caching cannot handle semantically similar queries, and context compression is complex to implement.

llm-inference-toolkit is an LLM inference middleware for production environments. Its core functions include: 1. Semantic response caching (vector embedding identifies similar queries and returns cached results directly); 2. Context compression engine (intelligently compresses historical conversations while retaining key information). It supports OpenAI-compatible APIs and connects to multiple service providers via litellm.

Semantic Cache: Convert input to vector embedding (text-embedding-3-small), use cosine similarity (threshold 0.92) to judge, and return cached results for similar queries. Context Compression: When tokens exceed 80% of the window, retain system prompts and recent conversations, and replace early conversations with summaries generated by gpt-4o-mini to ensure coherence.

Architecture flow: Request → Semantic cache layer → Context compression layer → LLM service layer → Cache write → Response. The cache supports local memory (for development) and Redis (for production), with a default TTL of 3600 seconds. Parameters (such as similarity threshold, compression model, etc.) can be configured via environment variables.

Customer service robot scenario: Semantic cache hit rate exceeds 70%, significantly reducing API costs; Long document/multi-turn conversation scenario: The compression scheme retains key information, avoids amnesia caused by truncation, and maintains the conversation context.

Deployment methods: Local (uv manages dependencies, hot reload); Docker (docker-compose one-click startup including Redis); API is compatible with OpenAI format, and can be accessed by modifying the base_url only. Example scripts are provided (cache hit rate, long conversation compression, chatbot).

llm-inference-toolkit provides cost optimization and performance enhancement solutions for LLM applications. Semantic cache + context compression makes building economical and intelligent LLM applications a reality. As attention to LLM costs and context increases, such tools will become more important and are worth evaluating and integrating by production-level teams.