CacheProbe: A Study on Audit Attacks Against Prompt Cache Isolation in LLM Gateway APIs

Core of the CacheProbe Research

This article focuses on the prompt cache isolation security issue in LLM gateway APIs, published by Gu et al. at ICML 2025 (arXiv source, released in May 2026). The research reveals that third-party API gateways like OpenRouter may bypass the prompt cache isolation mechanisms of LLM providers, leading to cross-user data leakage risks. The proposed CacheProbe audit method is based on timing attacks, which has important warning implications for AI security in multi-tenant environments.

Over the past year, prompt caching technology in LLM inference APIs has rapidly gained popularity, reducing latency and cutting costs by reusing KV caches. However, behind this technical convenience lie security risks: when multiple tenants share a cache, time differences between cache hits and misses (timing attacks) or metadata leaks may become channels for data theft.

Mainstream LLM providers (e.g., OpenAI) implement cache isolation by account/organization, but third-party gateways (e.g., OpenRouter) use shared organizational credentials to access underlying providers, which may lead to global cache sharing—user A's prompt cache may be hit by user B, causing cross-user cache pollution and violating data isolation expectations.

The CacheProbe audit method is based on two key observations:

1. Timing Side Channel: Cache hits result in faster responses, while misses are slower; attackers can infer other users' prompt prefixes through time differences.
2. Metadata Leakage: Some implementations leak cache status (e.g., hit or miss) via response headers or error messages.

Experiments targeting OpenRouter validated three hypotheses:

1. Shared credentials cause the underlying provider to treat all gateway traffic as from the same organization.
2. Insufficient isolation of end-users in cache key logic leads to cross-user sharing.
3. Attackers can detect isolation vulnerabilities via timing analysis. The results show that gateway architectures may weaken or invalidate underlying isolation mechanisms.

• End Users: When using LLMs via gateways, prompt prefixes may be probed; sensitive information should not be placed in prefixes, and adding random prefixes is recommended.
• Gateway Operators: Need to audit isolation mechanisms, assign independent credentials, implement additional isolation layers, and disclose risks.
• LLM Providers: Strengthen cache key logic, provide fine-grained control, and accept third-party audits.

Architecture Level

1. Credential Isolation: Assign independent credentials to users.
2. Cache Namespace Isolation: Embed user identifiers in cache keys.
3. Gateway-Level Caching: Use an independent caching system.

Application Level

1. Prompt Prefix Randomization.
2. Sensitive Information Postponement.
3. Response Timing Obfuscation.

Audit and Monitoring

1. Regular audits using CacheProbe.
2. Anomaly query pattern detection.
3. Public disclosure of cache security practices.

CacheProbe proves that timing analysis can detect isolation flaws in multi-tenant environments, reminding us that performance optimization must be considered alongside security. Future directions: Expand to more gateways/proxies, develop automated audit tools, and explore privacy-preserving caching mechanisms (e.g., differential privacy caching).