Keeping the KV cache warm — measuring prompt cache eviction across Anthropic, OpenAI, and Google

TLDR DevOps wrote a practical post on prompt cache eviction, comparing how Anthropic, OpenAI, and Google handle the KV cache in their LLM APIs. The cache lets repeated prompts reuse prior computation — if the cache stays warm, you pay less. If it evicts, you pay again.

The article walks through real measurements: how long each provider keeps the cache alive, what triggers eviction, and how much you save by keeping prompts close together in time. The numbers differ by provider, but the pattern is clear — eviction is a silent cost you don't notice until your bill does.

For us, this is the kind of infra detail that shows up in the margin. We run models through multiple providers and cache aggressively, so the difference between warm and cold matters more than most. If we can keep the cache hot across our API calls, the savings compound.

Why this matters for us: the cache is a quiet lever — warm cache means lower latency AND lower cost on every call we make, and the providers don't always tell you when it's about to expire.