Prompt Caching Guide: How to Reduce AI API Token Costs by 80%

Overview #

Prompt caching is a server-side inference optimization that stores the computed key-value (KV) attention states of long prefix prompts in GPU memory, allowing subsequent requests to reuse them with 50-90% cost discounts and 80% lower TTFT.

How Key-Value (KV) Cache Reusability Works #

In traditional inference, sending a 50,000-token system prompt requires the model to re-compute attention matrices on every request. With prompt caching, the inference cluster hashes the prefix tokens and caches the computed KV tensors. When another request arrives with the exact same prefix, the model skips prefill computation entirely.

Economic & Latency Impact #

On Anthropic Claude 3.5 Sonnet and Google Gemini 1.5 Pro, cached prompt tokens receive an 80% to 90% price discount compared to uncached input tokens. Additionally, Time to First Token (TTFT) drops from ~2.5 seconds down to ~300ms for large 100k-token prompts.

Code Example: Structuring Requests for Prompt Cache Hits #

// Ensure static context (documentation, API schema, rules) is placed 
// at the very beginning of the messages array:
{
  "model": "claude-3-5-sonnet",
  "messages": [
    {
      "role": "system",
      "content": "MASSIVE_STATIC_DOCUMENTATION_OR_CODEBASE..." // <-- Cached prefix
    },
    {
      "role": "user",
      "content": "Dynamic user query 1" // <-- Dynamic delta
    }
  ]
}

Frequently Asked Questions #

Q: How much does prompt caching save?

Prompt caching typically saves between 50% and 90% on input token costs while reducing Time to First Token by up to 80%.