LLM Caching

Maximize KV cache reuse to reduce costs and latency.

Core Concept

LLMs compute Key (K) and Value (V) vectors for each token during inference. These encode the model's "understanding" of context. Caching avoids recomputation.

Level 1: KV Cache (inference)     - Within one generation, reuse previous tokens' K,V
Level 2: Prompt Cache (API)       - Across requests, persist KV state server-side
Level 3: Prefix Sharing (batch)   - Across users/requests, share common prefixes

The Golden Rule

Static content first, variable content last.

[System prompt]         <- cacheable, same every request
[Tool definitions]      <- cacheable
[Few-shot examples]     <- cacheable (same order!)
[Reference documents]   <- cacheable if stable
[User message]          <- variable, at the end

Cache hits require the prefix (beginning) to match exactly. Any difference breaks caching for everything after.

Prompt Structure Template

┌─────────────────────────────────────┐
│  1. System instructions (static)    │  <- cache_control
├─────────────────────────────────────┤
│  2. Tool definitions (static)       │  <- cache_control
├─────────────────────────────────────┤
│  3. Few-shot examples (static)      │  <- cache_control
├─────────────────────────────────────┤
│  4. Documents/context (semi-static) │  <- cache_control if reused
├─────────────────────────────────────┤
│  5. Conversation history (growing)  │  <- cache after N turns
├─────────────────────────────────────┤
│  6. Current user message (variable) │  <- no caching
└─────────────────────────────────────┘

Anti-Patterns

Anti-Pattern	Why It Breaks Caching
Variable content early	Prefix changes every request
Randomizing few-shot order	Different order = different prefix
Timestamps in system prompt	Changes every request
User ID in prefix	Per-user cache = no sharing
Prompts < minimum threshold	Too small to cache (1024 tokens for Claude)
Shuffling tool definitions	Tool order is part of prefix

Cost Impact

Operation	Typical Pricing	Notes
Cache write	~1.25x input	One-time, stores KV state
Cache read	~0.1x input	90% savings on cache hit
No caching	1x input	Full recomputation every time

Example: 50k token system prompt, 100 requests

• Without cache: 50k × 100 × $3/1M = $15.00
• With cache: 50k × $3.75/1M + 50k × 99 × $0.30/1M = $1.67 (89% savings)

Provider References

• Anthropic Claude (recommended): references/claude.md
• Cohere: references/cohere.md
• Self-hosted (vLLM, SGLang, Ollama, HuggingFace): references/self-hosted.md
• OpenAI: references/openai.md
• Google Gemini: references/gemini.md

Cookbooks

Practical examples: references/cookbooks.md

Pattern	Key Insight
Web scraping agent	Same tools + system prompt, different URLs
RAG pipeline	Cache document chunks, vary queries
Multi-turn chat	Growing prefix, cache conversation history
Batch processing	Same prompt template, different inputs
Agentic tool use	Cache tool definitions + examples
Multi-tenant SaaS	Shared base prompt, tenant-specific suffix