LLM App Patterns

Core Philosophy

1. Retrieval quality is a ceiling on generation quality — No prompt engineering
   compensates for bad RAG. Fix retrieval before tuning prompts.
2. Workflows beat agents for predictability — Use agents only when the execution
   path is genuinely unknown at design time. Everything else should be code.
3. Measure before optimizing — Add cost attribution and eval metrics first.
   Optimization without measurement is guessing.
4. Schema failures cascade — Unstructured LLM output is a reliability tax.
   Constrain output at the token level; don't parse free text.
5. Stream by default — Token streaming is the lowest-effort UX improvement
   for any LLM interface. Users read while the model generates.

Quick-Start: Common Problems

"My RAG system gives wrong answers"

1. Measure faithfulness first — are answers grounded in retrieved context?
   -> See references/evaluation-observability.md (Faithfulness Check Pattern)
2. Check retrieval quality — are the right chunks being retrieved?
   -> See references/rag-patterns.md (RAG Evaluation Metrics)
3. Fix the pipeline stage that is failing; do not tune prompts to mask retrieval failures.

"I need structured data from LLM output"

1. Choose the right method (native structured outputs vs Instructor vs JSON mode).
   -> See references/structured-output.md (Method Comparison)
2. Design schemas for LLMs — use enums, Field descriptions, explicit bounds.
   -> See references/structured-output.md (Schema Design for LLMs)
3. Add retry logic with error context; LLMs correct mistakes when shown them.

"The app feels slow"

1. Enable token streaming immediately — reduces perceived latency to time-to-first-token.
   -> See references/streaming.md (Server Implementation Pattern)
2. Check proxy buffering — nginx/Cloudflare often buffer SSE by default.
   -> See references/streaming.md (Proxy Buffering table)
3. For actual latency: profile per pipeline stage; retrieval and reranking are often the bottleneck.

"LLM API costs are too high"

1. Implement prompt prefix caching first (60–90% reduction on large system prompts).
   -> See references/cost-optimization.md (Prompt Prefix Caching)
2. Add model routing — route simple requests to small models.
   -> See references/cost-optimization.md (Three-Tier Routing Pattern)
3. Add semantic caching for user-facing query endpoints.
   -> See references/cost-optimization.md (Semantic Cache)

"My agent loops, hallucinates tools, or gets stuck"

1. Implement loop detection — break on repeated (tool, args) pairs.
   -> See references/tool-use-agents.md (Loop Detection)
2. Set max_steps — always cap the ReAct loop.
   -> See references/tool-use-agents.md (Max Steps Guard)
3. Improve tool descriptions — models select tools by description, not name.
   -> See references/tool-use-agents.md (Tool Design Principles)

Decision Trees

RAG vs Fine-Tuning vs Prompting

Agent vs Workflow

Structured Output Method

Streaming Transport

Evaluation Minimum Viable Setup

Before shipping any LLM feature to production:

1. Run faithfulness and answer relevance on 50 examples (no golden labels needed).
   -> references/evaluation-observability.md
2. Add traces: log input, output, latency, token count per request.
3. Set up one judge metric as a regression gate in CI.
4. Collect failed cases from user feedback → build golden dataset over time.

Reference Files