Stacks AI

Comprehensive AI/LLM integration with 4 provider drivers, image generation, vision, RAG, MCP support, and personalization.

Key Paths

• Core package: storage/framework/core/ai/src/
• Configuration: config/ai.ts

Source Files

ai/src/
├── drivers/
│   ├── anthropic/index.ts # Claude driver
│   ├── openai/index.ts    # GPT + image + transcription + TTS
│   ├── ollama/index.ts    # Local LLM driver
│   └── bedrock/          # Amazon Bedrock, through the Converse API
├── image.ts              # Image generation & vision
├── speech.ts             # Provider-neutral text-to-speech & speech-to-text
├── search.ts             # RAG, embeddings, vector index
├── mcp.ts                # Model Context Protocol client
├── personalization.ts    # Sentiment, classification, recommendations
├── buddy.ts              # AI coding assistant
├── claude-agent.ts       # Claude CLI agent (local & EC2)
├── claude-agent-sdk.ts   # Claude Agent SDK driver
└── text.ts               # ask() / summarize(), on the Bedrock driver

Anthropic Driver

import { anthropic } from '@stacksjs/ai'

anthropic.configure({ apiKey: '...', model: 'claude-sonnet-5-5', maxTokens: 4096 })
const result = await anthropic.chat([{ role: 'user', content: 'Hello' }])
const stream = await anthropic.streamChat(messages, options)
const response = await anthropic.prompt('Summarize this text...')
const tokens = anthropic.estimateTokens(text)

OpenAI Driver

import { openai } from '@stacksjs/ai'

openai.configure({ apiKey: '...', model: 'gpt-4o', embeddingModel: 'text-embedding-3-small' })
const result = await openai.chat(messages, { temperature: 0.7 })
const stream = await openai.streamChat(messages)
const embeddings = await openai.embed('text to embed')
const image = await openai.generateImage('a sunset over mountains')
const transcription = await openai.transcribe(audioFile)
const speech = await openai.textToSpeech('Hello world')

Bedrock Driver

import { bedrock } from '@stacksjs/ai'

bedrock.configure({ model: 'global.anthropic.claude-sonnet-5-5', region: 'us-east-1' })
const result = await bedrock.chat([{ role: 'user', content: 'Hello' }], { system: 'Be brief.' })

• Every request goes through Bedrock's Converse API, so one shape serves Nova,
  Claude and Llama; the model is configuration. bedrock.streamChat() and
  streamChatEvents() stream through converse-stream (ts-cloud >= 0.16.44).
• Credentials come from the AWS credential chain, never config. Region:
  drivers.bedrock.region, AWS_REGION, AWS_DEFAULT_REGION, us-east-1.
• Default model amazon.nova-lite-v1:0: on demand, no profile, no agreement.
• Claude and the newer Nova/Llama models are invoked by inference profile
  id (global.anthropic.claude-sonnet-5-5, us.meta.llama4-...). The bare
  anthropic.claude-... id is refused by Bedrock. bedrockModels in
  @stacksjs/types lists the ids Bedrock serves, each in invocable form.
• buddy ai:access (or requestModelAccess()) makes the models in
  config/ai.ts invocable and reports each one.

Tool Calls

import { assistantTurn, toolResultsTurn } from '@stacksjs/ai'

const result = await client.generate(messages, { tools })
// result.toolCalls: [{ id, name, arguments }] - same shape from every driver
messages.push(assistantTurn(result), toolResultsTurn(
  result.toolCalls!.map(call => ({ toolCallId: call.id, name: call.name, content: run(call) })),
))

• content is the model's text, '' when it only called tools.
• The responseFormat structured-output tool is internal: its JSON is
  content, never a tool call.
• Ollama refuses toolChoice: 'required' / { name }; it cannot force a call.
• Streaming takes the same tools. streamChat() yields text and returns the
  full AIResult (with toolCalls) when it ends; streamChatEvents() yields
  { type: 'text' }, { type: 'tool_call', call } (whole, once its arguments
  have streamed) and a final { type: 'done', result }. Every driver,
  Bedrock included.

Provider-Neutral Client

Use the config-driven client for application features that can run against
Anthropic, OpenAI, Ollama, or Bedrock. default is a provider name or a model
id that implies one: claude-... Anthropic, gpt-... OpenAI, any Bedrock
model or profile id (amazon.nova-lite-v1:0, global.anthropic.claude-...)
Bedrock. Configuration inspection is safe to return from a
status endpoint because it never includes credentials.

import { createAIClient, getAIProviderConfiguration } from '@stacksjs/ai'
import aiConfig from './config/ai'

const configuration = getAIProviderConfiguration(aiConfig)
// { provider: 'openai', model: 'gpt-4o-mini', configured: true, source: 'environment' }

if (configuration.configured) {
  const client = createAIClient(aiConfig)
  const result = await client.generate([{ role: 'user', content: 'Draft a launch plan.' }])
}

Speech (Text-to-Speech, Speech-to-Text)

Provider-neutral, in ai/src/speech.ts. The driver is the driver option, else
default from config/ai.ts. Only openai implements speech (SPEECH_DRIVERS);
any other driver throws naming itself and the supported ones - do not catch that
and fall back to a different provider.

import { speechToText, storeSpeech, textToSpeech } from '@stacksjs/ai'

const speech = await textToSpeech('Hello', { voice: 'nova', format: 'mp3', speed: 1.1 })
// { audio: Uint8Array, mimeType: 'audio/mpeg', format, driver, model, voice }
await storeSpeech(speech, 'audio/hello.mp3', { disk: 'public' }) // disk name or adapter
const { text } = await speechToText(Bun.file('memo.m4a'), { language: 'en' })

Defaults: model gpt-4o-mini-tts, voice alloy, format mp3, speed omitted
(provider default 1.0); transcription model whisper-1. instructions is
rejected for tts-1 / tts-1-hd. OpenAI caps input at 4096 characters per
request. HTTP failures throw with the status and a body snippet. In tests, mock
globalThis.fetch and pass config explicitly (see ai/tests/speech.test.ts).

Ollama Driver (Local LLMs)

import { ollama } from '@stacksjs/ai'

ollama.configure({ host: 'http://localhost:11434', model: 'llama3' })
const result = await ollama.chat(messages)
const stream = await ollama.streamChat(messages)
const text = await ollama.generate('Write a poem')
const embeddings = await ollama.embed('text')
const models = await ollama.listModels()
await ollama.pullModel('llama3', (progress) => console.log(progress))
await ollama.deleteModel('old-model')
const info = await ollama.showModel('llama3')
const running = await ollama.isRunning()

Image Generation

import { generateImage, editImage, createImageVariation, analyzeImage, analyzeImages } from '@stacksjs/ai'

// Generate (DALL-E 3)
const result = await generateImage('a cat in space', {
  model: 'dall-e-3', size: '1024x1024', quality: 'hd', style: 'vivid', n: 1
})

// Edit (DALL-E 2)
await editImage(imageInput, 'add a hat', { mask: maskInput })

// Variations
await createImageVariation(imageInput, { n: 3 })

// Vision (GPT-4 Vision / Claude)
const analysis = await analyzeImage({ url: 'https://...' }, 'What is in this image?')
const multiAnalysis = await analyzeImages([img1, img2], 'Compare these')

Image inputs: { url: string }, { base64: string }, { file: string } (auto-converted)

RAG & Vector Search

import { createEmbedding, rag, VectorIndex, chunkText, indexText } from '@stacksjs/ai'

// Create embeddings
const embedding = await createEmbedding('text to embed')

// Vector similarity
cosineSimilarity(vecA, vecB)
dotProduct(vecA, vecB)
euclideanDistance(vecA, vecB)

// Text chunking
const chunks = chunkText(longText, { chunkSize: 500, overlap: 50 })

// Build index
const index = await indexText(text, { chunkSize: 500 })

// RAG query
const answer = await rag('What is X?', index, { model: 'claude-sonnet-4-20250514', maxTokens: 1000 })

// VectorIndex class
const idx = new VectorIndex({ provider: 'openai', model: 'text-embedding-3-small' })
await idx.add([{ id: '1', content: 'Hello', metadata: {} }])
const results = await idx.search('greeting', 5)
const results2 = await idx.searchByVector(queryEmbedding, 5)
idx.remove('1')
idx.clear()
idx.size      // number of documents
idx.ids       // all document IDs

MCP (Model Context Protocol)

import { MCPClient, MCPManager, connectStdio, connectHTTP } from '@stacksjs/ai'

// Single server
const client = new MCPClient({ name: 'my-server', transport: 'stdio', command: 'npx', args: ['my-mcp-server'] })
await client.connect()
const tools = await client.listTools()
const resources = await client.listResources()
const prompts = await client.listPrompts()
const result = await client.callTool('tool-name', { arg: 'value' })
const resource = await client.readResource('resource://path')
const prompt = await client.getPrompt('prompt-name', { arg: 'value' })
const anthropicTools = client.toAnthropicTools()
const openaiTools = client.toOpenAITools()

// Multiple servers
const manager = new MCPManager()
await manager.addServer({ name: 'server1', transport: 'stdio', command: '...' })
await manager.addServer({ name: 'server2', transport: 'streamable-http', url: '...' })
const allTools = manager.getAllTools() // cached tools from connected servers
await manager.callTool('server1/tool-name', args)
await manager.disconnectAll()

// Convenience functions
const client = await connectStdio('name', 'command', ['args'])
const client = await connectHTTP('name', 'https://server.com', headers)

Transport types: 'stdio' | 'sse' | 'streamable-http'

Personalization

import { analyzeSentiment, classifyText, summarize, recommend, createProfile } from '@stacksjs/ai'

const sentiment = await analyzeSentiment('I love this product!')
// { sentiment: 'positive', score: 0.95, confidence: 0.98, aspects: [...] }

const classification = await classifyText('Fix the login bug', ['bug', 'feature', 'question'])
// { label: 'bug', confidence: 0.92, allLabels: [...] }

const summary = await summarize(longText, { maxTokenCount: 100 })

const profile = createProfile('user-123', ['tech', 'gaming'])
recordInteraction(profile, { type: 'view', itemId: 'article-1', weight: 1.0 })
const recs = await recommend(profile, contentItems, { limit: 10 })
const interests = await extractUserInterests(profile, items)

Buddy AI Assistant

import { processCommand, buddyProcessStreaming, buddyStreamSimple, getAvailableDrivers } from '@stacksjs/ai'

const drivers = getAvailableDrivers()  // ['claude-cli-local', 'claude-cli-ec2', 'claude', 'claude-sdk', 'openai', 'ollama', 'mock']
const context = await getRepoContext('/path/to/repo')
await processCommand('Add error handling to auth.ts', 'anthropic')
// Streaming
for await (const chunk of buddyProcessStreaming('Refactor this function', 'openai', history)) {
  process.stdout.write(chunk)
}

Compact Project Context

Use the native context contract instead of sending dependency internals, lockfiles,
or an arbitrary repository dump to a coding model:

buddy ai:context
buddy ai:context --json
buddy ai:context --json --output storage/framework/runtime/ai-context.json
buddy ai:context --max-chars 4000 --model claude-sonnet-4

Programmatic callers use buildProjectContext(projectRoot, options). Existing
getRepoContext(projectRoot) calls return its compact text payload. The contract
is deterministic, reports a heuristic token estimate, and describes canonical
Model-View-Action roles, application overrides, instruction files, scripts,
dependency names, and representative application files. It excludes
node_modules, build output, caches, lockfiles, environment files, credentials,
private keys, and secret files by default.

The character budget applies to the prompt payload. JSON adds a versioned metadata
envelope for tools. Token estimates are planning heuristics, not provider billing
counts or evidence that model output is correct.

Claude Agent

import { createClaudeLocalAgent, createClaudeEC2Agent } from '@stacksjs/ai'

const agent = createClaudeLocalAgent({ cwd: '/project' })
for await (const chunk of agent.processCommandStreaming('Fix the tests')) {
  process.stdout.write(chunk)
}

// Remote EC2 agent
const remoteAgent = createClaudeEC2Agent({ ec2Host: '...', ec2User: '...', ec2Key: '...' })

Claude Agent SDK

import { createClaudeAgentSDKDriver } from '@stacksjs/ai'

const driver = createClaudeAgentSDKDriver({
  maxTurns: 10, cwd: '/project', permissionMode: 'auto',
  allowedTools: ['Read', 'Write', 'Edit', 'Bash'],
  customSystemPrompt: 'You are a Stacks expert'
})

for await (const chunk of driver.processStreaming('Build a user registration flow')) {
  process.stdout.write(chunk)
}
const sessionId = driver.getLastSessionId()
await driver.resumeSession(sessionId, 'Add validation')

config/ai.ts

{
  default: 'anthropic',                       // or 'openai', 'ollama', 'bedrock', a model id
  models: ['amazon.nova-lite-v1:0'],          // Bedrock models `buddy ai:access` enables
  deploy: false,
  drivers: {
    anthropic: { model: 'claude-sonnet-5-5' },
    bedrock: { model: 'global.anthropic.claude-sonnet-5-5' },
  },
}

Gotchas

• API keys should be in .env - ANTHROPIC_API_KEY, OPENAI_API_KEY
• Ollama requires a local server running on port 11434
• Image generation uses OpenAI DALL-E by default
• Vision supports both GPT-4V and Claude models
• VectorIndex is in-memory - not persisted between restarts
• RAG combines chunking + embedding + vector search + LLM generation
• MCP supports stdio (subprocess), SSE, and HTTP transports
• The buddy assistant has git integration (commit, push, apply changes)
• Claude Agent SDK wraps the Claude Code CLI for agentic workflows
• A Bedrock id as default selects the Bedrock driver; it never reaches api.anthropic.com
• Sentiment/classification use AI models - they're not rule-based

Usage, retries and multimodal messages

The public package also exposes onUsage/recordUsage/listUsageReporters
(and clearUsageReporters for teardown), estimateTokens/estimateMessageTokens,
sanitizePrompt, fetchWithRetry, and
buildMessageWithImages/normalizeMessagesForProvider. Use the exported
types and their utils/ implementations before composing a provider-neutral
message or reporting costs. Token estimates are heuristic, usage callback data
comes from provider results, and sanitizing a prompt is not authorization for
executing a suggested tool.

Provider capabilities differ: embeddings currently use OpenAI/Ollama; RAG supports
Anthropic/OpenAI/Ollama; speech uses OpenAI only. The shared chat client and
tool-turn helpers support the four chat providers but do not imply those other
features are implemented by Bedrock. VectorIndex is memory-only; its constructor
accepts EmbeddingOptions (provider/model), not an arbitrary vector dimensions
setting. Runtime credentials come from env/provider configuration, never a
browser-exported config response.

Tests: storage/framework/core/ai/tests/usage-and-tokens.test.ts,
vision.test.ts, retry.test.ts, driver-retry.test.ts,
tool-calls.test.ts and client.test.ts. Verify chosen model/provider
capabilities from config and registry rather than assuming a marketing model name
is usable for every API.

Serving MCP tools

Use the native server subpath so serving a tool does not load AI provider drivers:

import { createMCPServer, mcpTextResult } from '@stacksjs/ai/mcp-server'
import { route } from '@stacksjs/router'

const server = createMCPServer({
  name: 'example', version: '1.0.0',
  authenticate: async request => authenticateIntegrationKey(request),
  tools: [{
    name: 'account', description: 'Read the connected account',
    inputSchema: { type: 'object', additionalProperties: false },
    annotations: { readOnlyHint: true, openWorldHint: false },
    execute: async (args, { principal }) => mcpTextResult({ id: principal.id }),
  }],
})
route.post('/mcp', server.handle).parseBody(false).skipCsrf()
route.get('/mcp', server.handle)
route.delete('/mcp', server.handle).skipCsrf()

authenticateIntegrationKey is application-owned: return a principal or an HTTP
401/403 Response. Authenticate every request, enforce scopes and quotas, and
never accept ambient login cookies as integration credentials. Tool available
filters discovery and execution; domain operations must still check authorization.
@stacksjs/api/operation supplies defineApiOperation({ authorize, handle }) for
sharing that check across HTTP, MCP and background invocations.

This transport is stateless Streamable HTTP with JSON replies. It supports
2025-11-25, 2025-06-18 and 2025-03-26; GET/DELETE return 405. Clients send both
application/json and text/event-stream in Accept. It implements tools, static
resources, prompts and discovery pagination; it does not claim SSE subscriptions,
tasks, server requests, session ids, stdio or JSON-RPC batch support. parseBody(false)
is essential: the protocol owns its bounded read and error envelope. The router
keeps middleware, CSRF policy, CORS, security and request-id headers. JSON-RPC
error bodies keep their envelope instead of receiving a top-level request_id.

Schemas use an explicitly bounded vocabulary: type, enum, const, properties,
required, additionalProperties (boolean), items, minimum/maximum, min/maxLength,
min/maxItems and pattern, plus descriptive metadata. Unsupported keywords fail
registration; put additional domain validation in the authorized operation.
Use object schemas for input and optional structured output. mcpTextResult
serializes the same data in both text and structuredContent. Tool exceptions are
redacted, with server-side onError diagnostics. Expected domain failures may
return mcpTextResult(data, true). Do not advertise idempotency for a write unless
the underlying operation guarantees it. responseHeaders(principal) can expose
per-key rate limits. Browser requests require exact configured allowedOrigins. Requests without an Origin header
remain eligible for bearer authentication. The request Host is never an origin allowlist.

Tests: core/ai/tests/mcp-server.test.ts, core/router/tests/body.test.ts,
core/router/tests/protocol-body.test.ts and core/api/tests/operation.test.ts.

For OAuth-backed MCP hosts, set resourceMetadataUrl to the canonical RFC 9728
metadata URL; authentication failures keep their HTTP status and advertise it in
the Bearer challenge. Use native Stacks auth for registration, consent, PKCE,
audience checks, token exchange and revocation. Per-tool securitySchemes and
_meta are supported for host authentication metadata; they describe policy and
do not replace runtime authorization. Tool result _meta can carry protocol
extensions such as authentication challenges.

The server snapshots registered tools, schemas, prompt arguments, origins and
callbacks. Later changes to the supplied objects cannot relax an active server's
policy. Invalid schema types and bounds fail at registration; enum/const accept
JSON primitives only. String bounds count Unicode code points. Diagnostics and
response-metadata callback failures cannot reject the protocol handler.