langwatch
@langwatch Organization
Public Skills
scenarios
by langwatch
Test your AI agent with simulation-based scenarios. Covers writing scenario test code (Scenario SDK), creating platform scenarios via the langwatch CLI against a connected agent, reading the run parameters that agent declares so the scenarios and comparison runs turn its real levers, and red teaming for security vulnerabilities. Auto-detects whether to use code or platform approach based on context.
evaluations
by langwatch
Compatibility router for LangWatch evaluation requests. Use only when the user asks for evaluations without making it clear whether they mean pre-deployment experiments or production online evaluations. Routes the request to the focused companion skill and does not implement either workflow itself.
github
by langwatch
Open a real pull request. Clone a repo, branch, commit, push, and open a PR authored by the LangWatch app on behalf of the requesting user. Use when the user asks to open a PR, fix something in a repo and submit it, send a patch, raise a pull request, or otherwise land a code change on GitHub.
prompts
by langwatch
Version and manage your agent's prompts with LangWatch Prompts CLI. Use for both onboarding (set up prompt versioning for an entire codebase) and targeted operations (version a specific prompt, create a new prompt version). Supports Python and TypeScript.
tracing
by langwatch
Add LangWatch tracing and observability to your code. Use for both onboarding (instrument an entire codebase) and targeted operations (add tracing to a specific function or module). Supports Python and TypeScript with all major frameworks.
code-changes
by langwatch
Change the user's own program, on their machine or through GitHub. Use when a request needs a change to the user's code (instrument tracing, wire the SDK, fix the agent behind a failing scenario, add a run parameter to a connected agent, version a hardcoded prompt) and not when the platform alone can do it (create a scenario, an evaluation, a prompt version, read traces).
connect-agent
by langwatch
Connect the codebase's AI agent to LangWatch agent simulations, so test suites run against the real agent process. Adds a small connect function beside the service startup that calls the agent already in the codebase, which opens an outbound connection and registers the agent with its environment and its run parameters, confirms the agent is Online, and runs the first test suite. Falls back to an HTTP registration when the agent cannot import the SDK. Use when the user wants platform scenarios to test their real agent.
prompt-optimization
by langwatch
Improve a prompt on the evaluations workbench through a measured loop. Score the baseline first, then duplicate the target column, form a hypothesis from failing rows, edit the copy's prompt draft, run, compare pass rate and cost, and repeat until the numbers hold. Use when the user asks to optimize or improve a prompt, when they arrive from the workbench's "Optimize this prompt" menu item, or when they want their bot to answer better. Bootstraps a missing dataset or evaluator first.
agent-best-practices
by langwatch
Expert AI engineering consultant for your agent development practices. Audits your codebase, traces, evaluations, and scenarios against best practices, then guides you to close the gaps, starting from low-hanging fruit and going deeper. Use when you want to level up your agent's engineering quality.
context-sweet-spot
by langwatch
Investigates the context economics of your own coding-agent sessions in LangWatch. Reads real sessions to find where carrying a fat context stops paying for itself, measured in cache rebuilds, compactions and cost per turn, and delivers a report with the context size your sessions should stay under, with example sessions behind every claim. Use when coding-agent sessions feel expensive or degrade as they grow.
provider-cost-comparison
by langwatch
Prices your real LangWatch usage mix against other model providers. Exports your actual token mix per model, including the cache read and write split, fetches current price cards, and reprices the same month of usage under each candidate, with the cache sensitivity stated. Use when someone asks whether a cheaper provider or model would actually save money on your workload.
agent-improve
by langwatch
Turns production evidence into tested improvements for your AI agent. Forms hypotheses from real traces and analytics, explains the reasoning behind each one, then executes with the user: scenario tests that reproduce production failures, prompt and code changes as reviewable PRs, new evaluators and monitors that capture production signals, and experiments that settle open questions. Use when you want to know what to do next to improve your agent.
experiments
by langwatch
Create and run LangWatch experiments for pre-deployment batch testing. Use when the user wants to test an agent against a dataset, compare prompts or models, benchmark quality, detect regressions, or add a CI quality gate. Do not use for production monitoring or guardrails.
level-up
by langwatch
Take your AI agent to the next level with full LangWatch integration. Adds tracing, prompt versioning, evaluation experiments, and simulation tests in one go. Use when the user wants comprehensive observability, testing, and prompt management for their agent.
online-evaluations
by langwatch
Configure LangWatch online evaluations and guardrails for production traffic. Use when the user wants to score live traces or threads, monitor production quality, sample incoming traffic, or synchronously block unsafe requests and responses. Do not use for batch experiments.
setup-lw
by langwatch
Set up and troubleshoot the LangWatch CLI, covering login (cloud and self-hosted), endpoint configuration, project selection, and connection problems. Use when the CLI isn't authenticated, can't reach LangWatch, or talks to the wrong project.
drive-the-ui
by langwatch
Drive the page the user has open through live UI actions. List the actions a page accepts, call them with typed payloads, and read the live state including unsaved edits. Use when the user is looking at a page you can operate, such as the evaluations workbench, and a change should happen in front of them rather than behind their back.
lwql-charts
by langwatch
Author a saved analytics chart from a plain question and place it on a dashboard. Discovers the LangWatchQL analytics schema, writes and test-runs the SQL, saves it as a chart with a Vega-Lite specification, and places it where the team already looks. Use when asked to build, save, run, or dashboard a metric or chart.
agent-performance
by langwatch
Deep-dive diagnosis of how your AI agent behaves in production. Explores LangWatch analytics and traces end to end to map failure patterns, dissatisfied users, token cost hotspots, edge cases, behavior changes, and outliers, then delivers an HTML report where every finding links to real example traces. Use when you want to truly understand what your agent is doing in production.
debug-instrumentation
by langwatch
Debug and improve your LangWatch traces. Inspects production traces for missing input/output, disconnected spans, unlabeled traces, and missing metadata. Use when traces look broken or incomplete.
debug-with-langwatch
by langwatch
Root-cause production errors and misbehaving agent runs with LangWatch. Finds errored traces, inspects spans, checks monitor and evaluator scores, then narrows to a root cause. Use when something is failing or misbehaving in production (errors, bad answers, latency spikes).
eval-triage
by langwatch
Investigate failing experiments and evaluations with LangWatch. Triage a failing experiment run to the exact rows and evaluator scores that regressed, then to a root cause. Use when an experiment fails, scores drop, or evaluations regress.
test-compliance
by langwatch
Test that your AI agent stays observational and doesn't give prescriptive advice in regulated domains (healthcare, finance, legal). Creates scenario tests for boundary enforcement and red team tests for adversarial probing. Use when your agent advises but must not prescribe.
datasets
by langwatch
Generate realistic synthetic evaluation datasets by analyzing the user's codebase, prompts, production traces, and reference materials. Interactive and consultant-style. Asks clarifying questions, proposes a plan, generates a preview for approval, then delivers a complete dataset uploaded to LangWatch. Use when user asks to generate, create, or build a dataset for evaluation, testing, or benchmarking.
evaluate-multimodal
by langwatch
Evaluate multimodal AI agents that process images, audio, PDFs, or other files. Sets up evaluations using LangWatch's LLM-as-judge with image inputs, Scenario's multimodal testing, and document parsing evaluation patterns. Use when your agent handles non-text inputs.
generate-rag-dataset
by langwatch
Generate a synthetic evaluation dataset from your RAG knowledge base. Creates diverse Q&A pairs with expected answers and relevant context, ready for LangWatch experiments and platform import. Use when you need test data for your RAG pipeline.
langwatch
by langwatch
Read LLM traces back from LangWatch with the langwatch CLI. Use when asked what an agent, prompt or model call actually did in production, when debugging a failed or slow LLM run, when looking up a trace or session by id, or when checking whether this coding session's own activity was captured.
browser-test
by langwatch
"Validate a feature works by driving a real browser with Playwright MCP. No test files — just interactive verification."
browser-pair
by langwatch
"Collaborative headed browser session for UI work. Launch Playwright Chromium visible to the user, handle auth, then interactively drive the browser while the user watches and gives real-time visual feedback. Edit code and refresh to verify fixes live. Use when the user says 'browser pair', 'paired browser', 'let's look at this together', 'open chromium', or wants to iterate on UI with live visual feedback."
code-review
by langwatch
"Project-level code review: check changed files against LangWatch codebase rules (IDs, multitenancy, layering, naming, SRP)."
feature-map
by langwatch
"Maintain the canonical LangWatch feature map (/feature-map.json). Use when adding features, APIs, MCP tools, CLI commands, or skills — to update the central registry and keep surfaces in sync."
haven-setup
by langwatch
"Bring up the LangWatch dev stack via thuishaven (make haven up) — one-time proxy/CA setup, reusing existing local ClickHouse/Postgres/Redis, WSL2/no-colima fallback for langyagent, and the known gotchas that silently break it."
test-cli-usability
by langwatch
Write scenario tests that verify your CLI tool is usable by AI agents. Ensures commands work non-interactively, provide clear output, and don't hang on prompts. Use when you want to prove your CLI is agent-friendly.
github
by langwatch
Open a real pull request on the user's behalf — clone a repo, branch, commit, push, and open a PR authored by the requesting user. Use when the user asks to open a PR, fix something in a repo and submit it, send a patch, raise a pull request, or otherwise land a code change on GitHub.
langwatch-kanban
by langwatch
"Manage the LangWatch Kanban GitHub project board — sync statuses, view your board, find stale items, move issues, assign work."
launch
by langwatch
Create worktrees, tmux sessions, and Claude sessions for GitHub issues. Use when spinning up parallel implementation work.
watch-ci
by langwatch
"Watch CI for the current branch's PR. Blocks until CI completes, then fixes failures or addresses review comments. Loops until green."
plan
by langwatch
"Create a feature file with acceptance criteria before implementation. Use when no specs/features/*.feature file exists for the work."
orchestrate
by langwatch
"Orchestration mode for implementation tasks. Manages the plan → code → review loop. Use /orchestrate <requirements> or let /implement invoke it."
review
by langwatch
"Run parallel code reviews: uncle-bob-reviewer (SOLID/TDD), cupid-reviewer (CUPID properties), test-reviewer (pyramid placement), and pii-reviewer (security/secrets). Surfaces conflicts for orchestrator resolution."
challenge
by langwatch
"Stress-test an architecture proposal, plan, or technical idea. Invokes the devils-advocate agent to find weaknesses before you commit."
learn
by langwatch
"Learn from mistakes by updating AGENTS.md. Use when a mistake was made that should be prevented in future sessions."
test-review
by langwatch
"Review specs and tests for pyramid placement and quality."
langwatch
by langwatch
The platform for LLM evaluations and AI agent testing
e2e
by langwatch
"Generate and verify E2E tests for a feature. Explores live app, creates test plan, generates tests, runs and fixes until passing."
code
by langwatch
"Delegate implementation work to the coder agent. Provide requirements or feature file path."
implement
by langwatch
"Start implementation of a GitHub issue. Usage: /implement #123 or /implement <issue-url>"
sherpa
by langwatch
"Delegate repository, agent, or documentation questions to the repo-sherpa. Use for onboarding, DX improvements, or meta-layer changes."