Diagnose

A discipline for hard bugs. Skip phases only when explicitly justified.

When exploring the codebase, use the project's domain glossary to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.

Phase 1 — Build a feedback loop

This is the skill. Everything else is mechanical. If you have a fast, deterministic, agent-runnable pass/fail signal for the bug, you will find the cause — bisection, hypothesis-testing, and instrumentation all just consume that signal. If you don't have one, no amount of staring at code will save you.

Spend disproportionate effort here. Be aggressive. Be creative. Refuse to give up.

Ways to construct one — try them in roughly this order

1. Failing test at whatever seam reaches the bug — unit, integration, e2e.
2. Curl / HTTP script against a running dev server.
3. CLI invocation with a fixture input, diffing stdout against a known-good snapshot.
4. Headless browser script (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
5. Replay a captured trace. Save a real network request / payload / event log to disk; replay it through the code path in isolation.
6. Throwaway harness. Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
7. Property / fuzz loop. If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
8. Bisection harness. If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can git bisect run it.
9. Differential loop. Run the same input through old-version vs new-version (or two configs) and diff outputs.
10. HITL bash script. Last resort. If a human must click, drive them with scripts/hitl-loop.template.sh so the loop is still structured. Captured output feeds back to you.

Build the right feedback loop, and the bug is 90% fixed.

Iterate on the loop itself

Treat the loop as a product. Once you have a loop, ask:

• Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
• Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
• Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)

A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.

Non-deterministic bugs

The goal is not a clean repro but a higher reproduction rate. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.

When you genuinely cannot build a loop

Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do not proceed to hypothesise without a loop.

Do not proceed to Phase 2 until you have a loop you believe in.

Phase 2 — Reproduce

Run the loop. Watch the bug appear.

Confirm:

• The loop produces the failure mode the user described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
• The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
• You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.

Do not proceed until you reproduce the bug.

Phase 3 — Hypothesise

Generate 3–5 ranked hypotheses before testing any of them. Single-hypothesis generation anchors on the first plausible idea.

Each hypothesis must be falsifiable: state the prediction it makes.

Format: "If is the cause, then will make the bug disappear / will make it worse."

If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.

Check Stack Overflow for the specific error signature (exact exception message, stack trace shape, or symptom) before finalizing the ranked list — not instead of it. If the signature matches a well-corroborated SO answer (accepted, high-voted, version-consistent with your stack), fold it in as a ranked hypothesis with a note on its source; it doesn't skip Phase 1/2 or get treated as confirmed until it survives your own feedback loop. A generic search for the bug's symptom rather than its exact signature is usually noise — narrow the query to what makes this failure specific.

Show the ranked list to the user before testing. They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if the user is AFK.

MARVIN-specific gotcha classes — check these early when the symptom fits, before deeper instrumentation:

• Script works manually, fails/no-ops silently under launchd or cron: the scheduled job's PATH doesn't include the interactive shell's additions. A script that shells out to a CLI (e.g. claude) can resolve to nothing and have its error swallowed into the very output it's supposed to produce — confirmed 2026-07-02, where this masked a broken daily-digest cron for days because the "digest" was literally the error string.
• A hook fires on unrelated files/edits repo-wide: Claude Code hook matchers key on tool name (Write/Edit), not path — they do not scope by file location unless the hook script does its own path filtering. Confirmed 2026-07-02 on rebuild-manifest.py, which ran on every Write/Edit anywhere until path-scoped explicitly.
• Derived state (manifest, index, cache) goes stale after a raw deletion: hooks only fire on tool calls (Write/Edit), not on rm or other out-of-band filesystem changes — a manual rm on a tracked file leaves derived state pointing at something that no longer exists until a manual resync runs. Confirmed 2026-07-02: manifest.json stayed stale after deleting a skill file, with no hook to catch it.
• A trust/config flag in ~/.claude.json won't stick despite correctly re-accepting the dialog: a long-running claude process loads that file into memory at startup and periodically flushes its own state back to disk on its own schedule — this silently clobbers any out-of-band change (manual edit, or a dialog accepted in a different session) with the stale in-memory snapshot. The fix is ending the long-running session so a fresh process reads the current file, not repeating the accept dialog. Confirmed 2026-07-03, root-caused via PID/timeline correlation while debugging why claude remote-control workspace trust wouldn't persist.
• A deploy/install script silently drops files it should have copied: a copy step built around a glob like dir/* only reaches what's inside that one subdirectory — sibling top-level files and symlinks are missed with no error. Files cp'd rather than symlinked to their git-tracked original also drift silently and stop feeding cross-machine sync. Confirmed 2026-07-03 setting up a second machine — install_skills() missed brain-map/, retrospective-log.md, and resume-tailor's deps entirely; retrospective-log.md on the deployed machine was also a bare cp outside git until fixed to a symlink.
• A background/menu-bar app briefly flashes visible despite requesting .accessory activation policy: if it's a bare compiled binary rather than a proper .app bundle, NSApplication.setActivationPolicy(.accessory) runs too late — macOS has already registered the process under the default policy by the time the runtime call executes. Fix is a real .app bundle with LSUIElement set in Info.plist, not a runtime workaround. Confirmed 2026-07-03 on DesktopLive.
• A scheduled claude -p script produces a report that reads like a narrated failure ("I tried to write X but couldn't...") instead of the intended output: an unrestricted claude -p call for a read-only reporting/generation task can still attempt a Write or Edit, get silently denied by the harness, and then narrate that denial into the very stdout it's supposed to produce as clean output. Fix: pass --disallowedTools "Write,Edit" (or the minimal tool set actually needed) on any non-interactive claude -p invocation whose job is to produce text, not modify files. Confirmed 2026-07-06 in both daily_digest.py and research_digest.py — same root cause hit twice independently, fixed identically in both.
• Building anything (eval harness, client, integration) against a local model server's "OpenAI-compatible" API: compatibility is usually partial, not full — check which specific params are actually implemented before designing around the assumption. Confirmed 2026-07-07: mlx_lm.server has no echo support at all, which lm-eval's local-completions backend needs for log-likelihood tasks (leaderboard_mmlu_pro); the whole server-based harness was a dead end, discovered only after building it. Fix was dropping the HTTP layer and writing a small in-process adapter wrapping the library directly instead of assuming the server exposed everything the eval tool expected.
• A regex parses one log/state-file entry using a greedy .+ under re.DOTALL: DOTALL makes . match newlines too, so the greedy .+ keeps consuming past the intended entry boundary and bleeds into the next entry — corrupting every entry after the first, often silently. Confirmed 2026-07-14 in check_sync_log(): a greedy header regex under DOTALL swallowed subsequent log entries. Fix: use a negated-newline class ([^\n]+) for any field that should stay single-line, reserving DOTALL-sensitive greediness for the part of the pattern that actually needs to span lines.
• A monitoring check scans a log's text for a keyword (e.g. "CONFLICT") to detect an ongoing problem: once written, the keyword persists in the log forever, so the check keeps firing as a false positive long after the problem was resolved — text logs are a historical record, not current state. Confirmed 2026-07-14 building the SessionStart hook's conflict detector: sync-log.md still contained "CONFLICT" from an already-resolved merge, and a real stray stash from that old conflict was found sitting unresolved. Fix: query the live authoritative source directly (git status --porcelain conflict codes, MERGE_HEAD existence, git stash list) instead of grepping historical logs for keywords.
• A prose checklist in CLAUDE.md/SKILL.md gets converted into a deterministic hook that runs the same checks: leaving the old step-by-step prose in place alongside the new hook causes the model to re-perform checks the hook already ran and injected into context — wasted tool calls/context, and a real risk of silent disagreement between the hook's output and the model's own recheck. Fix: collapse the prose to a one-line pointer + an explicit "don't re-perform these by hand" instruction, and move full step rationale to a separate reference doc that tells editors to update it in the same commit as the script. Confirmed 2026-07-14 converting MARVIN's 11-step session-start checklist into session_start_report.py.
• A themed/dark CSS background silently doesn't render, but "looks right" in one specific browser: CSS custom properties scoped to a descendant selector (e.g. .viz-root { --page-plane: ... }) are invisible to an ancestor rule referencing them (e.g. html, body { background: var(--page-plane) }) — custom-property inheritance flows down the DOM tree, not up, so the ancestor rule silently resolves to nothing with no console error. Compounding factor: a browser with automatic dark-mode repainting (e.g. Comet) can incidentally produce the intended-looking dark background anyway, masking the bug entirely until viewed in a standards-compliant browser. Fix: scope custom properties at :root (or the highest consumer), not a scoped wrapper class. Confirmed 2026-07-17 on ~/.agents/brain-map/template.html.
• A JS animation library's high-level API (e.g. Motion's animate()) silently no-ops or looks frozen in a wallpaper-mode/embedded canvas rendering context: high-level animation helpers are built on requestAnimationFrame, which macOS suspends for windows rendered at the desktop-wallpaper window level — even though a separate native render loop (e.g. a Swift NSTimer-driven one) keeps calling into the JS layer on schedule. The library isn't broken; rAF itself just never fires there. Fix: use the library's low-level physics primitive directly (e.g. Motion's spring() generator) and sample it manually from the existing native-driven loop instead of the high-level rAF-based API. Confirmed 2026-07-17 on ~/.agents/brain-map/DesktopLive — hit twice independently (an earlier camera-rotation bug, then again for pulse/camera-nudge/fitScale spring animations), same root cause both times, same fix both times.
• A "cause" animation (a highlight/pulse) and its "effect" animation (a camera move, a transition) are tuned independently and one finishes well before the other: if the cause's visible duration is shorter than the effect's, the signal fades out mid-effect, so the effect reads as unmotivated ("why did it move — nothing happened") even though there was a real trigger. Fix: tune the cause's fade to comfortably outlast the effect's full duration, not just approximately match it. Confirmed 2026-07-17 on ~/.agents/brain-map/DesktopLive: a ~2.5s pulse fade against a 4.2s camera-ease left the camera moving with no visible cause for the last ~1.7s.
• Widening a shared instrumentation trigger's conditions for one consumer's benefit silently changes behavior for every other consumer keyed off the same signal: MARVIN's generic "heartbeat" pulse (from skill_activity.py) was widened to fire on nearly every tool call so the pulse itself would feel more alive, but an unrelated consumer (camera-nudge logic) listening to that same signal started re-triggering on every call too, fighting its own auto-rotate instead of only reacting to named-skill activity. The signal wasn't broken, it just had more than one purpose riding on it. Fix: split consumers onto the specific condition each actually needs (named-skill activity for the camera nudge, any activity for the pulse) rather than sharing one trigger across purposes. Confirmed 2026-07-17 on ~/.agents/brain-map/DesktopLive.
• Real algorithmic performance work (parallelization, caching, cancellation) is applied to a compiled-language hot path but the wins are smaller than expected, or a rewrite in a faster language is being floated as the next step: before optimizing further, confirm the build actually being measured is an optimized/release build, not the default debug build — an unoptimized compiled binary (Swift, C++, Rust) can dominate measured time by an order of magnitude more than any algorithmic change, making real optimizations look like they aren't working. Confirmed 2026-08-26 on killer-sudoku: parallel batch generation and cooperative solver cancellation were both implemented and measured against a ~10-38s baseline, and a C++ rewrite was being considered as the next lever, before discovering the entire session had been profiling the debug build — switching the run script to a release build alone dropped runtime to ~1s, dwarfing every prior optimization. Fix: at the start of any perf-diagnosis session on compiled code, confirm the invoked build/run command actually uses the optimized configuration before touching algorithms.
• An enum backs data that gets persisted (SwiftData, Codable, UserDefaults) and relies on implicit rawValues (case order): removing, inserting, or reordering a case silently reassigns every subsequent case's raw value, corrupting already-persisted records with no compile error and no crash — the app just starts reading/writing the wrong case for existing rows. Confirmed 2026-08-26 on killer-sudoku (ADR 0011): removing .beginner from a Difficulty enum with implicit raw values silently broke previously persisted SolveRecords. Fix: any enum backing persisted state must declare explicit rawValues up front, and case removal must leave the remaining values untouched (deprecate/reserve the slot rather than closing the gap).
• A Compose/Kotlin canvas animation (pulse, progress bar, transition) builds and runs with no errors but visually never advances: computing elapsed time by mixing a monotonic clock (System.nanoTime()) for one side of the subtraction with a wall-clock/epoch one (System.currentTimeMillis()) for the other produces a nonsense magnitude, usually already outside the animation's clamp range — so it silently renders in its finished (or never-started) state forever, with no exception to point at the mismatch. Confirmed 2026-08-26 on killer-sudoku's Kotlin/Compose Multiplatform port (BoardCanvas.kt): every board animation was dead because pulseProgress compared a nanoTime()-based now against an epoch-based start time; found via temporary println instrumentation only after two wrong guesses (font family, then a first mis-fix). Fix: pick one clock source for a given elapsed-time calculation — prefer System.nanoTime(), since it's monotonic — and use it consistently for both the captured start and every later now read.
• Keyboard input silently does nothing in a Compose Desktop/Multiplatform window, with no error or console output: requesting focus once at launch (e.g. a single focusRequester.requestFocus() in a LaunchedEffect(Unit)) doesn't survive the window losing and regaining focus, or the user clicking elsewhere in the same window — Compose Desktop doesn't automatically restore focus to that composable afterward. Confirmed 2026-08-26 on killer-sudoku's Kotlin port: the board never accepted key events after the very first interaction because focus was only requested once at launch. Fix: re-request focus on every relevant user interaction (e.g. on each board tap), not just at startup.
• Verifying visual/pixel parity between a UI port and its original across different rendering toolkits (Skia vs CoreGraphics, Compose vs SwiftUI, etc.): eyeballing side-by-side screenshots is unreliable for this — small offsets (a grid line a few pixels off, slightly different glyph metrics) look "close enough" but are real regressions, and a visual read can also give a false-positive parity verdict. Confirmed 2026-08-26 on killer-sudoku's Kotlin port: an eyeballed screenshot comparison gave a wrong verdict once. Fix: use pixel-level analysis (Pillow + numpy — grid-line detection, glyph bounding-box extraction) for cross-toolkit visual parity checks instead of relying on visual inspection alone.
• A long-uptime local daemon (rendering loop, live wallpaper, background service) drifts into a bad state over weeks in a way that's expensive to root-cause live (needs multi-day instrumentation) but already runs under a launchd agent with KeepAlive: rather than blocking the fix on live instrumentation, add a second, separate launchd agent whose only job is a scheduled pkill of the target process — the target's existing KeepAlive relaunches it within seconds, forcing a periodic restart with zero changes to the original agent's plist or code. This is a mitigation, not a root-cause fix — pair it with a scoped backlog entry (with the actual instrumentation plan) rather than treating "restart fixes it" as closing the investigation. Confirmed 2026-08-26 on ~/.agents/brain-map/DesktopLive: A/B'd two screenshots of the 22-day-old process 4s apart (genuine motion, not frozen, just zoomed in tight) against a fresh restart (immediately clean) to confirm drift vs. freeze before committing to this fix; root cause (suspected fitVelocity spring-physics accumulation) deliberately deferred, instrumentation plan logged in ~/.claude/suggestions.md.
• New code calls a real side-effecting module (ticket stage tracker, log writer, state file) and existing tests start writing to the REAL ~/.claude/ files: when the new call isn't dependency-injected, any pre-existing test that uses realistic-looking inputs (real ticket numbers, real paths) silently exercises it against live state, polluting production data with no test failure. Confirmed 2026-10-01 on MARVIN's ticket-stage work (ticket_stages.py/.js): hit repeatedly in one session, each time fixed after the fact (autouse pytest fixture redirecting the home/log dir; a vi.mock on the JS side). Fix: when adding a call to a side-effecting module, immediately add an autouse fixture/mock at the test-suite level (or inject the path), and grep existing tests that reach that code path, rather than waiting for the next pollution to surface.
• A classifier/lookup/search feature "works" but results are mediocre, and the primary backend (vector DB, embedding index) may be empty or unreachable: a silent fallback to a weaker path (e.g. keyword matching) hides the failure — no error, just lower accuracy. Confirmed 2026-10-01: route.py's embedding classifier had an empty ChromaDB collection and silently fell back to ~28%-accurate keyword matching; found only while benchmarking it against another approach (re-seeded via intent_classify.py --seed). Fix: when a feature has a fallback path, check which path actually served the result before trusting measured quality, and give the fallback a loud signal (log/Health check) so an emptied primary can't go unnoticed.
• A merge/CI gate runs tests in a fresh scratch worktree and reports a "massive error" on a PR that has no conflicts: a new worktree has no node_modules, so the test command fails wholesale on missing deps rather than on anything in the PR. Confirmed 2026-10-01 in webhook-server/merge.js (_defaultRunTests): PR #119 sat 30 days blamed on a merge conflict; real cause was no npm install before npx vitest run. Fix: install deps in the scratch worktree before running tests; when a gate fails on an unchanged-looking PR, read the gate's actual output before assuming conflict.
• An automated measure/metric (test counts, pass-rate, before/after delta) reads flat, zero, or "no change" for days or weeks and nobody questions it: suspect the measurer before the thing measured. A crashed or unparseable measurement coerced to 0 is indistinguishable from a real zero, so a broken harness looks like a healthy, unchanging system. Confirmed 2026-10-02 on MARVIN's ticket-pipeline measure, blind for ~a month with five independent causes stacked: child process fd limit (256) crashing the runner; a diff-scoped test command that skipped most tests; crash counted as 0 instead of raised; a parser that took the first "N passed" line instead of the final summary line; and a # in the worktree dir name crashing vitest. Fix: make measurement failure a distinct state (a MeasureError, not a number), report per-suite counts, parse the summary line only, run a fixed whole-suite command rather than a diff-derived one, and sanity-check the harness against a known-nonzero baseline before trusting any delta.
• A scheduled job installed on more than one machine "should be working" because health is green, or because the identical job works on the other machine: check which machine's health check actually reads that job's log. A central health check usually reads only its own machine's local logs, so a per-machine copy of the job elsewhere can fail every run and never appear in any report, and a working twin with the same plist hides it further. Confirmed 2026-10-06: the laptop's tidy-agent had thrown PermissionError every night since 2026-07-13 (about 12 weeks, 38 files waiting), while health-check ran only on the mac-mini, whose identical tidy-agent was fine. It was found by chance during storage work. Fix: for any job in health_checks.JOB_PLACEMENT that runs on more than one machine, make sure each machine's log is read somewhere (a per-machine check, or ship the log to the checker). Until then, read the job's own *.err.log on that machine directly before assuming it works.

Phase 4 — Instrument

Each probe must map to a specific prediction from Phase 3. Change one variable at a time.

Tool preference:

1. Debugger / REPL inspection if the env supports it. One breakpoint beats ten logs.
2. Targeted logs at the boundaries that distinguish hypotheses.
3. Never "log everything and grep".

Tag every debug log with a unique prefix, e.g. [DEBUG-a4f2]. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.

Perf branch. For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, performance.now(), profiler, query plan), then bisect. Measure first, fix second.

Phase 5 — Fix + regression test

Write the regression test before the fix — but only if there is a correct seam for it.

A correct seam is one where the test exercises the real bug pattern as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.

If no correct seam exists, that itself is the finding. Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.

If a correct seam exists:

1. Turn the minimised repro into a failing test at that seam.
2. Watch it fail.
3. Apply the fix.
4. Watch it pass.
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.

Phase 6 — Cleanup + post-mortem

Required before declaring done:

• Original repro no longer reproduces (re-run the Phase 1 loop)
• Regression test passes (or absence of seam is documented)
• All [DEBUG-...] instrumentation removed (grep the prefix)
• Throwaway prototypes deleted (or moved to a clearly-marked debug location)
• The hypothesis that turned out correct is stated in the commit / PR message — so the next debugger learns

Then ask: what would have prevented this bug? If the answer involves architectural change (no good test seam, tangled callers, hidden coupling) hand off to the /improve-codebase-architecture skill with the specifics. Make the recommendation after the fix is in, not before — you have more information now than when you started.