Writing Tests

A test earns its place by going red the moment a real defect returns — not by executing lines for a coverage number. Every rule below serves that.

Test behavior, not source text

Never grep the implementation as a proxy for what it does. assert "chmod 0444" in script passes whether or not that line ever runs, breaks on a harmless rename, and stays green against a broken implementation that still contains the token. Instead drive the real code path and assert the observable outcome: run the function or script with the real dependency where one can run (a real git in a scratch repo, a real file tree, the real parser, a real CLI against a local server), or where it genuinely cannot, under a stub that records how it was called (a fake install, docker, id on PATH) — and check what actually happened: the file written and its mode/owner, the argv+env a binary was exec'd with, the exit code, the bytes on stdout/stderr, the state left on disk.

The only legitimate exceptions parse data, not imperative logic: validating a config artifact the product ships and loads at runtime (a settings.json, an execpolicy.rules) with its real parser, or pinning a cross-file structural contract two files must agree on when no single process can observe both. A grep of imperative source to confirm it "does X" is never an exception.

A workflow under .github/workflows/ is imperative source, and this repository's own CI is where that rule broke down. A test that opens a workflow, a .github/scripts/ file or a git hook and searches its text for a step name, a flag, a marker comment or a regex is the banned shape, whatever it asserts. It is why a CI change costs a test change here: the tests pin the machinery's spelling, so a rename reds a suite that no defect reached. What the ban forbids is a search of TEXT nothing parsed. Three moves are allowed. Each reads a model some parser built. Parse the file with yaml.safe_load and assert on the DATA MODEL — a job's needs list, an if: string, a trigger's keys. Read a # required-check: or # memoize: marker through tests/_helpers.required_check_verdicts. That helper resolves the marker per job, never anywhere in the file. For a run: body, read its ARGV through .github/scripts/_shell_scan.commands. Assert on a command's words. A flag is a word, so a reflow or an added quote leaves the read alone. A rename still reds it. Best of all, drive the script the step runs and assert what it did. Two excuses license no text search: the cross-file-contract exception above, and "no single process can observe a workflow at run time". The parsed model IS that single process. "--flag" in run_body is the tell. "--flag" in command.words is not.

Fake the SERVICE, never the tool — and never invent its data

A change whose correctness depends on how an external dependency BEHAVES ships an end-to-end test that drives its whole path against that real dependency — write it FIRST, and let stubs fill only what it cannot reach. The trigger is a dependency whose behavior you would otherwise assert from belief: a CLI, a service, a kernel or filesystem semantic, a parser you do not own. Three kinds of change owe none. A change to pure logic — its unit test already drives the whole path, and there is no far side to be wrong about. A refactor that keeps behavior — the tests it must keep green ARE the end-to-end test, and a new one only re-pins what they pin. A fault branch reachable only by a failure you cannot induce for real (a chmod that root ignores, a vanished path, a tool absent from PATH) — inducing it is what a stub is legitimately for, and there the stub PRODUCES the fault; it never stands in for the tool's ordinary work. A stub answers with whatever the test author believed the real dependency returns, so when the bug under test IS that belief — the commonest kind in code that talks to an external API — the fixture encodes the wrong belief on both sides and the suite goes green through the defect.

• A tool that only needs a remote gets a REAL run against a local server — never a PATH stub of itself. Fake the far side, not the thing under your hand: a stub of gh/curl/aws certifies only your reading of its flags, exit codes and output shape — precisely the belief likeliest to be wrong — while a server can be wrong only about the protocol the real tool then rejects out loud. Copy tests/_fake_github.py — real gh against a localhost HTTPS GitHub, trusted via SSL_CERT_FILE. Everything else local looks unavoidable and is not either: a scratch git repo, a temp filesystem, unshare --net, a container, the library's own parser. Stub only what genuinely cannot run — a paid API, a capability the runner lacks — and say in one line why.
• A test that runs a real host tool judges only the output about its own subject. systemd-analyze verify also warns about the runner's own unit files, so an empty-stderr assertion went red in every merge-queue build once a runner image shipped executable units. Keep the lines that name the file the test wrote.
• Never stub a pure local computation — a hash, a checksum, a formatter, a compressor, a parser you depend on. It needs no network, no credential and no privilege, so "it cannot run here" is never true of one, and a stand-in that answers 0 deletes the very step the code under test exists to perform. Compute the expected value in the test from the same bytes the fixture serves (hashlib.sha256 over what your stub curl writes), and let the real tool run: then the two agree only when the code is right.
• A test that serves only AGREEING input pins nothing about a check. Wherever the code verifies something — a digest, a signature, a schema, a version bound — the passing case stays green after the verification is deleted, so it certifies nothing about the check. Drive the refusing direction too, and assert what did NOT happen (the binary never reached PATH, the archive was never extracted): that pins the ordering as well as the refusal.
• "It can't run here" is itself a belief, and it is usually wrong — PROBE IT before you accept it. Nobody sets out to encode a wrong belief; they conclude the real thing is unrunnable from the docs or from intuition rather than from an attempt. Point the real binary at a throwaway server or a scratch tree for ten minutes and read what it asks for. Judge that probe by its effect, never by its exit code: an install run against a deliberately wrong asset name exited 0 and installed nothing. Difficulty is not a reason either — the fixture that makes a real run possible is written once and copied by every later suite, while a stub's wrong belief is re-encoded by hand each time. When a probe genuinely proves the real thing unreachable, record at the stub what you ran and what it did.
• A capability this session lacks is not a reason to ship no end-to-end test — write it and commit it to the suite that HAS the capability (bin/checks/vm/*.bash and its live shards), name the run that will execute it, and treat the behavior as unverified until that run reports. Do not add a new blocking check to wire it: extend an existing surface, or say in the PR that wiring is the maintainer's call.
• When you must stub, feed it a RECORDED response — captured from the real dependency, committed beside the test, with its source endpoint/command written down so it can be re-captured. Scrub it first: a real authenticated response carries tokens, signed URLs and PII a hand-written one never had, and keeping only the fields the code reads is both the redaction and the smallest fixture. One typed from memory or from the docs is a guess in a fixture's clothes; one that cannot be captured (it needs a credential you lack) says so at the fixture, and every test resting on it counts as unverified until something exercises the real path.

The cost is asymmetric and that is the point: a recorded fixture can be wrong only about the past, an invented one about the present — in exactly the direction that hides the bug.

Never assert a constant equals its own literal

assert mod.TIMEOUT_S == 30, assert mod.STAGES == ("format", "elide") — the source-text antipattern with an import in place of a grep. It copies the definition into the test, so it stays green against an implementation that never reads the constant, reds on a retune that broke nothing, and buys a false mutation score (a mutant that changes the number dies against the copy while every real consumer goes unmutated).

Assert the constant's effect at the surface a consumer sees: a default binds (a step over refuses, a step under proceeds); an ordered set comes out in that order in the rendered table; a wire string reaches the wire when the real emitter runs; a tuning level yields its property (a 95% interval captures 95% of the distribution).

Two near-misses. A literal that arrives as observed output — assert uploaded == "chart.svg" after a real upload — is behavior. An expected value derived from an SSOT file the product also reads is a config contract (the data exception above). The tell is where the value comes from: read out of the module under test → delete; produced by running it → keep.

Deleting is often the whole fix — the effect is usually already covered. Check first, and name the covering test in the PR.

Prove the test can fail (non-vacuity)

• A bug-fix test isn't done until you've watched it fail on the unfixed code. Revert the fix (or run against git show <base>:<file>); confirm it goes red for the bug's signature, not a setup/import error; restore; confirm green. State "red on old → green on new" in your self-critique. When reverting is awkward, invert the branch the fix added and confirm a test goes red.
• For a bug in how code reads a tool's output, the red must come from the tool's REAL output, live or recorded (above), never from a stub's invented answer. A red that an invented answer produces proves only the stub author's belief. A test whose named bug the real tool cannot produce gets rewritten to the input that real output does break, or deleted. #6733's nil-Labels stub went red on a nerdctl exit code that nerdctl 2.3.5 never returns, and the test it fed then passed on every mutant.
• New code with no pre-fix counterpart: prove non-vacuity by mutating the new logic (flip a comparison, drop an emit, return the wrong status) and confirming a test catches it. CI's mutation gate does this for JS, Python and bash alike — the bash lane is .github/workflows/bash-mutation.yaml, a tree-sitter engine (tools/mutation/bash_mutate.py) at a floor of 0 survivors. It judges only a shell file that has a tools/mutation/bash-*.toml, so for a shell file with no config the invert-and-run check is still yours; uv run python tools/mutation/inventory.py says which those are.
• A mutation runs whatever the mutated code runs, so a test of a side-effecting entry point must be INCAPABLE of that side effect. A case that asserts "the refusal came before the command" by leaving the real binary reachable performs that command the moment you mutate the refusal. Stub the binary on PATH for EVERY case, not only the ones that expect it, and have the stub record its argv. An absent record is the stronger assertion. Mutating evals/parity/dispatch.py's authorization check dispatched four paid campaigns on main (#6697).
• Cap every local non-vacuity run at 2 minutes TOTAL across the whole change, and reach for CI's mutation lane past that. Time the targeted run BEFORE you mutate: a case whose green path already takes tens of seconds costs that again per mutant, and the run that overruns is the one whose test waits out a real sleep, a network timeout or a retry ladder instead of asserting the stop landed. That overrun is itself the finding — a test that passes by waiting is asserting nothing about what it waited for, so shorten the wait and assert the effect (the child is gone, the file is unlinked), then mutate the fast version.
• Assert the invariant a bug violated, not today's symptom, so the whole class can't recur by another path. When that principled test is hard to write, the difficulty usually points at a design smell.
• Rewriting a detector? Diff the OLD and NEW implementations' hit sets over the whole real corpus once, in the migration's own test file, and list every intended divergence — both implementations are green on a tree that has no violations, so a differential run is the only thing that shows which direction each one silently moved. Delete that test in the same commit that deletes the old implementation: it needs both sides to mean anything.
• A test driven from a list asserts the list is non-empty, at the one point that reads it. A parametrize over an empty sequence collects zero cases and reports a clean pass, and so does a for loop over nothing — so a helper that stops finding rows (the table was renamed, moved, or its parser broke) silently turns every test resting on it green. The read is where the guard belongs, not each consumer: assert rows, "read no rows from X — every case below would pass over nothing". Same shape wherever a suite discovers its own inputs — a glob of fixture files, a directory walk, a config query.
• A fix's own comment is the spec its test must be driven from. When the fix claims a generality property in a comment — "matched on the phrase, not the exact wording, so a reworded release still routes here", "names EVERY path level", "any of these retryable phrasings" — that sentence is the behavior under test, and a test exercising one example of it is instance-pinned: it passes while a regression narrows the code back to the single case the comment promised to generalize past. Drive the test from the claim (every reworded variant, every path level, every alternation member), not from the one input that first triggered the bug. "Comment promises more generality than the test exercises" is the cheapest reviewer tell for a hollow regression test.

Pinning a KNOWN DEFECT: assert the correct behavior and xfail it

A test that asserts a defect reads to the next session as the intended behavior. Two vm listing tests each pinned a real fail-open, and each survived months because the name, the assertion and the docstring all said the code was working as designed.

• Default: assert what CORRECT looks like, and mark the test @pytest.mark.xfail(reason="<why it is deferred, and where the fix lives>", strict=True). The suite documents intent instead of the bug, and strict=True is what closes it: the session that fixes the defect gets an XPASS failure until it deletes the marker, so the fix cannot land while the file still calls the behavior expected. This repo sets no xfail_strict in pyproject.toml, so strict=True is written per marker — without it an XPASS is silently green and the whole mechanism is inert.
• A frozen-record corpus is the one exception, because its whole job is to hold the bytes the old implementation actually produced — an xfail there fights the regeneration. Mark the scenario instead (pytest.param(..., marks=pytest.mark.xfail(...)) where the corpus is parametrized, or a DEFECT: line in the scenario comment where it is data).
• An xfail is a defect you are FIXING, so the usual reference is the PR that fixes it — never file an issue to hold one. A defect you can state as an assertion is a defect you can fix in this session; pin it only when something outside your reach blocks the fix, and say in the marker what. grep -rn "DEFECT:\|xfail" tests/ is then the list of known-broken behavior.

A mutation harness must assert its edit LANDED before it runs the test. A sed or patch whose anchor no longer matches edits nothing, the suite passes on unmodified code, and the harness reports SURVIVED — which reads as a test gap and sends you auditing tests that were fine. Any refactor that moves a line (a dataclass conversion, a rename) silently no-ops every mutant anchored on it. Diff the file after the edit, or fail the harness when the substitution count is zero.

An accusation needs evidence from the subject, never an absence

A check that concludes from a missing artifact reads a dead environment as a violation. An absent log, an unreadable output file, a process that exited before it wrote anything — each is evidence-shaped and proves nothing. Assert that the subject ran and produced the artifact, then judge what the artifact says. A check that skips the first step goes red loudest exactly when its own harness broke.

Read the subject's state before its output, and read it without touching it. Ask the runtime's own inventory — docker ps -a, a job list, a status endpoint — rather than entering the thing. Entering a stopped container starts it, so the probe destroys the ending it exists to observe.

Measure the SUBJECT, never a process-wide total. len(os.listdir("/proc/self/fd")) == before around a failing call counts the whole process's descriptor table to assert a property about one descriptor. Any unrelated open or close between the two reads then decides the verdict. A cyclic-GC collection is the usual one: its timing follows process-wide allocation, so the test passes alone and reds on a loaded xdist shard. Name the subject in the read instead: list the descriptors that resolve under the path the call should have released, and compare those. The same swap fixes a global counter, a thread total, a whole-directory file count and a total memory figure. A scoped read answers the empty list once it is aimed at a path nothing opens, so pin it against a state that MUST hold the subject, as A test driven from a list asserts the list is non-empty above asks — otherwise the swap trades a flaky red for a permanent silent green.

Fix a test race in the shared helper, never in the one test that raced. Wait on the event through tests/_helpers.py (wait_until, pid_alive). A private liveness check, a one-look-then-sleep, or a sleep before an assertion is a sibling copy: route it through the helper, and fix the helper when it lacks the case. A hold fixture that reports its event, then sleeps to model a holder that never releases, is exempt.

Write the readiness wait when you write the spawn, not after it races. A test that starts a child and then signals, reads or dials it starts it with tests/_proc_ready.spawn_ready, which returns once the child's marker file exists or its port accepts. The child writes the marker only after it arms its handler or binds its socket. A shell vehicle uses gb_wait_until from tests/scaled-poll.bash.

Bound a duration assertion by the time its failure takes, unscaled. elapsed < scale_timeout(N) grows with slow_runner_scale, so a slow runner can lift it past the failure it exists to catch: a 20-second bound doubled to 40 passed a 30-second pack that nothing stopped. Compare against the failure's own duration instead, such as elapsed < PACK_SECONDS.

Exclude your own guards' words from any failure-signature list. A defense usually refuses in the operating system's wording, so Permission denied from a root-owned file is the guard working. A crash-signature or error-string match that does not carve those out scores a correct refusal as a failure.

Would a generalizable invariant have caught it in advance?

For every bug you fix, before writing the instance regression, ask: what property of any correct run did this bug violate — and can a test assert that property without knowing today's symptom? The instance test ("this input now returns C") guards one input; the invariant guards the whole class. Reach for the invariant first; fall back to the instance test only when the invariant genuinely can't localize the regression.

The bugs that most need this are silent degradations — the system emitted a plausible-looking value (a —, a small cost, a token count) that was wrong and nothing failed. Three recurring shapes, and the invariant that catches each with no foreknowledge:

• Completeness / reachability — an output the report promises is never produced (a dashboard column scored only under a config the run never includes reads as "no data yet"). Invariant: every promised output is emitted by at least one cell of a complete run, and "unmeasured" is a distinct sentinel from "empty".
• Conservation / monotonicity — an input the system spends is never counted (a cost function that ignores a token category undercounts silently). Invariant: the total is strictly monotonic in every billable field — bump each field, assert the total moves.
• Economic asymmetry — a resource whose cost model is lopsided (write ≫ read) is used on its expensive side. Invariant: in steady state the cheap operation dominates, and a run that inverts the ratio fails.

When the invariant is hard to state, that difficulty usually names a design smell (a metric that can't be reached, a spend with no single accounting point) — fixing the design beats papering over it with one more example test.

A test that covers one site is the tell that the FIX covered one site. Before you write the regression, list every place that can violate the same property — the helper's other callers, the enumeration's other members, the next file added to that directory — and check the fix reaches all of them. When it does not, change the design so the class cannot recur (.claude/rules/development-workflow.md § Self-critique owns that step), then test the one definition the fix now has. A test parametrized over N hand-pasted sites is not that: it pins today's N and says nothing about N+1 — unlike the SSOT-driven enumeration Cover the whole input domain requires, where a new member with no case fails.

Cover the whole input domain

• An enumerated set (regex class, |-alternation, lookup table, allowlist) needs a case per member, not per line — 100% coverage fires the construct on one input and leaves the rest unverified, and a dropped member is invisible to coverage and often to mutation. Drive cases from the SSOT list (for...of a JS array so a new member without a case fails to compile).
• Parser / Unicode / character-property logic gets a fast-check property test over its real domain (joiner scripts, emoji parts, every invisible class, lone surrogates), pinning invariants — idempotence, output-is-a-subsequence, never-throws — not just ASCII examples.
• Code that writes into a user's filesystem gets driven through hostile pre-states — missing, regular file, directory, valid symlink, dangling symlink, wrong permissions — each asserting a well-defined outcome (post-condition met or a clear failure), never a silent exit-0 or a leaked raw cp:/ln: error. Test the reinstall-over-stale-state re-run too.
• Teardown or extract code that runs under a fd an outer harness may own needs a real-container e2e, because a fd collision is invisible to line, branch AND mutation coverage: the subshell aborts on the plumbing before the guarded command runs, so nothing errors and every in-process test still passes. Commit it to bin/checks/vm/*.bash, per the capability rule above.
• Redaction-test needles must be credential-shaped, never low-entropy — the engine skips placeholder-looking values (repeated-char runs, CAPS_WITH_UNDERSCORES metavariables, changeme), so a low-entropy fixture silently asserts nothing. Use the canonical shared needle from tests/secret-format-samples.json.

An e2e test that stubs its own subject is a unit test wearing a badge

An e2e test that monkeypatches or re-implements the component it names can pass
while the real boundary is broken. For each "end-to-end" test, ask: is the named
component actually executed, or stubbed? Drive the real component and assert an
observed side effect; reserve stubs for genuine external dependencies. Where a
real substitution can't run, pin the duplicated contract with a drift guard —
and name it as one (below).

A test of a collaborator is not a test of the wiring that reaches it. Calling a
production function directly skips everything between the entry point and that
function, so the suite stays green through two distinct failures. The call site
can vanish: the behaviour moves into the caller, no assertion goes red, and a
dead-code check then reports a function the suite still exercises. Or the
environment can differ: the direct call never sees an env key the suite's
conftest strips or a marker the collector applies, so an inert fix reports as
working. Both fixes are one fix — drive the live entry point (a pytest
subprocess over a fixture directory, the CLI, the collector) and assert its
observable effect: the skip and its reason, the emitted argv. Never exempt the
orphan instead.

Drift guards are a smell to NAME, not launder

A test that asserts two duplicated sources agree — a hand-maintained copy (a
literal list, an example map, a mirrored constant) that must match a separate
config/file/other-language copy — means you don't have an SSOT. The honest moves
are exactly two:

1. Kill the duplication — make one source authoritative, or generate the
   second copy at build time so it can't drift. When consumers share a language,
   check first whether one already reads the other at runtime; if so, hoist the
   value to a single sourced file and delete the guard.
2. Keep the guard and mark it in the open — only when a true single source is
   genuinely infeasible (a hard cross-language/cross-process boundary, an external
   value you don't control): mark it
   @pytest.mark.drift_guard("<why a true SSOT is infeasible>"), naming the
   concrete boundary.

The banned move is relabeling the guard to dodge that — calling a
copies-agree test an "SSOT contract" / "coverage contract" / "portability check"
so it reads as principled rather than duplicated. The tell you're laundering: the
framing makes duplication-with-a-guard sound like duplication eliminated
("pinned to the SSOT" when nothing was unified). This binds regardless of language
and regardless of whether a lint fires — "the check didn't catch it" is not a
defense; widen the check.

Contracts move with their data

• SSOT contract tests must change in the same commit as their data. When a
  deny/allow list, generated file, or doc has a round-trip test ("cases exactly
  cover the live config" / "committed output == regenerated output"), editing the
  source without updating the test is a silent CI break. Search for such a
  contract test before landing any change to the data it guards.
• Config-derived ordered lists: derive the test's expected order from the same
  config file. Reordering entries in the config silently breaks any test with a
  hardcoded copy of the order.
• When a fix repoints a dangling reference, add a repo-wide static scan for the
  whole class. After fixing a "referenced X was deleted" bug (file path, image
  tag, service name, config key, workflow filename), add a scan driven on
  git ls-files output that asserts every referenced X of that kind still
  resolves. These bugs hide on opt-in/cost-gated paths that no functional test
  exercises; a cheap static contract catches the entire class at once.

Stubs

• Don't write a stub. Drive the real thing. A stub encodes your reading of a dependency; the real dependency encodes its own — and the two drift the moment the real tool changes a flag, an exit code, or an error format. The stub then silently greens invocations the real tool would reject. Use the real binary against a fixture directory, a recorded interaction, or a container image pinned in CI. A stub is licensed only when the real thing genuinely cannot run in the test (a paid API, hardware, a wall-clock boundary you cannot fake) — and the stub definition site says which of those applies. "Faster to write" is not a reason.
• A stub that ANSWERS for a dependency sits on both sides of the test, so no assertion over it can refute the reading it encodes. A response fake supplies the input and defines the expected output, so the test agrees with your belief about the dependency however wrong that belief is. Count the suites resting on one stub: that is how many go green together on one wrong belief.
• When a stub is licensed, it must reject what the real tool rejects and consume what it consumes. A stub that accepts every flag pair certifies only your reading of the interface; one that exits without draining stdin under set -o pipefail causes the writer's write() to get EPIPE (rc 141) intermittently, independent of pipe-buffer size. Reproduce the argv/stdin/env behavior the caller depends on, and add cat >/dev/null in the body when it stands in for a pipe consumer.

Cross-platform

Host-facing code (setup.bash, bin/claude*, uninstall) hits BSD coreutils on macOS, so mark its tests @pytest.mark.cross_platform — the cross-platform-tests.yaml matrix then runs them on Linux and macOS; marking is all it takes. Keep them OS-agnostic (shutil.which, tmp_path, no GNU-only flags).

A cross_platform test is untimed until a main run publishes the refreshed durations map to R2 (the pytest gate's upload step — the map is never committed), so the cost-aware sharding (tests/_sharding.py) weights it at the p90 of known costs to keep it from clumping. That's still only an estimate: the WSL2 DrvFs legs run ~4× slower (FS/process ops cross the 9P bridge), so a subprocess/FS-heavy new test can overload one DrvFs shard and blow its timeout-minutes. A shard that times out or runs far longer than its siblings is a load-balance bug to fix at the root (the test is too slow/subprocess-heavy for a host test, or the durations/shard sizing need refreshing) — never a flake to re-run past. Prefer in-process assertions over spawning python3/jq/git per case, precisely because subprocess spawns are what the DrvFs bridge taxes.

Keep the suite honest and readable

• Never skip or weaken a test unless asked — including silently dropping an assertion while refactoring, and including loosening one to make a finding "go away". If a test is wrong, fix what it asserts and say so; don't quietly reduce what it proves. When you parametrize, every assertion and distinct input from the originals must survive (case-specific checks become per-case params).
• Anything a SECOND test file needs lives in conftest.py/_helpers.py and is imported, never re-pasted. Stubs and fixtures, and equally the readers: a parser of a command's output, a normalizer, a path or name derivation — and the RESET LIST for a module's process-global caches and latches, which belongs beside that state in the code it resets, never in a fixture. The trigger is the moment you are about to paste one into a second file — move it then, and change the first caller to import it. Nothing catches this for you: pylint's ignore-patterns drops every test_*.py in the tree, and ignore-paths drops all of tests/, so R0801 never sees a test-side clone, and a copy retyped in a different shape (a loop where the original was a comprehension) defeats a token detector too. Each copy then covers a different subset. A re-pasted reader drifts, and the stale copy keeps passing against a reading of the output the code stopped producing. A re-pasted reset list misses the cache added after it was written, and an xdist worker imports the module ONCE and runs many tests through it, so one test's write becomes the next test's answer — intermittently, because --dist load picks the worker by whoever is idle. --dist loadfile pins a file to one worker, which is what turns "sometimes, somewhere" into a stable answer.
• Prefer exact-equality assertions over in/truthiness, which pass for the wrong reason.
• A test that repeats another test's call shape with a different input is a parametrize case, never a pasted second test. Give a test no docstring or comment that restates its name.
• Prefer a real SSOT over a drift guard — make a duplicated value-set structurally unable to diverge (import the shared module, or generate the JSON from it) rather than testing that two copies still match; when a true SSOT is genuinely infeasible, mark the guard as one, per Drift guards are a smell to NAME, not launder above.
• Don't gate a load-bearing test behind skipif(which(tool) is None) — the tool is missing precisely in CI, so the gate becomes a silent no-op there. Install the tool in the job and let a missing binary error.

Python test idioms

• Resolve the repo root via git rev-parse --show-toplevel, not
  Path(__file__).resolve().parent.parent — depth-based parent-walking silently
  breaks when test files are moved.
• Don't add from __future__ import annotations unless you need runtime
  annotation introspection (typing.get_type_hints(), Pydantic) — dict[str, str]
  and X | None work natively in Python 3.9+.
• Running a module's __main__ via exec() executes real startup side
  effects, including direct os.environ[...] mutations that monkeypatch never
  recorded and therefore never restores. Under pytest-xdist those mutations leak
  to later tests on the same worker. Snapshot and restore the environment around
  such exec() calls, or pre-register every mutated key with monkeypatch first.
• A test that drops a package from sys.modules must put the original back.
  The re-import re-executes the module and builds a SECOND set of class objects,
  so an instance held from the first generation answers False to isinstance
  against the second, and a class held from it answers False to issubclass. The
  failure then surfaces in an unrelated test,
  on whichever pytest-xdist worker ran both — restore the entry in a teardown.
• Keep a platform-only import inside the test body, not at module top. A
  collector or push gate that imports every module carrying a marker runs on every
  platform, so a guest-only or Linux-only top-level import refuses the whole run
  rather than skipping one test. Import it under the skipif it belongs to.
• Stage a fixture you must chmod outside the shared temp root. Widening
  permissions on a directory the test does not own — pytest's basetemp, a
  worker's tmp dir — to let another uid reach a path is a shared-state bug, not a
  filesystem quirk: every other test under that root inherits the change. Make
  your own directory and grant on that.

Declare what you actually drive

.github/scripts/pytest-select.py maps a changed file to the tests that exercise it, from the collection-time import closure, the derived path-reference map (.github/scripts/pytest-data-closure.py, widened over each referenced shell file's declared source closure via .github/scripts/bash-source-closure.py), and each test's # covers: directive (grammar in tests/_xplat.py). The derived map already sees a file your test names through a resolvable path expression — a REPO_ROOT / "…" constant, a module-scope repo-path string, an import-time glob with a literal pattern, a helper symbol whose value comes from such a read — for files git tracks — so a directive there is redundant, and tests/test_covers_directive_derivation.py flags a redundant DATA-file, SHELL or PYTHON one (keep it only with a (derived-keep: <reason>) note on its line). A derived SHELL target is safe to delete only where the selector is its one consumer: keep the directive for a file in xplat_host_files() or a tests/*.{bash,sh} vehicle, because _xplat-files.sh, bash-mutation.yaml's prep job and per-module collection read # covers: under a bare python3 that has neither tree-sitter nor the virtualenv. A # covers: line is still needed where the reference is dynamic: a path built at runtime, an env-var target, a script reached only by subprocess with a computed argv. A .py target inside your test's own collection-time IMPORT closure is redundant, because by_import places it and _sources_covered_by reads the same closure — delete it. So is a load_script() target, which runs under the test's own tracer and which both derive, and a path-reference target coverage OMITS. A MEASURED .py the test runs in a child interpreter keeps its directive: no static route says which child turns measurement on. So does a .py in xplat_host_files(), for the bare-python3 readers above.

• Declare the file you drive, never the smoke test's import. A test that only imports a module to prove it still imports (stops before main()) is not that module's coverer — declaring it as one tells the selector a real behavioral bug is caught here, when only an import error would be. Put the directive on the test that actually calls the function or drives the CLI.
• A bare filename collides. # covers: README.md and a reference to packaging/homebrew/README.md are different files; write the full repo-relative path, and when auditing test source for undeclared references, match the whole joined path, not a matched basename.
• A glob target (scripts/*.py, .claude/rules/*.md) covers every file the pattern matches TODAY — it is sound only when the test's own logic is equally general (it re-reads the directory, or asserts a property every member must hold), never as a shortcut to mark a whole directory "handled" by one narrower test.
• A test that DISCOVERS its subjects at run time needs no directive for the tree it walks. .github/scripts/pytest-walk-closure.py derives the domain from an rglob/glob/iterdir/os.walk over a repo path, a git ls-files/grep/ls-tree argv, a tests/ helper that does either, and a whole-tree contract base class, and the selector adds the test for every changed file in that domain. Print what it derived with uv run .github/scripts/pytest-walk-closure.py <test>. Write a glob # covers: only for a listing it cannot see: one inside a module outside tests/ that the test runs over the real tree, or one under a directory it cannot fold.
• New test, new external file it reads or runs: add the directive in the same commit. uv run pytest tests/test_pytest_select.py -k covers -q exercises the grammar; select() returning SUBSET for your new file (.github/scripts/pytest-select.py <file> <your_test>) confirms it took.

Does the health check need a new entry?

When your change adds a runtime prerequisite, a new failure mode, or a state a user could land in and not know why a launch degrades, ask whether glovebox doctor (bin/subcommands/doctor) needs a matching check — and if so, add it in the same change. A test proves the code works; a doctor entry is how a user discovers it's broken on their machine. Triggers: a new required CLI/daemon/device, a new credential or sign-in the launch depends on, a new symlink or service, a new "silently degraded to host execution" path. The doctor is read-only and must skip (not fail) when an optional tool is absent, but a genuinely required prerequisite that's missing is a fail with the exact fix command. Treat "did I owe a doctor check?" as a standing item on every feature/fix.

Before pushing: fast and targeted only

Never run a full suite while you still have work in flight. Run only the targeted fast tests for the files you changed and their obvious neighbors: timeout 60 uv run pytest tests/test_foo.py -k <name>, stopped early with -x, under the wall-clock budget root CLAUDE.md § Testing sets.

The local post-push verification run is discouraged and must never be waited on (the git-workflow skill) — a session that fires one at all fires and forgets it, and it gates nothing. Never run the full suite locally (root CLAUDE.md § Testing), and mutation testing stays CI-only at every point. The three per-file coverage floors are the exception. scripts/background-coverage.sh measures this branch's changed files under coverage.py, c8 and kcov. It is the checker for the 100% every measured file owes (§ Coverage floors). Run it on its own, never in the same command as a push, and never wait on it (.claude/rules/development-workflow.md § Commands, which owns this). Read its verdict with uv run python scripts/background-verdict.py --name background-coverage before the PR is called done. It gates nothing. A Python or bash miss it names is a candidate, because both run a subset of CI's tests. When a test outside the maps already drives the line, that test takes a # covers: line naming the file. Otherwise the test is yours to write in this change.

Coverage floors

Each floor below is enforced per file, and must be met with representative behavioral tests, not a test per residual branch — a branch that only a hollow test can reach is usually code worth simplifying (or mark it # kcov-ignore-line with a justification).

• Python sources: the floor BLOCKS on the run that executes the suite for a PR, and reports on a push. Two runs block: the pull request run, which every push to a ready PR repeats over its selected tests, and the merge-queue batch, because its tree merges the entries ahead of it. .github/scripts/py-coverage-ratchet.py reads each file against the one committed floor [tool.coverage.report].fail_under in pyproject.toml sets, with config/py-coverage-baseline.json recording the files below it. pytest-combine-coverage.sh fails the gate on that run and prints a ::warning:: on a push one. Two shortfalls stay warnings even on a PR, because the run did not measure them: a SUBSET pytest-scope run's percentage is a lower bound, and an incomplete union measures only the shards that reported. That is by design. So the session that changed a file is the only thing that closes its gap before it merges, and the verdict to read is scripts/background-coverage.sh's, not the check's.
• A measured Python file owes 100%, line AND branch — never the floor. A file resting on the floor has stopped being measured for anything, and the floor's own miss budget forgives a few uncovered lines. So cover the WHOLE file you touch, not only the lines your diff added — every file this branch already changes. A branch destination coverage.py reports as partial (342->347) counts: it is a path no test takes. A task whose subject IS coverage is TOTAL: close every file the run reports under 100%, never the one shortfall that blocked. The ratchet names one file because that is the file it measured, so a paydown that stops there leaves the rest of the list for a session nobody asks. Read the list off the pytest job's own coverage table — every row whose percentage is not 100, partial branches included — and take each to 100%. FAN THE PAYDOWN OUT: split the list into groups of about six files and launch ONE BACKGROUND worktree sub-agent PER GROUP, every one of them in a single tool block. Working the list yourself, file after file, is the failure — the files share nothing, so nothing about the list is serial, and the session's own turn is the one resource a group does not need. Each agent takes the tier root CLAUDE.md § Delegation sets for single-file judgement work, its own group's files with the lines the log named, and one verification command: timeout 60 uv run pytest <the file's tests> -k <cases>. Keep only the core root CLAUDE.md names, and carry on with your own task while they run. Keep a smaller paydown in the session, and every file in the core root CLAUDE.md keeps un-delegated.
• JS hooks (.claude/hooks/*.mjs): c8, 100% lines/branches/functions (pnpm test:coverage).
• Every shell file in the tree, whatever its extension or shebang: enrolled in kcov (tests/_kcov.py), 100% real lines. Discovery is the tree's one shell predicate, _treecheck.is_shell, so a #!/bin/sh script and a shebang-less # shellcheck shell=bash library are enrolled exactly like a .bash one. Each discovered file is enrolled, gated through the driver or the sourcing wrapper that runs it (KCOV_GATED_VIA_VEHICLE, KCOV_TRACED_WITH_SOURCER), or carries its own # kcov-exclude: <reason> line — a new shell file reds the gate until it is one of the three.
• A Python script a test runs as a subprocess is measured when the child's env carries tests/_fake_github.coverage_env(). Add it to that test's env. Never write an in-process copy of a subprocess test to earn coverage.
• Mutation proves the assertions bite. Stryker (JS) runs on CI only — never in this sandbox. Python has no mutation lane. The bash lane is the exception: tools/mutation/bash_mutation_run.py scores each mutant against a private git snapshot, so it is safe to run locally on one module.
• A generative suite is owed by every untrusted-input parser and every security-critical Python module tests/test_fuzz_coverage.py lists, and two gates hold that obligation rather than a percentage: .claude/hooks/fuzz-coverage.test.mjs for the JS surface, tests/test_fuzz_coverage.py for the Python one. Adding a parser without one reds the gate; the alternative is an entry on that file's documented exclusion list, with a reason. A change to a decision such a module makes extends its property suite in the same change: state the invariant the change keeps, and give it a @given case. The gate sees only that a suite exists, so review judges whether it is strong enough.

Cover every new file, and never opt one out silently. Adding a file to a coverage exclusion is a real loss of a safety net — surface the argument and get sign-off first, and you still owe a behavioral test of the invariant that mattered. Enrollment mechanics and kcov blind spots (empty case arms, /dev/kvm probes) live in `docs/mutation-testing.md`.

One example

Weak (source grep): assert "install -o root -g root -m 0444" in entrypoint_src — green even if the install never runs, red on a harmless reformat.

Strong (behavioral): put a fake install on PATH that records mode owner group src dest to a capture file, run the real bring-up, then assert the captured record for the config path is 0444 root root — including the re-run-over-an-existing-file pre-state. It fails exactly when the mode/owner regresses, and survives a rename. (install is stubbed because the test cannot chown to root; nothing about its reply is invented — the assertion is on the argv the real bring-up produced.)