claudeloop testing philosophy

Codex skill mirror of .claude/skills/claudeloop-testing/SKILL.md. When this guidance changes, update Claude skill, Cursor rule, and .agents/skills/ in the same PR.

Layout

tests/domain/           # pure unit + Hypothesis property tests
tests/application/      # fakes for every port (fakes.py), zero real I/O
tests/infrastructure/   # adapters (incl. scripted test-agent unit tests)
tests/cli/
tests/live/             # opt-in free + paid live tiers
tests/live/system/      # marker: system — real FS/git/CLI + scripted agent

Run: pytest (skips live and system via addopts).
System harness: pytest -m system.
Live free: pytest -m live. Paid: pytest -m "live and paid" --run-paid-live.
See docs/guides/live-testing.md.

Coverage: pytest --cov-report=term-missing (already the default via
addopts in pyproject.toml).

Test-only agent gate (composition root only; not a user feature):
CLAUDELOOP_ALLOW_TEST_AGENT=1 + CLAUDELOOP_TEST_AGENT_SCRIPT=<json>.

Coverage is per-layer, not global — and why

There is deliberately no blanket --cov-fail-under in pyproject.toml.
CI enforces coverage separately per layer: 100% on domain/ and
application/ (zero I/O, zero third-party deps — no excuse for an untested
branch, and this is the code deciding whether an unattended multi-day run
keeps going, waits, or gives up), a lower floor on infrastructure//cli/.
When adding domain code, run coverage scoped to it and treat anything less
than 100% as a defect, not a number to negotiate down.

Fakes, never unittest.mock.Mock, for ports

Every port in application/ports.py gets a real class
implementing the same Protocol — checked by mypy --strict against the
port shape. A Mock has no such check; a port method rename silently
breaks nothing in test code, possibly not even at runtime. If you're
tempted to from unittest.mock import Mock for a port, write a small
FakeXxx class instead.

FakeClock / FakeSleeper — never sleep for real in a test

domain/waiting.py is designed entirely around instants
(next_probe_instant() -> datetime), never durations, specifically so
tests never call real time.sleep(). Pattern (tests/application/fakes.py):

clock = FakeClock(start=NOW)
sleeper = FakeSleeper(clock)  # sleep_until(instant) jumps clock straight there, no real delay

This is what lets a test simulate a seven-day rate-limit wait or the
credit-top-up scenario (already covered at the domain layer —
test_credit_topup_sequence_resumes_after_several_failed_probes in
tests/domain/test_loop.py) in milliseconds. Do not use
unittest.mock.patch("time.sleep") — a real fake port is one thing to
reason about; a patched stdlib call is something every test touching timing
has to remember to apply consistently.

Hypothesis property tests are mandatory for numeric/time-based logic

Not a nice-to-have. test_property_credits_probe_never_in_the_past_and_never_exceeds_ceiling
in tests/domain/test_waiting.py caught a real production-shaped bug during
development: probe_count=29, factor=3.0 overflowed timedelta's max
magnitude via unclamped exponential backoff — an example-based test at any
hand-picked probe_count would never have found this. When you add a
numeric config field (an interval, ceiling, factor, budget cap), write a
@given(...) test asserting the invariant holds across the entire valid
input space, not a handful of examples. Two existing property tests to use
as templates: the one above, and
test_property_never_proposes_instant_beyond_max_wait.

Golden fixtures from real transcripts

Prefer capturing real observed SDK/API payloads as test fixtures over
inventing synthetic ones — a real credits_required 429 rejection from a
development transcript is already used this way; the shape of a real error
payload is rarely exactly what you'd guess.

# pragma: no cover — reserved, and always justified inline

Every use must carry a comment explaining why the branch is unreachable,
not just that it is. Two real examples already in the codebase:

if candidate < now:  # pragma: no cover — unreachable: all config intervals are
    candidate = now  # validated positive in __post_init__, so every branch above
    # already yields candidate >= now. Kept as a defensive invariant guard.

A bare # pragma: no cover with no reasoning will be rejected in review —
grep the codebase for the pattern above before adding a new one, and match
that level of specificity.

Full reference

docs/contributing/testing.md.