Integ Test Picker

Decide which integration tests to run right now, ranked by two signals: how
stale the last run is (from the committed ledger) and whether recent changes
touch the area a test exercises. It RECOMMENDS only; the orchestrator runs
the chosen tests via /run-integ.

The ranking is a running ORDER, not a budget (AGENTS.md → "Cost is not a
tiebreaker"): when several tests plausibly cover the touched code, run all of
them, and prefer a broad-set fixture whenever the change is cross-cutting.
Priorities decide what runs FIRST; a plan too long for one session is handed
forward per "Running a large plan across sessions" — never silently
truncated, and the wrap-up names exactly which tests were not run.

Inputs

• Optional positional base-ref (default origin/main): the diff base. Use
  the last release tag when picking post-merge; origin/main for a branch.

Data sources

1. Ledger docs/_generated/integ-last-run.tsv (committed, one row per
   test; written by /run-integ on every run):

   • Stale: age > the integ-destroy gate TTL (14 days) — past that
     the marker expires, so a clean result no longer proves today's AWS
     behavior.
   • Expiring soon: age 12–14d. The ledger accumulates in sweep-shaped
     cohorts (one sweep stamps many rows with one timestamp), so a strict >14d
     boolean answers "nothing" or "everything" — always report this tier.
   • Failing: result == FAIL — always a candidate.
   • Never-run: a fixture directory with NO ledger row — highest
     staleness.

2. Recent changes: git diff <base-ref>...HEAD --name-only, mapped to
   tests via the table below.

Steps

1. Discover the universe + ledger state:

   LEDGER="docs/_generated/integ-last-run.tsv"
   now=$(date -u +%s)
   # Union-merge can leave duplicate rows — LAST row per test wins. The
   # header is MULTIPLE `#` lines; skip them all (a single NR==1 guard let
   # later header lines through as phantom stale rows).
   DEDUPED="$(mktemp)"
   awk -F'\t' '/^#/{next} {last[$1]=$0} END{for (t in last) print last[t]}' "$LEDGER" > "$DEDUPED"
   # never-run: fixtures with no ledger row
   comm -23 \
     <(ls -d tests/integration/*/ | sed 's#tests/integration/##;s#/##' | sort) \
     <(awk -F'\t' '{print $1}' "$DEDUPED" | sort)
   # resolve each row's age ONCE, then slice for the views below.
   AGES="$(mktemp)"
   awk -F'\t' -v now="$now" '{
     cmd="date -u -j -f %Y-%m-%dT%H:%M:%SZ \""$2"\" +%s 2>/dev/null || date -u -d \""$2"\" +%s 2>/dev/null";
     cmd | getline t; close(cmd);
     printf "%s\t%d\t%s\t%s\n",$1,int((now-t)/86400),$3,$6
   }' "$DEDUPED" > "$AGES"
   
   # age histogram — makes a clustered cohort visible even when NOTHING is >14d
   awk -F'\t' '{print $2}' "$AGES" | sort -n | uniq -c |
     awk '{printf "%6d tests at age %sd%s\n",$1,$2,($2>14?"   <-- STALE":($2>=12?"   <-- crosses the TTL in "(15-$2)"d":""))}'
   
   # stale (>14d) or failing:
   awk -F'\t' '$3=="FAIL" || $2>14 {printf "%s\tage=%sd\tresult=%s\t%s\n",$1,$2,$3,$4}' "$AGES"
   
   # expiring soon (12-14d, not already failing) — the cliff cohort:
   awk -F'\t' '$2>=12 && $2<=14 && $3!="FAIL" {printf "%s\tage=%sd\texpires in %dd\n",$1,$2,15-$2}' "$AGES"
   
   # one-line cliff summary for the step 4 header:
   awk -F'\t' '$2>=12 && $2<=14 {n[15-$2]++} END{for (d in n) printf "%d\t%d tests expire in %dd\n",d,n[d],d}' "$AGES" |
     sort -n | cut -f2-

   (The date line handles both BSD/macOS -j -f and GNU -d.)

2. Map recent changes to tests (a changed path pulls in the listed tests;
   when in doubt, include the broad set):

   Changed path	Integ tests to run
   src/deployment/deploy-engine.ts, src/deployment/intrinsic-function-resolver.ts, src/deployment/retry.ts, src/deployment/retryable-errors.ts, src/deployment/rollback-executor.ts, src/analyzer/dag-builder.ts, src/analyzer/template-parser.ts, src/cli/commands/{deploy,destroy,destroy-runner}.ts, src/provisioning/{provider-registry,register-providers}.ts	BROAD set (bench-cdk-sample, lambda, microservices, multi-resource, multi-stack-deps, drift-revert, drift-revert-vpc, remove-protection, export) — cross-cutting deploy/destroy. Also cross-stack-references for any cross-stack/exports change.
   src/state/**, src/state/export-index-store.ts	schema-v<N>-to-v<N+1>-migration (latest), cross-stack-references, import-value-strong-ref
   src/provisioning/providers/<Svc>*	the matching service integ (iam-managed-policy, dynamodb-*, s3-*, rds-*, kms-encryption, stepfunctions, eventbridge, sns-sqs-event, cognito, wafv2, route53, …). Custom Resource changes → custom-resource-provider.
   src/provisioning/cloud-control-provider.ts	cc-api-fallback, cc-api-fallback-transitions, recreate-via-cc-api
   src/local/**, src/cli/commands/local-*.ts	the matching local-* test; a cross-cutting src/local/ change → at least local-invoke + local-start-api.
   src/cli/commands/{export,import,retire-cfn-stack}*	export, export-nested-stack, migrate-from-cfn, import-nested-stack, import-value-strong-ref
   src/synthesis/macro-*	macro-expansion
   package.json (cdk-local bump)	local-* cluster (the bump's blast radius)

   Mark every name an agent session cannot run, and offer a substitute. A
   fixture drives itself from verify.sh, or from run.sh invoked directly;
   with neither, /run-integ falls back to a flow needing a bare cdkd deploy,
   which the harness refuses — legal only when a human drives the shell. Such a
   name is a MAINTAINER-only recommendation, never a lane's. Several BROAD-set
   entries are in this state, so derive the split rather than recalling it:

   for t in NAME1 NAME2; do   # the names about to be recommended
     d=<LANE_TREE>/tests/integration/$t
     [ -f "$d/verify.sh" ] || [ -f "$d/run.sh" ] || echo "MAINTAINER-ONLY: $t"
   done

3. Rank the union of {changed-area} ∪ {failing} ∪ {stale >14d} ∪
   {expiring soon} ∪ {never-run}:

   • P0: changed-area AND (stale OR failing OR never-run).
   • P1: changed-area but recently-green — verify no regression.
   • P2: not changed-area but stale / failing / never-run / expiring soon —
     coverage hygiene. Prefer the BROAD set plus a spread of providers, and
     ORDER them rather than capping: a long tail is handed forward, and whatever
     is not run is named in the wrap-up. An expiring-soon test ranks below an
     already-stale one and above a recently-green one; within a tier, oldest
     first.
     Bias up for AWS-coupled, deletion-sensitive, multi-resource paths; pure
     docs / test / skill changes often need NO integ.

4. Render the plan:

   Recommendation: run N integ tests (P0: ..., P1: ..., P2: ...)
   Base: <base-ref> (<changed-file-count> files changed)
   Ledger: <count> stale (>14d), <count> failing, <count> never-run,
           <count> expiring soon (12-14d) — <N> tests expire in <M>d
   Age histogram:
     <the uniq -c output from step 1, cliff rows annotated>
   
   P0 (changed + stale/failing/never-run):
     /run-integ <name>    # <why: which changed path + age/result>
   P1 (changed, recently green):
     /run-integ <name>    # verify no regression in <area>
   P2 (coverage hygiene):
     /run-integ <name>    # stale <age>d / expires in <M>d / never-run
   
   Not scheduled this pass (recently green + untouched): <count> tests — list a few.
   (A count, not a cap: these are DEPRIORITIZED, not excluded.)

   Never render "zero stale" on its own — if a cohort sits at 12–14d, the
   headline is the cliff ("0 stale, but 105 tests expire in 1d").

Running a large plan across sessions

A large plan may span sessions. Treat it as a relay whose hand-off points
are batch boundaries:

• Batches of ~4–5 tests. After EACH batch: (a) verify the account is
  orphan-clean (aws s3 ls s3://<bucket>/cdkd/ --recursive | grep state.json
  returns 0; no live NAT / RDS / OpenSearch / Redshift / ElastiCache / EC2),
  (b) commit the ledger.
• A backgrounded batch loop can have a test's subshell die between deploy and
  destroy, leaving a full orphan (a NAT GW included) — the post-batch state scan
  is what catches it, and the orphan's stack NAME often differs from the fixture
  directory (read it from the deploy log or synth).
• Stop only at a batch boundary: commit the ledger, report "ran N more (list),
  account clean, ~M remain", and leave a memory with the remaining list and
  findings.
• The sweep is DONE only when /pick-integ shows no stale tests left; the
  committed ledger is the source of truth.
• A FAIL that never reached the fixture's assertions is not a cdkd bug —
  open the log before recording it. Two classes: fixture staleness (AWS retired
  an engine / instance tier the fixture hardcodes), and a fixture whose own
  previous run blocks the next because it pins a name AWS holds a cooldown on
  (export recreates one FIXED S3 bucket, and S3's same-name cooldown of up to
  ~58 min answers OperationAborted before phase 1). Record it as such and
  queue the fixture fix. A run whose only purpose is to refresh the
  integ-destroy marker takes a SIBLING from the broad set instead; the SWEEP
  does not — the FAIL row keeps it a candidate, so re-schedule it after the
  window rather than counting it covered.

Important

• Tests run serially — one AWS account; mind VPC / EIP / NAT limits.
• The ledger is only as good as its discipline; treat an absent or
  impossibly-old row as stale.
• "Recently green + untouched" tests are scheduled LAST, not dropped — surface
  the count either way.
• Pure docs / .claude/skills / test-only diffs usually need NO integ; say so
  rather than padding the list.