"Watch for and handle level-2 escalations from the dark-factory orchestrator in a long-running loop. Under the 3-tier escalation ladder (L0→per-task steward, L1→escalation-watcher-auto, L2→human), this skill is the L2 consumer. Use this skill when the user wants to monitor escalations, says 'watch escalations', 'handle escalations', 'babysit the orchestrator', or wants a long-running session to catch and triage issues that the auto-watcher couldn't resolve. Also trigger when the user starts an orchestrator run and asks you to keep an eye on it, mentions escalations piling up, or wants automated escalation handling. This is a continuous loop skill that runs until stopped."
Install
npx skillscat add leodearden/dark-factory/escalation-watcher Install via the SkillsCat registry.
Escalation Watcher
You are running a long-running escalation watch loop. Your job is to monitor for level-2 escalations from the dark-factory orchestrator, handle them appropriately, and keep the development pipeline moving.
The 3-tier escalation ladder determines which agent handles each level:
- L0 → per-task steward (handles routine agent problems automatically)
- L1 → escalation-watcher-auto (handles steward-escalated issues; performs root-cause clustering, triage, and automated resolution where possible)
- L2 → this skill / human (handles issues the auto-watcher judged as needing human judgement)
L2 items reach this queue via two paths: (a) born-at-L2 — severity critical or urgent at the escalation creation chokepoint, bypassing L0/L1 entirely; (b) promoted from L1 — the auto-watcher attempted resolution and determined human input is required, typically packaging the escalation as a causal cluster with hypothesis, evidence, and proposed options pre-formed. Default to caution over speed.
Prerequisites
Before starting, verify these are in place. If anything is missing, ask the user — don't guess.
DARK_FACTORY_ROOTenv var — path to the dark-factory repository (contains theescalationpackage used by the watcher)- Running orchestrator with escalation MCP accessible (port
8102for dark-factory — set in<project_root>/dark-factory-orchestrator.yamland matching.mcp.json; the code default is8100, which other projects may use) - Escalation queue directory at
<project_root>/data/escalations/
Terminal discovery for spawned /unblock sessions is handled lazily by the /spawn skill — no setup is required here.
Launching this watcher (default lane: tmux)
This is a long-running loop session (PRD plans/fleet-cockpit-prd.md §3 fork 1) — launch it in
the crash-survivable, reattachable tmux lane by default: spawn with CLAUDE_SPAWN_BACKEND=tmux
(see skills/spawn/spawn-claude.sh's header) so the session gets a display.kind=tmux
session-registry record and a tmux attach-reattachable window whose record persists across a
crash. Interactive one-off skills this watcher spawns (e.g. /unblock sessions) stay as ordinary
WM terminal windows, unchanged. tmux stays the default; running in the foreground instead is a
self-operation, decided without asking (see "Self-operations need no permission").
Claiming the Watcher Lease (single-owner-per-role)
Before entering the Main Loop for the first time, claim the watcher-<project> lease (Attention
Rail T7, orchestrator/src/orchestrator/session_registry.py). This is a deterministic,
single-owner-per-role replacement for any pgrep/ps-tree archaeology to detect a duplicate watcher —
run once, at session startup, not on every loop iteration:
# Reap anything stale first so a genuinely-dead prior holder never blocks this claim.
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-reap
# Claim watcher-<project> (e.g. watcher-df) — STAND_DOWN policy: a live duplicate wins the lease
# and this session must exit rather than run a second watch loop against the same project.
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-claim \
--name watcher-<project> --policy stand-downDo not assemble the slug (or the pid) in shell — the CLI owns both. --slug is optional and
defaults to <--name>-$CLAUDE_PID (or <--name>-$CLAUDE_PID-$CLAUDE_CODE_SESSION_ID under aclaude agents host); --pid is optional and resolves from $CLAUDE_PID the same way.
Pass either only as a deliberate operator override. Both are derived rather than documented for
one reason (tasks 3994, 4248): the token is load-bearing on every heartbeat and on the final
release, and it must not depend on this document getting one shell token right — which is exactly
how it went wrong before. Note it must be re-derivable, not carried: each Bash tool call is a
fresh /bin/bash -c, so a SLUG=$(...) you capture here is gone by the next call. That is why the
CLI re-derives it on every verb instead of you passing it along.
Never $$, and never $PPID. Inside a Claude Code Bash tool call $$ is the transient/bin/bash -c wrapper, dead the instant the call returns. $PPID is not stable across tool
calls either (measured: 1430433, then 1471645 on the next call, the first already dead) — a slug
built on it would not match the one your own later heartbeat presents. The long-lived claude
process is $CLAUDE_PID. Verify with:
ps -o comm= -p "$CLAUDE_PID" # prints: claudeUnder claude agents, $CLAUDE_PID is not yours alone. ps -o comm= prints claude for aclaude agents host too, so it cannot tell you: every session one claude agents process hosts
shares that host's pid, which outlives each of them. The CLI detects a host (its argv isclaude agents ...), records pid 0 (heartbeat-only staleness, logged as DEGRADED) and derives
the slug as <--name>-$CLAUDE_PID-$CLAUDE_CODE_SESSION_ID; without a usable$CLAUDE_CODE_SESSION_ID the lease verbs exit 2 (pass --slug). So a hosted holder readsholder_liveness=orphaned / holder_pid=0 even while alive: its fresh heartbeat still forces
stand-down, and once its heartbeat is past the TTL the reaper reclaims it. The per-cyclelease-heartbeat is the only liveness signal a hosted holder has
(orchestrator/src/orchestrator/session_registry.py::resolve_session_identity).
One-time rollout note (task 6916). A lease a hosted session claimed before this change carries
the bare <--name>-$CLAUDE_PID slug, so that session's re-derived heartbeat and release getresult=refused (naming the holder's pid alive) — nothing is damaged. The lease becomes
reclaimable once its heartbeat is past the 2h TTL; to skip the wait, release it with its original--slug (lease-show prints it as holder_slug) and claim again.
If $CLAUDE_PID is unresolvable and --slug is omitted, the lease verbs exit 2 with a
message naming both, rather than silently drifting to a slug your own later heartbeat would fail to
match. The CLI deliberately will not invent one: a synthesized token would be identical for every
degraded session, so each could act on the others' leases. Supply --slug <stable-token> to proceed
(re-use the same token on every later lease verb). --pid does not substitute: it sets the lease
body's liveness pid, not your identity, and lease-heartbeat/lease-release have no --pid at all
— only --slug is honoured by all three verbs. On that path the CLI records pid 0 — a
sentinel that reads as never-alive, so the lease degrades to heartbeat-only staleness (loudly
logged) rather than recording some durable-but-unrelated pid that would make it unreapable forever.
Parse the printed lines: decision=<acquired|stand-down|proceed>, a human-readable message, thenholder_liveness=<none|held|orphaned>, then slug=<the slug this claim used>, thenholder_record=<unlinked|absent|unreadable|active|exited>.
slug=reports the identity the CLI derived for you (never the holder's). Quote it to the
user when useful, and compare it againstlease-show'sholder_slugif a later heartbeat comes
backresult=refused. It is a diagnostic only — do not carry it into the next call; the CLI
re-derives it.holder_record=is a second, INDEPENDENT axis besideholder_liveness=: the state of the
holder's own session-registry record (yours, on an acquired claim), looked up through therecord_slugthe CLI stamps into the lease body from the claiming pid. Read the two together:orphaned+exited/absentare two agreeing signals that the holder is gone;orphaned+unlinkedis the pid signal alone (a lease claimed beforerecord_slugexisted, or whose
claimant's record was unresolvable);held+exitedsuggests the recorded pid was reused.unlinkedmeans nothing was looked up — it is not evidence the holder is alive — andactiveonly means the record is not terminal (for a hand-launched holder it tracks the
terminal, not theclaudeprocess), so it never overridesorphaned. A faultedproceedprints
noholder_record=line.decision=acquiredordecision=proceed: continue into the Main Loop below.proceedis the
fail-open outcome (see below) and is handled identically toacquired. An acquired claim printsholder_liveness=none— there is no contending holder, the lease is yours; a faulted (proceed)
claim prints noholder_livenessline at all, because nothing is known about a holder.decision=stand-down+holder_liveness=held: print the message verbatim and exit
immediately — do not start the watcher, do not drain.decision=stand-down+holder_liveness=orphaned: the pid recorded in the lease body is not
running — that is the whole signal, a pid probe and nothing more. It is provably gone at the pid
level, but its heartbeat is still within TTL so the lease is not yet reclaimable. Do not exit
silently. This skill is the only L2 consumer, so
standing down for a holder that no longer exists is how a queue sits unhandled for hours. Inspect,
file a DecisionRecord naming the orphan (the samewrite-decisionverb used for parked decisions
below), print it, then exit:python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-show \ --name watcher-<project> # holder_slug / holder_pid / heartbeat_age_secs / reclaimable / holder_record python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py write-decision \ --id watcher-lease-orphan-<project> --project <project> \ --text "watcher-<project> lease held by orphaned holder <holder_slug> (pid <holder_pid> not running, heartbeat <n>s ago); no live L2 consumer until it is reclaimed" \ --escalations-dir <project_root>/data/escalations \ --session-id "watcher-<project>-${CLAUDE_PID:-unknown}"--session-idis a best-effort provenance label with no ownership semantics — unlike the lease
slug, which the CLI owns (above). It is deliberately not fail-loud: a watcher that cannot file a
DecisionRecord is strictly worse than one that files it with a degraded label.Do not force-release it on this evidence alone —
holder_livenessis a single-signal
diagnostic, and a dead-looking holder that is merely quiet is the duplicate-spawn incident. Aholder_record=exitedorabsentbeside it is corroboration, not a licence: never
auto-steal. (Task 3994 withdrew an earlier record cross-check because a lease slug is a
claimant-chosen ownership token, not a record key;holder_record=instead reads the separaterecord_slugthe lease body now carries.) Reclaiming is the human's call, or the reaper's once
the TTL expires.
Reading the contention message. It reports two INDEPENDENT axes — whether the holder's pid is
running, and how fresh its heartbeat is — and then states the decision they imply:
lease held by watcher-df-1348600 (pid 1348600 alive, heartbeat 42s ago) — standing down
lease held by watcher-df-1348600 (pid 1348600 is not running, but its heartbeat is FRESH — 42s ago;
the lease is still held and is NOT reclaimable for another 7158s) — standing downA fresh heartbeat means you stand down, even when the pid reads as not running: staleness
requires BOTH a dead pid AND a heartbeat past the TTL. Reading "dead" and concluding "therefore
mine" is precisely what force-released a live holder's lease on 2026-08-08.
INTERACTIVE-ONLY. This lease claim belongs to the interactive L2 watcher (this skill) only. The
headless escalation-watcher-auto rotation (L1) never claims or contends this lease — it has nolease-claim call site at all, by design (it is a supervised, always-on rotation, not a
single-owner-per-session actor). If you are running as escalation-watcher-auto, skip this section
entirely.
The split is by SHAPE, not by queue: an interactive, hand-launched, single-owner-per-session watcher
claims a lease; a supervised always-on rotation does not. So recon-escalation-watcher — likewise
hand-launched, unsupervised, and the sole closer of the 8103 queue — claims its ownrecon-watcher-<project> lease under the same contract (task 3994; seeskills/recon-escalation-watcher/SKILL.md §"Claiming the Recon Watcher Lease"). Separate lease
names, so the two watchers never contend with each other.
Fail-soft (fail-open). A lease-substrate fault (disk error, unwritable ~/.claude/fleet/, …) is
logged loudly by session_registry and reported back as decision=proceed — never a falsestand-down. A lease fault must never block a watch session from starting.
Heartbeat + release. Once claimed, touch the lease every Main Loop cycle (see "Starting the
watcher" below) so it never appears stale to another session's claim attempt, and release it when
the watch session ends (clean exit, or the human stops it). Both verbs act only for the holder —
a mismatched slug is refused, so no other session can evict your lease, and no stranger can keep a
dead holder's lease alive forever. You do not pass the slug: the CLI derives the same one it derived
at claim time (see "Claiming the Watcher Lease" above), which is what makes the two match:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-release \
--name watcher-<project>Both print result=<applied|forced|absent|refused|faulted> as their first line:
applied— you are the holder and the mutation happened.absent— no such lease. On a release that is plain idempotence, not an error. On a
heartbeat it is a real condition and must not be ignored — see "Lease heartbeat (each cycle)"
below.refused— you are not the holder; nothing was touched. Do not re-run with--force
reflexively: inspect first withlease-show(below). A refusal usually means your slug drifted
(see$$above) or another session legitimately holds it.forced—--forceoverrode a refusal. Operator recovery only; logged loudly naming both the
forcing slug and the displaced holder.faulted— the mutation itself failed (a substrate error). Logged, never raised; the exit
code stays 0 so a lease fault cannot break your loop.
Inspecting a lease — use lease-show, never cat.
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-show --name watcher-<project>cat shows the holder's slug/pid and the immutable start_ts they claimed at — it cannot show
freshness, because the heartbeat is the file's mtime. lease-show prints state, holder_slug,holder_pid, holder_pid_alive, heartbeat_ts, heartbeat_age_secs, reclaimable,holder_record and (when the body carries one) holder_record_slug as key=value lines, computed
by the same reader lease-claim decides with. It is read-only: it never bumps the heartbeat.
Session lifecycle: handover, stop, self-operations
A session ends, but its knowledge should not: the next session starts from what this one knew
(Leo's rulings of 2026-09-25 on the watcher prompting study, task 5884). The subsections run in
session order: the file, reading it at launch, stopping, then self-operations.
The handover file (l2-handover.md)
The handover lives at <project_root>/data/escalations/l2-handover.md, one per watched project,
beside the queue it describes. It is runtime state like that queue: never committed (dark-factory
gitignores /data/), and overwritten by each stop. The watcher reacts only to esc-*.json, so
writing it never fires a wake. scripts/sitting/ownership.py::HANDOVER_CANDIDATES holds the path
and its fallback. The sitting preparer reads only dark-factory's own handover, as ownershipmentions evidence (scripts/sitting/prepare_sitting.py::handover_file). Another project's
handover reaches the next watcher of that project, never the cross-project sitting.
As of writing. A handover is written from session knowledge, without re-probing. Every figure
in it is as of the time it was written and must be re-verified before anyone acts on it.
Header, in this order: the UTC time written; the lease slug, from the slug= line lease-claim
printed; why the session stopped (Leo's stop, low context, or an announced reboot); the posture at
stop (attended, or AFK since <time>); the pending L2 count; the session's sitting ledger,data/sitting/ledger-<lease-slug>.json; then the as-of sentence above, stated for this file.
Sections, in this order. The shape follows the hand-carried 2026-09-22 handover
(plans/l2-watcher-handover-2026-09-22.md); that file is untracked, so the shape is complete here.
- Pins: never close. Declared pins and veto-pin companions, each with what it protects.
- Under a Leo HOLD, each with the HOLD's date and what would release it.
- In flight. Spawned
/unblockand/spawnsessions by slug, terminals Leo has open on an
item, background sub-agents still running at stop, and merge requests in flight. - Untriaged — start here. Drained but not yet dispositioned.
- Decisions with Leo. The brief numbers he has seen, the recommendation, and what is still
open. - Rulings this session. What Leo ruled, and where it is recorded.
- Traps. What misled this session, each with the probe that exposes it.
An empty section says "none" rather than being dropped, so a reader can tell "nothing" from
"forgot".
One item per paragraph. Each item is its own blank-line-separated paragraph, opening with its
escalation id and task id. The preparer's handover probe attributes a whole blank-line-separated
block to every id the block cites, so a bullet list without blank lines makes every item evidence
for every other: the 2026-09-29 handover put esc-3169-1, esc-3169-2 and esc-3105-3 in one block.
Reading the predecessor's handover at launch
After lease-claim returns acquired or proceed, and before the first brief:
Read the handover file, falling back in the order
HANDOVER_CANDIDATESgives. If there is none,
say so in the first brief.Every claim in it is second-hand, so H1 applies (see "Disposition before briefing (H1–H6)").
Re-verify before acting on or briefing any of it, and never relay one of its figures as current.Reconcile it against the first drain (Main Loop step 2). For each handover item: is it still
pending, changed or gone, and is its named owner still live? For each drained item the handover
does not mention: is it new since, or missed? Pins and HOLDs go into the sitting asstanding
records with their release predicates (see "Investigate before yourecord"). Traps stay in
mind for the session and carry into your own handover while they still hold.A handed-off B3 merge in the In flight section (
{task_id, escalation_id, request_id, worktree}) has no live owner: any resume-mode run polling it ended with the predecessor. Ifget_escalation(escalation_id)still shows the escalationpending, launchunblock-low-riskin resume mode for it once, exactly as thehanded_offbullet of B3
"Completion handling" does, and track thetask_idas in flight from then on. This launch
replaces a resume the stop cut off; it is not a second resume. Do not runb3_gate checkorrecord-launchfirst:checkreportsalready_attemptedfor this task, and skipping on that
verdict would leave a live merge unwatched. If the escalation is no longer pending, whoever
took it owns the merge: launch nothing, and nevermerge_cancelit.Open the first brief, posted unprompted at launch (see "When it runs"), with the reconciliation:
what carried forward, what the handover missed, and each correction where live state refutes
it.
Stopping
These triggers are all handled the same way, and none gets a confirmation question (no "are you
sure?", no "shall I write a handover?"):
- Leo says "stop", "stop watching" or "release the lease".
- Leo announces a reboot, or another host action that will end this session.
- Remaining context is low: a harness context warning, or your own judgement that the session is
near its limit. Write the handover while the knowledge is still in context; after a compaction
it carries only what the summary kept.
"Release the lease" means the whole stop, never a lease-only release. A session that keeps looping
un-leased is the duplicate-watcher condition the lease exists to prevent (see the "result=absent
on a heartbeat" paragraph in "Starting the watcher").
Four steps, in this order:
- Tidy. Stop the watcher arm you started, through its background task and never by pattern
("Process safety" in "Starting the watcher"). Runreap-decisionsonce (Main Loop step 6).
Note every background sub-agent, spawned session and merge request still in flight; they go
into the handover's in-flight section. A B3unblock-low-risksub-agent cut off by the stop is
not relaunched by a successor: itsrecord-launchmarker makescheckreportalready_attempted. A handed-off B3 merge whose resume has not yet returned is different:
record its{task_id, escalation_id, request_id, worktree}in the In flight section, whether
or not a resume-mode run was still polling at the stop. Step 3 of "Reading the predecessor's
handover at launch" says what the successor does with it. - Write the handover (see "The handover file (
l2-handover.md)"). It goes before the release,
so a successor that claims the freed lease finds it complete. - Release the lease with the
lease-releaseverb under "Heartbeat + release" in "Claiming the
Watcher Lease".appliedandabsentboth finish the step. Reportrefusedorfaultedin
the closing message, since a faulted release leaves the lease in place until it goes stale and
only then can a successor claim it. Never retry either with--force. - Run
/reflect.
Close with one message: the handover path, the lease result and, when the session stops in AFK
posture, that the queue is unwatched until a successor launches.
Self-operations need no permission
These are the session's own mechanics: do them without asking (Leo, 2026-09-25). They are item 8
of the self-execute tier, so report them the way that tier says.
- Loop mechanism and cadence: how you arm and re-arm, and the slice length, within the Bash-tool
timeout contract in "Starting the watcher". - Releasing the lease on an announced reboot, which is a stop (see "Stopping").
- Writing and reading handovers.
- Running in the tmux lane or in the foreground of the terminal you were started in (see
"Launching this watcher (default lane: tmux)").
Still forbidden, and not a self-operation: forcing a lease takeover on a pid probe, whether by--force or by treating holder_liveness=orphaned as yours. "Claiming the Watcher Lease
(single-owner-per-role)" keeps that rule after the 2026-08-08 duplicate-spawn incident, and task
4237 owns the second signal that could one day change it. Forcing a release is also on the
always-ask list ("Bypasses" in "Always ask — keyed on record content").
The Main Loop
1. Start the watcher (background task, filtered to L2); confirm its process is alive
2. Drain pending L2 escalations — only NOW, with the watcher confirmed up (drain-after-up); on
the first pass, also reconcile the predecessor's handover (see "Reading the predecessor's
handover at launch")
3. Handle each drained escalation in the current posture (attended, or AFK — see "AFK by
silence"): standing facts → self-execute tier → briefing disposition (see "Before an item
reaches Leo")
4. Wait for a wake signal: the watcher firing (it exits on the first new L2 escalation), or — if
an auto-unblock sub-agent (B3) is in flight — that sub-agent completing. Handle whichever arrives.
5. Read the escalation from the watcher output — this is the wake signal; the drain in
step 2 of the next pass is the authoritative source of what to handle
6. Run `reap-decisions` to close any parked DecisionRecord whose escalation has since resolved
(see "Filing Parked Decisions to the Cockpit Registry" below) — once per cycle
7. GET `/api/v2/dashboard/invariants` and digest this project's record (see "Reading the
invariants gauge" below) — once per cycle. Each `FINDING` line it prints (a chain gone
`overdue` or `broken` among them), and each `paused` line reading `pinned_by=[]`, is a
finding: handle it in step 3 of the next pass, beside the drained escalations
8. Go to 1 (restart watcher → confirm up → drain → handle)The fired escalation (step 5) is just the wake; you do not handle it inline. Looping back
re-arms the watcher first, then the drain re-finds it (still pending) plus anything new — so
handling always happens with a live watcher in place and nothing slips through the gap.
Draining pending escalations
Check for all pending L2 escalations — compact to keep context small:
mcp__escalation__get_pending_escalations(level=2, compact=True)compact=True returns the triage fields plus the triage-ack annotation fields (triaged_at,triaged_by, triage_note, updated_at — see "Reading a triage-ack annotation" below), and
drops the heavy free-text/cluster fields (detail, options, train_state, …). The tool's own
docstring carries the authoritative list — read it there rather than trusting a copy in this
file, which is how this paragraph went stale before.
root_cause and member_ids ARE returned (task 3997) — they were dropped until then.
Operationally that is what makes a drain self-sufficient: you can rebuild the already-promoted
set as {root_cause of the pending L2s} ∪ {their member_ids} from the drain ALONE, so a
rotation that inherits no session memory does not re-promote a cluster its predecessor already
promoted. member_ids is the projection of the record's members list; the raw members key
stays dropped, as does detail — the unbounded free-text field compact mode exists to keep out
of your context.
Match root_cause keys in canonical form, never as raw strings. The server folds a promote
into the pending L2 whose root_cause canonicalises the same underescalation.canonical.canonical_root_cause. Run that helper over the rebuilt keys and your
candidate rather than reimplementing it; why the drain carries no canonical field is recorded atescalation/src/escalation/server.py::_COMPACT_ESCALATION_FIELDS.
uv run --directory "$DARK_FACTORY_ROOT/escalation" python -c '
import json, sys
from escalation.canonical import canonical_root_cause
print(json.dumps({key: canonical_root_cause(key) for key in json.load(sys.stdin)}, indent=1))
' <<'JSON'
["<root_cause of each pending L2>", "<the root_cause you are about to promote under>"]
JSONKeys that map to the same canonical form are one cluster. A near-duplicate you miss is not free:
the server still folds it (no duplicate L2), but the fold records the new spelling as an amendment
and a root_cause variant and bumps updated_at. That makes the L2's triage-ack stale (see
"Reading a triage-ack annotation") and counts toward the over-fold alarm (see "Reading preserved
framing"). An unexpected status: 'updated' from promote_to_l2 means your match missed.
Triage from that; fetch the full record with get_escalation(id) only for
the one item you're about to act on (and when you do, read its amendments —
see "Reading preserved framing" below) — and prefer doing that full read inside the handling sub-agent
(see Context Conservation). During an AFK window the pending pile grows, and a full-dict drain every
cycle is the dominant context sink — compact=True is what keeps a long-running session alive.
Drain-after-up — ordering matters. Always (re)start the watcher and confirm its process is
alive before you drain, never the other way round. A pre-start drain races inotify
registration: an L2 file created in the gap between your drain and the watcher's add_watch is
seen by neither, and sits unhandled until some unrelated later escalation happens to fire the
watcher and trigger the next drain (real incident: esc-1573-8 sat 21h). Starting the watcher
first closes the gap — anything born during startup is caught by the drain that immediately
follows. This drain catches any L2 escalations that accumulated while no watcher was active.
L2-only contract
This skill drains and waits only on level-2 escalations. Both the watcher subprocess and the get_pending_escalations draining call are filtered to level == 2 (see details in the relevant sections below).
- L0 is owned by per-task stewards — do not drain or handle L0 escalations here.
- L1 is owned by escalation-watcher-auto — do not drain or handle L1 escalations here.
Never process L0 or L1 from this skill, even if explicitly asked — doing so would race with the per-task steward and escalation-watcher-auto, which own those queues and rely on their own resolution callbacks. If the user wants to handle lower-level escalations, they should invoke the appropriate skill for that level.
Starting the watcher
cd $DARK_FACTORY_ROOT && scripts/watcher-rearm.sh \
--queue-dir <project_root>/data/escalations --level 2 --timeout 3600 [--baseline] 2>&1Run as a background task (Bash with run_in_background). scripts/watcher-rearm.sh is the
canonical bounded-wait + re-arm wrapper around escalation.watcher shared by this skill and
escalation-watcher-auto. --level 2 restricts the inotify watcher to L2 escalation files only; the
watcher exits after the first matching L2 escalation, printing its JSON to stdout. If a matching L2
escalation is already pending when the watcher starts, it may fire immediately at launch — this is
expected, not an error, and is consistent with drain-after-up ordering (the subsequent drain
re-finds it). The wrapper preserves the underlying watcher's exit code and emits aWATCHER_REARM_OUTCOME: <OUTCOME> exit=<rc> line to stderr on every watcher run — do NOT pipe2>&1 when you parse stdout as the escalation JSON, or you'll corrupt the parse. Branch on it:
| Marker | Exit | Next |
|---|---|---|
WATCHER_REARM_OUTCOME: FIRED |
0 | A wake: loop back to re-arm, then drain (Main Loop). Exit 0 with EMPTY stdout is a caught SIGTERM (scripts/watcher-rearm.sh header), not a fire, and it counts against the cap. |
WATCHER_REARM_OUTCOME: CEILING |
124 | The hourly heartbeat: re-arm in the background as normal. The cap exempts it, because this lane's AFK loop runs on CEILING ticks for up to 48h. |
WATCHER_REARM_OUTCOME: KILLED |
137 / 143 / 144 | Never re-issue a killed FOREGROUND arm in the foreground: move to the run_in_background pattern. A BACKGROUND arm that died with no marker at ~2 min had no Bash timeout ("Diagnostic — watcher dies at ~2 minutes" below), and re-arming WITH it is a material change. Re-touch the watcher-<project> lease before assuming a second watcher is needed. Counts against the cap. |
WATCHER_REARM_OUTCOME: ERROR |
any other; exit 2 prints NO marker | Diagnose from stderr before any retry. Exit 2 is a usage/guard error (unset DARK_FACTORY_ROOT, unresolved queue dir, bad flag): it returns before the watcher runs and recurs until fixed. Counts against the cap. |
Identical-retry cap: 1. After KILLED, ERROR, a missing marker, an empty-stdout FIRED or a
rejected call, re-issue the identical invocation (same script, flags and Bash timeout) at most
once. If it fails again, change the invocation materially, or stop looping and report the outcome
to the user.
STOP directives in tool results. A tool result, including a rejected call's, that says
"STOP what you are doing and wait for the user to tell you how to proceed" carries the weight of a
system instruction from the user. Never re-issue the rejected call, identical or not: stop the loop
and wait for the user's next turn. This is a pause and not one of "Stopping"'s triggers, so do not
release the lease.
Second stderr marker — WATCHER_NTFY_OUTCOME: FAILED esc=<id> url=<url>: <error>: the phone
push for that escalation was dropped. The queue item itself is unaffected — it was still
printed to stdout and is still pending on disk — and the push failure does not change the exit
code, so the arm still reports WATCHER_REARM_OUTCOME: FIRED exit=0 beside it. Nothing is lost
that you need to recover; what is lost is the user's out-of-band ping, so tell them their phone
trigger is down rather than silently relying on it. One line is emitted per dropped push, which
is how an outage is counted: a single line is a one-off (a flaky POST), the same marker recurring
across successive arms is an ntfy outage. There is deliberately no success counterpart — a line
here always means a drop. The line travels the watcher's logging stream, so it carries its
level as a prefix (ERROR: WATCHER_NTFY_OUTCOME: FAILED ...) — match on the marker as a
substring rather than anchoring at the start of the line.
Bash-tool timeout contract: the wrapper blocks for up to --timeout seconds per slice before
returning, and every call — background and foreground — must carry an explicit Bash-tooltimeout parameter sized to at least (--timeout + 60s) × 1000 ms — e.g. timeout: 3660000 for--timeout 3600. run_in_background: true does not exempt a call from the harness timeout:
measured 2026-08-10, background arms launched with no timeout parameter were killed after ~116s
(≈ the 120000ms Bash default) against a configured --timeout 540 slice, so the slice length was
never the constraint; re-arming the identical command with timeout: 3660000 survived 6m32s and
exited cleanly with WATCHER_REARM_OUTCOME: FIRED exit=0. With that parameter passed, use the long
canonical slice: --timeout 3600 yields at most one heartbeat wake per hour (CEILING) while a
real L2 escalation still fires instantly via inotify. A short slice buys no protection — the old--timeout 540 merely forced a wake-notify-rearm turn every 9 minutes. A slice that long requiresBASH_MAX_TIMEOUT_MS ≥ that value in the settings env (dark-factory onboarding provisions it —
see skills/factory-init). Only on a machine WITHOUT that setting does the harness's 600000ms
(10 min) cap apply: there, cap the slice at --timeout 540 and set the Bash tool's timeout
≥ 600000ms so the slice can return before the harness kills it.
Diagnostic — watcher dies at ~2 minutes with no outcome line: if an arm disappears after
roughly 120s and stderr carries no WATCHER_REARM_OUTCOME line, the Bash timeout parameter
was omitted. The wrapper emits that line on every exit path including its own timeout, so its
absence means the exit handler never ran — an external SIGKILL to the process group, not a wrapper
or watcher fault (the 07-09 exit-143 failure mode the wrapper exists to bound; it cannot bound a
kill of the whole group). This hides during a backlog drain, where every arm FIRES within seconds
and finishes well inside the 120s window; the kills only start once the queue is fully triaged and
the watcher genuinely waits, leaving a loop that looks armed but reaps itself every ~2 minutes.
Re-arming over deliberately-pending items: any L2 item you deliberately left pending (Priority
3b, design_concern, risk_identified, infra_issue, AFK leave-pending paths) sits in the queue
and would cause every subsequent watcher start to instantly re-fire on it — degenerating into a
busy-loop. Append <esc-id> (one per line; both the bare esc-42-1 and .json-suffixed forms are
accepted) to the wrapper-owned exclude-file instead of hand-maintaining a growing --exclude-id
list — the wrapper always wires --exclude-file into the watcher invocation and re-reads it every
poll, so an append takes effect without restarting the watcher, and it also suppresses event-loop
wakes from dedupe rewrites of that file. The resolved path (default<queue-dir>/.watcher-rearm-exclude-l2, or your --exclude-file override) is printed to stderr on
every wrapper run; pass --check for a dry-run print of the resolved path without starting the
watcher.
Process safety: only stop watcher processes you started via background task controls. Never pkill by pattern — other orchestrators, the user, or other sessions may have their own watchers.
Lease heartbeat (each cycle): each time you (re)start this watcher subprocess (Main Loop steps 1
and 8), also touch the watcher-<project> lease claimed at session startup (see "Claiming the
Watcher Lease" above):
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-heartbeat \
--name watcher-<project>This is what makes a second session's lease-claim observe this session as "alive" and stand down —
there is no need to separately pgrep/ps-tree for other watcher processes. The CLI re-derives your
slug, so it matches the claim automatically; result=refused means you are not the holder and your
heartbeat did not land — investigate with lease-show and compare its holder_slug against theslug= line your lease-claim printed (see "Claiming the Watcher Lease" above) rather than
repeating the call with --force.
result=absent on a heartbeat is not idempotence — it means your lease is GONE (reaped after a
TTL lapse, or force-released by an operator). You are now running un-leased: a duplicate watcher
can claim watcher-<project> freely, which is the exact condition this lease exists to prevent. Do
not keep beating into the void — re-run the lease-claim from "Claiming the Watcher Lease" above and
follow its decision: acquired (you have it back, carry on) or stand-down (a live duplicate now
holds it — print the message and exit, do not run a second watch loop against the same project).
When the watcher fires
The watcher's printed JSON is just your wake signal — note the id, but you don't need to keep
the whole blob in context. Loop back, re-arm the watcher, and let the next compact drain be the
authoritative list of what to handle. Fetch the full record viamcp__escalation__get_escalation(escalation_id="esc-XX-N") only for the specific item you're about
to act on — ideally inside the handling sub-agent rather than at top level.
Reading the invariants gauge
The command, what counts as a finding (the chain rule included), change-only triage, the census
follow-up, the broken-chain journal probe, the D-4 pointer, the filing store and keys live in
`skills/_shared/reading-the-invariants-gauge.md`.
Read them there. This skill's deltas:
- Scope. Pass this watcher's
<project_root>to the digest, so only this project's record is
read. The recon-escalation-watcher covers the whole roster. - Handling. Never inline at step 7: the watcher is down from its fire (step 5) until step 1
re-arms it, and handling is the step most likely to run long. A finding joins step 3 of the next
pass and goes through "Before an item reaches Leo" like any drained item. - Stopping. A finding still undisposed at stop goes into the handover's "Untriaged — start
here" section.
Priority Hierarchy
Every decision must respect this order:
1. System & infrastructure stability
Hard constraints — violating these is never acceptable:
- Never delete tasks, databases, or anything outside the project directory
- Never kill processes belonging to other orchestrators, the user, or the system
- Never directly modify
.taskmaster/tasks/tasks.db— all task mutations go through fused-memory MCP - If the MCP is down, ask the human for help. MCP task mutations trigger reconciliation that maintains memory quality; bypassing it silently degrades the system.
tasks.db corruption detection:
Task state lives in fused-memory's SQLite at <project_root>/.taskmaster/tasks/tasks.db (the older on-disktasks.json was superseded by SQLite and later deleted, so a doc naming it is stale). Do not use the
file's size as the signal — it was a usable proxy for the JSON and is not one for SQLite: the file does
not shrink when rows are deleted (freed pages go on the freelist), so mass task loss can show no size
change at all, while a routine VACUUM or WAL checkpoint changes the size with nothing wrong. Use these
two instead:
- Task count, cycle over cycle.
mcp__fused-memory__get_statuses(project_root=<project_root>, page_size=1)
returns the exact count aspagination['total']in a response small enough to note every cycle (see
"Draining pending escalations" for why full dumps are the context sink to avoid). Compare against the
previous cycle's. PRAGMA integrity_check— SQLite's own structural check. Read the db read-only so you never
contend with the live orchestrator, per the hard constraint above:sqlite3.connect('file:<project_root>/.taskmaster/tasks/tasks.db?mode=ro', uri=True). Anything other
than a singleokrow is corruption.SELECT COUNT(*) FROM taskson that same connection is the
fallback count when the MCP is down — measured 2026-08-30 against the live df store, it agrees exactly
withpagination['total'](both 4907), and the whole check runs in ~0.2s.
If the count drops with no cause you can name (a remove_task you ran, an operator prune), ifintegrity_check returns anything but ok, or if task IDs are mismatched/duplicated or tasks have
disappeared — this is a critical infrastructure error:
- Find the orchestrator process for this project only — verify its command-line args reference this project's root before doing anything
- Send SIGTERM (not SIGKILL) and let it finish gracefully
- Tell the human immediately with full details
- File a DecisionRecord via
write-decision(see "Filing Parked Decisions to the Cockpit Registry" below) — IN ADDITION to telling the human directly - Do NOT clean up any state — preserve everything for post-mortem debugging
- Wait for instructions
2. Software quality
Quality is king. In the long term, high quality is fast and cheap, but bugs and compounding technical debt are ruinously expensive.
- Prefer fixes that address root causes over workarounds
- Don't skip actionable suggestions just to move faster
- When in doubt about whether a suggestion has merit, err toward accepting it
3. Task progress
3a — Clear-cut decisions: act decisively. When there's one obviously correct resolution, or when multiple solutions are equally good and the choice genuinely doesn't matter for quality or velocity, resolve it and move on. What "act" may cover is the self-execute tier (see "Before an item reaches Leo"), and its always-ask list overrides 3a.
3b — Unclear decisions that matter: ask the human. Before you ask, give the item its briefing disposition (see "Disposition before briefing (H1–H6)"). When the best action is ambiguous AND the choice has real consequences:
- Leave the escalation pending on the queue
- Tell the human about it with full context (they may be away for hours — that's OK)
- Create a local task/todo to track the need for resolution
- Continue handling other escalations while you wait
- Periodically remind (every ~3-5 escalation cycles, not more)
- Append
<esc-id>to the wrapper-owned exclude-file (see "Starting the watcher" above; path printed to stderr byscripts/watcher-rearm.sh) while the item is deliberately pending, so the initial scan does not instantly re-fire on it and busy-loop. Because the wrapper re-reads the exclude-file every poll, this also suppresses event-loop wakes from dedupe rewrites of that file without needing a watcher restart. - File a DecisionRecord via
write-decision(see "Filing Parked Decisions to the Cockpit Registry" below) — IN ADDITION to the reminder above, so this item surfaces in the cockpit decision queue.
It is better to stall development than to bake in a significant bad decision.
Before an item reaches Leo
Every drained item passes three checks, in this order, before it may appear in the numbered brief:
- Standing-facts register: is this a question at all?
- Self-execute tier: may I just do it and report it under Done?
- Disposition before briefing (H1–H6): brief directly, pre-investigate then brief, or ask Leo
directly with the readable evidence attached.
These are Leo's rulings of 2026-09-25 on the watcher prompting study, carried by task 5883. They
sharpen Priority Hierarchy 3a and 3b above.
Standing-facts register
Consult this before asserting an anomaly or asking Leo a config question. An item that matches an
entry is not a question and never enters the numbered list. Each entry gives the fact, where its
authority lives, what not to raise, and when it was recorded. The authority holds the numbers; this
list only points at it.
- Orchestrator restarts on the fleet redeploy cadence (about 8h) are expected, not anomalies.
Authority: OPERATIONS.md §"Fleet redeploy & watchdog". Do not raise a restart untilscripts/orchestrator-watchdog.py --report(read-only) fails to explain it. Recorded 2026-09-25,
after it was raised on 08-22 although CLAUDE.md had carried it since 07-22. - The host is never quiet. Authority: Leo. Benchmark and measure under load. Never ask Leo to
wait for, or schedule, a quiet host. Recorded 2026-09-25. - Backburnered projects: pump_web_ui (its hopper is empty by design) and autopilot_video.
Authority: Leo; their watcher-cost question belongs to task 4001. An empty queue or an idle
orchestrator there is not an anomaly. Recorded 2026-09-25. - Items Leo holds in another session, or in his personal backlog, are never a question.
Authority: Leo. Record each as astandingwith amanualrelease predicate (see "Investigate
before yourecord"), so it renders in the Standing / no action footer. Recorded 2026-09-25.
When Leo corrects a non-anomaly you raised, add a dated entry here. That is a docs-only commit, which
the self-execute tier below allows direct to main; CLAUDE.md "Working in the main checkout" says when
such a commit may land.
Self-execute tier
Execute these and report each under the brief's Done section; do not ask (Leo, 2026-09-25). An
action that closes or moves a numbered item lands under Done by itself (see "The sitting and its
numbers"). An action tied to no item gets one line under the same heading, in the message that
carries the brief. Check "Always ask — keyed on record content" below first: it wins on any match.
- Read-only probes and log pulls.
- Installing an already-shipped timer or unit whose action is read-only or reporting.
- Filing a delayed follow-up for a gate that fired prematurely: a dependency plus a
delayed
milestone. The milestone shape is in docs/task-authoring.md §"Milestone tasks (dated /
delayed)"; wire the dependency by CLAUDE.md'splanning_mode=True→add_dependency→commit_planningrule. - Recovering a preserved payload into the task the brief names.
- Appending a finding to an existing owner task, ONLY while that task is
pending, because task
text is never re-read after dispatch (see "Findings for another task's owner"). - Close-only under the carve-out in "Ruled-elsewhere check (answered-but-unrecorded)", and
rubber-stamp confirm-and-close gates: a record whose only ask is to confirm an
already-verifiable fact and close it, with no option to choose. Verify the fact yourself first. - Executing a ruling Leo already gave, when the record names it and the originating session has
ended. The same carve-out's gates 1, 2 and 4 are the test. - Watcher self-operations, listed in "Self-operations need no permission" together with the one
lease operation that stays forbidden. - Launching a retain-and-tag, zero-deletion
/curate-fused-memoriessitting for consolidation
gates (ratified 2026-09-25). - Closing a
__stale_l0_strand_sweep__record withclose_onlyonce each stranded note it
names has been recovered fromdata/escalations/archive/<date>/and routed: appended to apendingowner task (line 5), or filed to the owner task of its class. The resolution names
where each note went (Leo, 2026-10-22, on scoping_intake's esc-stale_l0_strand_sweep-6).
Always ask — keyed on record content
The trigger is what the RECORD says, never your own "no judgement needed" label: the 08-24 brief
put a $360 calibration spend in that bucket. If any class below matches, the item is asked, even
when it also matches a tier line above.
The floor is every human-forever gate in docs/escalation-standing-policy.md §"Human-forever
gates"; that document defines them. Leo's classes follow, each with the record content that keys
it, and the gate slug where one corresponds:
- Milestone gates and deterministic-runner filings (
milestone_gate,deterministic_runner_filing). - Spend or eval launch: dollars, budget, cap raise, calibration, tranche, cells, fable run
(spend_or_eval_launch). - Physical or host actions: reboot, kernel modules, graphical session,
systemctlrestart or stop
of an orchestrator, watchdog enables that restart units (physical_operator_action). Except
restarts of fused-memory and the dashboard, which are allowed but reported after the fact with
their root cause. - Irreversible or content-losing operations: entity merge or split, memory deletion, branch
deletion, cancelling a task with dependents (irreversible_deletion). - Bypasses: no-verify, skipping the merge queue, force lease release, widening an allowlist past a
guard. Except direct-to-main docs-only commits, which are allowed; CLAUDE.md "Working in the
main checkout" says when one may land. - Behavioural config: concurrency, timeouts, thresholds, alarm retunes, merge breadth, scheduler.
- A record whose
pin_declared_byfield is set, that is under a Leo HOLD, or that names Leo's
other session or his personal backlog. - Priority, pin-queue placement, cross-cluster dependency sequencing.
- Scope changes: acceptance criteria, cancel-and-refile or re-scope.
Choosing Fable for an H3 investigation seat is your call. The "fable run" key matches a record that
asks to launch one.
Disposition before briefing (H1–H6)
Every item that is neither a standing fact nor self-executed gets exactly one of three dispositions
before it may enter the numbered brief (Leo, 2026-09-25):
- Brief directly.
- Pre-investigate, then brief.
- Ask Leo directly, with the readable evidence attached.
Six rules decide which. Each carries the measured cases it was drawn from, so you can recognise the
next one.
- H1 — second-hand premise ⇒ investigate first. The deciding claim came from the record text, a
handover or a prior session, not from something this session verified now. Use a verifier
sub-agent with a refute mandate. Exemplars: recon gate 652 ("migrate" accepted, then refuted the
next day); esc-5485-8 (the premise was a rebase artifact); task 4803 ("undefer" while folded into
5255). - H2 — options you cannot write a ramification line for ⇒ investigation-shaped, never a bare
pick. Exemplars: esc-5588-8 ("file a trace task or close" hid 300 prompt replays in 30 days);
red-tier host levers asked with no options; xdist flags proposed without ramifications when task
1907 held the answer. - H3 — design or architectural scope, or anything adjacent to a Leo HOLD ⇒ pre-investigate to
"ready to discuss", never to "resolved". Opus by default; Fable only where an Opus attempt
struggled or the discussion is strategy-level. Exemplars: esc-5601-7, esc-5620-6, esc-4811-3,
esc-3169-1. - H4 — a recommendation, evidence this session verified itself, and a reversible action ⇒ brief
directly, one numbered line. Exemplar: 48 such items in the month to 2026-09-23, all taken. - H5 — inputs only Leo has ⇒ ask directly, do not investigate, but gather the readable
measurement first. Such inputs: attention or priority, spend, risk appetite, reversal of his
own ruling. Exemplars: esc-3637-1; the $360 calibration; merge_verify_breadth, where the brief
lacked "scoped admitted reds". - H6 — two or fewer options on a design-shaped item ⇒ run a cheap option-space seat first. Its
brief: "what else, including nothing, deferring gated on X, questioning the premise". Exemplars:
over-delivery-gap; esc-4131-9 (ruled D, never offered); esc-5332-7; steward-budget-C.
When several rules fire. H1's premise check always runs first. H6's option-space seat runs
before or inside an H3 investigation. H5 decides "ask directly" over H2, H3 or H6 investigation,
because investigating cannot supply an input only Leo has. H4 applies only when no other rule
fires. Leo's own investigation template (2026-09-21), "investigate and either resolve directly if
clear-cut, or get ready to discuss in depth", is the brief for H1 and H2 seats, within two limits:
"resolve directly" never crosses "Always ask — keyed on record content", and an H3 seat stops at
ready to discuss.
Sub-agent or /spawn (Leo's rule, 2026-09-25)
- If handling will take a multi-turn conversation with Leo,
/spawna session. That keeps the
conversation un-interleaved and out of this session's context, which is reserved for system
supervision. The category handlers' interactive/unblockspawns below are this case. - If it ends in a single report and a single ruling, use a sub-agent (the
Agenttool) and deliver
its result into this session. That means fewer terminals; Leo has run more than 15 at once.
Join a completed spawn per "Joining a completed spawn: read result.md, don't explore".
No cap on items awaiting Leo
The brief lists every item awaiting Leo; the queue exists to show them all. Leo ruled against the
study's proposed cap on 2026-09-25.
Filing Parked Decisions to the Cockpit Registry (C8)
Fleet Cockpit C8 (plans/fleet-cockpit-prd.md): every time this skill parks a decision for the
human — Priority 3b, an AFK-mode deferral, a B3 gate abort/drift-pending outcome, or aninfra_issue/recon_* "tell the human" — also file it to the cockpit decision registry, IN
ADDITION to (not instead of) the in-session note and the afk-digest.md line. The registry is
what makes the cockpit decision queue (C5b) a return-triage surface, and the 05:30 return brief,data/return-brief.md, reads it too: that brief is the primary pull-on-return surface and
supersedes afk-digest.md. The digest is retained (demoted to a generated history view, not
removed in this batch), so nothing that already reads it breaks.
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py write-decision \
--id <stable-id> --project <project> --text "<one-line question>" \
--escalations-dir <project_root>/data/escalations \
[--task-id <task_id>] [--escalation-id <escalation_id>] \
[--session-id "watcher-<project>-${CLAUDE_PID:-unknown}"] \
[--severity <esc.severity>]--id: a stable id you can recompute idempotently for the same pending item — the
escalation id (esc-42-1) is usually the natural choice. Re-filing the same id overwrites the
prior record rather than duplicating it.--idis project-LOCAL (task 4835). The verb files the record as<canonical project>-<id>(e.g.dark_factory-esc-42-1) and prints that id: use the
PRINTED id for cross-links and forclose-decision/reopen-decision. Keep passing
the bare escalation id — never hand-prefix the project. A record filed before this change under a
bare id is continued in place by your re-file, so do not change an existing--idtemplate
(such aswatcher-lease-orphan-<project>above): rows already filed under it would stop receiving
your re-files. That includes a template that already hand-prefixes the project (df-esc-<id>):
keep it as it is, and the verb continues thedf-esc-…row in place. "Never hand-prefix"
governs only a template you are writing new; dropping an existing prefix moves your re-files to
a fresh row that ignores the old row's answered or dropped state.
You no longer have to pre-check before filing (task 3559). Decision ids are fleet-global, so
another watcher (notably the recon watcher, which runs its own queue) may already have filed a
decision for the same underlying human gate under this id. The verb now handles that for you: if
anopenrecord already exists at this id and was filed from a different--escalations-dir,
your filing enriches it instead of overwriting — non-empty fields survive, fields the first
filer left empty are filled from yours,severitytakes the max of the two and is never
downgraded, andfiled_at/state/manual_booststay with the first filer (so you cannot reset
an operator's cockpit boost or re-open a closed row). Just file; you do not need to look first.
One field is exempt from "empty ones are filled":--escalation-id. It travels with--escalations-diras a pair and is never filled from the other filer across a queue boundary,
becauseesc-<taskid>-<n>ids are unique only within one queue and the reaper joins on the
(queue, id) pair — a borrowed id would resolve against an unrelated escalation in the adopting
queue. Two things that means for you:- Always pass
--escalation-idalongside--escalations-dir: they name the id and the namespace
it lives in, and a filing that supplies the queue but not the id cannot upgrade a legacy
unstamped record. - If a legacy record already holds a different escalation id under an unknown queue, your
filing is refused with aWARNINGand the record deliberately stays a visible cockpit row
rather than being silently repointed — that is the fail-OPEN direction, and the remedy is the
back-fill (scripts/backfill_decision_queue_stamp.py), which actually investigates provenance.
(Observed with
esc-5914-1, where both queues surfaced the same reify gate; that duplicate
landing on one id is the correct outcome — one question, one cockpit row — but only if the
second filer doesn't degrade the first one's record.)
Two deliberate limits, both drawn on the queue axis. A re-file from the same queue is
still a plain idempotent whole-file overwrite of everything that is yours — text, severity and
the task/session/escalation ids all land verbatim, downgrades and emptied fields included,
because that is the restart promise above and you are the sole authority on your own escalation.
What it does not touch is the record's custody:filed_at,manual_boostandstate
stay with the record, at any state (task 3872). So your restart cannot bump a row to the top
of the age ordering, cannot drop a boost an operator set between your two filings, and cannot
re-open a row an operator already answered or dropped — within one queue anesc-<taskid>-<n>id is unique, so your re-file is the same gate the human already dealt with
rather than a new ask. (Unique, but not absolutely: an id can be re-minted inside one queue
after a lost sequence counter, which is precisely why the hold is announced with aWARNINGfor
you to adjudicate rather than applied silently — see below.) Across queues those id namespaces
genuinely collide, so the second
limit is that a filing from a different queue against a closed record is still a plain
overwrite that re-opens it: there it may truly be an unrelated new question, and holding it
closed would hide a live gate instead of surfacing it.Across projects, the same local id is two rows.
esc-<taskid>-<n>numbering restarts
per project, soesc-42-1under two different--projectvalues names two unrelated gates, and
because the stored id carries the project they land on two rows (dark_factory-esc-42-1,reify-esc-42-1). A refusal remains only when the qualified id is already heldopenby
another project's legacy hand-prefixed record (say a dark_factory record someone filed by
hand under the idreify-esc-42-1): anERROR, nothing written, rc still 0, because merging would hide your ask
inside that row and overwriting would delete it. Your ask still reaches the human through the
in-session note / afk-digest line this filing accompanies; a different--idgets it a cockpit
row too.- Always pass
--project: the project's canonical token — thememory.project_iditsdark-factory-orchestrator.yamldeclares. For dark-factory that isdark_factory, notdf
and notdark-factory. The value is normalized at the CLI boundary (case-folded,-and_
equivalent,dfaliased todark_factory), so a stale spelling can no longer create a hidden
partition — but pass the canonical token anyway, so what you type matches what the cockpit shows
and no rewrite warning is logged. The project prefix on a stored id (dark_factory-esc-3524-1)
is added bywrite-decisionfrom the folded--project, so never type it into--id;
hand-prefixed ids likedf-esc-3524-1are legacy, from before the verb qualified ids. Typing a
project spelling into an id is what led humans to pass--project df, and so to a three-way
split of one project's decisions (41 open dark-factory rows spread acrossdark_factory/df/dark-factory, each invisible to a reap scoped to either of the others).- Caveat — check your project's existing rows before trusting the declared token. Folding
merges spellings that differ only by case or separator; only an entry inshared.project_id.PROJECT_ID_ALIASEScan bridge a project whose filed decisions fold to something other
than its declaredmemory.project_id, and todaydf → dark_factoryis the only such entry.
solar-challenge is the known standing case: its config declaresmy_solar_challenge, but
its decisions are filed undersolar-challenge/solar_challenge(which fold together, but
not ontomy_solar_challenge), so reaping it with the declared token matches zero rows —
passsolar_challengethere. That guidance is permanent, not provisional: task 3813
decided the alias question and declined it (the fold left no split to heal, and the
identity question is an open human gate in that project marked "Do NOT auto-act"), recording
the evidence inPROJECT_TOKEN_ALIASES_DECLINED. You no longer have to remember this
unaided —write-decisionandreap-decisionsboth warn if you passmy_solar_challenge, so the mismatch announces itself instead of returning a silent
zero-row no-op. To check any other token, runreap-decisions --project <token> --escalations-dir <queue> --expect-matchesby hand: it warns
when no record, in any state, carries the folded--projecttoken, and lists the folded tokens
that do. (It replaces acollections.Counterone-liner that listed RAW spellings,
which the reaper folds together, so it showed splits that were not there.) Leave the flag off
the per-cycle Main Loop reap: a project that has simply never filed would warn every cycle.
- Caveat — check your project's existing rows before trusting the declared token. Folding
--text: the one-line question a human needs to answer — the same summary you'd otherwise
only give in-session or in the digest.--task-id/--escalation-id: thread through whatever you have — the blocked task and the
escalation this resolves — so the cockpit can cross-link the decision to its source.--session-id: a provenance label saying which watcher filed the row — not a link, since
its lease-token shape is not a session-registry key. The cockpit's Enter-to-focus followsrecord_slug, your session record's key, whichwrite-decisionstamps itself from$CLAUDE_PID;
there is no shell token to add. If it cannot be resolved the decision is still filed, just
unlinked.--severity: pass through the parked escalation's own severity (esc.severity—info/blocking/critical/urgent). This now weights the cockpit decision-queue rank, so a
freshly-filedcritical/urgentpark surfaces at the top of the queue instead of being buried
under stale awaiting-input sessions.--escalations-dir(mandatory): the escalation queue your--escalation-idbelongs
to — for this watcher,<project_root>/data/escalations. It must name the SAME queue you later pass toreap-decisions(below). Decision records are fleet-global while an escalation id
(esc-<taskid>-<n>) is unique only within one queue, and a project can run several
(dark_factory also runsdata/reconciliation/escalationsover the same id namespace), so this is
what lets the reaper join a decision back to the right per-queue id namespace instead of matching
an unrelated same-named escalation. Stored normalized, so any spelling of the same directory
works. You can no longer file without it (task 3559): omit the flag and the verb exits 2 with
nothing filed, and pass it empty and it refuses with a loud error and prints no id — so if the id
doesn't come back on stdout, your filing did not land. There is no watcher for which this is a
burden: it is the same directory you already pass toreap-decisions.
It is mandatory even for a sentinel park with no--escalation-id(a lease orphan, a
pipeline stall): task 4835 decided against exempting them, because the same-queue custody hold
keys on stamp equality, so an unstamped sentinel later re-filed with a stamp would re-open a row
an operator had dropped.
There is a third value the field can hold:<unknown>(session_registry.UNKNOWN_QUEUE) —
"this record's owning queue was investigated and could not be determined". You never write it —
and that is now enforced, not just asked:write-decisionrejects--escalations-dir <unknown>outright, because a record stamped that way is refused by every
reaper and could only ever be closed by hand. It stays valid on the back-fill path
(set_decision_escalations_dir), which is where task 3640 wrote it, for legacy records whose
escalation id resolved in several queues at
once. It is not a respelling of the queue-less''state:''means nobody told us and
falls back to project-only scoping, while<unknown>means we looked and could not tell and
the reaper refuses to close it at all.The verb files
state=openfor a new record, but it never re-opens a row an operator already
answered or dropped when you re-file from your own queue (task 3872). What you will see: the
filed id still comes back on stdout — that signal is unchanged, and your filing did land, since
your text/severity/ids were written — plus aWARNINGon stderr naming the state it held. That
warning means a human dealt with this gate while it sat parked, so adjudicate it rather than
re-filing blindly on your next restart. If this same gate genuinely needs a human again,
re-open the row in place; the warning names the exact command:python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py reopen-decision \ --id <printed id> --project <project> --escalations-dir <project_root>/data/escalationsIt clears the row's closing evidence, so a later
close-decisioncan quote fresh evidence, and
it exits non-zero when it refuses (the record at that id is another project's or queue's) or
finds no record. It cannot hold open a row whose escalation is already resolved or dismissed:
the next reap closes it again, and the verb warns on stderr when that is the case. Such a gate
needs a new escalation, filed under its own id. File under a new id only for a genuinely
different ask. Either way, do not
try to force the row open by re-filing.The verb prints the filed id on success for your own cross-link (e.g. into the digest line). It
is fail-soft — a registry fault is logged and swallowed, never raised, so filing a decision can
never crash the watch loop or block the park itself.
This is additive at every "leave pending" / "tell the human" / "park" moment below — do the
existing action exactly as documented, and also run write-decision once per parked item.
Closing parked decisions on resolve
A filed DecisionRecord stays state=open — and therefore visible in the cockpit decision queue —
until its escalation reaches a terminal status. The watcher that files a decision is almost never
the one that resolves it: resolution typically happens later, via a spawned /unblock session, an
L2 cascade, or the human acting directly on the dashboard. So closing a parked decision is a
separate, recurring step, not something the write-decision call itself can do.
Once per Main Loop cycle (see step 6 above), run:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py reap-decisions \
--project <project> --escalations-dir <project_root>/data/escalationsThis closes (answered/dropped) any state=open, escalation_id-bearing decision whose
escalation has since resolved (resolved → answered) or been dismissed (dismissed →dropped) — regardless of who resolved it: this session, /unblock, an L2 cascade, or the human.
The join is scoped on two axes, project and queue: a decision stamped (viawrite-decision --escalations-dir) with a queue other than the --escalations-dir you pass
here is skipped outright, so your reaper can never close the recon watcher's decisions against
your own same-named escalations. A decision filed without --escalations-dir — every record
predating that flag — falls back to project-only scoping and therefore has no such protection:
it can still be closed by whichever queue's reaper reaches it first. Task 3640 back-filled that
pre-existing open population, and task 3559 made the flag mandatory at the verb, so the unprotected
set can no longer regrow through write-decision — which is what makes the back-fill terminal
rather than a recurring chore. It can now only shrink.
On a MODE 2 cross-queue collapse — both queues surfacing the same human gate onto one
decision id (see the recon watcher's MODE 1 / MODE 2 taxonomy) — the stamp that survives is the
first filer's queue, since a second filing enriches rather than overwrites. The field holds one
queue, not a list, so the other queue's reaper still skips that record; the trade is that the
outcome is now deterministic (first filer) instead of depending on who happened to write last. The
verb logs a warning naming both queues when it discards one. The single stamp is a decided
limit, not a pending fix (task 4835; see plans/4835-decision-plumbing-decisions.md).
A decision stamped <unknown> is refused, not closed: its owning queue was investigated and
could not be determined, so no reaper may close it and it stays a visible cockpit row until a
human closes it. The reaper never defaults to closing on doubt — that direction is deliberate,
since an over-held decision is a triageable row while a falsely-closed one is invisible.
If unstamped open records ever reappear, the re-runnable remedy isscripts/backfill_decision_queue_stamp.py (dry-run by default; --verify exits non-zero while
any open record still lacks a stamp).
The project axis matches on the canonical token (see --project above), so ONE run per
queue closes every historical spelling of that project — there is no need to re-run the verb
once per token (df, dark_factory, dark-factory) as was necessary before. This holds as long
as the token you pass folds into the same bucket its rows carry, which is not automatic for every
project — read the --project caveat above before assuming a zero-row reap means "nothing to
close". Folding only ever
merges spellings of the same project; it never merges two different projects (e.g.solar_challenge and solar_challenge_platform stay separate), so this widens what a reap
closes without ever letting one project's reaper close another's decisions.
To repair the legacy population — records filed before write-decision canonicalized --project
— run the one-shot backfill (dry-run first):
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py \
migrate-decision-projects --dry-run # preview; writes nothing
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py \
migrate-decision-projectsIt rewrites only the project field: state and filed_at are preserved, so an already-answered
row is never reopened, and a record's id (and its cockpit cross-links) is never rewritten. It
is idempotent — a re-run once the fleet is clean prints nothing — so it is also the repair tool if
a record is ever hand-edited.
It is read-only with respect to escalations (it only ever writes the decision's own state field)
and fail-soft, exactly like write-decision — a registry fault is logged and swallowed, never
raised, so it can never crash the watch loop. A decision filed with no escalation_id (e.g.
the tasks.db-corruption park) is never auto-closed this way and needs explicit human closure.
Likewise, a decision whose escalation_id never resolves to a status — the escalation was purged
by archive retention pruning, or never existed — also stays open forever and needs the same
explicit human closure; until then, every cycle repeats a full scan of the escalations archive
looking for it.
Merge Submissions — Bounded Submit, Then Poll
An unbounded foreground mcp__escalation__merge_request blocks until the merge worker finishes
rebasing, running the full verify suite, and CAS-advancing main. On a large/slow repo (e.g. reify)
such a call could take 30+ minutes — made in the foreground it would freeze the entire watch
loop for that long: no draining, no watcher re-arm, a born-at-L2 critical sits unseen (real
incident: esc-2831-78 wedged a reify watcher >30 min on a direct foreground retry-land). The watch
loop's latency budget must stay bounded.
Protocol invariant: every merge_request call passes an explicit bounded wait_secs;
completion is awaited only via merge_status polling (15 s → 60 s backoff using eta_seconds).
Because no call can block >100 s, top-level submission is safe BY PROTOCOL.
§7.3 Submit → poll mechanics:
Submit with an explicit bounded
wait_secs(use100):mcp__escalation__merge_request( task_id=..., branch=..., worktree=..., description=..., wait_secs=100 )A return within the window yields a terminal outcome shape (
status∈done | conflict | blocked | already_merged | done_wip_recovery | unknown_branch | unmerged_state | stash_failed | wip_halted | wip_recovery_no_advance | error | superseded).The six worker-internal outcomes (
wip_halted,done_wip_recovery,wip_recovery_no_advance,unmerged_state,stash_failed,error) are rare;merge_statuscollapses all of them exceptdone_wip_recoverytoblockedwhen
observed by polling — handle them asblocked. (failed, which this list named
until task 4829, is not a value the server ever returns; the real one iserror.)A timeout yields a non-terminal shape with
status: 'queued'|'attached'and arequest_idto poll; its full key set is stated inmerge_request's own docstring
(escalation/src/escalation/server.py::merge_request), not restated here.Both are a successful, durable submission — the entry survives disconnect (PRD D2);
intent persists even if the MCP session drops mid-bounded-wait.status='attached'on a coalesced submission means the merge is already queued under the
existing entry'srequest_id— already covered; do not re-submit.
If non-terminal, poll until resolution:
mcp__escalation__merge_status(request_id=...)Back off 15 s → 60 s, using
eta_secondsas the hint when present. Terminal states:done | conflict | blocked | abandoned | superseded.(
already_merged, which this list named until task 4829, is a submit status the
server collapses todonewhen observed by polling — see step 1 andescalation/src/escalation/server.py::_map_terminal_state.)
After an orchestrator restart,{state: 'unknown', hint: 'check git log main'}→ fall back togit log main(PRD I3).To abandon a queued entry before it is picked up:
mcp__escalation__merge_cancel(request_id=...)Returns
{cancelled: bool, state, reason?}. On success (cancelled: true) the entry is
dropped without halting the queue;merge_statussubsequently returnsstate: 'abandoned'.
Tracking in-flight merges: record {task_id, escalation_id (if any), request_id} and
never submit a second merge for a task_id that already has one in flight. The coalescedstatus='attached' response is the backstop — if you see it, the merge is already covered.
AFK Mode (extended unattended operation)
When the human will be away for an extended period (hours to days) and cannot adjudicate 3b
decisions, switch posture from "stall and ask" to "keep the pipeline moving, defer the judgement,
and leave a clean trail." The posture needs no confirmation; "AFK by silence" below says when it
switches on and off. Three behavioural shifts:
Defer, don't wedge. For a 3b item (ambiguous AND consequential), stalling the whole queue for
days helps no one. Where the decision can be safely postponed without baking anything in:- Queue a follow-up task capturing the decision to be made (two-phase
submit_task→resolve_ticket), and - Hold the task, choosing by the record's severity:
Not
info:resolve_issue(..., action='park'). The task landsblocked, held under
the open L2, because a pinning escalation vetoes the stranded-blocked sweep.info: never park;resolve_issuerefuses it (park_on_info). An info record never
pins (escalation/src/escalation/pins.py::classify_pins; intended, Leo 2026-10-09). The sweep
re-pends a parked task within a tick, as with esc-5708-3, reify esc-7881-5 and
esc-4681-3 (16 s). Leave the escalation pending and defer the task:set_task_status(id, status='deferred', deferral={'kind': 'until_condition', 'condition': 'Leo rules <esc-id>', 'reason': '<one line>'}), and confirm the response
echoesdeferral. No echo means the fused-memory server predates the deferral gate
and the hold is unrecorded: fall back toupdate_task(id, metadata={'x_deferral': {'escalation_id': '<esc-id>', 'until_condition': 'Leo rules <esc-id>', 'reason': '<one line>'}}, metadata_mode='merge'), which the deferral migration maps, and
re-stamp with the call above after fused-memory restarts;- confirm with
get_taskabout 30 s later that the task isdeferredwith its record.
A deferral keeps a not-yet-running task out of dispatch. It does not stop a live run, and
an info filer was told to keep driving, so leave anin-progresstask alone (leave
pending + digest). When the ruling lands, carry it out, then re-pend the task withset_task_status(id, status='pending')unless the ruling says otherwise; that removes the
record. Task 6579 makes this hold the typeduntil_escalationkind. Recipe:docs/task-authoring.md§9 "Deferring a task". And
- File a DecisionRecord via
write-decision(see "Filing Parked Decisions to the Cockpit
Registry" below) — IN ADDITION to the follow-up task, so the parked decision surfaces in the
cockpit decision queue.
This is parking a decision for later human review — NOT making it. Only park when the task has no
half-merged or destructive state. The Priority Hierarchy bar still holds: better to defer than to
bake in a bad decision — when in real doubt, fall back to "leave pending + digest."
- Queue a follow-up task capturing the decision to be made (two-phase
Don't spawn unattended terminals. The interactive
/spawn→/unblockpath needs a human at
a terminal; while AFK those sit idle and the task stays blocked anyway. So in AFK mode:task_failure/review_issues: run the low-risk auto-unblock gate first (see the
Low-risk auto-unblock gate (B3) subsection for the full
gate procedure and applicability rule). If the gate does not launch (abort / over-cap /
already-attempted) OR the launched sub-agent aborts, leave the escalation pending and add
it to the digest — do NOT spawn an interactive/unblock.wip_conflict/unmerged_state/stash_failed/dependency_discovered-with-no-task /design_concern/risk_identified/infra_issue/recon_*: leave pending + digest. These need a human;
a terminal nobody attends just clutters. Append<esc-id>to the wrapper-owned exclude-file
(see "Starting the watcher" above) for each item left pending so the initial scan does not
busy-loop on it.- Either way, also file a DecisionRecord via
write-decision(see "Filing Parked Decisions
to the Cockpit Registry" below) for each item left pending — IN ADDITION to the digest entry.
Batch into a digest, don't ping per-item. Reminding "every 3-5 cycles" is noise when nobody is
reading. Maintain a single rolling manifest at<project_root>/data/escalations/afk-digest.md
(overwrite each cycle) listing every pending item: id, task_id, category, severity, age, and a
one-line "why it's waiting / what decision is needed." On return the human reads one file, and
the primary one is nowdata/return-brief.md: the 05:30 cross-project return brief supersedes
this digest as the pull-on-return surface (see "Sitting preparer (prepare-sittingmode)").
Keep maintaining the digest, which is retained for the reasons given under "Filing Parked
Decisions to the Cockpit Registry (C8)" below. If
phone push is configured (--ntfy-urlon the watcher command), a born-at-L2critical/urgent
still pushes immediately — those are the only items worth interrupting an AFK human for.
AFK by silence
On. Either of two triggers switches AFK posture on, and neither needs a confirmation question:
an explicit "I'll be away", at once; or 4 hours with no human turn. Leo can be away for up to 48
hours, weekends included, without saying so (Leo, 2026-09-25, task 5884).
Measuring the silence. A human turn is a message Leo types. A watcher fire, a CEILING tick
and a background sub-agent completion are not human turns. At launch and at each human turn, writedate -u +%FT%TZ to <project_root>/data/escalations/l2-last-human-turn, overwriting it. At each
wake, compare against that file, not against a time you remember: a compaction can drop it from
context. A quiet queue still wakes you once per slice (the Bash-tool timeout contract in "Starting
the watcher"), so the switch lags the 4-hour mark by at most one slice.
What changes. The three shifts above: defer, don't wedge; no unattended interactive terminals;
a DecisionRecord and the brief instead of per-item pings. The 3b reminder cadence also stops.
What keeps running. Pre-investigation sub-agents are not terminals. Keep launching the
ownership and in-flight checks of "Investigate before you record" and the H1–H3 and H6 seats of
"Disposition before briefing (H1–H6)", so items are ready to brief on return. A /spawn for a
multi-turn conversation with Leo (see "Sub-agent or /spawn") waits for his return; meanwhile a
sub-agent takes the item to "ready to discuss".
Off. The next human turn ends AFK posture. That turn is a return: post the unprompted brief
(see "When it runs") and say when AFK posture began.
Low-risk auto-unblock gate (B3)
When this gate applies (see Applicability below), before leaving the item for the human, run the
mechanical gate to check whether the at-block-time dry-run investigation found a low-risk fix:
Gate check — run from $DARK_FACTORY_ROOT:
.venv/bin/python -m orchestrator.b3_gate check \
--task-id <task_id> \
--worktree <worktree> \
--project-root <project_root> \
--category <task_failure|review_issues> \
--config <project_root>/dark-factory-orchestrator.yaml
--tagnote: Bothcheckandrecord-launchdefault--tagtomaster— the
taskmaster tag under which the watched project stores its tasks. If the watched project
uses a non-mastertag, supply--tag <tag>to both verbs. Without it,checkwill
silently find no proposal row and returndriftoraborton every call (the behavior is
fail-safe — it never launches — but the watcher will appear stuck with no signal that the
tag was wrong).
Parse the JSON output: verdict (fresh|drift|abort), reason, cap_remaining,already_attempted, head_sha, main_sha, age_seconds, age_state.
age_state (parsed|unparseable|absent) says WHY age_seconds is null when it is.
The implication runs ONE way: a non-parsed age_state can never accompanyverdict == "fresh" — the gate never certifies a proposal fresh without a parsedinvestigated_at — but it does NOT imply abort. The age check is deliberately last, so an
earlier, more specific check can return drift first: a proposal missing its sha anchor
returns drift with age_state: "absent". Branch on verdict alone; read age_state as the
diagnostic explaining a null age_seconds, never as evidence of which verdict you got.
A fourth value,
no_clock, exists in the gate for in-process callers that pass no clock.checkalways resolves one, so the CLI documented here never emits it.
Decision table:
| Condition | Action |
|---|---|
already_attempted == true OR cap_remaining == 0 |
Leave escalation pending + digest line; do NOT launch |
verdict == "abort" |
Leave escalation pending + digest line carrying the gate's reason |
verdict == "drift" |
Drift path (see Drift path section below) |
verdict == "fresh" |
Record-launch + launch (see below) |
Every "leave escalation pending + digest line" outcome above also files a DecisionRecord viawrite-decision (see "Filing Parked Decisions to the Cockpit Registry" below) — IN ADDITION to
the digest line, not instead of it.
On fresh — record-launch then launch:
Before launching, run from $DARK_FACTORY_ROOT:
.venv/bin/python -m orchestrator.b3_gate record-launch \
--task-id <task_id> \
--worktree <worktree> \
--project-root <project_root> \
--config <watched-project orchestrator config>If record-launch returns already_attempted: true (concurrent or restart race), skip the
launch: leave pending + digest line. Otherwise it durably records the launch — this is the
don't-double-launch marker.
Then launch the unblock-low-risk skill as a NON-INTERACTIVE background sub-agent (theAgent tool, general-purpose, run_in_background: true — NOT /spawn), passing task_id,escalation_id, project_root, the worktree path, and the latest proposal, and instructing it
to read and follow skills/unblock-low-risk/SKILL.md. It applies the fix scoped tofiles_referenced, runs the verify suite, and merges via the queue — or aborts cleanly.
Background, not foreground — why. The unblock-low-risk sub-agent runs a full
apply → verify → submit → poll cycle in its own context — verify alone can take several minutes on
a large repo. Run in the foreground (Agent without run_in_background), that entire cycle
occupies the watch loop's context, making it unresponsive to incoming L2 escalations throughout.
Backgrounding keeps the foreground lean and responsive: record the launch (above), then immediately
loop back to re-arm the watcher and drain. The harness re-invokes you with the sub-agent's result
when it completes — that completion is itself a wake signal (Main Loop step 4), handled below.
Record the launch; don't double-launch. The durable b3_gate record-launch call above
serializes concurrent and restart races. Stash {task_id, escalation_id, background-task-id} in
your context. The escalation stays pending until the background sub-agent resolves it, so the
next drain WILL re-find it — before the gate check for any task, check your context: if thattask_id already has an in-flight or this-cycle-completed unblock-low-risk sub-agent, skip it.
A task whose B3 run returned handed_off counts as in flight while its resume-mode run is
outstanding.
The durable rolling-24h merge cap is enforced by b3_gate charge inside the unblock-low-risk
sub-agent immediately before its merge-submit — a charge refusal causes the sub-agent to ABORT.
The watcher consults only check's cap_remaining to skip launches that charge would refuse.
Drift path — one-shot re-investigation:
On verdict == "drift", spawn ONE read-only background sub-agent runningskills/unblock-auto/SKILL.md in the worktree (Agent tool, general-purpose,run_in_background: true), passing task_id, worktree, and the block reason. Capture the git
anchor at re-investigation start:
head_sha=$(git -C <worktree> rev-parse HEAD)
main_sha=$(git -C <worktree> rev-parse main)
investigated_at=$(date -u +%Y-%m-%dT%H:%M:%S+00:00)Take investigated_at from that command rather than writing one yourself. It MUST carry a UTC
offset: the gate subtracts it from an aware clock, so a naive timestamp
(2026-09-16T12:00:00) raises, yields age_state: "unparseable", and hard-ABORTs the re-gate
— dead-ending the very recovery path this section exists to complete.
When the sub-agent returns {proposal_text, files_referenced, risk_label}, build a proposal
entry mirroring _build_entry success-path keys and append it viamcp__fused-memory__update_task(id=<task_id>, project_root=<project_root>, metadata={"dry_run_proposals": [entry]}, append=true):
{
"proposal_text": "<from sub-agent>",
"risk_label": "<from sub-agent>",
"files_referenced": ["<from sub-agent>"],
"block_reason": "<original block reason>",
"investigated_at": "<$investigated_at from above — UTC ISO-8601 WITH offset>",
"timestamp": "<ISO now>",
"head_sha": "<captured above>",
"main_sha": "<captured above>"
}A malformed entry is fail-safe by construction — it simply fails the next check (b3_gate check
is the single shape validator). Then re-gate once: re-run b3_gate check. If fresh →
record-launch + launch; if drift again (a second drift in the same handling cycle) → leave
pending + digest (drift-reinvestigated outcome — main is moving inside the task's footprint; a
human should look) and file a DecisionRecord via write-decision. At most one re-investigation
per handling cycle.
Completion handling:
On the sub-agent's completion (you're notified asynchronously — match the result to a recorded
launch by task_id / background-task-id):
outcome == "merged": it has already set the task done and resolved the escalation. Add a
digest entry. In attended mode, also emit an immediate in-session report: one-line summary +
merge sha + diff pointer.outcome == "aborted": it changed nothing terminal and left the escalation pending. Keep thetask_idin your context as completed this cycle (do NOT re-launch it), record the abort reason
in the digest, and move on — do NOT retry, and do NOT spawn an interactive/unblockin AFK
mode; it waits for the human. If the abort reason indicates drift/staleness and the one-shot has
not been used this cycle, route through the drift path once. Also file a DecisionRecord viawrite-decisioncarrying the abort reason as--text(see "Filing Parked Decisions to the
Cockpit Registry" below). A resume-modeaborted(seehanded_offbelow) differs in two
ways. It never takes the drift path: the re-gate ends inrecord-launch, which can only reportalready_attemptedfor this task. And when its reason is that the escalation is no longer
pending, write only the digest line and file no DecisionRecord: whoever took the escalation
owns it, and the merge carries on.outcome == "handed_off": neither a failure nor an abort. The mergerequest_idit returns
was picked up by the merge worker and is still live (merging or verifying); the sub-agent did
not cancel it, did not change the task status, and left the escalation pending. Nevermerge_cancelit, and never submit another merge for thattask_id: it is an in-flight
merge, so record its{task_id, escalation_id, request_id}per "Tracking in-flight merges". Launchunblock-low-riskagain as a NON-INTERACTIVE background sub-agent in resume mode, exactly
as B3 is launched above (Agenttool, general-purpose,run_in_background: true), pointing it
at the "Resuming a handed-off merge (resume mode)" subsection ofskills/unblock-low-risk/SKILL.mdand passingtask_id,escalation_id,project_root,worktreeandrequest_id. Do not runb3_gate checkorrecord-launchfor it: the slot
is already charged, andcheckwould reportalready_attempted. Keep thetask_idin flight
until that run completes; its completion is handled by this same list. Resume once per B3
launch: if a resume-mode run itself returnshanded_off(its 3 h backstop expired with the
merge still live), do not resume again — leave the escalation pending, write the digest line,
and file a DecisionRecord viawrite-decisionwhose--textnames therequest_idand its
last state. Still never cancel. A resume-modemergedis handled by themergedbullet, and a
resume-modeabortedby theabortedbullet, including its resume-mode exceptions.
The sub-agent re-checks the gate defensively and refuses anything not unambiguously low-risk; treat
its abort as authoritative.
Applicability:
B3 applies in AFK mode always. In attended mode it applies when the watched project's orchestrator
config UnblockAutoConfig.attended_b3_enabled (<project_root>/dark-factory-orchestrator.yaml →unblock_auto.attended_b3_enabled, default false; dark-factory and reify set it true since
2026-09-25) is true OR the human enabled it for this
session via a session override. A session override wins in either direction — a human may turn it on
even if config is false, or off even if config is true.
Digest line format (written into <project_root>/data/escalations/afk-digest.md — the single
shared B3 outcome ledger for both AFK and attended modes; the "afk" prefix reflects the file's
original AFK-only scope, but it is now the unified record for all B3 outcomes regardless of session
mode; AFK shift 3 manages it):
- Merged:
B3 <task_id> — merged: <one-line summary> (sha: <merge_sha>) - Aborted:
B3 <task_id> — aborted: <reason> - Drift-reinvestigated, second drift:
B3 <task_id> — drift re-investigated; re-gate: drift-again → pending - Drift-reinvestigated, relaunched:
B3 <task_id> — drift re-investigated; re-gate: fresh → launched - Handed off:
B3 <task_id> — handed off: <request_id> still <state> at the polling ceiling → resume launched - Resume backstop:
B3 <task_id> — resume backstop: <request_id> still <state> → pending (not cancelled)
afk-digest.md is retained (Fleet Cockpit C8) — it is not removed in this batch, but it is
demoted to a generated history view: data/return-brief.md, the 05:30 return brief, is the
primary pull-on-return surface that supersedes it, and it reads the cockpit decision queue (C5b),
fed by write-decision (see "Filing Parked Decisions to the Cockpit Registry" above), for any of
the above that left a decision open (i.e. every outcome except "Merged", a first "Handed off",
whose resume is still running, and a resume-mode "Aborted" on an escalation someone else already
took).
When a line is derived from a triage ack. If a digest line above, or a write-decision --text
registry entry (see "Filing Parked Decisions to the Cockpit Registry" above), draws on an existingtriage_note rather than fresh investigation, carry forward the predicate and probe it names — not
just its conclusion (see "Reading a triage-ack annotation" above). A conclusion-only line forces a
returning human, or the next rotation, to re-derive the disposition from scratch instead of
re-checking a named, machine-checkable predicate.
Merge-failure disposition vocabulary (skew ⇒ port, don't debug)
Merge-gate verify failures now carry a disposition — a classification of whose
fault the failure is, orthogonal to the failure's category (plans/merge-skew-attribution-prd.md,
task β, orchestrator/src/orchestrator/merge_lane/disposition.py). It is a closed enum:
| Disposition | Meaning |
|---|---|
main_red |
The failure already reproduces on main tip (preexisting-main-break probe fired) — reported through the existing "fix main:" dedup path, not new. |
integration_skew |
The branch's own pre-merge verify was green, but a commit landed on main since the branch's merge-base that touches the files the failing test(s) depend on. The branch is semantically stale against that landing — nothing is wrong with the branch's own diff. |
branch_bug |
No landed commit is implicated — the failure is the branch's own bug. |
indeterminate |
The classifier couldn't reach a verdict (evidence inconclusive, or an internal error — fail-open). Treat exactly like today's undifferentiated failure. |
Where it surfaces: integration_skew does get its own escalation
category: the workflow layer files the block with category='integration_skew'
and suggested_action='port_landed_change' (workflow.py, INTEGRATION_SKEW
disposition branch). The task's block reason also carries an appended suffix of
the form integration_skew: port landed commit(s) <sha[, sha...]> touching <files> — do not hunt your own diff, and the same disposition +
implicated commits + overlap files are available verbatim in merge_status'sfailure_diagnostic field. Look for that category/suffix/field before you (or a
spawned /unblock session) start reading the branch's own diff for a bug that
isn't there.
Triage rule — the load-bearing part:
integration_skew⇒ PORT the named landed commit(s) into the branch. The fix
is an agent edit that brings the branch's content in line with what already
landed on main — cherry-pick, reapply, or hand-port the relevant hunks of the
named sha(s) into the files the diagnostic lists. Do not debug the branch's
own diff; the branch's logic was fine when it was cut, and re-running verify or
hunting for a regression in the branch's own change is wasted effort.- Skew is NOT a flake, and must not be auto-filed into flake stats. It reads
superficially like a transient failure (retry-after-rebase would make it pass),
but it has a deterministic, name-able cause. Any flake-tracking/auto-file path
(e.g. reify 5142 / DF 2358's flaky ledger) must filter ondisposition == 'integration_skew'and exclude it, and a human doing manual triage should
likewise never wave it off as "just flaky." main_redandbranch_bugneed no special handling beyond what their existing
category sections already say;indeterminatedegrades to today's behavior —
treat it as an undifferentiated failure.
This vocabulary matters most for task_failure and wip_conflict (see those
sections below) — check the block reason / failure_diagnostic for a disposition
before spawning /unblock or trying the low-risk auto-unblock gate.
Handling Escalations by Category
For every escalation, read the suggested_action field. It's a free-text hint — sometimes a conventional verb, sometimes natural language. First determine the escalation's L2 origin, then interpret the hint accordingly:
Born-at-L2 (severity critical or urgent at creation — bypassed L0 and L1 entirely):
Neither the per-task steward nor the auto-watcher has seen this record. Read suggested_action as the originating agent's own annotation — a starting point, not evidence of prior triage. investigate_and_retry here means what it says: a retry may well succeed since no automated attempt has been made yet.
Promoted-from-L1 (the auto-watcher attempted resolution and escalated to human):
manual_intervention— The auto-watcher explicitly gave up. This is authoritative: the issue genuinely needs human judgment. Always respect it.investigate_and_retry— Misleading for promoted items. The item has already passed through both the per-task steward (L0) and the auto-watcher (L1) and persisted through their combined triage and retry budgets. Treat as a deeply persistent problem, not transient. Don't just retry.triage_suggestions/fix_review_issues— Routing hints confirming what the category tells you. No new information.- Free-form text (e.g., "Restore Value::Frame from previous commits") — Valuable diagnostic context about what the escalating agent thought would help. Read it as a starting point for investigation, not as instructions — the agent was stuck, so its diagnosis may be incomplete.
A promoted cluster's root_cause and evidence are the auto-watcher's narrative, not the record.
Before ruling on one that joins two or more failures of a task, or restating it to the human, count
those failures per `skills/_shared/counting-failure-events.md`.
Every handler below ends in an action or a question to Leo. The self-execute tier bounds the
action, and the question passes through its briefing disposition first (see "Before an item
reaches Leo").
Additive-context convention for spawned /unblock prompts. Several categories below spawn an
interactive /unblock session with a prompt of the form /unblock <task_id> (esc <escalation_id>, <category>, <severity>: <summary>). Only the leading /unblock <task_id> token is load-bearing:/unblock's own Step 0 (Locate the task) extracts <task_id> from it, and Step 1 (Gather context)
re-derives all context — escalation, task status, git state — fresh from TASK_ID
(skills/unblock/SKILL.md). The trailing (esc ...) context is purely additive, there only to
orient the human reading the terminal before /unblock runs, so it stays non-load-bearing even if
a section's summary below drifts out of date.
Two mechanical caveats when building this prompt string:
- Which number is the task id. The parenthetical introduces a second number (
<escalation_id>);
Step 0 must take only the number immediately following/unblockas<task_id>—<escalation_id>
always sits behind theesctoken inside the parenthetical, never in the leading position, so the
leading number is unambiguous. - Escape or truncate
<summary>before interpolating. Escalation summaries are free text and may
contain single quotes, parens, or newlines. Collapse<summary>to a single line, and either
shell-escape inner single quotes as'\''per/spawn's Arguments section (skills/spawn/SKILL.md)
or truncate it to a short slug — otherwise an unescaped quote in the summary breaks the
single-quoted<prompt>argument passed tospawn-claude.sh, and the spawn fails or truncates.
Each call site below notes this briefly rather than repeating the full explanation.
Reading a triage-ack annotation
A pending record — most often an L2 promoted from L1, but any pending item in principle — may
carry a triage-ack annotation: triaged_at, triaged_by, triage_note (plus updated_at, the
record's own last-substantive-change marker). This means an earlier rotation — almost alwaysescalation-watcher-auto, since stamping is how that skill records "I assessed this without
resolving or promoting it" — already looked at the item and left a handoff note so the next reader
doesn't have to re-derive its disposition from scratch. Read it exactly like suggested_action
above — a starting point, not a verdict.
Re-verify (re-run the probe yourself) instead of trusting the note when either:
triaged_atis older than roughly 6 hours, or- the record changed since triage —
updated_atis notNoneand is newer thantriaged_at(e.g. the L2 cluster gained a new member viapromote_to_l2after the stamp was
written, its severity was promoted, or a later fold carried in NEW FRAMING — see "Reading
preserved framing" below).updated_atdefaults toNone(never bumped) until the record's first real content
change, so a triaged record with no changes since still readsupdated_at = None— treat that as
"not newer thantriaged_at", never as an ordering comparison betweenNoneand a timestamp
string.
A well-formed triage_note names a machine-checkable predicate (e.g. `task-604 status==done`) and the **probe** used to check it (e.g. `probe: get_task 604 -> status=done`) — re-run that same probe before trusting the predicate still holds. Atriage_note that states only a conclusion, with no predicate or probe behind it, is untrusted
prose, not a substitute for investigation. This is exactly the esc-2584 failure mode: a
conclusion-only recommendation ("resume will close it") was taken at face value, got refuted, and
cost two churn cycles and five separate resolve_issue calls before the item was actually closed.
A machine-checkable predicate can still be vacuous: never accept — or write — a predicate about
the record's own status. "still a member of pending esc-X" / "esc-X still pending" passes
every rule above (named predicate, honest probe, cheap re-run) and carries zero information,
because it is trivially true for as long as the item stays parked. esc-6107-7's members were
re-probed on exactly that predicate ~30 times over 7.6 days while the question had been ruled,
implemented, and measured elsewhere. A parked item's predicate must be about the world: has
the answer appeared? Concretely — a member already ruled (see "Ruled-elsewhere check" below), an
esc-id-citing correction block in the cluster's task descriptions, or the subject's branch tip
advancing while the task is blocked/parked (tip-advance on an in-progress subject is ordinary
work, not a signal).
triaged_by is server-enforced (escalation/src/escalation/server.py::stamp_triage) when the
stamping connection sends X-Escalation-Identity (the auto-watcher does), the same attribution
contract this skill documents for resolved_by (see "Recognizing the supervised auto-watcher's
resolutions" below); for a header-less interactive session it is a convention, not a guarantee —
see "Shadow-mode standing-policy rulings (measurement only)".
Refuting a premise: the claim's execution context, or it is not a refutation
A check that does not reproduce an escalation's premise refutes it only if it ran in the premise's
execution context; anywhere else it is a non-reproduction. The rule covers the probes you re-run and
the resolution text you write. It also covers spawn briefs: never hand a spawned session a premise
marked disproved, or marked "do not re-derive", unless the refuting check ran in that context, and
state the context each cited check ran in. Specimen: on esc-legibility-trickle-reify-3 a brief
refuted a boot-time timer's PATH premise from a post-login interactive shell. The premise was right,
and the auto-watcher's triage note had said its probe did not verify the job's own PATH. Single
normative statement of the rule:orchestrator/src/orchestrator/agents/premise_refutation_guidance.py::PREMISE_REFUTATION_GUIDANCE.
Reading preserved framing (amendments)
A pending L2 is a cluster, and escalation-watcher-auto re-promotes the same cluster every
time it finds more L1s matching that root cause. Each of those folds carries its ownroot_cause/evidence/options/summary — the promoting rotation's current read of the problem,
which is often sharper than the first one. Until task 3997 all of it was discarded on the floor
(measured: 336,875 characters). It is now kept.
When you pull the full record with get_escalation(id), read amendments alongside the record's
own framing. The two are different things and the distinction is the whole point:
- The record's OWN
root_cause/detail/options/summaryare the original framing,
from the promote that minted the L2. They are immutable — a fold never overwrites them, so the
decision context a human started reading cannot shift under them. amendmentsis an append-only list of what later folds carried in — or what an explicitamend_escalationwrote (a ruling made elsewhere) — oldest first, each with theagent_rolethat
submitted it, which is how the two sources are told apart, and a queue-stampedtimestamp. The
last entry is the most recent read of the record. When a fold entry disagrees with the
record's own framing, that disagreement is the signal — either the cluster drifted, or root-cause
matching folded in something that does not belong. An explicit amend that disagrees is usually a
ruling: read it as the answer to the question, not as drift.- Framing byte-identical to what the record already says is not re-recorded, from a fold or an
explicit amend alike, so every entry present is a genuine reframing rather than an echo.
A set of counters says what was NOT kept — check them before treating any of these lists as
complete. Each is a durable record field you can read straight off get_escalation(id), so the
loss is always assertable from the record rather than being log-only:
amendments_truncated > 0— older entries were shed at the cap (oldest-first). The record's own
original framing is unaffected; only intermediate reframings were lost.amendments_chars_elided > 0— individual fields were long enough to be clipped at the per-field
cap. Elision is marked in-band ([... N char(s) elided ...]), so a field ending in that marker is
the head of what was submitted, not all of it.root_cause_variants_truncated > 0— the oldest distinct pre-canonical root-cause spellings
were shed at the 20-entry cap (oldest-first). The TRUE distinct count islen(root_cause_variants) + root_cause_variants_truncated. Weigh this one heavily: over-folding
is the exact failure this set exists to catch, so a non-zero value means the over-fold evidence is
under-reported precisely when there is most of it.dedupe_children_truncated > 0— the oldest non-head child ids were shed at the 200-entry
cap, which is head-preserving (the first 20 are always kept). The TRUE provenance total islen(dedupe_children) + dedupe_children_truncated.dedupe_count— the load-bearing recurrence
signal — is not capped and is unaffected, so a truncateddedupe_childrennever understates
how often the cluster recurred, only which ids you can name.
A sustained burst of truncation files its own info infra escalation (under the syntheticl2-amendment-truncation task anchor, not against any real task) saying either the cap is too low
for the live fold rate or root-cause matching is over-folding unrelated clusters into one L2. If you
see one, the named L2s are where to look.
Ruled-elsewhere check (answered-but-unrecorded)
The most expensive queue failure measured to date is a pending L2 whose question has already been
answered — ruled by a human or an /unblock session — with the ruling recorded on a different
surface: a twin L2 over the same member, a sibling task's description, a branch commit. Measured
2026-08-22→24: five instances (3 design_concern, 1 dependency_discovered, 1 infra_issue —
the class is category-agnostic), 30.2 answered-yet-open days total; worst case, a complete and
verified fix (task 3875) did not ship for 6.8 days after Leo said "task 3875 is released".
All five shared one fingerprint: the surviving record and the record carrying the ruling had the
identical single member id — one L1 promoted to L2 more than once; ruling one promotion
cascades down to the member but never sideways to the other L2s built on it. The probe, per
pending L2 (also runnable across all queues via scripts/member-chain-sweep.py, read-only):
- For each id in
.members, locate the member across the archive (find <queue-dir> -name '<member-id>.json'— every ruled member is archived by definition), never just the live dir. - If the member's status is terminal, read its
resolutionTEXT — status alone is useless
(both shapes below showdismissed):- substantive ruling text ⇒ this L2 is ANSWERED. Recover the ruling verbatim from the record
named inresolved_by(l2-cascade:<id>) rather than re-deriving it. "DUPLICATE of esc-X (survivor, stays open)"⇒ the opposite: a dedup pass deliberately
kept THIS record live; the question may be genuinely open.
- substantive ruling text ⇒ this L2 is ANSWERED. Recover the ruling verbatim from the record
When the probe fires, the item's ask changes from "human must decide" to "human must ratify and
propagate" — a much cheaper request. Present it that way, with the recovered ruling attached.
Unless it clears the carve-out below, in which case the ask changes to NOTHING and you close it
yourself: ratification is only needed for a ruling that is not already Leo's own, executed, and
orphaned by a terminated session.
This check is REPORT-ONLY — with exactly ONE carve-out (below), it must never close anything on
its own. A record can be a deliberately-preserved PIN whose value is its existence, not its
question: esc-3105-3 scores 15/15 ruled members on this probe and must NOT be closed (it is the
last hold on task 3105 / task 3546's mu-gate specimen; its sibling 3371 was destroyed by a bulk
close cascade on 2026-08-08; companion esc-3105-5 carries the DO-NOT-CLOSE flag asroot_cause = veto-pin-do-not-close:3105). From the member chain alone, a pin and an answered
question are indistinguishable. The machine-readable marker now exists (task 4377): a record whosepin_declared_by is non-empty has been declared load-bearing, and resolve_issue refuses
every non-park action on it — and on any L2 whose cascade would close it — with{'code': 'declared_pin_refused', 'declared_pins': [...]}. It rides every compact row, so read it
on the drain; pin_declared_reason (the free-text why) is only on the full record viaget_escalation. Read what pin_declared_by NAMES and consult it — acknowledge_declared_pins
exists to spend a pin deliberately, not to clear an inconvenient error. None of that changes this
check: it stays REPORT-ONLY, and an unmarked record is still not proof that nothing relies on
it — the marker is opt-in, so absence means "not declared", not "safe". Reading the record and its
companions before proposing any disposition remains the protection. Note also who can WRITE the
marker: declare_pin is operator/steward-only today — it is not in the rotation's allowed tools
(orchestrator/src/orchestrator/harness.py::_WATCHER_ALLOWED_TOOLS) — so when this probe finds a
likely pin that carries no pin_declared_by, the output is a REPORTED candidate pin naming what
appears to rely on it, for a human to declare. esc-3105-3 itself is still in that state.
Carve-out: mechanically actioning a ruling Leo has ALREADY made. You do not need Leo's
permission a second time to do the bookkeeping on a decision he has already made and that has
already been executed. When a session ruled a record, did the world-facing work, and then
terminated without closing the record, closing it is mechanical, not a new decision — close it
yourself and report afterwards. This is safe only because the checklist below demands positive
documentary evidence of a specific human ruling, which a pin by construction never has: the
report-only default exists because a pin and an answered question are indistinguishable FROM THE
MEMBER CHAIN ALONE, and nothing here takes the member chain as its authority. esc-3105-3 /
esc-3105-5 fails at item 5 and keeps working exactly as it does today.
All six gates must hold. Any miss ⇒ report-only, unchanged.
- The ruling is Leo's OWN and EXPLICIT — never inferred. It must be attributable in writing:
a task description, a commit message, a fused-memory ruling record, or another escalation'sresolutiontext that names him ruling it. An agent's recommendation, atriage_note's
conclusion, asuggested_action, or your own read that "this looks settled" does NOT qualify. - The ruling names THIS record — by escalation id, or it unambiguously answers this record's
specific question. A ruling on an adjacent topic in the same programme does not qualify. - The ruling was EXECUTED, and that is verifiable NOW. The world-facing change — task
retargeted or rewritten, dependencies wired, commit landed, config flipped — must be observable
at close time, not merely promised in the ruling text. Go look; never take the ruling's own word
for its own execution. - The originating session has TERMINATED. Do not race a live session. Peer sessions are
enumerable via theListAgentstool; a session that ran out of context, was closed, or whose
work landed hours ago with the record still open is terminated for this purpose. If it may still
be running, leave the record and note it. - The record is NOT a pin.
pin_declared_byis empty,pins_recoveryis empty,root_cause
is not aveto-pin-do-not-close:*key, and no DO-NOT-CLOSE companion record exists for the same
task. This is the protection that must not be weakened — if any of the four is unclear, treat
the record as a pin and stop. (A non-emptypin_declared_byis refused byresolve_issue
regardless;acknowledge_declared_pinsis never the answer on this path.) - The sideways check has been run —
get_pending_escalations(task_id=...)for the subject
task, dispositioning any twin L2 sharing a member in the same sitting (see "At every resolve,
look sideways before moving on" under "Resolving Escalations" below).
When all six hold:
- Resolve with the appropriate C1 action — usually
resume;close_onlywhen the record is a
re-report of an already-ruled class and nothing about the task should change. - Write the recovered ruling into the
resolutionverbatim-in-substance, and name where the
ruling lives (briefing artifact id, commit sha, task id) so the next reader does not re-derive
it — and record which residual questions are owned by which tasks, rather than letting the close
imply those closed too. - Report the action to Leo afterwards. No permission needed; the close is never silent.
Worked example — esc-3881-3 (design_concern, info, task 3881) asked for sign-off on
retargeting task 3881 away from a named consumer (_split_cross_project_task_nodes) that does not
exist. Leo ruled it option C on 2026-09-01 via the "The Identity Seam" briefing (artifactb9b4c172-6853-4a12-b242-1a139bd4bb50) and the ruling was fully executed — task 3881's description
now opens RETARGETED 2026-09-01 — option C of esc-3881-3, ruled by Leo via "The Identity Seam" briefing, its scope was rewritten to the safe-A shape, and deps were wired to 3669/3672/4932/4985
— but that session ran out of context before closing the record, so the L2 sat pending with its
question already answered. pin_declared_by empty, pins_recovery empty, root_cause the substantivedesign-concern:3881:… key rather than a veto-pin key, no DO-NOT-CLOSE companion, sole memberesc-3881-2 cascade-closing cleanly: all six hold, so the watcher closes it and reports.
Expect orphans; they are not anomalies. That same sitting's docs commit (3057ffedfd) recorded
the esc-3673-1 / esc-3375-1 half of the ruling and never mentioned 3881 at all — which is exactly
how the record got orphaned. A ruling's propagation is routinely PARTIAL, so a ruled-and-executed
record left open is the expected residue of a large sitting, not a sign that something went wrong.
Everything that FAILS the checklist stays report-only: the ask is still "human must ratify and
propagate", and if you park it, park with a world-facing predicate naming where the candidate
ruling lives — never a predicate about the record's own status (see "Reading a triage-ack
annotation" above).
If you run a dedup/consolidation pass over pending L2s (the 2026-08-19 sweep was such a pass,
done by hand): before designating any survivor, run this check on the shared members. Never keep
an already-answered record open as the survivor — close it against the recovered ruling, recording
where the ruling lives. The 08-19 pass did the opposite on three clusters and manufactured three
of the five instances above.
Sitting preparer (prepare-sitting mode)
The sitting preparer (scripts/sitting/) turns every question awaiting Leo, across every project's
queue and the cockpit decision registry, into one brief he answers by number. It reads the stores
of record and never writes them; its only files are under data/sitting/, plusdata/return-brief.md. Every mutation below is an agent action from a pre-built payload, taken
only for an item Leo has answered or one that clears the carve-out in "Ruled-elsewhere check
(answered-but-unrecorded)" above, whose gates it evaluates and does not restate. Run it from the
project root as uv run --frozen --project shared python scripts/sitting/prepare_sitting.py <subcommand>.
When it runs
- Nightly at 05:30, from
return-brief.timer: headless on Fable, recommend-only and read-only.
This is the nightly form: investigate, thenrecordpreparations, which it does through
batch files in its per-run inbox (data/sitting/inbox/<run>/). The render that follows
writesdata/return-brief.md(OPERATIONS.md §12, "Cross-project return brief (05:30)"). Its
autonomous closes are windowed onclosed_at, and a close with noclosed_atis listed as
undated, not dropped. - On demand, whenever Leo asks for it.
- Unprompted, at three points: on watcher launch; on every return, meaning a human turn after 2h
or more of silence, the human turn that ends AFK posture (see "AFK by silence"), or any status or
"what do you need" question; and after each applied ruling batch.
The sitting and its numbers
- At launch:
new-sitting --ledger data/sitting/ledger-<lease-slug>.json --seed data/sitting/ledger-nightly.json. The seed carries forward the numbers Leo read in the return
brief. - Each 05:30 render seeds its numbering from the night before. An item keeps its number from night
to night, so a session launched on an earlier night still agrees with a newer page on every item
a render had numbered before that launch. An item your session numbered itself after launch can
carry a different number on a later page. Theresolve-answersecho table names each record id,
so check it there. - For every brief or re-brief:
brief --ledger <that ledger>. - A number belongs to its item for the whole session and is never reused. An applied item keeps its
number under Done. An item under a HOLD, a pin or another owner renders in the Standing / no
action footer instead of the numbered list.
Investigate before you record
record --from <file|-> validates and merges preparations: JSON, one object or a list, in the shape
of scripts/sitting/preparation.py::to_json_payload, where item is the key of the item'sbrief --json row. A refused payload writes nothing. Before recording an item:
- Do the work the script cannot:
git show task/<id>:<file>for work on an unmerged branch, and a
grep forspawned_fromfollow-ups. - Read every ownership
mentions; the handover file is prose, so it is evidence, never routing.
Anything the handover or Leo places under a HOLD, as Leo-owned, or as owned elsewhere is recorded
as astandingwith its release predicate (task_status_is,escalation_closedormanual). - A "nothing owns this" claim cites the probes that came back empty. An
unavailableprobe is not
an empty one. - Every option carries its
ramification. Every item carries a recommendation with its evidence
chain, or an explicitno_leanwith its reason. - Put every escalation and task id you mention in
cites, so the brief glosses it. - The item's briefing disposition decides whether it is recorded now, after investigation, or as a
direct ask (see "Disposition before briefing (H1–H6)").
Gate facts are yours to supply
The script measures only what the stores state outright: the pin markers, the DO-NOT-CLOSE
companions, and the member chain. Carve-out gates 1-4 (ruling, names_this_record, executed,session_terminated) and pins_recovery arrive as the preparation's gate_facts. Takepins_recovery from the get_pending_escalations compact row, where the server annotates it at read
time; it is not on disk. Each fact is {held, evidence, source_kind}, and ruling holds only from a
documented source kind (scripts/sitting/gates.py::DOCUMENTED_RULING_SOURCES).
Every fact defaults to unknown, and unknown fails closed. That is why --apply-closes alone closes
nothing: an item becomes closeable only when all six gates hold on recorded evidence. Without the
flag, would-be closes are listed under "Recommended closes (recommend-only)".
Docket page or numbered list
When the brief reports a docket threshold (6 or more decisions, 3 or more heavy ones, or--multi-sitting), publish the brief --docket-json rows as a page with the db capability. Read
Leo's rulings back from it and echo each one VERBATIM into the transcript (item, option, note,
timestamp) before resolving. Below the threshold, the numbered terminal list is the surface. Both
enter through resolve-answers, with --source docket or --source terminal.
Applying Leo's answers, in order
- Run
resolve-answers --ledger <ledger> --answer 'N=OPTION[:note]' ...and paste its echo table.
It exits 3 when a token does not resolve: ask back on each such token and apply nothing for it. stamp_triagewithpayloads.append_markers(existing_note, <x_prepared>, <x_agreed>), while the
record is still pending (see "Sitting markers" below; the REPLACES caution applies).resolve_issue(..., resolution_turns=<the value resolve-answers printed>).- The
update_taskx_rulingpayload. Say so when it discloses a value it supersedes. - The pre-built
close-decisionpayload, quoting the deciding evidence.close-decisionrequires--projectand--escalations-dirnaming the record you READ, and refuses any other record, so
run the payload rather than hand-typing the verb. A non-zero exit stops that item's apply; report
it to Leo. - One
add_memoryasdecisions_and_rationale, plus a second write carrying the rationale to a
Mem0 category, because the Graphiti route keeps the conclusion and distills the reasoning away.
The order is deliberate. The markers go on while the record is still pending, and resolve_issue
runs before close-decision, so a server refusal such as declared_pin_refused or park_on_info
stops the apply before the registry records an answer. On park_on_info the record is
info-severity: do not retry the park; hold the task with a deferral (AFK shift 1) and report it to
Leo.
An L2 with no DecisionRecord is filed, then closed, underscripts/sitting/payloads.py::sitting_decision_id: <project>-<esc id>, or <project>-recon-<esc id> for the recon queue, which is the id write-decision stores for the project-local id the
payload files. Decision ids are fleet-global while escalation numbering restarts per project; see
"Across projects, the same local id is two rows" in "Filing Parked Decisions to the Cockpit
Registry (C8)".
Findings for another task's owner
A finding is appended to its owner task only while that task is pending. Otherwise it is put to
Leo, or filed through the pre-built submit_task follow-up; payloads.route_finding_to_owner builds
whichever applies.
Trial and kill criteria
- Stay recommend-only for the first two sittings.
- Leave the trial only at 90% or more agreement AND a 60% or greater fall in sitting time, read from
summary --ledger <ledger>and the return brief's trial line. - One wrong autonomous close in 50 reverts to recommend-only: stop passing
--apply-closes.
Sitting markers: x_prepared and x_agreed
The sitting preparer (scripts/sitting/) records its trial instruments on the escalation itself,
as marker lines in triage_note, stamped only when an item Leo has answered is applied — never
before he answers:
x_prepared— what the sitting recommended: an option label, or an explicit no-lean with its
reason.x_agreed— Leo's answer, whether it matched that recommendation, and how many turns it took.
Each is one line of sorted JSON built by scripts/sitting/payloads.py
(render_prepared_marker / render_agreed_marker); a reader takes the last line of each kind.stamp_triage replaces the whole note (see "CAUTION: stamp_triage REPLACES triage_note"
below), so send payloads.append_markers(existing_note, ...), which keeps the existing note —
including any world-facing predicate line — verbatim. These are distinct from x_shadow_ruling:,
which records what the adjudicator would have ruled, never what Leo did.
review_suggestions (info)
This handler is unreachable at L2. Review suggestions reach live workflows as curator tickets
via_route_review_suggestions_to_curatorin workflow.py (call site ~line 3064), with no
escalation file written; they fall back to level-0 steward escalations filed around
workflow.py:6272 and consumed by_next_escalationin steward.py. They do not reach this
queue. This stub is kept only to document whyreview_suggestionsmust not be re-added here.
review_issues (blocking)
Blocking issues found during code review — the review cycle exhausted without the agent fixing them. The task agent is stopped.
This is distinct from review_suggestions (info-level, non-blocking). Review issues are real problems that prevented the task from merging.
Spawn an interactive /unblock session via the /spawn skill: invoke /spawn with prompt="/unblock <task_id> (esc <escalation_id>, review_issues, <severity>: <summary>)", terminal_title="unblock:<project>#<task_id> <short-slug>" (e.g. unblock:df#2085 routing-mechanism; abbreviate the project token per the emergent convention), cwd=<project_root>, skip_permissions=true. Leave the escalation pending — /unblock resolves it when the human finishes. The human needs to see the specific blocking issues and decide how to fix them. The trailing (esc ...) context is additive only (see the additive-context convention note above).
If the low-risk auto-unblock gate applies — see Low-risk auto-unblock gate (B3) — try it first.
task_failure (blocking)
Merge conflicts, verification failures, build breaks. The task agent is stopped and waiting.
Before investigating, check the block reason / merge_status.failure_diagnostic for adisposition — see Merge-failure disposition vocabulary
above. If it reads integration_skew, the fix is porting the named landed commit(s), not
debugging the branch's own diff, and it must not be counted as a flake.
Spawn an interactive /unblock session so the human can investigate and resolve it: invoke /spawn with prompt="/unblock <task_id> (esc <escalation_id>, task_failure, <severity>: <summary>)", terminal_title="unblock:<project>#<task_id> <short-slug>" (e.g. unblock:df#2085 routing-mechanism; abbreviate the project token per the emergent convention), cwd=<project_root>, skip_permissions=true. Leave the escalation pending — the /unblock skill resolves it when the human finishes. Track the spawned session so you can report its status if asked. The trailing (esc ...) context is additive only (see the additive-context convention note above).
If the low-risk auto-unblock gate applies — see Low-risk auto-unblock gate (B3) — try it first.
wip_conflict / unmerged_state / stash_failed (blocking, halt-owner)
These escalations mean the merge queue is globally halted — no other task can merge until exactly one of them (the "halt owner") is resolved. The orchestrator records which escalation owns the halt on the merge worker (_halt_owner_esc_id); resolving that specific escalation via MCP un-halts the queue. Resolving any other escalation — even another wip_conflict — will NOT release the halt (fixed 2026-04-19; prior code relied on a category heuristic that caused phantom-L1 bugs like esc-1888-57).
Three flavours:
wip_conflict— the merge queue tripped on uncommitted work inproject_root. Three sub-variants distinguishable from thedetail:- WIP overlaps the merge diff (merge did not land; workflow will retry after resolution).
- Stash pop conflicted after the merge landed (merge IS on main; WIP preserved on
wip/recovery-<task>-<ts>). - Stash pop conflicted on CAS-failure path (merge did NOT land; WIP on recovery branch; task blocks).
unmerged_state—project_rootalready had UU/AA/DD markers before the merge attempted to advance (pre-existing corruption, not caused by this merge).stash_failed— the merge queue could not parkproject_root's dirty tracked WIP before advancing (task 2758). Like the other two this is a shared main-checkout-hygiene fault rather than a fault of the merging task, so the queue halts once instead of failing task after task.
As with task_failure, check for a disposition in the block reason / failure_diagnostic
before assuming this is a raw conflict to resolve mechanically — see
Merge-failure disposition vocabulary
above. integration_skew still needs a port of the named landed commit(s), not just conflict
resolution, and is never a flake.
Never auto-resolve — manual_intervention is authoritative. The human has to inspect project_root:
- For
wip_conflict: recovery branch named in the detail preserves the user's WIP; they may need to cherry-pick or reapply before resolving. - For
unmerged_state: rungit statusinproject_root; UU/AA/DD files needgit mergetool, manual edit, orgit resetdepending on intent. - For
stash_failed: inspect the main checkout's uncommitted work and commit it — or get its owner to. Do not reach forgit stash: CLAUDE.md forbids it in any dark-factory checkout,project_rootor task worktree, becauserefs/stashis a single ref shared by every worktree in the checkout (it is not per-worktree) and the merge worker's advance path consumes the same stack — so a stash you push can be popped out from under you, and a pop on a clean tree can apply an unrelated task's WIP into it (incident 13674d3c68). Park WIP as commits on a branch.- Sub-variant: if the detail/log instead shows
CRITICAL: stale refs/dark-factory/merge-park present, that's a staleMERGE_PARK_REFleft over from a prior crash — it holds real, unrecovered WIP that the code deliberately refuses to overwrite. Committingproject_root's current dirty files is not enough here: the stale ref itself is what's blocking the next park attempt, so the halt recurs immediately on the next advance. Recovery means inspecting and draining that ref first — e.g.git branch <recovery-name> refs/dark-factory/merge-parkto make its WIP reachable, thengit update-ref -d refs/dark-factory/merge-parkto free it — before cleaning upproject_root.
- Sub-variant: if the detail/log instead shows
Spawn an interactive /unblock session via /spawn (prompt="/unblock <task_id> (esc <escalation_id>, <wip_conflict|unmerged_state|stash_failed>, <severity>: <summary>)", terminal_title="unblock:<project>#<task_id> <short-slug>" — e.g. unblock:df#2085 routing-mechanism; abbreviate the project token per the emergent convention — cwd=<project_root>, skip_permissions=true) so the human can see the recovery branch, inspect project_root, and resolve the escalation when finished. The trailing (esc ...) context is additive only (see the additive-context convention note above).
Phantom-halt check: if the orchestrator log shows "Merge queue un-halted: halt owner <esc.id> resolved" but the escalation file still has status: pending, that is a bug — report to the human; do not silently dismiss. (Historical context: pre-fix, this was a common symptom of the category-match un-halt bug.)
scope_violation (info or blocking)
Agent discovered it needs to touch files beyond its assigned scope.
Resolve with action='resume', carrying the scope grant as granted_files — a single call, no task-metadata edit:
mcp__escalation__resolve_issue(
escalation_id="...",
resolution="Scope expanded to include [<files>]; resuming.",
action='resume', # re-pends the task AND folds the grant — see below
granted_files=["<project-relative file path>", ...], # file-level paths, not module names
resolved_by="escalation-watcher"
)granted_files is now delivered on the re-pend path, not just to a live workflow. It is persisted durably on the escalation record either way, and there are two consumers:
- Live workflow (per-task steward's path).
orchestrator/src/orchestrator/workflow.py::TaskWorkflow._collect_granted_files→_set_task_scope, from a single call site inworkflow.py::TaskWorkflow._drive, reached only after_wait_for_resolution()returns for a still-alive workflow process. That is why the same wording is correct inorchestrator/src/orchestrator/agents/roles.py::STEWARD. - No live workflow (this queue's normal case). The
scope_violationitems reaching an L2 queue are usually on ablocked— or strandedin-progress— task whose workflow slot is gone. Resolution takes the orphan/cascade branch (orchestrator/src/orchestrator/harness.py::Harness._on_escalation_resolved) intoharness.py::Harness._cascade_unblock_member, which since task 3540 callsharness.py::Harness._fold_granted_files_on_rependbetween the re-block guard and the status write. That helper unionsgranted_filesacross every resolved escalation for the task (the whole history, order-preserving — the same union_collect_granted_filescomputes) and writes it to bothplan.json'sfilesandmetadata.files, before the row goes re-pendable.
So resume + granted_files on a blocked or stranded task with no live claimant now widens the scope the redispatched agent actually works to. plans/task-escalation-state-graph-prd.md D8 and docs/task-escalation-state-spec.md E9 list this as semantics to be built; 3540 built it.
Three caveats that survive, so state them accurately rather than promising more than the fold delivers:
- The fold is best-effort. A failed plan write or a failed
update_tasklogs a WARNING and the task re-pends anyway, against the unwidened scope — deliberately, because withholding the re-pend would park the task with its escalation already closed and nothing left to advance it. Re-checkplan.filesafter the redispatch rather than assuming the widening took. - No plan, no widen. A task that never reached the architect (no
plan.jsonin either.task-meta/<worktree>/or the legacy<worktree>/.task/) has no scope to widen; the fold logs a DEBUG and skips, and the re-pend still happens. - The re-block guard outranks the grant. If the guard withholds the flip, nothing is re-pended, so nothing is widened.
Module locks are a separate question from working scope, and the fold does not touch them directly: orchestrator/src/orchestrator/scheduler.py::Scheduler._get_modules is cache-first (its own docstring: "deterministic short-circuit > in-memory cache > metadata.files > fallback"), and the narrow _module_cache entry from the task's prior dispatch survives an escalation — blocked is in orchestrator/src/orchestrator/task_status.py::ACTIVE_TASK_STATUSES, so scheduler.py::Scheduler._phase_stale_sweep does not evict it. Lock widening is re-established branch-side instead, by workflow.py::TaskWorkflow._reconcile_scope_locks — the choke point every path that (re)establishes plan.files runs through (_plan(), _apply_revalidation_skip(), _run_simple_task(), _set_task_scope()). That is also why writing plan.json is the load-bearing half of the fold: _reconcile_scope_locks persists metadata.files = plan_files on every successful refinement, so a metadata-only widen would be narrowed straight back down on the next dispatch, while a plan-level widen propagates the other way.
Do not hand-write metadata.files yourself to simulate the fold. Beyond being redundant now, a metadata-only write is self-reverting for exactly that reason, and it is not consulted for lock derivation while the module cache is warm.
Do not try to widen scope by writing modules into task metadata. Lock derivation (Scheduler._get_modules) reads metadata.files and has never read that key, so such a write is a silent no-op that reports success. A lock conflict on the grant is handled orchestrator-side — the task requeues rather than resuming under another task's lock.
dependency_discovered (info or blocking)
Agent found it depends on work that isn't done yet.
- Check if the prerequisite is an existing task that isn't Done yet.
- If yes: add the dependency via
mcp__fused-memory__add_dependency, then re-pend — the
dependency gate will hold the task until the prerequisite completes:mcp__escalation__resolve_issue( escalation_id="...", resolution="Added dependency on task <dep_id>. Task re-pends; held by dependency gate until dep completes.", action='resume', # flips blocked→pending; dependency gate holds dispatch until dep is done resolved_by="escalation-watcher" ) - If no matching task exists: spawn an interactive
/unblocksession via/spawn(prompt="/unblock <task_id> (esc <escalation_id>, dependency_discovered, <severity>: <summary>)",terminal_title="unblock:<project>#<task_id> <short-slug>"— e.g.unblock:df#2085 routing-mechanism; abbreviate the project token per the emergent convention —cwd=<project_root>,skip_permissions=true). The trailing(esc ...)context is additive only (see the additive-context convention note above).
design_concern (info or blocking)
Architectural or design questions. These already failed steward auto-resolution — they're genuinely ambiguous.
Always escalate to the human, except for the narrow self-close case defined in "Standing rule:
accept verified info-level design deviations" below — check that subsection first:
- Present the concern with full context, after its briefing disposition — H3 and H6 usually fire
here (see "Disposition before briefing (H1–H6)") - Leave the escalation pending — the open escalation record IS the durable record that something
needs doing - Create a local todo for this session only — it does not survive session end and is not the
record
3a. File (or confirm one already exists for thisesc-id) a cockpit DecisionRecord viawrite-decision— this, not the todo, is what makes the item recoverable across sessions and
after this session ends (same registry as the Priority-3b instructions above). Skipping this
step is how esc-3223-4/-5 kept task 3223 blocked for 11 days: the question never reached the
cockpit queue the human actually reads. - Continue handling other escalations while waiting
- Append
<esc-id>to the wrapper-owned exclude-file (see "Starting the watcher" above) while this item is pending
While parked, the item must remain falsifiable. Steps 2+5 park the record and gag its wake
channel — deliberate, but it makes the parked set a mutation-blind pocket: a task-side ruling
never touches the record, so nothing bumps updated_at when the world answers the question. On
rotations that revisit the parked set, re-verify with world-facing probes only (see "Reading a
triage-ack annotation" and the "Ruled-elsewhere check" above) — never a predicate about the
record's own pending status. If a probe fires, the ask flips from "human must decide" to "human
must ratify and propagate": recover the ruling, present it for ratification, and propagate it into
the record with amend_escalation (an append-only amendment; it changes no status, severity or
level). This applies equally to risk_identified parks below.
A scope-not-delivered record closes only two ways. That is a record stating that ONE ITEM of
a task's scope was not deliverable by that task; it closes only by one of the two closure forms indocs/escalation-standing-policy.md, never by a bare accept. That policy is IN FORCE now (Leo,
2026-09-21, esc-4811-3) — not one of the shadow-mode candidate classes further down this file —
and it grants no one authority to close such a record.
Standing rule: accept verified info-level design deviations (Leo, 2026-09-17)
The watcher may close a design_concern itself, without parking it, only when ALL of these
hold:
- Kind. Severity
info, neverblocking/critical/urgent, and never amilestone_gate. - After the fact. It asks the human to ratify a deviation an agent has already made. The
agent is an architect, planner or amendment pass, and the deviation is from a frozen plan, a
reviewer suggestion or a prescribed approach. It is not a choice about what happens next. - Evidence checked. The deviation's reason is measured or documented, AND the watcher has
checked it itself against the diff, plan, code or test output. Never take it from the record's
own text. - Nothing waits on it. The subject task is not blocked on this record, and the record is not
a pin:pin_declared_byis empty androot_causeis notveto-pin-do-not-close:*. - Accept is enough. After checking, the answer is "accept as done" with no follow-up work.
- It overrides nothing of the human's. Accepting does not:
- contradict an existing ruling;
- touch a HOLD or a milestone gate;
- drop scope a task was meant to deliver;
- change a persisted, public or cross-project contract away from what a PRD or ruling
specified; - touch security or sandboxing;
- cancel or delete work.
When all hold:
resolve_issue(action='close_only'), which leaves the task untouched. Theresolutionnames
the deviation and the evidence checked, and says "accepted under the standing rule (Leo,
2026-09-17)".- Append a one-line dated note to the task's
details(update_task(..., append=True)) so the
acceptance is visible from the task record. An escalation's resolution is not reachable from the
task. - Report it to the human in the next message as one line: id, what was accepted, what was
checked. A cockpit DecisionRecord already filed for it closes throughreap-decisions.
Otherwise, meaning any condition fails, the evidence cannot be checked, or the recommendation
is anything but accept, use the normal park procedure above.
Limits. A second such escalation on the same task goes to the human as a pattern instead: a
stream of deviations on one task suggests its planning is off. There is no daily cap: Leo discarded
it on 2026-09-26 ("Discard the cap. You can ratify these without limit."). The human can revoke
this rule at any time.
Worked examples (2026-09-17).
- Accepted: esc-4876-8. A planner measured that of three suggested graphiti levers only
entity_typesreaches the dedupe decision. The watcher confirmed against the installedgraphiti_corethatresolve_extracted_nodestakes nocustom_extraction_instructionsand
thatprompts/dedupe_nodes.pynever uses it. - Excluded: esc-4811-3. "Fixture expansion (plan item 4) is not deliverable" is a scope item
not delivered, bearing on a reason behind the human's write_triage HOLD. That fails condition 6
even though it is info-level and well-evidenced, and its task was blocked on it (condition 4).
It is ascope-not-deliveredrecord, so how it may be closed is governed by the closure policy
above rather than by this rule.
risk_identified (info)
An agent flagged a risk during development. Risk assessment requires human judgment.
Escalate to the human. Tell them; create a session-only todo (an attention aid — the pending
escalation, not the todo, is the durable record); file (or confirm) a cockpit DecisionRecord viawrite-decision for this esc-id, exactly as in design_concern step 3a; continue with other
work. Append <esc-id> to the wrapper-owned exclude-file (see "Starting the watcher" above) while
this item is pending.
cleanup_needed (info, rarely blocking)
Technical debt or cleanup discovered during development.
Info: queue as a follow-up task using the two-phase pattern:
suggestion_hash = hashlib.sha256( (escalation['detail'] or escalation['summary'] or escalation['id']).encode() ).hexdigest()[:16] # Case A — escalation id already in scope; see _shared/ticket-failure-handling.md # Phase 1: submit — returns immediately with a ticket id submit_result = mcp__fused-memory__submit_task( project_root="<project_root>", title="<cleanup description>", description="<what needs cleaning up, with file paths and specifics>", priority="medium", metadata={ "source": "escalation-info", "escalation_id": escalation_id, "suggestion_hash": suggestion_hash, # (escalation_id, suggestion_hash) is the idempotency key "spawn_context": "steward-triage", # sparse is fine — the architect widens scope at plan time. File paths only (a directory is rejected); use [] to defer entirely. "files": ["<path/to/file.py>"], }, ) ticket = submit_result["ticket"] # Phase 2: block until the curator decides resolve = mcp__fused-memory__resolve_ticket( ticket=ticket, project_root="<project_root>", timeout_seconds=<see skills/_shared/ticket-failure-handling.md> ) if resolve["status"] in ("created", "combined"): task_id = resolve["task_id"] elif resolve["status"] == "refused": # Deliberately NOT in the tuple above: a refusal has no task_id. # A deterministic guard rejected the candidate; no task was created. # Record resolve["reason"] in the escalation resolution note. # Do NOT retry and do NOT record a task id. note_refused(resolve["reason"]) elif resolve["status"] == "awaiting_curation": # Deliberately NOT in the tuple above: the ticket is parked with no # task_id yet. Do NOT re-submit. Resolve the escalation with # "queued for curation (ticket <ticket>)". note_parked(ticket) elif resolve["status"] == "failed": # Record reason in escalation resolution note; skip this item. # See skills/_shared/ticket-failure-handling.md for the retryable/terminal reason matrix. handle_failure(resolve["reason"])Resolve via
mcp__escalation__resolve_issueonce the ticket resolves.Blocking (rare): spawn an interactive
/unblocksession via/spawn(prompt="/unblock <task_id> (esc <escalation_id>, cleanup_needed, <severity>: <summary>)",terminal_title="unblock:<project>#<task_id> <short-slug>"— e.g.unblock:df#2085 routing-mechanism; abbreviate the project token per the emergent convention —cwd=<project_root>,skip_permissions=true). The trailing(esc ...)context is additive only (see the additive-context convention note above).
infra_issue (blocking)
Infrastructure problems — database connectivity, MCP failures, service outages.
Priority 1 — system stability:
- Tell the human immediately with full details
- Leave the escalation pending
- Do NOT attempt automated infrastructure fixes
- File a DecisionRecord via
write-decision(see "Filing Parked Decisions to the Cockpit
Registry" above) — IN ADDITION to telling the human directly - Wait for human instructions
- Append
<esc-id>to the wrapper-owned exclude-file (see "Starting the watcher" above) while this item is pending
recon_* categories
recon_failure, recon_backlog_overflow, recon_stale_run, recon_integrity_issue — these are all fused-memory reconciliation problems.
Reconciliation is infrastructure that affects memory quality across the entire system. Tell the human with full details. Track as a todo. These may indicate systematic issues that need root-cause investigation rather than point fixes. Also file a DecisionRecord via write-decision (see "Filing Parked Decisions to the Cockpit Registry" above).
Shadow-mode standing-policy rulings (measurement only)
docs/escalation-standing-policy.md proposes classes of L2 that an adjudicating session could one
day rule without waiting for the human. None of them is adopted. This section adds one thing to
your loop and it is not an action: for an L2 you would rule under that policy, record what you
would have ruled, then handle the record exactly as its category section above says — which forrisk_identified and design_concern still means escalating to the human and filing the cockpit
DecisionRecord. The stamp changes nothing about what you do.
Never stamp a record you are going to rule yourself
This rule comes before the mechanics because skimming past it is how the measurement goes bad.
A design_concern you close under "Standing rule: accept verified info-level design deviations
(Leo, 2026-09-17)" above gets no shadow stamp. You are the adjudicator there, so there is no
independent decision to compare your proposal against.
The reason, in one line you can check: the weekly count reads resolved_by back throughescalation/src/escalation/classify.py::classify_resolver_tier, where escalation-watcher
classifies as human — exactly like a Leo ruling. A stamp plus a self-close is therefore the
session agreeing with itself, and it would push a class toward its own adoption threshold on the
strength of your own actions.
The count does catch it: such a record is bucketed self_resolved and dropped from every rate. So a
violation costs the sample, not the truth — but it still costs the sample.
Stamping
Write the proposal as one x_shadow_ruling: line inside triage_note:
x_shadow_ruling: {"class": "risk_identified_branch_behind_main", "proposed_action": "close_only", "evidence": "git merge-base --is-ancestor main task/4821 -> rc=0; branch is not behind", "confidence": 0.9}Call it as stamp_triage(escalation_id=..., triaged_by=..., triage_note=...). The class must be
one of the first-tranche slugs and proposed_action one of the reversible-action slugs, both
enumerated in docs/escalation-standing-policy.md. evidence quotes the deciding probe output
verbatim — not a conclusion about it. confidence is in [0.0, 1.0].
A payload outside those vocabularies is discarded by the reader, so it is a lost sample rather than
a loud error. This one is thrown away:
x_shadow_ruling: {"class": "risk_identified_branch_behind_main", "proposed_action": "restart", "evidence": "looks fine", "confidence": 0.9}restart is a C1 action but it is not reversible, so it is not in the reversible-action list and
the whole payload is dropped.
CAUTION: stamp_triage REPLACES triage_note, it never appends
Verified in escalation/src/escalation/queue.py::stamp_triage: passing a non-empty triage_note
overwrites the existing one wholesale.
So on an already-triaged record you must re-send the previous note's content with thex_shadow_ruling: line appended on its own line. Send the marker alone and you destroy the earlier
predicate and probe. (Omitting triage_note entirely is the safe freshness-bump form — it leaves
the existing note untouched.)
Re-stamping therefore leaves the note carrying two marker lines, which is expected and safe:
the weekly count reads the last one as the record's ruling and treats the earlier lines as
superseded. Append the new marker below the old one rather than editing the old one in place — and
if the newest line is malformed the record is counted in rejected_stamps, never scored against
the stale proposal above it.
The marker goes on its own line of a note that still satisfies the freshness contract in
"Reading a triage-ack annotation" above: a named world-facing predicate plus the probe used to check
it. A shadow stamp is not a substitute for that predicate — and per that same subsection, a
predicate about the record's own status is vacuous. A well-formed stamped note looks like:
task-4821 branch tip not behind main | probe: git merge-base --is-ancestor main task/4821 -> rc=0
x_shadow_ruling: {"class": "risk_identified_branch_behind_main", "proposed_action": "close_only", "evidence": "git merge-base --is-ancestor main task/4821 -> rc=0; branch is not behind", "confidence": 0.9}Never stamp a human-forever gate
docs/escalation-standing-policy.md lists all seven. Two are detectable from the record itself andescalation/src/escalation/shadow_ruling.py::mechanically_gated finds them: a milestone_gate
category and the orchestrator-deterministic role. The other five — model admission, physical
operator actions, irreversible deletions, spend or eval launches, and a post-breakerresume_scheduler — have no signal on the record, so they are your judgement. A stamp on any of
them is reported as gated_stamps and excluded from every rate.
Two facts about attribution and timing
Attribution here is a convention, not a guarantee (see "Reading a triage-ack annotation"
above): this session sends no identity header, so triaged_by is whatever you pass. Pass the
same identity string you resolve with, or triaged_by and resolved_by never compare and theself_resolved check silently never fires.
Stamp before the record is resolved. stamp_triage refuses anything that is not pending, so a
stamp written after the close is simply not written.
The weekly count
uv run --directory escalation python -m escalation.shadow_ruling \
--queue-dir <project_root>/data/escalationsRead it as: agreed / diverged over the comparable denominator printed beside the rate;not_comparable for proposals whose action is task-side and leaves no resolution_action to check
against; non_human_resolver for a record no human resolved at all — a cascade, a sweep, the
steward — counted per class, because the aggregate resolver_tiers line says only which tier took
the sample; gated_stamps, self_resolved and rejected_stamps for stamps excluded from every rate
— those three are counted over the same window as the rate. A class whose records are mostlyself_resolved is not a class with a small sample — it is not measurable yet, and a non-zerorejected_stamps means the count could not read that many markers at all.
unresolved_lifetime is the exception and says so in its name: a pending record has noresolved_at to window on, so that number is the standing backlog at sweep time, not a count from
the window in the header.
A class adopts only when task 3346 has landed and it has met the threshold indocs/escalation-standing-policy.md. Until both hold, keep stamping and keep escalating.
Context Conservation
You're in a long-running session — conserve your context window aggressively. Over a multi-day AFK
window this is the difference between one durable session and repeated restarts. When context runs
low anyway, stop (see "Stopping"): the handover carries forward what the next session needs.
Read compact, expand lazily:
- Drain with
get_pending_escalations(level=2, compact=True)— never pull full dicts just to triage. - Don't keep the watcher's wake-signal JSON in context; triage from the compact drain.
- Pull the full record (
get_escalation(id)) for only the one item you're about to act on, and
prefer doing that read inside the handling sub-agent so the heavydetail/evidencenever lands
at top level.
Delegate to sub-agents:
- Researching escalation context for ANY category that needs code reading (e.g.
task_failure,design_concern): have the sub-agent fetch the full escalation, read the code/reviews, and return
only a compact verdict + recommended action — not the raw material - The low-risk auto-unblock sub-agent (
unblock-low-risk) — run it in the background
(run_in_background: true) so its full apply→verify→submit→poll cycle stays in its own context,
keeping the watch loop lean and responsive; it returns only a small JSON result when it completes.
After ahanded_off, that cycle includes its resume-mode poll, also in the background, so the
watch loop never polls a B3 merge at top level - ANY other merge submission (e.g. retrying the land of a done-but-unmerged task) — submit
top-level using the bounded submit→poll protocol; see "Merge Submissions — Bounded Submit, Then Poll" - Creating follow-up tasks (once you've decided what to create, have a sub-agent do the MCP calls)
- Choosing between a sub-agent and a
/spawned session: see "Sub-agent or/spawn"
Keep in top-level context:
- The watch loop itself (your core job)
- Decision-making about how to handle each escalation
- Communication with the human
- Tracking pending human decisions and spawned
/unblocksessions
When delegating, give the sub-agent complete context — paste the escalation JSON and explicit instructions. The sub-agent cannot see your conversation history or MCP state.
Tracking Pending Human Decisions
Maintain awareness of escalations waiting for human input. When the human returns or asks for
status, answer with the sitting preparer (see "Sitting preparer (prepare-sitting mode)" above)
rather than a hand-assembled list: brief --ledger <session ledger> is the numbered list, with each
item's options, recommendation and age, and its standing footer says what is waiting on someone
else. The brief lists every item awaiting Leo, with no cap (see "No cap on items awaiting Leo").
Remind about unresolved items roughly every 3-5 escalation handling cycles — enough to keep them visible without being noisy.
Joining a completed spawn: read result.md, don't explore
When a /spawn-launched /unblock background task completes, that's the join step of the
fan-out/join pattern (fan-out was the spawn; the background task's completion is your join signal).
Exit codes are liveness-only — present-vs-died-silent, not a semantic outcome signal, including
the observed 129-on-clean-exit race (see skills/spawn/SKILL.md's Verification section) — so never
infer success, failure, or completeness from the background task's exit code.
Instead, read the session's result file for the structured outcome:
- Locate the session-registry record for the spawn (the slug you launched it under, or the
matching record under~/.claude/fleet/sessions/for the task/title you spawned). - Read
record.result_file— equivalently~/.claude/fleet/sessions/<slug>/result.md— for theoutcome: done|blocked|abandoned|handed-off|in-progress,changed(commits/branches/task ids touched),
andaction_neededfields the spawned session wrote before ending. - Use that structured outcome to decide your next step (resolve the escalation, follow up, escalate
further) —result.mdis the authoritative outcome source, so this replaces exploring the
task/worktree yourself to reconstruct what happened. - An
in-progressoutcome on a finished spawn (the background task has completed, so the record
is terminal) means the session ended at checkpoint, before its final write. Treat it like an
absent file and fall back to exploring the task/escalation state.
Fail-soft: the write is best-effort — if result.md is absent, empty, or unparsable (the
spawned session ended without writing it, or the registry write itself faulted so the record never
got a result_file), fall back to the existing exploration (task/escalation state via MCP, the
worktree itself) rather than blocking on the file.
Resolving Escalations
Via MCP (always prefer this):
mcp__escalation__resolve_issue(
escalation_id="esc-XX-N",
resolution="<text injected into the agent's briefing when it resumes>",
action='resume', # default least-destructive intent; see C1 table below
resolved_by="escalation-watcher"
)C1 — action semantics (single source of truth)
Scope: this is resolve_issue's HANDLER-side action parameter (server.py::RESOLVE_ACTIONS).escalate_blocker's response also carries an action key, but that is an orthogonal
filer-facing vocabulary (terminate_cleanly / keep_driving,escalation.models.FILER_ACTIONS, described in DESIGN.md) which merely shares the key name —
you never receive one, since this skill resolves escalations and never files them, and none of
its values may be passed to resolve_issue.
action |
Record disposition | Live workflow | Task status effect | Intent |
|---|---|---|---|---|
resume (default) |
resolved |
resumes; resolution text injected (L0 live path) | blocked → pending (any task-attached level ≥ 1, incl. memberless born-at-L2) |
"Here's the answer — continue." |
restart |
resolved |
killed (soft-cancel → grace → hard) | → pending (from in-progress or blocked) |
"This run is off-course — re-run fresh." |
park |
kept open at L2 | killed | → blocked (from any non-terminal status) |
"Stop; human decides later; held blocked under an open L2." Never on an info record: info never pins, so the sweep re-pends the task within a tick. Defer the task instead (AFK shift 1). resolve_issue refuses it with code park_on_info and changes nothing; the response's hint and task_id give the deferral to use instead. An info park that is accepted means that orchestrator's escalation server predates task 6578: defer the task anyway. |
abandon |
dismissed |
killed | → cancelled |
"Never run again." |
close_only |
dismissed |
untouched | none | "Record is noise/duplicate — change nothing." |
C1 notes:
- Terminal task statuses (
done,cancelled) are never overwritten by any action. - The removed
terminateparameter now raises a hard error naming the five actions above. - L2 cluster cascade: the action applies uniformly to the L2 and every member task.
queue.resolve()cascades members viaresolved_by='l2-cascade:<L2-id>'; the harness member callback reads the parent action from the queue read API. Foraction='park':queue.park()keeps the L2 and all member L1s open (status=pending); each member task endsblocked, covered by its still-open member L1 escalation — the stranded-blocked sweep skips each because Fix #1b finds the open L1. That cover holds only where the member L1 pins. Aninfomember never pins by itself, so treat its task as unheld (see theparkrow). - Legacy in-process callers with
resolution_action=None:dismiss=Truemaps toclose_only;dismiss=Falsemaps toresume.
Where the resolution text actually goes. It reaches the working agent only in the L0
steward-resolved path, where a workflow is still live and waiting (_wait_for_resolution →build_resume_prompt). That is not the usual L2 case. For the escalations this skill resolves:
- L2 cluster (has member L1s),
action='resume': the resolution cascades to each member L1,
flipping the member taskblocked→pending. It re-dispatches into a fresh workflow that does
not read your resolution text — the harness propagates status only. Don't rely on the string
reaching the agent. If the agent needs specific guidance, either spawn an interactive/unblock
(drive the worktree directly) or write durable guidance into fused-memory / task metadata, which
the fresh workflow's briefing memory-search may surface. - Memberless born-at-L2 (a direct
critical/urgentblocker with no L1 members): under D7
(task β),action='resume'on a memberless born-at-L2 now flipsblocked→pending— the orphan
flip accepts any task-attachedlevel >= 1. The resolution text is recorded for audit only and
does not reach the agent (no live workflow); write durable guidance into fused-memory / task
metadata instead. To re-run fresh useaction='restart'(→pendingfrom scratch); to park for
later useaction='park'(→blocked, held under an open L2); to abandon permanently useaction='abandon'
(→cancelled); to close the record without touching the task useaction='close_only'.
Either way, still write a clear, specific resolution (file paths, function names, the decision and
why): it is the audit record and the human-readable trail even when no agent re-reads it.
L2 cluster cascade (live). When a resolved L2 represents a causal cluster (member L1
escalations packaged by the auto-watcher), resolving the L2 here cascades to close its L1 members
via the escalation server — this skill resolves only the L2 itself, never each member directly. The
action applies uniformly across the cluster. The cascade is implemented in queue.resolve(): it
recurses over esc.members, resolving each with resolved_by='l2-cascade:<L2-id>', and the
auto-watcher files clusters via promote_to_l2. For design details, seeplans/escalation-l2-tiering.md.
You may still occasionally see multiple unclustered L2s that share a root cause — the auto-watcher
deduplicates by exact root-cause string, so near-miss hypotheses file separately. When you do, scan
them for shared files, summaries, or task IDs and handle related ones together, noting the
relationship in your resolution text.
At every resolve, look sideways before moving on. The ruling you write reaches only the record
you name (plus its downward member cascade — never sibling L2s). The resolve_issue response
carries that census as related_pending: the records still pending after your resolve that share
its task, or are L2s sharing a member with it (or clustering it). If the key is ABSENT, the census
could not be computed — fall back to running get_pending_escalations(task_id=...) for the subject
task and scanning for other pending L2s sharing any member id. Either way, disposition them in the
same sitting — close them against the same ruling, or park them with a world-facing predicate
naming where the ruling lives. Never auto-close from the census alone: it reports candidates, and
a pin looks exactly like an answered question on member evidence. An entry with same_task: true
and shared_member: null under a synthetic anchor task id (a slug such asl2-amendment-truncation, not a task number) is the same KIND of alarm, not necessarily the same
incident — read it before applying your ruling to it. A ruling recorded on one twin
while another survives is the answered-but-unrecorded class (see "Ruled-elsewhere check" above);
all five measured instances were minted exactly this way, in sittings that ruled the record in
front of them and never looked sideways.
Recognizing the supervised auto-watcher's resolutions (not a rogue actor)
You may see resolved_by="orchestrator-escalation-watcher-auto" stamped on archived L0/L1
records, or agent_role="escalation-watcher-auto" on a promote_to_l2 call that filed an L2.
Both are the trusted, supervised identity of the dark-factory orchestrator's own autonomous
auto-watcher (spawned per rotation by the watcher-supervisor, task 1326; runs theescalation-watcher-auto skill). Per the connection-capability guard
(plans/escalation-connection-capability-guard-prd.md), the orchestrator wires that watcher's MCP
connection with X-Escalation-Identity: orchestrator-escalation-watcher-auto, and the escalation
server stamps resolved_by from that header on resolve_issue's resolve/park path —
server-attributed, not something the watcher agent can spoof or drift. promote_to_l2 has no such
override: its agent_role is a plain tool argument, and the auto-watcher skill always passesagent_role="escalation-watcher-auto" (the literal tool-arg value) on every promote call, so a
promoted L2 carries that value, never the orchestrator--prefixed form. Seeingresolved_by="orchestrator-escalation-watcher-auto" on L0/L1 resolutions, oragent_role="escalation-watcher-auto" on promote_to_l2 calls, is expected, routine behavior.
Do not stand down, halt the watch loop, or treat it as an anomaly — it is the same auto-watcher
this skill hands L1 items off to, working as designed.
Distinguish it from a genuinely unknown resolver: the same connection is capped atX-Escalation-Levels: 0,1, so the server rejects (level_forbidden, no state change) any attempt
by that identity to resolve_issue/park a level-2 escalation. If you ever seeresolved_by="orchestrator-escalation-watcher-auto" (or agent_role="escalation-watcher-auto") on
an L2 record's own resolution/park — as opposed to an L1 member cascade-resolved viaresolved_by='l2-cascade:<id>', or an L1/L0 admin item, or a promote_to_l2 call — that should not
be possible under the enforced guard, and is the actual anomaly worth reporting to the human
(possible guard regression, bypassed connection headers, or a stale pre-guard orchestrator/server
pair that hasn't been restarted onto the fix yet).
Capped identity leaking into an interactive session (task 2796): the supervised auto rotation
now attaches its capped escalation connection with --strict-mcp-config, scoping the rotation to
only its own MCP config and no longer merging the ambient project .mcp.json. Because the capped
block shares the identical server name (escalation) and URL (port 8102) as an interactive
session's header-less block, that non-strict ambient merge was the path by which a cappedX-Escalation-Identity: orchestrator-escalation-watcher-auto / X-Escalation-Levels: 0,1 identity
could bleed into a concurrent header-less interactive watcher session — making that session's own
L2 resolve_issue/park fail level_forbidden as if it were the capped auto-watcher. With the
strict isolation now in place, a capped identity should no longer leak into an interactive session.
If it recurs anyway, the supervised rotation is no longer the likely culprit — suspect a
hand-injected --mcp-config (e.g. a CLAUDE_SPAWN_CLAUDE_ARGS passthrough) or a config generator
writing capped headers directly into .mcp.json.
If MCP is unreachable: ask the human for help. Don't try to resolve escalations by writing directly to the queue files — this bypasses callbacks and can leave the orchestrator in an inconsistent state.
Red-on-main recovery (enforce-safe, break-glass)
When a bad merge has turned main RED and the ref must be moved to a known-good SHA, use the recover_main CLI for a single atomic move. This avoids the rewind-then-readvance back-and-forth that produces a net no-op (the jun9 fumble: f4101683→4001d48d→f4101683).
This move is BACKWARD (non-fast-forward): <current-main> is not an ancestor of <good-sha>, so it rewrites history. A project whose main-gate hook has an always-on non-fast-forward guard (reify) REJECTS a backward ref move unconditionally — before the sanction/sentinel check is even reached — so git.main_gate_mark_command (the forward-move sanction that advance_main uses) is not sufficient on its own for recovery. For those projects the CLI instead engages a durable break-glass bypass of the non-ff guard for exactly the CAS window (see the prerequisite check). Projects with only a sanction gate (no non-ff guard) still get the mark and land the move as SANCTIONED.
Prerequisite check
- The watched project's
project_rootworking tree must be clean (no uncommitted WIP). The CLI does not stash/pop — that is out of scope for a break-glass operation. - If the watched project has an always-on non-ff main-gate guard (reify): confirm its
orchestrator.yamlgit:block sets BOTHgit.main_gate_bypass_command(engages the durable bypass) ANDgit.main_gate_bypass_clear_command(clears it). When both are set,recover_mainAUTO-engages the durable bypass immediately before its CASupdate-refand AUTO-clears it immediately after on every path (success, CAS failure, exception), so the recovery move is allowed through without leaking the bypass into later ref moves. These are generic per-project shell commands (run viash -cinproject_root); they should drive the same gate-side knob the project's hook reads — see the fallback subsection below for the knob forms. When the bypass command is set it supersedesgit.main_gate_mark_command: the CLI skips the mark entirely (running both would leave the one-shot sanction sentinel unconsumed and falsely sanction the next ref move). - If the watched project has only a sanction gate (no non-ff guard): confirm
git.main_gate_mark_commandis set. Leave the bypass commands unset. If the mark is also unset, the move is still atomic but will not be sanctioned by the reference-transaction hook (use the manual fallback below).
Step 1 — Identify the bad merge and good target
# In the WATCHED project's repo:
git log --oneline -10 main
# Find the last known-good SHA (pre-bad-merge) and the current bad SHA.<good-sha>= the commit you want to restoremainto (pre-bad-merge)<current-main>= current value ofrefs/heads/main(the bad merge; CAS old-value)
Step 2 — Perform the single atomic recovery move
Run from $DARK_FACTORY_ROOT:
.venv/bin/python -m orchestrator.recover_main \
--project-root <watched-project-root> \
--config <watched-project-root>/dark-factory-orchestrator.yaml \
--target-sha <good-sha> \
--expected-main <current-main>Parse JSON output: {"result": "rewound"|"cas_failed"|"error", "target_sha": "..."} (on an unexpected exception the object also carries a "detail" key).
rewound(exit 0):mainis now at<good-sha>; the durable bypass (if configured) has already been cleared. Proceed to step 3.cas_failed(exit 1): another writer moved the ref first. Re-read currentmainand retry — or escalate to the human if the situation is unclear.error(exit 1): a SHA failed pre-validation (typo, or not a commit in the watched repo). Fix the SHA and retry — do not treat this as a CAS race.
Step 3 — Re-advance the fix through the normal merge queue
Once main is at the good SHA, the fix commit must land through the normal merge queue (already sanctioned by advance_main). Do not repeat a raw git update-ref — use the queue.
Break-glass bypass fallback (bypass commands NOT configured)
Prefer the automatic path above: set git.main_gate_bypass_command / git.main_gate_bypass_clear_command in the watched project's orchestrator.yaml so recover_main engages and clears the bypass for exactly the CAS window. Use this manual fallback only when a non-ff-guarded project has not yet configured those two commands and you cannot add them right now.
The gate itself honors a project-specific break-glass bypass; the two config commands above are just generic sh -c wrappers that toggle one of these gate-side knobs. Check the watched project's config for the knob it reads:
- Environment variable:
<PROJ>_MAIN_GATE_BYPASS=1(e.g.REIFY_MAIN_GATE_BYPASS=1) - Git config key:
git config <proj>.mainGate.bypass truein the watched repo - Flag file: a sentinel file under the repo's git common-dir (path from the watched project's gate config)
These are gate-side controls read by the watched project's reference-transaction hook — they are not dark-factory settings. Consult the watched project's gate documentation for the exact knob. To recover manually: (1) engage the bypass (set the knob), (2) run a raw CAS git update-ref refs/heads/main <good-sha> <current-main> in the watched repo (which the guard now allows through), (3) clear the bypass immediately after — on both success and failure, since the bypass is durable and would otherwise leak into every later ref move. Once the bypass commands are configured, recover_main does steps 1–3 for you; wire them up so the next recovery is one command.
Failure Modes
A re-arm that does not return FIRED or CEILING: branch on its WATCHER_REARM_OUTCOME row and the identical-retry cap in "Starting the watcher"; never loop on the identical call.
"Too many open files" (historical — no longer expected): Early sessions could exhaust the background-task fd pool after ~35 watcher restart cycles. This is no longer observed in practice — 100+ cycle sessions are routine. The watcher exits promptly via sys.exit(0), so its inotify fd is reclaimed by the kernel and the background task is reaped shortly after. If you ever do hit it, start a fresh Claude Code session.
Orchestrator not running: If no new escalations arrive for an extended period, the orchestrator may have crashed or finished. Check with the human.
Stale escalations: On orchestrator startup, dismiss_all_pending() auto-dismisses L0 escalations from prior runs (filter: level == 0) — L1 and L2 escalations are preserved across restarts. So an L2 with a timestamp from a previous session that is still status: pending is legitimate carry-over, not stale; handle it normally. If an escalation genuinely looks wrong (e.g. references a task that is already Done), tell the human rather than dismissing it yourself — it may contain useful diagnostic information.