"Watch and close fused-memory's RECONCILIATION escalation queue (port 8103) in a long-running loop. This is the consumer for the recon queue — the integrity/operational findings the reconciliation harness files, NOT the orchestrator's task-pipeline escalations (those are escalation-watcher's job, ports 8100/8102). Use when the user says 'watch recon escalations', 'monitor the fused-memory escalation queue', 'babysit reconciliation', mentions the 8103 queue, recon_integrity_issue / recon_failure / recon_stale_run findings piling up, or wants the reconciliation queue triaged and closed. This is a continuous loop skill; the watcher is the SOLE closer of the recon queue (recon never resolves its own findings). This is NOT for orchestrator task escalations, blocked tasks, merge failures, or worktrees."
Install
npx skillscat add leodearden/dark-factory/recon-escalation-watcher Install via the SkillsCat registry.
Recon Escalation Watcher
You are running a long-running watch loop over fused-memory's reconciliation
escalation queue (port 8103, dir <project_root>/data/reconciliation/escalations/).
Your job is to triage and close the integrity and operational findings the
reconciliation harness files, keeping the queue a small, meaningful signal.
This is a sibling of escalation-watcher, not the same skill. It shares the
MCP poll-loop scaffolding but the subject and semantics are different:
| escalation-watcher | recon-escalation-watcher (this) | |
|---|---|---|
| Queue | orchestrator (8100/8102) | reconciliation (8103) |
| Subject | task-pipeline blockers | memory/task integrity & recon ops |
| Tiering | L0→L1→L2 ladder | none — recon files flat, no levels |
| On resolve | resumes/abandons an agent in a worktree | just marks the finding handled — no agent, no worktree, no resume |
| Closer | steward / auto-watcher / human | this watcher is the SOLE closer (recon never resolves its own) |
There is no merge queue, no worktree, no steward, no L0/L1/L2 here. You never
spawn /unblock. Most findings carry a synthetic recon-<runid> task_id that
identifies the reconciliation run, not a real Taskmaster task.
Prerequisites
Verify these before starting. If anything is missing, ask the user — don't guess.
DARK_FACTORY_ROOT— path to the dark-factory repo (default/home/leo/src/dark-factory). Theescalationpackage (watcher) lives here.- This session's
escalationMCP must point at 8103, andfused-memoryat- The stock repo
.mcp.jsonpointsescalationat 8102 — wrong queue.
Launch via the recon-watch config (see "Launching" below); confirm the
binding from the store-identity block onget_pending_escalationsreadingkind=reconciliation, which holds even on an empty queue (how to read it:
the empty-result bullet in "Failure Modes" below).
- The stock repo
- The 8103 server is up — it runs inside
fused-memory.service. Ifget_pending_escalationserrors, the service is down → tell the user
(priority 1, system stability); do not try to start it yourself.
Launching (run a SEPARATE Claude session pointed at 8103)
MCP connections are per-process, so a dedicated session leaves any 8102
escalation-watcher session untouched. Use the launcher config that names BOTH
servers — the watcher needs fused-memory (8002) for its fix/file actions andescalation (8103) for the queue:
claude --strict-mcp-config --mcp-config "$DARK_FACTORY_ROOT/recon-watch/mcp.json" \
"/recon-escalation-watcher"Default lane: tmux. This is a long-running loop session (PRDplans/fleet-cockpit-prd.md §3 fork 1) — run it in the crash-survivable, reattachable tmux lane
by default: wrap the invocation above via skills/spawn/spawn-claude.sh withCLAUDE_SPAWN_BACKEND=tmux (see that script's header) so the session gets a display.kind=tmux
session-registry record and a tmux attach-reattachable window. A killed watcher is reattachable
with tmux attach, and its record persists across the crash. Interactive one-offs are unaffected
— this watcher never spawns interactive sessions of its own (see the table above: no /unblock,
no worktree).
recon-watch/mcp.json (created by the setup; both servers required):
{
"mcpServers": {
"escalation": { "type": "http", "url": "http://127.0.0.1:8103/mcp" },
"fused-memory": { "type": "http", "url": "http://127.0.0.1:8002/mcp" }
}
}Claiming the Recon Watcher Lease (single-owner-per-role)
Before entering the Main Loop for the first time, claim the recon-watcher-<project> lease
(Attention Rail T7, orchestrator/src/orchestrator/session_registry.py) — a deterministic
single-owner-per-role check that replaces any pgrep/ps-tree archaeology for a duplicate recon
watcher. Run once, at session startup, not per cycle:
# Reap anything stale first so a genuinely-dead prior holder never blocks this claim.
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-reap
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-claim \
--name recon-watcher-<project> --policy stand-downWhy this watcher takes a lease (task 3994). recon-watch/run.sh execs a hand-launchedclaude session with no supervisor and nothing preventing a second invocation, and this skill is
the SOLE closer of the 8103 queue (recon never resolves its own findings) — so two of these
running at once is actively harmful: both would triage and close the same findings. That is the
interactive single-owner-per-session shape, exactly like escalation-watcher. It is unlikeescalation-watcher-auto, whose no-lease status is justified (seeskills/escalation-watcher/SKILL.md §"Claiming the Watcher Lease", INTERACTIVE-ONLY) by its being
a supervised, always-on rotation rather than a single-owner actor. The lease name is the one task
2289 specified and that build_lease_name('recon-watcher', '<project>') already produces.
Do not assemble the slug (or the pid) in shell — the CLI owns both. --slug is optional and
defaults to <--name>-$CLAUDE_PID (or <--name>-$CLAUDE_PID-$CLAUDE_CODE_SESSION_ID under aclaude agents host); --pid is optional and resolves the same way. Pass either only
as a deliberate operator override. Both are derived rather than documented because the token is
load-bearing on every heartbeat and on the final release, and must not depend on this document
getting one shell token right (tasks 3994, 4248). It has to be re-derivable, not carried: each
Bash tool call is a fresh /bin/bash -c, so a SLUG=$(...) captured here would be gone by the next
call — hence the CLI re-derives it on every verb.
Never $$, and never $PPID. Inside a Claude Code Bash tool call $$ is the transient/bin/bash -c wrapper, dead the instant the call returns, which makes the liveness guard inert.$PPID is not stable across tool calls either (measured: 1430433, then 1471645 on the next call,
the first already dead), so a slug built on it would not match your own later heartbeat. The
long-lived claude process is $CLAUDE_PID — verify with ps -o comm= -p "$CLAUDE_PID", which
prints claude.
Under claude agents, $CLAUDE_PID is not yours alone. ps -o comm= prints claude for aclaude agents host too, so it cannot tell you: every session one claude agents process hosts
shares that host's pid, which outlives each of them. The CLI detects a host (its argv isclaude agents ...), records pid 0 (heartbeat-only staleness, logged as DEGRADED) and derives
the slug as <--name>-$CLAUDE_PID-$CLAUDE_CODE_SESSION_ID; without a usable$CLAUDE_CODE_SESSION_ID the lease verbs exit 2 (pass --slug). So a hosted holder readsholder_liveness=orphaned / holder_pid=0 even while alive: its fresh heartbeat still forces
stand-down, and once its heartbeat is past the TTL the reaper reclaims it. The per-cyclelease-heartbeat is the only liveness signal a hosted holder has
(orchestrator/src/orchestrator/session_registry.py::resolve_session_identity).
One-time rollout note (task 6916). A lease a hosted session claimed before this change carries
the bare <--name>-$CLAUDE_PID slug, so that session's re-derived heartbeat and release getresult=refused (naming the holder's pid alive) — nothing is damaged. The lease becomes
reclaimable once its heartbeat is past the 2h TTL; to skip the wait, release it with its original--slug (lease-show prints it as holder_slug) and claim again.
If $CLAUDE_PID is unresolvable and --slug is omitted, the lease verbs exit 2 naming both,
rather than silently drifting to a slug your own later heartbeat would fail to match; the CLI will
not invent one, because a synthesized token would be identical for every degraded session and let
each act on the others' leases. Pass --slug <stable-token> to proceed, re-using it on every later
lease verb — --pid does not substitute (it is the lease body's liveness pid, not your identity,
and the heartbeat/release verbs have no --pid; only --slug is honoured by all three).
On that path the CLI records pid 0, a never-alive sentinel that degrades the lease to
heartbeat-only staleness (loudly logged) rather than recording an unrelated durable pid that would
make the lease unreapable forever.
Parse the printed lines: decision=<acquired|stand-down|proceed>, a human-readable message, thenholder_liveness=<none|held|orphaned>, then slug=<the slug this claim used>, thenholder_record=<unlinked|absent|unreadable|active|exited>.
slug=reports the identity the CLI derived for you (never the holder's). Quote it to the
user when useful, and compare it againstlease-show'sholder_slugif a later heartbeat returnsresult=refused. It is a diagnostic only — do not carry it into the next call; the CLI
re-derives it.holder_record=is a second, INDEPENDENT axis besideholder_liveness=: the state of the
holder's own session-registry record (yours, on an acquired claim), looked up through therecord_slugthe CLI stamps into the lease body from the claiming pid. Read the two together:orphaned+exited/absentare two agreeing signals that the holder is gone;orphaned+unlinkedis the pid signal alone (a lease claimed beforerecord_slugexisted, or whose
claimant's record was unresolvable);held+exitedsuggests the recorded pid was reused.unlinkedmeans nothing was looked up — it is not evidence the holder is alive — andactiveonly means the record is not terminal (for a hand-launched holder it tracks the
terminal, not theclaudeprocess), so it never overridesorphaned. A faultedproceedprints
noholder_record=line.decision=acquiredordecision=proceed: continue into the Main Loop.proceedis the
fail-open outcome — a lease-substrate fault is logged loudly and reported asproceed, never a
falsestand-down, so a lease fault can never block the only consumer of this queue. An acquired
claim printsholder_liveness=none(no contending holder — it is yours); a faultedproceed
prints noholder_livenessline at all, because nothing is known about a holder.decision=stand-down+holder_liveness=held: another recon watcher is live. Print the
message verbatim and exit immediately — do not drain, do not start the watcher.decision=stand-down+holder_liveness=orphaned: the pid recorded in the lease body is not
running — that is the whole signal, a pid probe and nothing more — but its heartbeat is still
within TTL so the lease is not yet reclaimable. Do not exit silently — this skill is the
queue's only closer, so a silent
stand-down for a holder that no longer exists means nothing closes 8103 at all. Inspect, file a
DecisionRecord naming the orphan (thewrite-decisionverb this skill already uses for parked
decisions), print it, then exit:python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-show \ --name recon-watcher-<project> # holder_slug / holder_pid / heartbeat_age_secs / reclaimable / holder_record python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py write-decision \ --id recon-watcher-lease-orphan-<project> --project <project> \ --text "recon-watcher-<project> lease held by orphaned holder <holder_slug> (pid <holder_pid> not running, heartbeat <n>s ago); the 8103 queue has no live closer until it is reclaimed" \ --escalations-dir $DARK_FACTORY_ROOT/data/reconciliation/escalations \ --session-id "recon-watcher-<project>-${CLAUDE_PID:-unknown}"--session-idis a best-effort provenance label with no ownership semantics — unlike the lease
slug, which the CLI owns (above). It is deliberately not fail-loud: a watcher that cannot file a
DecisionRecord is strictly worse than one that files it with a degraded label.Never force-release on this evidence alone:
holder_livenessis a single-signal pid probe, and a
dead-looking holder that is merely quiet is the duplicate-spawn incident. Aholder_record=exitedorabsentbeside it is corroboration, not a licence: never
auto-steal. (Task 3994 withdrew an earlier record cross-check because a lease slug is a
claimant-chosen ownership token, not a record key;holder_record=instead reads the separaterecord_slugthe lease body now carries.) Reclaiming is the human's call, or the reaper's once
the TTL expires.
Reading the contention message. It names two INDEPENDENT axes — whether the holder's pid is
running, and how fresh its heartbeat is:
lease held by recon-watcher-df-1348600 (pid 1348600 alive, heartbeat 42s ago) — standing down
lease held by recon-watcher-df-1348600 (pid 1348600 is not running, but its heartbeat is FRESH —
42s ago; the lease is still held and is NOT reclaimable for another 7158s) — standing downA fresh heartbeat means you stand down even when the pid reads as not running: staleness needs
BOTH a dead pid AND a heartbeat past the TTL.
A leftover loop.sh never touches this lease (task 5885). A scripts/watcher-rearm.sh tree
under a ~/.claude/recon-watch/loop.sh ancestor never claims, heartbeats or releasesrecon-watcher-<project>, so it changes no lease verdict; see "One watcher: the session's own"
under "Starting the watcher".
Heartbeat + release. Touch the lease every Main Loop cycle (see "Starting the watcher" below),
and release it when the session ends. Both verbs act only for the holder — a mismatched slug is
refused, so no other session can evict your lease or keep a dead one alive. You do not pass the slug:
the CLI derives the same one it derived at claim time, which is what makes the two match:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-release \
--name recon-watcher-<project>Both print result=<applied|forced|absent|refused|faulted> first: applied = done; absent = no
such lease — plain idempotence on a release, but a real condition on a heartbeat (see "Lease
heartbeat (each cycle)" below); refused = you are not the holder and nothing happened — inspect withlease-show --name recon-watcher-<project> rather than reflexively re-running with --force
(operator recovery only, logged loudly naming both parties); faulted = a substrate error, logged
and swallowed. Use lease-show, never cat: the body carries an immutable start_ts and cannot
show freshness, because the heartbeat is the file's mtime.
The Main Loop
0. Reap orphans: `derive_orphaned_recon_escalations.py --apply` (see "Reaping
orphans" below) — BEFORE every drain, so the drain never sees them
1. Drain all pending recon escalations
2. Start the watcher: `scripts/watcher-rearm.sh` (background task, recon queue
dir, NO --level, --timeout 3600)
3. Wait for it to exit. THREE exit paths — tell them apart, don't assume a fire:
FIRED (exit 0, escalation JSON on stdout) → go to 4
CEILING (exit 124, NO escalation payload) → the bounded slice just
expired; SKIP step 4 and fall through to the drain (5)
KILLED / ERROR (any OTHER rc: 137|143|144, or 2 for a usage/env
failure) → STOP and report to the human; do NOT re-arm
4. Read the escalation from watcher output; fetch full detail via MCP
5. Reap orphans again (step 0), then drain any other pending escalations
6. Handle each
7. Run `reap-decisions` to close any parked DecisionRecord whose escalation has since resolved
(see "Filing Parked Decisions to the Cockpit Registry" below) — once per cycle
8. GET `/api/v2/dashboard/invariants` and digest every root's record (see "Reading the
invariants gauge" below) — once per cycle. Each `FINDING` line it prints (a chain gone
`overdue` or `broken` among them), and each `paused` line reading `pinned_by=[]`, is a
finding: triage it with The Action Set's judgement before re-arming (9)
9. Go to 2Distinguishing the exits at step 3. The wrapper emits a machine-readableWATCHER_REARM_OUTCOME: <FIRED|CEILING|KILLED|ERROR> exit=<rc> line to
stderr on every run — use it rather than inventing your own vocabulary. ACEILING wake carries no escalation payload; reading it as a fired-but-empty
escalation is the mistake this step exists to prevent. It is not an error and
needs no report — just re-drain and re-arm.
KILLED (137/143/144) and ERROR (any other rc) are not re-arm paths.
Report to the human and stop the loop: something killed the watcher or the
launch itself failed, and neither self-heals by trying again. The one you are
most likely to hit is exit 2 — a usage/env guard, most often an unsetDARK_FACTORY_ROOT (thrice-observed startup failure,plans/confusion-census-2026-07-24.md §1.5). It returns instantly, before
the watcher is ever invoked, so falling through to "re-drain and re-arm" turns
a mis-configured launch into a hot busy-loop of failed invocations — and this
queue's sole closer would never actually be watching. Note that exit 2
prints a plain stderr diagnostic and no WATCHER_REARM_OUTCOME marker
(scripts/watcher-rearm.sh header, lines 59-63), so check the exit code
first; do not infer "no marker" means the run is still in flight.
Also: exit 0 with EMPTY stdout is a non-fire, never proof an escalation was
printed. escalation.watcher installs a SIGTERM handler that converts a
SIGTERM into a clean sys.exit(0), so a killed watcher surfaces asFIRED exit=0 with nothing on stdout (scripts/watcher-rearm.sh header,
lines 65-72). Check for non-empty stdout before treating exit 0 as a fire —
this queue's sole closer must not silently skip a cycle on a caught signal.
Reaping orphans (standing authorization — Leo, esc-5793-2, 2026-09-23)
You are authorized and required to run the orphan reaper with --apply
on every pass through steps 0 and 5 — at loop start and after every watcher
wake, FIRED or CEILING. The --timeout 3600 ceiling therefore bounds how long
an orphan waits to about an hour. No per-record approval, no cockpit record, no
park. This IS the closing step that the Stage-1 orphaned_recon_escalation
flag asks for (task 3052). Nothing else closes these records: no timer, no cron,
and the harness never resolves this queue (A7b).
cd $DARK_FACTORY_ROOT && set -a && \
eval "$(systemctl --user show fused-memory -p Environment --value | tr ' ' '\n' | grep -E '^(CONFIG_PATH|DASHBOARD_KNOWN_PROJECT_ROOTS)=')" && \
set +a && uv run --project fused-memory \
python fused-memory/scripts/derive_orphaned_recon_escalations.py --applyWithout the service unit's DASHBOARD_KNOWN_PROJECT_ROOTS, the reaper exits
4 and skips every project except dark_factory. Always gate on the exit code (table in the
playbook row below): 0 → report reaped and continue; 3/4 → a partial
scan, so continue the loop but tell the human once; 1 → wrong cwd, fix and
re-run. The reaper closes only records whose subject is done/cancelled or
has no task row. It leaves live, ambiguous and unresolvable records
alone, so running it every cycle cannot churn.
What does NOT veto a reap. Some things look like "leave this alone" but
aren't:
- Text on the subject task saying its paired recon record "self-clears" or
"must not be resolved by hand". Nothing self-clears; this step is how it
clears. - A ruling recorded on the orchestrator queue under the same
esc-<id>-<n>.
The two queues mint ids independently, so the same id names two different
records. - A Mem0
stage1_flag_suppressionrecord.filter_suppressedreads onlyrecon_ledger, so such a record suppresses nothing.
Task 5793 was filed on all three of these misreadings and was cancelled.
Draining
mcp__escalation__get_pending_escalations() # NO level arg — recon is flatPriority order when several are pending: blocking severity first; then within
a severity, highest dedupe_count first (recurrence = persistence = signal).
Starting the watcher
cd $DARK_FACTORY_ROOT && scripts/watcher-rearm.sh \
--queue-dir $DARK_FACTORY_ROOT/data/reconciliation/escalations --timeout 3600Lease heartbeat (each cycle): each time you (re)start this watcher subprocess, also touch therecon-watcher-<project> lease claimed at session startup, so a second session's lease-claim
observes this one as alive and stands down. No other process heartbeats this lease, so this call is
its only keep-alive:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py lease-heartbeat \
--name recon-watcher-<project>result=refused means your heartbeat did not land (you are not the holder) — investigate withlease-show, comparing its holder_slug against the slug= line your lease-claim printed; don't
re-run with --force.
result=absent on a heartbeat means your lease is GONE, not that the call was a harmless no-op:
it was reaped after a TTL lapse or force-released by an operator, and you are now draining the 8103
queue un-leased, so a duplicate can claim recon-watcher-<project> freely. Re-run thelease-claim from "Claiming the Recon Watcher Lease" and obey its decision — acquired = carry on,stand-down = a live duplicate holds it now, so print the message and exit rather than continuing as
a second closer of the same queue.
Run as a background task (run_in_background). scripts/watcher-rearm.sh is
the canonical bounded-wait + re-arm wrapper around escalation.watcher, shared
with the sibling escalation-watcher skill; task 3530 made its --level
optional precisely so this flat queue could reuse it. The watcher uses inotify
and exits after the first new escalation file, printing its JSON to stdout — so
do not pipe 2>&1 when you parse that stdout as the escalation JSON, or the
wrapper's stderr WATCHER_REARM_OUTCOME line lands in your parse and corrupts it.
The shared wrapper contract is documented once — read it there, not here.skills/escalation-watcher/SKILL.md §"Starting the watcher" covers everything
identical for every caller: the preserved exit codes and theWATCHER_REARM_OUTCOME marker, the Bash-tool timeout sizing rules, and the
wrapper-owned exclude-file mechanics (one esc-id per line, bare or.json-suffixed; re-read every poll, so an append needs no restart; it also
suppresses event-loop wakes from dedupe rewrites; the repeatable --exclude-id
does the same for a single id but does not scale). scripts/watcher-rearm.sh's
own header is the authoritative copy. Only this queue's deltas are below.
Delta 1 — omit --level. Recon escalations have no level field worth
filtering, and omitting the flag selects the watcher's match-all mode. Because
an omitted flag is otherwise invisible, the wrapper declares the level it
resolved on stderr at the top of every run — level=<all> here, level=2 for
the sibling. Read it back from the output rather than assuming.
Delta 2 — the exclude path has no level suffix. With no --level the
wrapper-owned default is <queue-dir>/.watcher-rearm-exclude: the levelled
sibling's -l<level> suffix dropped, not a separate recon-only name. You
never name this path yourself — the wrapper creates it if absent, always wires
it into the watcher invocation, and prints the resolved path to stderr on every
run (--check dry-runs that resolution without starting the watcher). An
explicit --exclude-file still overrides it. Being a dotfile keeps it out of
the watcher's own esc-*.json glob (_initial_scan, watcher.py:85; the same
pattern gates EscalationQueue's reads,escalation/src/escalation/queue.py:431 and :509) — and, a constraint unique
to this directory, it must not end in .json, because the dashboard's
read-only recon subsection globs the broader *.json (not esc-*.json) over
this same path (dashboard/src/dashboard/data/escalations.py:89); a.json-suffixed name would surface there as a phantom escalation record.
Delta 3 — as sole closer, you depend on the exclude file's failure mode.
The initial scan emits the oldest matching pending escalation and exits
immediately, so any item you deliberately left pending (Priority 3 "leave
pending, tell the human") re-fires on every watcher start and degenerates into
a busy-loop. Append with echo <esc-id> >> <path>: _read_exclude_file's
docstring (watcher.py:128-132) blesses a single short-line append as atomic
on POSIX, and a torn multi-write append is self-healing on the next poll.current_excludes() (watcher.py:224-225) re-reads the file on the initial
scan (watcher.py:246) and on every event-loop poll (watcher.py:269), so the
parked set grows mid-run with no restart and nothing beyond the ids you listed
is ever masked. The reader is fail-open — a missing or unreadable file
yields an empty set, retried next poll (watcher.py:134-140) — so a lost
exclude file degrades to re-firing on parked items: noisy, never silent, which
is the only acceptable failure direction for this queue's sole closer (contrast--baseline below, which fails silent). Blank lines and # comments are
skipped (watcher.py:143-146), so the file can be annotated.
Slice length (recon delta on the shared timeout rules). In the background
the wrapper is exempt from the Bash tool's foreground timeouts, so this skill
uses the long canonical slice --timeout 3600: at most one heartbeat wake per
hour (CEILING) while a real escalation still fires instantly via inotify. Only
foreground calls (e.g. debugging) are governed by the harness timeouts —
there, size the Bash timeout to at least (--timeout + 60s) × 1000 ms, or cap
the slice at --timeout 540 on a machine without a raised BASH_MAX_TIMEOUT_MS;
the sibling skill's §"Starting the watcher" has the full rules.
--baseline is NOT safe for this loop — do not reach for it here,
even though escalation.watcher accepts it as a flag. _snapshot_pending_ids()
(watcher.py:151-158) freezes the pending-id set exactly once, at launch,
strictly before add_watch arms the inotify watch (watcher.py:238 runs
before watcher.py:243) — and unlike --exclude-file, that snapshot is
never re-read afterward (current_excludes(), watcher.py:225; the code's own comment atwatcher.py:227-237 reasons only about the narrower snapshot→add_watch
race, not this one). This loop's handling phase is long — draining,
handling dozens of parked records, and filing a cockpit DecisionRecord for
each takes real wall-clock time between one watcher restart and the next — so
any escalation recon files during that window is already on disk by the
time the following restart takes its --baseline snapshot. That new
escalation gets folded straight into the snapshot and is excluded from both
the initial scan (watcher.py:246) and the event loop (watcher.py:269-286)
for that run's entire lifetime, despite never having been drained or
triaged by anyone. It is worse than a one-cycle miss: the snapshot is
retaken at every restart while the item is still pending, so it stays
masked cycle after cycle until some unrelated, newer escalation happens to
fire the watcher and the next drain re-finds it. Bounding the wait does
not fix this. This skill's invocation now carries --timeout 3600, so a
quiet queue does force a restart-and-redrain at least hourly — but the--baseline snapshot is retaken at every one of those restarts while
the item is still pending, so each redrain re-masks it. The bound caps how
long any individual masking window lasts; it does not stop the mask from
recurring. Net effect: with --exclude-id / --exclude-file, an escalation filed during
the handling window is not on the exclude list, so it fires at the next
watcher start like any other pending item — normal, expected, and loud.
With --baseline that same new escalation is instead swallowed by the
launch snapshot into a silent, open-ended delay in this queue's sole
closer — open-ended in the item's own handling, not in the slice length,
which --timeout 3600 does bound.
With PARK now the default disposition for reconciliation_stale_gate_backlog /reconciliation_stale_human_operator, the parked set is structurally large
— these two categories make up the entire pending queue today (dozens of
records; see the dated census in the playbook row below for the current
count) — so hand-listing one --exclude-id per record does not scale; use--exclude-file for it, not --baseline. The siblingskills/escalation-watcher/SKILL.md documents --baseline as an available
flag too, but that is not a counterexample: its Main Loop restarts the
watcher first and drains only after confirming it's up, and its last step isGo to 1, so every restart there is immediately followed by a fresh,
authoritative drain — that skill says as much at its lines 93-94, that the
fired escalation "is just the wake ... the drain re-finds it (still
pending) plus anything new." That drain-after-every-restart ordering — not
its --timeout 3600, which this skill now shares — is what makes--baseline survivable there and not here. Tellingly, that skill's own
re-arming guidance for its parked set does not reach for --baseline
either — it routes through the same wrapper-owned --exclude-file
(skills/escalation-watcher/SKILL.md:165-175, scripts/watcher-rearm.sh,
default path <queue-dir>/.watcher-rearm-exclude-l2 there,<queue-dir>/.watcher-rearm-exclude here). --exclude-file — not--baseline — is what is actually consistent across both watcher skills.
One watcher: the session's own (task 5885). This skill runs exactly one watcher: the Main
Loop's step-2 background task. The queue directory is the durable record and the drain is
authoritative; the watcher is only a wake signal, so a detached re-arm loop whose fire journal no
session reads adds nothing. The hand-written, never-tracked ~/.claude/recon-watch/loop.sh was
retired for that reason — do not recreate one. A watcher the harness reaps needs no second watcher
to cover it, because step 3 sees the reap: as KILLED, or, when only the python child caught a
SIGTERM, as the exit-0-with-empty-stdout non-fire (both under
"Distinguishing the exits at step 3"). If a process probe still shows a scripts/watcher-rearm.sh
on this queue under a loop.sh ancestor, it is that leftover. Its tree holds no claude process and closes nothing. Mention it to
the human once and leave it running, per the rule below.
Process safety: only stop watcher processes you started via background task
controls. Never pkill by pattern.
Reading the invariants gauge
The command, what counts as a finding (the chain rule included), change-only
triage, the census follow-up, the broken-chain journal probe, the D-4 pointer and
the filing store and keys live in
`skills/_shared/reading-the-invariants-gauge.md`.
Read them there. This skill's deltas:
- Scope. Omit the root argument, so every root in the roster is read:
fused-memory, like this queue, is fleet-wide. An escalation-watcher reads only
its own project. - Handling. At step 8, before re-arming: this loop already drains and handles
with the watcher down (steps 5 and 6). Apply The Action Set's judgement, but
there is NO escalation record, so never callresolve_issue. Note a
verify-fixed or accept-as-known finding in the accepted-as-known list kept in
top-level context ("Context Conservation"). file-a-real-task uses the shared
file's Case B keys in place of an escalation id. Tell the human about a finding
that needs one, and file no cockpit DecisionRecord:reap-decisionscloses a
decision only once its escalation is terminal, so one with no escalation behind
it would stay open forever ("Closing parked decisions on resolve"). Apaused
chain's pinning escalation is an orchestrator-queue L2 that the
escalation-watcher handles, not this queue's. - A failed read is reported to the human (Priority Hierarchy 1).
The Action Set
For each finding, decide among four closures. Read summary, detail (usually
JSON with description/affected_ids/actionable), and dedupe_count.
verify-fixed — Check current state via
mcp__fused-memory__search,get_entity,get_task. If the finding is already true-resolved (the edge
exists, the task is in the expected state, the contamination is gone), close
it:resolve_issue(..., action='resume', resolution="Verified fixed: <what you checked>").accept-as-known — The finding is non-actionable or an accepted state (e.g.
a deliberately-deferred task, a known intractable item, an auto-recovered
stale run). Dismiss it:resolve_issue(..., action='close_only', resolution="Accepted as known: <why>").file-a-real-task — The finding is genuinely actionable dev work. File it,
then resolve the escalation. Two-phase pattern:suggestion_hash = hashlib.sha256( (escalation['detail'] or escalation['summary'] or escalation['id']).encode() ).hexdigest()[:16] # Case A — escalation_id already in scope; see _shared/ticket-failure-handling.md sub = mcp__fused-memory__submit_task( project_root="<project_root>", title="<title>", description="<what + specifics>", priority="medium", metadata={"source": "recon-watcher", "escalation_id": escalation_id, "suggestion_hash": suggestion_hash, # (escalation_id, suggestion_hash) = R4 idempotency key "spawn_context": "steward-triage"}, ) res = mcp__fused-memory__resolve_ticket(ticket=sub["ticket"], project_root="<project_root>", timeout_seconds=<see _shared/ticket-failure-handling.md>) if res["status"] in ("created", "combined"): task_id = res.get("task_id") # `combined` can also omit task_id — see below if task_id: resolve_issue(..., action='resume', resolution="Filed task <task_id>: <title>") else: # `combined` = folded into an existing task, a normal success not # an error — but an idempotency-hit combine can resolve with no # surviving task id. Record the reason instead of a fabricated id. resolve_issue(..., action='resume', resolution="Filed (combined): <res['reason']>") elif res["status"] == "refused": # Guaranteed (not just possible, as with `combined` above) to omit # task_id: a deterministic guard (cancelled-premise blocklist / recon # premise registry) rejected the candidate because its premise is # dead. This is a SUCCESS, not a failure: do NOT retry (a retry just # re-hits the same guard) and do NOT record a task id. resolve_issue(..., action='close_only', resolution="Ticket <res['reason']>") # res["reason"] already reads "refused (no task created): ..." elif res["status"] == "awaiting_curation": # Parked for curation: no task_id yet, NOT a failure. Do NOT re-submit. resolve_issue(..., action='resume', resolution="Queued for curation (ticket <sub['ticket']>)") elif res["status"] == "failed": # Retryable (`server_restart`, `timeout`): apply # _shared/ticket-failure-handling.md's retry policy (retry # `resolve_ticket` first under `timeout`; re-submit the pair once # under `server_restart`) within its 2-submit/2-resolve cap, then # re-enter this dispatch on the retry's outcome. Once the reason is # terminal — or that budget is exhausted — do NOT call resolve_issue: # leave the escalation pending. Record res["reason"], tell the human, # file a cockpit DecisionRecord via `write-decision` (Priority # Hierarchy item 3 below / C8), and append this esc-id to the # wrapper-owned exclude file (`<queue-dir>/.watcher-rearm-exclude`). # See _shared/ticket-failure-handling.md for the full retryable/ # terminal reason matrix and R4 idempotency guidance. else: # Unrecognized status (e.g. `cancelled`, written by cancel_ticket — # the ticket status column has no CHECK constraint, and tools.py's # docstring documents only the four statuses above). Same # follow-through as the `failed` arm above (tell the human, file a # DecisionRecord, append this esc-id to the exclude file; do NOT call # resolve_issue) — plus record the raw `res`, since the status itself # is the anomaly here.This recipe spells out every arm where a sibling can get away with less
because this watcher is the SOLE closer of the recon queue — the next
action after the status check ARCHIVES the finding, so a status-blind
"Filed task " both fabricates a completion claim and drops the finding
with no other consumer left to re-find it.fix-directly via fused-memory — For memory-integrity findings you can
safely repair yourself, use the fused-memory write tools, then resolve:mcp__fused-memory__update_edge— correct a stale/wrong Graphiti edge factmcp__fused-memory__delete_memory— remove a duplicate/incorrect memorymcp__fused-memory__merge_entities— consolidate duplicate entity nodesmcp__fused-memory__refresh_entity_summary— rebuild a stale summary
Thenresolve_issue(..., action='resume', resolution="Fixed directly: <tool + what changed>").
Caution: fixing directly mutates the knowledge graph. When the right
repair is ambiguous or wide-reaching, prefer file-a-real-task or ask the
human — quality over speed.
Resolution-text convention: action='resume' → status resolved (you took
action). action='close_only' → status dismissed (accepted-as-known, no action).
Both archive the record. Be specific in the note — it is the only audit trail.
Runtime-behaviour findings: a verify-fixed or accept-as-known close on a
finding about runtime behaviour needs its check run in that runtime's execution
context; a check from your own session that fails to reproduce it is a
non-reproduction, not a refutation. The resolution note names the context the
check ran in. Single normative statement of the rule:orchestrator/src/orchestrator/agents/premise_refutation_guidance.py::PREMISE_REFUTATION_GUIDANCE.
Per-Category Playbook
recon_integrity_issue(info) — memory/task consistency findings. Run the
action set: verify-fixed → fix-directly → file-a-real-task → accept-as-known.
Highdedupe_count= a persistent intractable item; if you've accepted it as
known before, accept-as-known again briefly. See the caveat below — these
re-fire every cycle until the recon-side gating fix lands.recon_stale_run(info) — "Run stale, recovered". The harness already
self-recovered. accept-as-known (dismiss) unless several cluster, which
signals a stuck recon loop → tell the human (and file a DecisionRecord, see
"Filing Parked Decisions to the Cockpit Registry" below).recon_failure/recon_backlog_overflow(blocking) — a reconciliation
run failed or the queue is overflowing. This is infrastructure: tell the
human with full detail, leave pending, do NOT attempt automated fixes. Also
file a DecisionRecord viawrite-decision.infra_issue(blocking) — DB/MCP/service problems. Priority 1 — system
stability: tell the human immediately, leave pending, do not auto-fix. Also
file a DecisionRecord viawrite-decision.risk_identified(info) — needs human judgment. Tell the human, track as
todo, continue. Also file a DecisionRecord viawrite-decision.dependency_discovered— if a real prerequisite task exists, note it; else
file-a-real-task. Then resolve.cleanup_needed— file-a-real-task (two-phase), then resolve.reconciliation_stale_gate_backlog(blocking) — a task has satblockedon a human-gated milestone gate for 48h+ (Stage 1's aging
detector,fused-memory/src/fused_memory/reconciliation/stage1_stall_detector.py).
Default: PARK. Leave it pending, file a cockpit DecisionRecord viawrite-decision(see "Filing Parked Decisions to the Cockpit Registry
(C8)" below), and append its esc-id to the wrapper-owned exclude file
(<queue-dir>/.watcher-rearm-exclude) so the watcher skips it — the
file is re-read every poll, so this takes effect without a restart —
not--exclude-id, which does not scale at this queue's size
(see "Re-arming over deliberately-pending items" above).
Also stamp the record itself — once, on first park; skip re-stamping on
repeat cycles.stamp_triageunconditionally overwritestriaged_atto
now on every call (escalation/src/escalation/queue.py:949-958, set at
line 977), and
the "Drain-side shortcut" note below already has you re-affirming PARK on
this same record every cycleget_pending_escalations()returns it — so
the same restraint applies to the MCP call itself: skip it whentriaged_atis already set and the record is unchanged since (updated_at
isNone, or not newer thantriaged_at— never orderNoneagainst a
timestamp string directly). Re-stamping on every pass would overwrite the
"first parked at" time each rotation and destroy the aging signal the
stamp exists to create. This reuses, verbatim, the sibling skill's
freshness rule —skills/escalation-watcher-auto/SKILL.md, "Triage-ack
freshness contract".On first park, use a predicate+probe note, not a bare conclusion — same
convention as that freshness contract, and for the same reason (a
conclusion-only note is untrusted prose the next rotation can't check;
that section documents real churn — esc-2584, two refuted cycles, five
separateresolve_issuecalls — caused by skipping this):mcp__escalation__stamp_triage( escalation_id, triage_note="PARK: task-650 status==blocked, gate_escalated_at=2026-06-20T..." " | probe: get_task 650 -> status=blocked | decision: <printed decision id>", )(registered on the 8103 server,
escalation/src/escalation/server.py:1120).
This is metadata-only:stamp_triagetouches none of the fieldshas_open_l1reads (status,level, and optionallycategory;escalation/src/escalation/queue.py:489-504), so a stamped record still
counts as an open L1 and the re-file rule stays suppressed. It turns the
record'striaged_atfrom "never looked at" into "deliberately parked" —
a durable rotation-to-rotation handoff note plus a machine-readable marker
in the record JSON, which is what that field should mean now that PARK is
a sanctioned default disposition rather than a gap (decided under task
3526; ratified specifically for the two aging categories below — do not
read it as a blanket rule for every other Priority-3 "leave pending" case
in this skill). Not an analytics dependency today: dashboardtriage_segments
(dashboard/src/dashboard/data/escalation_analytics.py:381-390) is fed
only fromconfig.escalations_dir/known_project_roots
(dashboard/src/dashboard/app.py's_analytics_project_dirs), never fromreconciliation_escalations_dir— this queue isn't in that pipeline at
all — and even if it were,triage_segmentssamples only terminalresolved/dismissedrecords, so a PARKed record, which stayspending
by this very playbook's design, could never contribute a sample either
way. Wiring the recon queue into dashboard analytics is unfiled follow-up
work, not something this stamp does on its own.
Why park, not resolve: recon files a fresh gate-backlog escalation for
a task only whenhas_open_l1(task_id, category='reconciliation_stale_gate_backlog')is false
(maybe_escalate_stalled_gate_backlog, same module, lines 483-492) — the
open pending record itself IS the dedup. Resolving it re-arms the filing
rule, and if the task is stillblockedwith ametadata.gate_escalated_at
stamp older than 48h, recon re-filesesc-<task>-N+1on the very next
Stage 1 cycle (extract_stalled_gate_backlog_task_ids, same module, lines
198-237). This is not theoretical: resolving-to-tidy causes measured
re-file churn, so do not resolve this category just to shrink the
queue. Observed indata/reconciliation/escalations/archive/(as of
2026-08-02):esc-650-1resolved 2026-07-25T12:58Z →esc-650-2filed
and resolved by 2026-07-25T17:03Z the same day (~4h round trip);esc-646-1dismissed 2026-08-01T14:10Z →esc-646-2filed and still
pending now;esc-3361-1resolved 2026-08-02T10:40Z →esc-3361-2filed
and still pending now. As of 2026-08-02 the pending queue held 32
records, 100% this category, with only one (esc-648-1) ever
triage-stamped; by 2026-09-02 the same census read 124 (see the dated
census in the terminal-subject branch below) — treat every count here as a
snapshot, not a fixture: it moves with every filing/resolve cycle, so
re-censusdata/reconciliation/escalations/yourself if the number matters
to your decision. Resolve only when the underlying task will genuinely
stop qualifying for re-selection — you completed the gate and can close
the task, or recon will reconcile the task to a terminal status next
cycle regardless.A terminal-or-missing subject is exactly that case, and cannot churn
(task 3052).extract_stalled_gate_backlog_task_idsselects only tasks
withstatus == 'blocked'(same module,if task.get('status') != 'blocked': continue), so a subject that isdone,cancelled, or absent
from its project's task store can never be re-selected and the filing rule
can never re-arm. The churn argument above does not apply to it. Resolve
such a record withresolution_class='moot-terminal-subject'(a member ofescalation/src/escalation/models.py::RESOLUTION_CLASSES, which validates
it at call time); do not park it. This is the one sanctioned exit from
the PARK default — the default itself is unchanged and stays correct for
every still-blockedsubject.How to find them, two ways. (i) Stage 1 now computes the set every full
cycle: read the report statsorphaned_recon_escalations_terminalandorphaned_recon_escalations_missing, and the emittedorphaned_recon_escalationflags, each of which names the escalation id,
the subject's project, and the observed subject status. The stats count the
WHOLE shared queue, but since task 5813 a cycle's flags name only the
orphans whose subject belongs to THAT cycle's project, so no single report's
flags list every orphan. The fleet-wide view is the reaper in (ii), which
the main loop already runs with--applyevery cycle. (ii) On demand,
re-derive it live. Run from the project root —--queue-dirdefaults to
the relative./data/reconciliation/escalations, so the cwd is what decides
which queue is read; from anywhere else the script refuses withTargetStoreMissingrather than manufacturing an empty queue and reporting a
false clean run (seefused-memory/src/fused_memory/utils/target_store_preflight.py). Pass an
absolute--queue-dirif you cannot control the cwd.cd /home/leo/src/dark-factory python fused-memory/scripts/derive_orphaned_recon_escalations.py # dry run (default) python fused-memory/scripts/derive_orphaned_recon_escalations.py --apply # closes themRead the exit code, not just the JSON.
0is a clean scan —reaped: 0
then genuinely means nothing to do.1is the refusal above (wrong--queue-dir/cwd; the reason is one line on stderr).3means at least one
project's task store could not be READ, so its records were classified as
nothing at all — re-run once the store is readable, since the set is
re-derived every run.4means a record could not be scoped to a known
project (the registry gap below). A3or a4is a partial scan wearing a
clean-looking report: do not treat that run'sreapedcount as the
whole story.
The main loop now runs (ii) with--applyon every cycle, under the
standing authorization in "Reaping orphans" above. Run it by hand only
when the loop isn't running.
Detection is recon-side only; you are the sole closer — no
reconciliation stage ever callsqueue.resolve()on this queue (the A7b
invariant above_RECON_DEDUP_CONFIGinfused-memory/src/fused_memory/reconciliation/harness.py).The scale, as a dated snapshot — re-derive, never drain a roster. As of
2026-09-02 the pending queue held 124 records of this category
(100% of pendingreconciliation_stale_*), alllevel=1, across seven
projects: dark_factory 56, reify 48, autopilot_video 7, solar_challenge 5,
know_live 3, solar_challenge_platform 3, pump_web_ui 2. Trend: 108 measured
2026-08-30 (of which 104 were orphaned under this rule), 117 at task-3052
planning, 124 on 2026-09-02. The queue accumulates continuously, so a list
of ids goes stale between the moment it is written and the moment you act
on it — always re-derive live; never work a copied roster.The
ambiguousbucket is NOT an orphan either. Task ids are per-tag, not
global (PRIMARY KEY (tag, id)infused-memory/src/fused_memory/backends/sqlite_task_backend.py), so the same
id in two tags is normal — and a record names only itsproject_id, never a
tag. When a subject id carries differing statuses across tags the subject
cannot be identified, so the record is countedorphaned_recon_escalations_ambiguous, never flagged, and never reaped by--apply. Check by hand which tag the record's subject actually lives in
before touching it.The
unresolvablebucket is NOT an orphan. A record whoseproject_id:
detail line is unparseable, or whose project is absent from the known-projects
registry, is countedorphaned_recon_escalations_unresolvableand
deliberately never flagged — the sweep cannot see that project's task store,
so it holds no evidence either way. Verify that project's tasks by hand
before touching such a record. Empirically 0 of the 124 fail the parse today
and the deployedDASHBOARD_KNOWN_PROJECT_ROOTScovers all seven projects
present, so a non-empty bucket signals a registry gap worth investigating
rather than a reapable record.You cannot drive the gate itself terminal from
here: the gate is a born-at-L2milestone_gateescalation filed on the
target project's own orchestrator queue, not this one — resolving the
recon surfacing changes nothing about the gate or the task. If you find a
satisfied-but-unclosable gate (the work is done but you have no path to
close the task), say so explicitly in the cockpit decision text as a
NO-OP marker, so the next human triaging the queue doesn't have to
re-derive that conclusion.reconciliation_stale_human_operator(blocking) — a task has stayed
flaggedhuman_operator_requiredforSTAGE1_HUMAN_OPERATOR_STALL_THRESHOLD
= 5 Stage 1 cycles that survived dedup, filed once per task while it has
no open level-1 escalation (maybe_escalate_stalled_tasks,fused-memory/src/fused_memory/reconciliation/stage1_stall_detector.py).
Same aging/park shape asreconciliation_stale_gate_backlogabove —
default: PARK, file a cockpit DecisionRecord, append its esc-id to
the wrapper-owned exclude file<queue-dir>/.watcher-rearm-exclude
(not--exclude-id— see
that row above for why), and stamp_triage it the same way (see the
gate-backlog row above for the call, the once-on-first-park cadence, the
predicate+probe note shape, the safety argument, and why it matters); see
that row above too for the mechanism and churn evidence, not repeated
here. Asymmetry to know:
this path dedups on an un-categorizedhas_open_l1(task_id)insidemaybe_escalate_stalled_tasks(same module, line 385), so an openreconciliation_stale_gate_backlogL1 on
the same task CAN suppress a would-be HOR escalation, while the
reverse can't happen (the gate-backlog lookup is category-scoped).
The terminal-or-missing-subject RESOLVE branch in the gate-backlog row
above covers this category too — the task-3052 reaper reads both
categories, so adone/cancelled/absent subject here is likewise resolved
withresolution_class='moot-terminal-subject'rather than parked; as of
2026-09-02 this category had zero pending records, so that coverage is
future-proofing, not a backlog waiting to be worked.
Practically: a parked gate-backlog record may be masking a HOR condition
on that same task — don't read "no pending HOR record for this task" as
"no HOR condition for this task". Same resolve-to-tidy trap as above,
for a different reason: the stall marker that counts cycles never
resets (same module docstring, lines 48-59) — resolving a HOR escalation
while the task is still flaggedhuman_operator_requiredcan cause an
immediate re-file on the very next cycle, since the accumulated cycle
count is already at or past the threshold.
Drain-side shortcut — the exclude file only quiets the watcher, not the
drain: Main Loop step 5's get_pending_escalations() has no exclusion
mechanism of its own and returns every pending record each cycle, parked
or not. For an esc-id you've already parked in a prior cycle, don't
re-read and re-reason about it from scratch each time: the cockpit--id is idempotent on the esc-id ("Filing Parked Decisions to the
Cockpit Registry (C8)" below — re-filing the same id overwrites rather
than duplicates), so if you recognize you've already filed a decision for
it, just re-affirm PARK and move on instead of re-deriving the decision
text.
A note on tiering for these two categories: the comparison table above
already says "Tiering | none — recon files flat, no levels" — that
stays architecturally true, but it is tempting to reach formcp__escalation__promote_to_l2 anyway when a finding feels L2-worthy.
Doing so from a recon-watch session actually succeeds: the recon
harness builds its 8103 server with the same create_escalation_server(...)
the orchestrator uses
(fused-memory/src/fused_memory/reconciliation/harness.py:2016), sopromote_to_l2 is registered there too, and the identity gate
(escalation/src/escalation/authority.py:65) only denies identified
callers outside PROMOTE_ALLOWED — a header-less recon-watch/mcp.json
session is never identified, so the call goes through and mints a real L2
file. It works, and that is exactly the problem: nothing automated ever
consumes it. The orchestrator harness supervises escalation-watcher-auto
as its own subprocess (_start_watcher_supervisor,orchestrator/src/orchestrator/harness.py:10572), and the rotation prompt
it builds binds the watched dir to self.config.escalation.queue_dir
(_run_watcher_rotation, harness.py:10618, queue-dir line at :10645
— data/escalations behind 8100/8102); it never opensdata/reconciliation/escalations. That
does not mean the recon queue is invisible to humans — the dashboard's
Escalations tab does render it, as a read-only reconciliation subsection
built from this same directory (build_escalation_queues,dashboard/src/dashboard/data/escalations.py) — the real gap is that a
promoted record has no automated L2 triage consumer and no L2 cascade
actor, so it just sits there as extra queue noise nobody is watching for.
This watcher is the actual human-facing consumer of recon findings: the
sanctioned way to raise one to a human is the cockpit DecisionRecord
(§"Filing Parked Decisions to the Cockpit Registry (C8)" below), whose C5b
decision queue is a surface a human actually works — not promote_to_l2.
Priority Hierarchy
- System & infrastructure stability — never touch anything outside the
project dir; never kill other processes; if the 8103 server or fused-memory is
down, ask the human. Never edit queue JSON files by hand — resolve only viamcp__escalation__resolve_issue. - Software / memory quality — prefer root-cause repair over papering over.
When a direct fix is ambiguous or risky, file a task or ask, rather than
guessing at a graph mutation. - Throughput — clear-cut closures: act decisively. Ambiguous-and-consequential:
leave pending, tell the human, track it, move on. For each item deliberately
left pending, append its esc-id to the wrapper-owned exclude file
(<queue-dir>/.watcher-rearm-exclude;echo <esc-id> >> <path>) so the
initial scan does not instantly re-fire on it and busy-loop. Also file a
DecisionRecord viawrite-decision(see "Filing Parked Decisions to the
Cockpit Registry" below) — IN ADDITION to telling the human.
Filing Parked Decisions to the Cockpit Registry (C8)
Fleet Cockpit C8 (plans/fleet-cockpit-prd.md): every time this skill leaves a finding pending
for the human — recon_failure/recon_backlog_overflow, infra_issue, risk_identified, a
clustered recon_stale_run, or the general Priority 3 "leave pending, tell the human" case —
also file it to the cockpit decision registry, IN ADDITION to telling the human in-session.
This is the same registry escalation-watcher files to (skills/escalation-watcher/SKILL.md,
"Filing Parked Decisions to the Cockpit Registry"), so the cockpit decision queue (C5b) becomes
the primary return-triage surface across both watchers:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py write-decision \
--id <stable-id> --project <project> --text "<one-line question>" \
--escalations-dir $DARK_FACTORY_ROOT/data/reconciliation/escalations \
[--task-id <task_id>] [--escalation-id <escalation_id>] \
[--session-id "recon-watcher-<project>-${CLAUDE_PID:-unknown}"] \
[--severity <esc.severity>]--id: a stable id you can recompute idempotently for the same pending item — the
escalation id (e.g.esc-recon-abc-1) is the natural choice. Re-filing the same id overwrites
the prior record rather than duplicating it.--idis project-LOCAL (task 4835). The verb files the record as<canonical project>-<id>(e.g.dark_factory-esc-42-1) and prints that id: use the
PRINTED id for cross-links (including thedecision:field of a PARK
triage note) and forclose-decision/reopen-decision. Keep passing
the bare escalation id — never hand-prefix the project. A record filed before this change under a
bare id is continued in place by your re-file, so do not change an existing--idtemplate
(such asrecon-watcher-lease-orphan-<project>above): rows already filed under it would stop receiving
your re-files. That includes a template that already hand-prefixes the project (df-esc-<id>):
keep it as it is, and the verb continues thedf-esc-…row in place. "Never hand-prefix"
governs only a template you are writing new; dropping an existing prefix moves your re-files to
a fresh row that ignores the old row's answered or dropped state.
You no longer have to pre-check before filing (task 3559). Decision ids are fleet-global, so
the orchestrator'sescalation-watchermay already have filed a decision under this id for the
same underlying human gate. The verb now handles that for you: if anopenrecord already exists
at this id and was filed from a different--escalations-dir, your filing enriches it
instead of overwriting — non-empty fields survive, fields the first filer left empty are filled
from yours,severitytakes the max of the two and is never downgraded, andfiled_at/state/manual_booststay with the first filer (so you cannot reset an operator's
cockpit boost or re-open a closed row). Just file; you do not need to look first.
One field is exempt from "empty ones are filled":--escalation-id. It travels with--escalations-diras a pair and is never filled from the other filer across a queue boundary,
becauseesc-<taskid>-<n>ids are unique only within one queue and the reaper joins on the
(queue, id) pair — a borrowed id would resolve against an unrelated escalation in the adopting
queue. Two things that means for you:- Always pass
--escalation-idalongside--escalations-dir: they name the id and the namespace
it lives in, and a filing that supplies the queue but not the id cannot upgrade a legacy
unstamped record. - If a legacy record already holds a different escalation id under an unknown queue, your
filing is refused with aWARNINGand the record deliberately stays a visible cockpit row
rather than being silently repointed — that is the fail-OPEN direction, and the remedy is the
back-fill (scripts/backfill_decision_queue_stamp.py), which actually investigates provenance.
Two deliberate limits, both drawn on the queue axis. A re-file from the same queue is
still a plain idempotent whole-file overwrite of everything that is yours — text, severity and
the task/session/escalation ids all land verbatim, downgrades and emptied fields included,
because that is the restart promise above and you are the sole authority on your own escalation.
What it does not touch is the record's custody:filed_at,manual_boostandstate
stay with the record, at any state (task 3872). So your restart cannot bump a row to the top
of the age ordering, cannot drop a boost an operator set between your two filings, and cannot
re-open a row an operator already answered or dropped — within one queue anesc-<taskid>-<n>id is unique, so your re-file is the same gate the human already dealt with
rather than a new ask. (Unique, but not absolutely: an id can be re-minted inside one queue
after a lost sequence counter, which is precisely why the hold is announced with aWARNINGfor
you to adjudicate rather than applied silently — see below.) Across queues those id namespaces
genuinely collide, so the second
limit is that a filing from a different queue against a closed record is still a plain
overwrite that re-opens it: there it may truly be an unrelated new question, and holding it
closed would hide a live gate instead of surfacing it.Across projects, the same local id is two rows.
esc-<taskid>-<n>numbering restarts
per project, soesc-42-1under two different--projectvalues names two unrelated gates, and
because the stored id carries the project they land on two rows (dark_factory-esc-42-1,reify-esc-42-1). A refusal remains only when the qualified id is already heldopenby
another project's legacy hand-prefixed record (say a dark_factory record someone filed by
hand under the idreify-esc-42-1): anERROR, nothing written, rc still 0, because merging would hide your ask
inside that row and overwriting would delete it. Your ask still reaches the human through the
in-session note / afk-digest line this filing accompanies; a different--idgets it a cockpit
row too.- Always pass
--project: the project's canonical token — thememory.project_iditsdark-factory-orchestrator.yamldeclares. For dark-factory that isdark_factory, notdf
and notdark-factory. The value is normalized at the CLI boundary (case-folded,-and_
equivalent,dfaliased todark_factory), so a stale spelling can no longer create a hidden
partition — but pass the canonical token anyway, so what you type matches what the cockpit shows
and no rewrite warning is logged. The project prefix on a stored id (dark_factory-esc-3524-1)
is added bywrite-decisionfrom the folded--project, so never type it into--id;
hand-prefixed ids likedf-esc-3524-1are legacy, from before the verb qualified ids. Typing a
project spelling into an id is what led humans to pass--project df, and so to a three-way
split of one project's decisions (41 open dark-factory rows spread acrossdark_factory/df/dark-factory, each invisible to a reap scoped to either of the others).- Caveat — check your project's existing rows before trusting the declared token. Folding
merges spellings that differ only by case or separator; only an entry inshared.project_id.PROJECT_ID_ALIASEScan bridge a project whose filed decisions fold to something other
than its declaredmemory.project_id, and todaydf → dark_factoryis the only such entry.
solar-challenge is the known standing case: its config declaresmy_solar_challenge, but
its decisions are filed undersolar-challenge/solar_challenge(which fold together, but
not ontomy_solar_challenge), so reaping it with the declared token matches zero rows —
passsolar_challengethere. That guidance is permanent, not provisional: task 3813
decided the alias question and declined it (the fold left no split to heal, and the
identity question is an open human gate in that project marked "Do NOT auto-act"), recording
the evidence inPROJECT_TOKEN_ALIASES_DECLINED. You no longer have to remember this
unaided —write-decisionandreap-decisionsboth warn if you passmy_solar_challenge, so the mismatch announces itself instead of returning a silent
zero-row no-op. To check any other token, runreap-decisions --project <token> --escalations-dir <queue> --expect-matchesby hand: it warns
when no record, in any state, carries the folded--projecttoken, and lists the folded tokens
that do. (It replaces acollections.Counterone-liner that listed RAW spellings,
which the reaper folds together, so it showed splits that were not there.) Leave the flag off
the per-cycle Main Loop reap: a project that has simply never filed would warn every cycle.
- Caveat — check your project's existing rows before trusting the declared token. Folding
--text: the one-line question a human needs to answer.--task-id/--escalation-id: thread through the syntheticrecon-<runid>task id (if
any) and the escalation id, so the cockpit can cross-link the decision to its source.--session-id: a provenance label saying which watcher filed the row — not a link, since
its lease-token shape is not a session-registry key. The cockpit's Enter-to-focus followsrecord_slug, your session record's key, whichwrite-decisionstamps itself from$CLAUDE_PID;
there is no shell token to add. If it cannot be resolved the decision is still filed, just
unlinked.--severity: pass through the parked escalation's own severity (esc.severity—info/blocking/critical/urgent). This now weights the cockpit decision-queue rank, so a
freshly-filedcritical/urgentpark surfaces at the top of the queue instead of being buried
under stale awaiting-input sessions.--escalations-dir(mandatory): the escalation queue your--escalation-idbelongs
to — for this
watcher,$DARK_FACTORY_ROOT/data/reconciliation/escalations. It must name the SAME queue you
later pass toreap-decisions(below). Decision records are fleet-global while an escalation id
is unique only within one queue, and this project runs two of them over the sameesc-<taskid>-<n>namespace, so this is what lets the reaper join a decision back to the right
per-queue id namespace instead of matching an unrelated same-named orchestrator escalation.
Stored normalized, so any spelling of the same directory works. You can no longer file without
it (task 3559): omit the flag and the verb exits 2 with nothing filed, and pass it empty and it
refuses with a loud error and prints no id — so if the id doesn't come back on stdout, your
filing did not land. It is the same directory you already pass toreap-decisions.
It is mandatory even for a sentinel park with no--escalation-id(a lease orphan, a
pipeline stall): task 4835 decided against exempting them, because the same-queue custody hold
keys on stamp equality, so an unstamped sentinel later re-filed with a stamp would re-open a row
an operator had dropped.
There is a third value the field can hold:<unknown>(session_registry.UNKNOWN_QUEUE) —
"this record's owning queue was investigated and could not be determined". You never write it —
and that is now enforced, not just asked:write-decisionrejects--escalations-dir <unknown>outright, because a record stamped that way is refused by every
reaper and could only ever be closed by hand. It stays valid on the back-fill path
(set_decision_escalations_dir), which is where task 3640 wrote it, for legacy records whose
escalation id resolved in several queues at
once. It is not a respelling of the queue-less''state:''means nobody told us and
falls back to project-only scoping, while<unknown>means we looked and could not tell and
the reaper refuses to close it at all.The verb files
state=openfor a new record, but it never re-opens a row an operator already
answered or dropped when you re-file from your own queue (task 3872). What you will see: the
filed id still comes back on stdout — that signal is unchanged, and your filing did land, since
your text/severity/ids were written — plus aWARNINGon stderr naming the state it held. That
warning means a human dealt with this gate while it sat parked, so adjudicate it rather than
re-filing blindly on your next restart. If this same gate genuinely needs a human again,
re-open the row in place; the warning names the exact command:python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py reopen-decision \ --id <printed id> --project <project> --escalations-dir $DARK_FACTORY_ROOT/data/reconciliation/escalationsIt clears the row's closing evidence, so a later
close-decisioncan quote fresh evidence, and
it exits non-zero when it refuses (the record at that id is another project's or queue's) or
finds no record. It cannot hold open a row whose escalation is already resolved or dismissed:
the next reap closes it again, and the verb warns on stderr when that is the case. Such a gate
needs a new escalation, filed under its own id. File under a new id only for a genuinely
different ask. Either way, do not
try to force the row open by re-filing.The verb is fail-soft (a registry fault is logged and swallowed, never raised) — filing a
decision can never crash the watch loop or block the "leave pending" action itself.
Closing parked decisions on resolve
A filed DecisionRecord stays state=open — and therefore visible in the cockpit decision queue —
until its escalation reaches a terminal status. The watcher that files a decision is almost never
the one that resolves it: resolution typically happens later, via the human acting on the
dashboard or an L2 cascade. So closing a parked decision is a separate, recurring step, not
something the write-decision call itself can do.
Once per Main Loop cycle (see step 7 above), run:
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py reap-decisions \
--project <project> --escalations-dir $DARK_FACTORY_ROOT/data/reconciliation/escalationsNote the queue dir: recon escalations live under data/reconciliation/escalations/ (the
flat recon queue on port 8103), not data/escalations/ (the levelled orchestrator queue thatescalation-watcher reaps). A mis-pointed reaper does not merely no-op. Escalation ids
(esc-<taskid>-<n>) are unique only within a queue and both queues share that namespace, so
pointing --escalations-dir at data/escalations here makes the reaper falsely close this
watcher's decisions against unrelated orchestrator escalations that happen to share an id — the
decision vanishes from the cockpit queue while its own escalation is still pending. Observed withesc-3036-1 (task 3528): a blocking recon gate was silently closed and sat invisible for ~7 days;
15 ids were resolved in both queues at time of measurement.
Passing --escalations-dir on write-decision is what makes the reaper skip cross-queue records:
it stamps the owning queue on the decision, and a reaper scanning a different queue leaves that
decision alone. Keep the two dirs straight regardless — a queue-less legacy record (filed before
that flag existed, i.e. the pre-existing population only) falls back to project-only scoping and
has no such protection. Task 3640 back-filled that population, and task 3559 made the flag
mandatory at the verb, so no new queue-less record can be created through write-decision. That is
what makes the back-fill terminal rather than a recurring chore: the unprotected set can now only
shrink.
This queue is FLEET-WIDE, and that widens the collision surface.data/reconciliation/escalations holds reconciliation escalations for reify, autopilot_video,
solar_challenge, know_live and pump_web_ui as well as dark_factory. So an esc-<taskid>-<n> filed
here routinely collides with an id in one of those projects' own orchestrator queues — e.g.esc-5773-1 exists in both this queue and reify/data/escalations. Such collisions are
expected, not anomalous, and they are why --escalations-dir is now mandatory on everywrite-decision rather than merely advisable — the stakes reach beyond dark_factory: without it, a
record filed here is false-closable by a reaper running in a project you have never touched.
A decision stamped <unknown> is refused, not closed: no reaper may close it, and it stays a
visible cockpit row until a human closes it. The reaper never defaults to closing on doubt — an
over-held decision is a triageable row, a falsely-closed one is invisible. If unstamped open
records ever reappear, the re-runnable remedy is scripts/backfill_decision_queue_stamp.py
(dry-run by default; --verify exits non-zero while any open record still lacks a stamp).
The project axis matches on the canonical token (see --project above), so ONE run per
queue closes every historical spelling of that project — there is no need to re-run the verb
once per token (df, dark_factory, dark-factory) as was necessary before. This holds as long
as the token you pass folds into the same bucket its rows carry, which is not automatic for every
project — read the --project caveat above before assuming a zero-row reap means "nothing to
close". Folding only ever
merges spellings of the same project; it never merges two different projects (e.g.solar_challenge and solar_challenge_platform stay separate), so this widens what a reap
closes without ever letting one project's reaper close another's decisions. Note this is
orthogonal to the queue axis above: canonicalizing the project token does not relax the
cross-queue guard, so a mis-pointed --escalations-dir remains just as hazardous as described.
To repair the legacy population — records filed before write-decision canonicalized --project
— run the one-shot backfill (dry-run first):
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py \
migrate-decision-projects --dry-run # preview; writes nothing
python3 $DARK_FACTORY_ROOT/orchestrator/src/orchestrator/session_registry.py \
migrate-decision-projectsIt rewrites only the project field: state and filed_at are preserved, so an already-answered
row is never reopened, and a record's id (and its cockpit cross-links) is never rewritten. It
is idempotent — a re-run once the fleet is clean prints nothing — so it is also the repair tool if
a record is ever hand-edited. It is fleet-global, so it only needs running once, not once per
watcher.
Two collision modes exist and must not be conflated:
- MODE 1 — unrelated same-id escalations (
esc-3036-1): two different escalations that merely
collide in the shared id namespace. Closing across queues here is a straight bug, and the queue
stamp is what prevents it. - MODE 2 — same-subject duplicates (
esc-5914-1): both queues surfacing the same underlying
human gate. These must collapse to ONE cockpit decision — a human asked the same question
twice is its own regression. That is why the queue is recorded as a field on the record rather
than namespaced into the decision id (recon:esc-…/orch:esc-…would double-file the same
question). A second watcher must never downgrade an existing record's severity — and since task
3559 the verb enforces that rather than asking you to remember it: a filing from a different
queue against anopenrecord enriches it, never clobbering or downgrading (see--idabove).
The queue stamp is scalar, so the collapsed record keeps the first filer's queue and is
therefore reapable only by that queue's reaper; the verb logs a warning naming both queues when
it discards the other. The single stamp is a decided limit, not a pending fix (task 4835;
seeplans/4835-decision-plumbing-decisions.md).
This closes (answered/dropped) any state=open, escalation_id-bearing decision whose
escalation has since resolved (resolved → answered) or been dismissed (dismissed →dropped) — regardless of who resolved it. It is read-only with respect to escalations (it only
ever writes the decision's own state field) and fail-soft, exactly like write-decision — a
registry fault is logged and swallowed, never raised, so it can never crash the watch loop. A
decision filed with no escalation_id is never auto-closed this way and needs explicit human
closure. Likewise, a decision whose escalation_id never resolves to a status — the escalation
was purged by archive retention pruning, or never existed — also stays open forever and needs
the same explicit human closure; until then, every cycle repeats a full scan of the escalations
archive looking for it.
Caveat: recon re-files until the go-forward fix lands
As of 2026-05-27 the recon harness still escalates non-actionable info findings
into 8103 every cycle (the content-fingerprint dedup, A7a/A7b, is ineffective —
the records have no stable identity). The 5,958-item historical pile was bulk-
dismissed (Direction 3); the queue starts at ~90. Until the recon-side fix lands
(stop escalating non-actionable info findings — tracked as a separate task),
expect a steady trickle of fresh recon_integrity_issue items. Handle them
efficiently — most are accept-as-known. If the trickle is heavy, remind the
human that the upstream fix is the real lever; you are holding the line, not
solving the source.
Context Conservation
Long-running session — conserve context. Delegate to sub-agents: researching
a finding's current state (search/get_entity/get_task reads), and executing
file-a-real-task MCP calls once you've decided. Keep in top-level context: the
loop, closure decisions, human communication, and which findings are accepted-as-
known so you don't re-investigate them each cycle.
Failure Modes
- "Too many open files" after many watcher restart cycles → fd pool
exhaustion from accumulated background tasks; tell the user to start a fresh
session. get_pending_escalationsempty when you expect items → read theTHIS ENDPOINT SERVES:block ending that tool's own description in your
tool listing (rendered byescalation/src/escalation/server.py::_render_store_identity_block).kind=reconciliation→ this queue answered; nothing is pending.kind=project(e.g.project=dark_factory) → wrong store, an orchestrator
queue: relaunch per "Launching" above. No block → the server cannot vouch
for its store; tell the human rather than read[]as absence.- 8103 server unreachable → fused-memory.service is down. Priority 1: tell
the human; do not start services yourself.