Reference pattern for parallel sub-agent dispatch, fan-out/fan-in, concurrency control, retry with backoff, and convergence gates. Consumed by other skills — not invoked directly.
Install
npx skillscat add daniel-ospina/agent-infra/parallel-orchestrator Install via the SkillsCat registry.
⛔ This skill MUST be read in full — not skimmed. Formal review gates depend on its workflow.
Skipping steps silently bypasses quality checks. Missing gates = undetected breakages.
Parallel Orchestrator (Reference Pattern)
Standardized fan-out/fan-in pattern for parallel sub-agent dispatch in pi. All orchestrator skills (codebase-audit, content-strategy-agent, prototype-review, epic-executor) should reference this pattern instead of reimplementing concurrency control.
When to Use This Pattern
- Dispatching 2+ independent sub-agents that can run simultaneously
- Review cycles with parallel reviewers
- Batch processing where items don't depend on each other
- Any workflow where sequential dispatch would add unnecessary wall-clock time
Not for: Tasks with data dependencies (output of A feeds into B), tasks touching the same files, or single-agent operations.
The Pattern
1. Fan-Out Dispatch
Use subagent({ tasks: [...] }) for true parallel execution:
subagent({
tasks: [
{ agent: "worker", task: "Review diff for correctness..." },
{ agent: "worker", task: "Review diff for security..." },
{ agent: "worker", task: "Review diff for accessibility..." },
]
})The subagent tool blocks until ALL tasks complete. All tasks run concurrently — wall-clock time ≈ slowest task, not sum of all tasks.
If you must NOT block the orchestrator (long-running lanes, work you can overlap with planning), dispatch with task({ ..., background: true }) instead — it returns immediately with a run handle. See the Background Execution Note below.
2. Concurrency Control
MAX_CONCURRENCY = 16 # hard cap — never exceed (bounded by fan-in context + worktree contention, NOT API limits — direct DeepSeek API is concurrency-only: 500 v4-pro / 2,500 v4-flash, #317)
STAGGER_MS = 200 # delay between task launches
RETRY_BACKOFF = [1000, 2000, 4000] # exponential backoff + jitter ±200msRules:
- Never launch more than
MAX_CONCURRENCYtasks in a singlesubagentcall - If you have N > 16 tasks, batch them: first 16 → wait → next 16
- Stagger launches within each batch: add a brief pause between constructing task prompts
- On rate-limit errors: retry with exponential backoff. After 3 retries, surface the failure
3. Structured Output Format
ALL parallel sub-agents MUST return results in a parseable, deduplicatable format:
FINDING: [P0|P1|P2|P3] Brief title
- File: path/to/file.ts:line
- Category: [Security|Bug|Config|SupplyChain|Database|UX|Accessibility|SEO|Content|...]
- Description: 1-2 sentences
- Fix: Specific suggested approachOr for binary results:
RESULT: PASS|FAIL
- Detail: <explanation>4. Fan-In Synthesis
After all sub-agents return:
- Collect all results into a single list
- Deduplicate — if two agents flag the same issue (same file+line or ≥70% similar description), merge into one (keep higher severity)
- Sort by severity (P0 first)
- Count by category
5. Convergence Gate
For review cycles (review → fix → re-review), stop when:
is_clean: true— all reviewers return zero issues- Convergence — issue count is shrinking but not reaching zero (surface remaining to human)
- Safety cap — at 10 cycles, exit with remaining issues documented
Never loop infinitely. Always have a cap.
Pre-Warming Pattern
For tasks where a slow prerequisite can run in parallel with planning:
# In Step 0, before planning begins, start typecheck in the background:
npm install --silent 2>&1 | tail -1 && npx tsc --noEmit > /tmp/typecheck-preflight.txt 2>&1 &
# Bounded: this is a NON-pi background job — it needs the deadline watchdog + log
# redirect (above) but NO [task-heartbeat] markers (pi-runtime-only). Never launch
# a nested pi this way: nested pi launches carry the FULL bounded template
# (TASK_HEARTBEAT=1 + PI_MODE=print + mktemp log + kill -0 watchdog) — see the
# Background Execution Note below.
# Deadline watchdog for ANY backgrounded job (portable; no GNU timeout on macOS):
# ( sleep 1800; if kill -0 $! 2>/dev/null; then kill $! 2>/dev/null; fi ) &
# Before starting implementation, check the result:
if [ -f /tmp/typecheck-preflight.txt ] && [ -s /tmp/typecheck-preflight.txt ]; then
echo "⚠️ Pre-flight typecheck errors:" && cat /tmp/typecheck-preflight.txt
elif [ -f /tmp/typecheck-preflight.txt ]; then
echo "✅ Pre-flight typecheck clean"
fiThe orchestrator checks the background result before starting implementation. If FAIL, surface errors. If still running, warn and proceed.
Integration
Consumed by:
- codebase-audit — Phase 2 fan-out/fan-in
- content-strategy-agent — Reviewer cycle dispatch
- prototype-review — Review cycle dispatch
- epic-executor — Issue batch dispatch
- executing-plans — Step 0 pre-warming, Phase 3 verification dispatch
Uses:
- subagent tool (built-in) — provides
subagentwithtasksarray for parallel dispatch. It always blocks until completion. For non-blocking work, use thetask({ background: true })primitive (task_status/task_collect) — see Background Execution Note below. A bounded bash&is still the tool for NON-pi background jobs (typechecks, builds).
Background Execution Note
The subagent tool blocks until completion. The task tool now has a first-class non-blocking mode (#1662) — do NOT hand-roll a bounded bash launch just to avoid blocking on a pi lane:
task({ prompt, background: true })
→ { run_id, pid, pgid, log_path } # returns IMMEDIATELY, never awaits
task_status({ run_id }) # alive | wedged | gone | done
task_collect({ run_id }) # the final message + exit status, once done- Liveness comes from the child's
[task-heartbeat]markers, never from%CPU.%CPU == 0means waiting, not finished — a lane blocked in a long tool call is exactly 0 % CPU (docs/ops/fleet-liveness.md).task_statusreads the same markers the watchdog uses (pid probe + fresh markers), and returns non-blockingly. task_collectnever blocks. While the lane runs it returns a not-finished notice; polltask_statusinstead. Reaping loop: polltask_statusbetween other work → calltask_collectonce it reports a terminal status (done|failed|cut) orgone.gonemeans the process exited but its terminal record is not yet persisted — the child can exit seconds before the parent's settle write lands, sotask_collectkeeps returning not-finished until the settle grace (5 min) expires, then salvages it fail-closed tofailed. Re-poll rather than treating agonelane as finished. A lane leftwedgedwith a stale log is likewise terminated bytask_collect(same fail-closed salvage) instead of being polled forever.- It is ONE spawn per background lane — no zero-output retry and no provider-failover hop chain. A lane that fails surfaces as status
failed; re-dispatch it (the #208 resume contract: assume-dead, not done; design waves as resumable). - Reused machinery, not a parallel universe: the background path spawns through the same
spawnSubAgent, soTASK_DETACHED's own pgid, the settle-path sweep,TASK_HARD_CAP_MS, and the state-aware watchdog all still apply. A capped lane still returns partials viatask_collect.
⛔ REVIEW GATE IS A PARENT STEP — the
task_collectanalogue ofwatch. Sub-agents run withAGENT_SKIP_REVIEW_GATE=1FORCED, so the review dispatch cannot be satisfied from inside the child; the PARENT enforces the review ceremony centrally (#825). A background lane makes that step easy to forget:task_status,task_collect, and the dispatch handle all carryreview_gate: "parent-required". Aftertask_collectreturns a terminal result you MUST run the review ceremony for that lane before treating it as complete. A background lane that is collected but never reviewed is UNREVIEWED CODE BY DEFAULT.
For a NON-pi background job (pre-warming a typecheck, a long build, a vendored tool), the bounded bash template below is still correct — it needs the deadline watchdog + log redirect, but NO [task-heartbeat] markers (those are pi-runtime-only). Never launch a nested pi that way: use task({ background: true }) (or, if you truly must, the full bounded template in ### Bounded nested pi launch).
Check a non-pi background result before it is needed.
⛔ Fleet policy — load-aware scheduling (#209). Before dispatching a heavy wave or rerun batch, defer while the machine is under load:
bash scripts/wait-for-load.sh 8 600(shared helper, promoted from the wt-291 gated-rerun pattern). The task watchdog's load-aware bounds — the first-message bound and the marker-gap cut bound (#1070) — scale with loadavg through the fixed bands declared indocs/ops/load-policy.md§6, which is the contract for those numbers (not restated here, so the two cannot drift);TASK_LOAD_SCALE_OFF=1bypasses the scaling. A load storm therefore no longer cuts live sub-agents for first-message latency, and an explicitTASK_HEARTBEAT_CUT_GAP_MSis honoured verbatim and never rescaled.
⛔ Assume dispatched sub-agents may die (#208). The parent task tool now returns partial results within a bounded hard cap (TASK_HARD_CAP_MS, default 6h) when a sub-agent is cut — but a cut is still a cut: design dispatch waves as RESUMABLE (each sub-agent commits WIP early, the orchestrator re-dispatches on a cut and reconciles from the commit/partial result). Never design a wave where one lost sub-agent strands the rest.
⛔ HARD RULE (#206): never launch an unbounded nested
pi. Any background/nested pi launch MUST carry all three bounds — a hard wall-clock cap (use the portable bounded template in the### Bounded nested pi launchblock below — GNUtimeoutdoes not exist on macOS, #206), log-file redirect (> file 2>&1; pipe stdout is block-buffered and looks frozen, #202), and a liveness marker (periodic timestamped heartbeat line). The 2026-08-11→12 incident: an unboundedpi -plaunched to escape the worktree guard sat for 105,150s before manual abort. The sanctioned guard escape is the terminal one-liner (cd <repo> && git checkout main && git pull --ff-only), never a nested pi.
For parallel dispatch of independent blocking tasks, use subagent({ tasks: [...] }) — the tool blocks but all tasks run concurrently.
Partial-Failure Handling
When dispatching parallel sub-agents, not all may succeed. Handle gracefully:
Pattern
- Collect all results — successes and failures
- Classify: ✅ Success, ⚠️ Timed out, ❌ Failed
- If some succeeded: Proceed. Flag gaps: "⚠️ reviewer-3 timed out — skipped"
- If ALL failed: Return structured error: "❌ All reviewers failed (3/3)"
- Aggregate multiple gaps into single summary
All fan-out orchestrator skills (code-review, plan-review, prototype-review, test-review, content-strategy-agent, codebase-audit) implement this pattern.
Teardown Contract (worktrees/branches) — MANDATORY for parallel dispatch
Issue #195: an aborted 6-task parallel dispatch left orphaned worktrees and fix/* branches with NO teardown (observed 2026-08-12; .worktrees/ accumulated 60+ dead entries). Nothing owned the lifecycle. Since pi exposes no abort hook the dispatcher can rely on, teardown is record-first: every artifact a dispatch creates is written to a manifest before dispatch, so an abort leaves an explicit teardown trail instead of silence.
The contract — every orchestrator that spawns sub-agents which create branches/worktrees MUST:
RECORD before dispatch (each worktree/branch the sub-agent will create):
bash scripts/record-worktree.sh add --branch fix/801-signup-rate --worktree .worktrees/fix-801-signup-rate --dispatch d-<id>(Records append to
~/.pi/agent/worktrees.jsonl— one JSONL line per artifact: ts, branch, worktree, dispatch, repo. Writes are atomic tmp+mv; a failed write warns but NEVER blocks the dispatch.)DONE on clean completion — after the fan-in succeeds and worktrees are removed, the orchestrator removes its own records:
bash scripts/record-worktree.sh done --dispatch d-<id>ABORT leaves the record — on user abort, sub-agent hang, or crash, do NOT clean records. The surviving record IS the teardown manifest: the next sweep flags it.
SWEEP after every dispatch (and periodically) —
scripts/scan-orphans.shreads the manifest + git state and classifies each record:- ORPHAN — branch local-only, no open PR, record stale (>1d by default) → safe to remove
- DIRTY-ORPHAN — worktree holds uncommitted changes → manual review;
--applyrefuses without--force-dirty - RECENT — recorded, not yet stale → listed, never removed
- LIVE — branch pushed or has an open PR → ignored
- GHOST — branch/dir already gone → record pruned
- UNRECORDED — informational only (pre-contract worktrees/branches); NEVER auto-removed
bash scripts/scan-orphans.sh # dry-run: teardown list, deletes nothing bash scripts/scan-orphans.sh --apply # explicit removal of ORPHANs + ghost records--applyremoves the worktree dir, deletes the local branch (nevermain/master, never a pushed branch, never a dirty worktree without--force-dirty), and prunes the record.Document the wiring — see
scripts/scan-orphans.sh --helpfor the full contract and the issue-workflow skill's Worktree Gate for the dispatcher-side rules.
Why not an abort hook? Research for #195 found no reliable cross-session abort/cancel hook in the pi SDK. A deterministic, no-LLM record + sweep is robust to kills, crashes, and lost sessions — the record survives the process that wrote it.
Cut / Assume-Dead Handling (resume contract)
When a worker is cut — killed externally (provider cut, OOM, watchdog kill) or detected dead — treat it as assume-dead, not done (#208):
- Assume-dead: a result with
killed: true, reason: "cut"(task tool) orstopReason: "cut"(subagent ext — limitation: only signal-death maps to cut there, no tool state) means the worker is dead, not done — never block on it, never treat it as success, classify it as ⚠️ Cut. - Bounded wait: parent waits are bounded (watchdog bound + margin); a cut returns partials + reason within the bound — if a dispatch exceeds it, treat as cut.
- Partial-results recovery: read the partial output (task tool content/details; subagent ext result messages) and the F6 result cache (
cachePath) for whatever the worker produced before the cut. - Commit-early contract: workers persist incrementally (docs, results, DB rows) so a cut loses at most the last uncommitted step; re-dispatch may re-run partial writes — workers must be idempotent.
- Re-dispatch on cut: retry 3× with backoff (the SAME cadence as the fallback row below), then surface to human with options. Each re-dispatch is cheap: F6 caches partials on disk.
#195 pairing: a killed mid-write worker may leave
index.lock/ partial worktree state — run the #195 worktree-recovery path before re-dispatching into that worktree.
Bounded nested pi launch — the ONLY sanctioned form
# Bounded nested pi launch — the ONLY sanctioned form of nested pi:
LOG="$(mktemp /tmp/nested-pi.XXXXXX)"
PI_MODE=print TASK_HEARTBEAT=1 pi -p "<prompt>" > "$LOG" 2>&1 &
launch_pid=$!
echo "launched $launch_pid → $LOG"
# Deadline watchdog — portable sleep-deadline + kill -0 liveness probe
# (no GNU timeout on macOS): SIGTERM at 1800s; SIGKILL after a 60s grace.
( sleep 1800; if kill -0 "$launch_pid" 2>/dev/null; then kill "$launch_pid"; pkill -P "$launch_pid" 2>/dev/null; sleep 60; kill -9 "$launch_pid" 2>/dev/null; fi ) &
# Liveness check + abort trigger — [task-heartbeat] markers land on stderr
# every 30s (2>&1 merged). No marker within 60s → the process is dead or
# blocked at the OS level → ABORT. Markers present → the launch is ALIVE and
# the 30-min deadline watchdog is the only remaining bound — do NOT kill here:
sleep 60
if grep -q '\[task-heartbeat\]' "$LOG"; then
echo "ALIVE — bounded by the 30-min deadline watchdog"
else
echo "NO MARKER — ABORTING (process dead/blocked at OS level)"
kill "$launch_pid" 2>/dev/null
pkill -P "$launch_pid" 2>/dev/null
echo "ABORTED — surfacing to user"
exit 1
fi(docs(skills): portable bounded-launch template + drift guard for the never-unbounded-pi rule (#206))
Anti-Patterns
| Anti-Pattern | Why It Matters |
|---|---|
| Dispatching 20+ agents at once | Context pressure on fan-in (orchestrator ingests all results) + worktree/merge contention. Cap at 16, batch the rest |
| No staggered launches | Burst dispatch can spike provider load / hit account-level concurrency caps. 200ms stagger smooths load |
| Sub-agents touching same files | Merge conflicts, inconsistent state. Partition file ownership before dispatch |
| No structured output format | Can't deduplicate or sort results. Enforce format in agent prompts |
| Infinite review cycles | Always cap at 10 cycles. Surface remaining issues to human |
| Waiting for background tasks synchronously | Defeats the purpose. Check background results before they're needed |
Unbounded nested/background pi launch (no timeout, no log redirect, no liveness marker) |
A gate-stalled pi can hang the fleet ~29h (#206). Always use the bounded template (PI_MODE=print TASK_HEARTBEAT=1, mktemp log, sleep 1800 + kill -0 watchdog, abort on no-marker); the terminal one-liner is the ONLY guard escape. |
| No fallback for subagent tool failures | If the subagent tool is unreachable (network, API down), the skill hangs. Always have a fallback: retry 3× with backoff, then surface to human with options to (a) retry, (b) proceed sequentially, (c) abort. |
| Spawning worktrees/branches with no teardown record | Aborted dispatch silently orphans them (#195). Record every artifact before dispatch (record-worktree.sh add), remove on clean completion (done), sweep with scan-orphans.sh — abort leaves the record as the teardown manifest. |
| Blocking on a cut/dead sub-agent | Parent waits are bounded; a cut must return partials + reason within the watchdog bound, then the orchestrator re-dispatches instead of hanging. |
Continue following the workflow as mandated by this skill. Do not skip steps.