daniel-ospina

parallel-orchestrator

Reference pattern for parallel sub-agent dispatch, fan-out/fan-in, concurrency control, retry with backoff, and convergence gates. Consumed by other skills — not invoked directly.

daniel-ospina 0 2 Updated 1w ago
GitHub

Install

npx skillscat add daniel-ospina/agent-infra/parallel-orchestrator

Install via the SkillsCat registry.

SKILL.md

⛔ This skill MUST be read in full — not skimmed. Formal review gates depend on its workflow.
Skipping steps silently bypasses quality checks. Missing gates = undetected breakages.

Parallel Orchestrator (Reference Pattern)

Standardized fan-out/fan-in pattern for parallel sub-agent dispatch in pi. All orchestrator skills (codebase-audit, content-strategy-agent, prototype-review, epic-executor) should reference this pattern instead of reimplementing concurrency control.

When to Use This Pattern

  • Dispatching 2+ independent sub-agents that can run simultaneously
  • Review cycles with parallel reviewers
  • Batch processing where items don't depend on each other
  • Any workflow where sequential dispatch would add unnecessary wall-clock time

Not for: Tasks with data dependencies (output of A feeds into B), tasks touching the same files, or single-agent operations.

The Pattern

1. Fan-Out Dispatch

Use subagent({ tasks: [...] }) for true parallel execution:

subagent({
  tasks: [
    { agent: "worker", task: "Review diff for correctness..." },
    { agent: "worker", task: "Review diff for security..." },
    { agent: "worker", task: "Review diff for accessibility..." },
  ]
})

The subagent tool blocks until ALL tasks complete. All tasks run concurrently — wall-clock time ≈ slowest task, not sum of all tasks.

If you must NOT block the orchestrator (long-running lanes, work you can overlap with planning), dispatch with task({ ..., background: true }) instead — it returns immediately with a run handle. See the Background Execution Note below.

2. Concurrency Control

MAX_CONCURRENCY = 16           # hard cap — never exceed (bounded by fan-in context + worktree contention, NOT API limits — direct DeepSeek API is concurrency-only: 500 v4-pro / 2,500 v4-flash, #317)
STAGGER_MS = 200               # delay between task launches
RETRY_BACKOFF = [1000, 2000, 4000]  # exponential backoff + jitter ±200ms

Rules:

  • Never launch more than MAX_CONCURRENCY tasks in a single subagent call
  • If you have N > 16 tasks, batch them: first 16 → wait → next 16
  • Stagger launches within each batch: add a brief pause between constructing task prompts
  • On rate-limit errors: retry with exponential backoff. After 3 retries, surface the failure

3. Structured Output Format

ALL parallel sub-agents MUST return results in a parseable, deduplicatable format:

FINDING: [P0|P1|P2|P3] Brief title
- File: path/to/file.ts:line
- Category: [Security|Bug|Config|SupplyChain|Database|UX|Accessibility|SEO|Content|...]
- Description: 1-2 sentences
- Fix: Specific suggested approach

Or for binary results:

RESULT: PASS|FAIL
- Detail: <explanation>

4. Fan-In Synthesis

After all sub-agents return:

  1. Collect all results into a single list
  2. Deduplicate — if two agents flag the same issue (same file+line or ≥70% similar description), merge into one (keep higher severity)
  3. Sort by severity (P0 first)
  4. Count by category

5. Convergence Gate

For review cycles (review → fix → re-review), stop when:

  • is_clean: true — all reviewers return zero issues
  • Convergence — issue count is shrinking but not reaching zero (surface remaining to human)
  • Safety cap — at 10 cycles, exit with remaining issues documented

Never loop infinitely. Always have a cap.

Pre-Warming Pattern

For tasks where a slow prerequisite can run in parallel with planning:

# In Step 0, before planning begins, start typecheck in the background:
npm install --silent 2>&1 | tail -1 && npx tsc --noEmit > /tmp/typecheck-preflight.txt 2>&1 &
# Bounded: this is a NON-pi background job — it needs the deadline watchdog + log
# redirect (above) but NO [task-heartbeat] markers (pi-runtime-only). Never launch
# a nested pi this way: nested pi launches carry the FULL bounded template
# (TASK_HEARTBEAT=1 + PI_MODE=print + mktemp log + kill -0 watchdog) — see the
# Background Execution Note below.
# Deadline watchdog for ANY backgrounded job (portable; no GNU timeout on macOS):
# ( sleep 1800; if kill -0 $! 2>/dev/null; then kill $! 2>/dev/null; fi ) &

# Before starting implementation, check the result:
if [ -f /tmp/typecheck-preflight.txt ] && [ -s /tmp/typecheck-preflight.txt ]; then
  echo "⚠️ Pre-flight typecheck errors:" && cat /tmp/typecheck-preflight.txt
elif [ -f /tmp/typecheck-preflight.txt ]; then
  echo "✅ Pre-flight typecheck clean"
fi

The orchestrator checks the background result before starting implementation. If FAIL, surface errors. If still running, warn and proceed.

Integration

Consumed by:

  • codebase-audit — Phase 2 fan-out/fan-in
  • content-strategy-agent — Reviewer cycle dispatch
  • prototype-review — Review cycle dispatch
  • epic-executor — Issue batch dispatch
  • executing-plans — Step 0 pre-warming, Phase 3 verification dispatch

Uses:

  • subagent tool (built-in) — provides subagent with tasks array for parallel dispatch. It always blocks until completion. For non-blocking work, use the task({ background: true }) primitive (task_status / task_collect) — see Background Execution Note below. A bounded bash & is still the tool for NON-pi background jobs (typechecks, builds).

Background Execution Note

The subagent tool blocks until completion. The task tool now has a first-class non-blocking mode (#1662) — do NOT hand-roll a bounded bash launch just to avoid blocking on a pi lane:

task({ prompt, background: true })
  → { run_id, pid, pgid, log_path }      # returns IMMEDIATELY, never awaits
task_status({ run_id })                  # alive | wedged | gone | done
task_collect({ run_id })                 # the final message + exit status, once done
  • Liveness comes from the child's [task-heartbeat] markers, never from %CPU. %CPU == 0 means waiting, not finished — a lane blocked in a long tool call is exactly 0 % CPU (docs/ops/fleet-liveness.md). task_status reads the same markers the watchdog uses (pid probe + fresh markers), and returns non-blockingly.
  • task_collect never blocks. While the lane runs it returns a not-finished notice; poll task_status instead. Reaping loop: poll task_status between other work → call task_collect once it reports a terminal status (done | failed | cut) or gone. gone means the process exited but its terminal record is not yet persisted — the child can exit seconds before the parent's settle write lands, so task_collect keeps returning not-finished until the settle grace (5 min) expires, then salvages it fail-closed to failed. Re-poll rather than treating a gone lane as finished. A lane left wedged with a stale log is likewise terminated by task_collect (same fail-closed salvage) instead of being polled forever.
  • It is ONE spawn per background lane — no zero-output retry and no provider-failover hop chain. A lane that fails surfaces as status failed; re-dispatch it (the #208 resume contract: assume-dead, not done; design waves as resumable).
  • Reused machinery, not a parallel universe: the background path spawns through the same spawnSubAgent, so TASK_DETACHED's own pgid, the settle-path sweep, TASK_HARD_CAP_MS, and the state-aware watchdog all still apply. A capped lane still returns partials via task_collect.

⛔ REVIEW GATE IS A PARENT STEP — the task_collect analogue of watch. Sub-agents run with AGENT_SKIP_REVIEW_GATE=1 FORCED, so the review dispatch cannot be satisfied from inside the child; the PARENT enforces the review ceremony centrally (#825). A background lane makes that step easy to forget: task_status, task_collect, and the dispatch handle all carry review_gate: "parent-required". After task_collect returns a terminal result you MUST run the review ceremony for that lane before treating it as complete. A background lane that is collected but never reviewed is UNREVIEWED CODE BY DEFAULT.

For a NON-pi background job (pre-warming a typecheck, a long build, a vendored tool), the bounded bash template below is still correct — it needs the deadline watchdog + log redirect, but NO [task-heartbeat] markers (those are pi-runtime-only). Never launch a nested pi that way: use task({ background: true }) (or, if you truly must, the full bounded template in ### Bounded nested pi launch).

Check a non-pi background result before it is needed.

⛔ Fleet policy — load-aware scheduling (#209). Before dispatching a heavy wave or rerun batch, defer while the machine is under load: bash scripts/wait-for-load.sh 8 600 (shared helper, promoted from the wt-291 gated-rerun pattern). The task watchdog's load-aware bounds — the first-message bound and the marker-gap cut bound (#1070) — scale with loadavg through the fixed bands declared in docs/ops/load-policy.md §6, which is the contract for those numbers (not restated here, so the two cannot drift); TASK_LOAD_SCALE_OFF=1 bypasses the scaling. A load storm therefore no longer cuts live sub-agents for first-message latency, and an explicit TASK_HEARTBEAT_CUT_GAP_MS is honoured verbatim and never rescaled.

⛔ Assume dispatched sub-agents may die (#208). The parent task tool now returns partial results within a bounded hard cap (TASK_HARD_CAP_MS, default 6h) when a sub-agent is cut — but a cut is still a cut: design dispatch waves as RESUMABLE (each sub-agent commits WIP early, the orchestrator re-dispatches on a cut and reconciles from the commit/partial result). Never design a wave where one lost sub-agent strands the rest.

⛔ HARD RULE (#206): never launch an unbounded nested pi. Any background/nested pi launch MUST carry all three bounds — a hard wall-clock cap (use the portable bounded template in the ### Bounded nested pi launch block below — GNU timeout does not exist on macOS, #206), log-file redirect (> file 2>&1; pipe stdout is block-buffered and looks frozen, #202), and a liveness marker (periodic timestamped heartbeat line). The 2026-08-11→12 incident: an unbounded pi -p launched to escape the worktree guard sat for 105,150s before manual abort. The sanctioned guard escape is the terminal one-liner (cd <repo> && git checkout main && git pull --ff-only), never a nested pi.

For parallel dispatch of independent blocking tasks, use subagent({ tasks: [...] }) — the tool blocks but all tasks run concurrently.

Partial-Failure Handling

When dispatching parallel sub-agents, not all may succeed. Handle gracefully:

Pattern

  1. Collect all results — successes and failures
  2. Classify: ✅ Success, ⚠️ Timed out, ❌ Failed
  3. If some succeeded: Proceed. Flag gaps: "⚠️ reviewer-3 timed out — skipped"
  4. If ALL failed: Return structured error: "❌ All reviewers failed (3/3)"
  5. Aggregate multiple gaps into single summary

All fan-out orchestrator skills (code-review, plan-review, prototype-review, test-review, content-strategy-agent, codebase-audit) implement this pattern.

Teardown Contract (worktrees/branches) — MANDATORY for parallel dispatch

Issue #195: an aborted 6-task parallel dispatch left orphaned worktrees and fix/* branches with NO teardown (observed 2026-08-12; .worktrees/ accumulated 60+ dead entries). Nothing owned the lifecycle. Since pi exposes no abort hook the dispatcher can rely on, teardown is record-first: every artifact a dispatch creates is written to a manifest before dispatch, so an abort leaves an explicit teardown trail instead of silence.

The contract — every orchestrator that spawns sub-agents which create branches/worktrees MUST:

  1. RECORD before dispatch (each worktree/branch the sub-agent will create):

    bash scripts/record-worktree.sh add --branch fix/801-signup-rate --worktree .worktrees/fix-801-signup-rate --dispatch d-<id>

    (Records append to ~/.pi/agent/worktrees.jsonl — one JSONL line per artifact: ts, branch, worktree, dispatch, repo. Writes are atomic tmp+mv; a failed write warns but NEVER blocks the dispatch.)

  2. DONE on clean completion — after the fan-in succeeds and worktrees are removed, the orchestrator removes its own records:

    bash scripts/record-worktree.sh done --dispatch d-<id>
  3. ABORT leaves the record — on user abort, sub-agent hang, or crash, do NOT clean records. The surviving record IS the teardown manifest: the next sweep flags it.

  4. SWEEP after every dispatch (and periodically) — scripts/scan-orphans.sh reads the manifest + git state and classifies each record:

    • ORPHAN — branch local-only, no open PR, record stale (>1d by default) → safe to remove
    • DIRTY-ORPHAN — worktree holds uncommitted changes → manual review; --apply refuses without --force-dirty
    • RECENT — recorded, not yet stale → listed, never removed
    • LIVE — branch pushed or has an open PR → ignored
    • GHOST — branch/dir already gone → record pruned
    • UNRECORDED — informational only (pre-contract worktrees/branches); NEVER auto-removed
    bash scripts/scan-orphans.sh            # dry-run: teardown list, deletes nothing
    bash scripts/scan-orphans.sh --apply    # explicit removal of ORPHANs + ghost records

    --apply removes the worktree dir, deletes the local branch (never main/master, never a pushed branch, never a dirty worktree without --force-dirty), and prunes the record.

  5. Document the wiring — see scripts/scan-orphans.sh --help for the full contract and the issue-workflow skill's Worktree Gate for the dispatcher-side rules.

Why not an abort hook? Research for #195 found no reliable cross-session abort/cancel hook in the pi SDK. A deterministic, no-LLM record + sweep is robust to kills, crashes, and lost sessions — the record survives the process that wrote it.

Cut / Assume-Dead Handling (resume contract)

When a worker is cut — killed externally (provider cut, OOM, watchdog kill) or detected dead — treat it as assume-dead, not done (#208):

  1. Assume-dead: a result with killed: true, reason: "cut" (task tool) or stopReason: "cut" (subagent ext — limitation: only signal-death maps to cut there, no tool state) means the worker is dead, not done — never block on it, never treat it as success, classify it as ⚠️ Cut.
  2. Bounded wait: parent waits are bounded (watchdog bound + margin); a cut returns partials + reason within the bound — if a dispatch exceeds it, treat as cut.
  3. Partial-results recovery: read the partial output (task tool content/details; subagent ext result messages) and the F6 result cache (cachePath) for whatever the worker produced before the cut.
  4. Commit-early contract: workers persist incrementally (docs, results, DB rows) so a cut loses at most the last uncommitted step; re-dispatch may re-run partial writes — workers must be idempotent.
  5. Re-dispatch on cut: retry 3× with backoff (the SAME cadence as the fallback row below), then surface to human with options. Each re-dispatch is cheap: F6 caches partials on disk.

#195 pairing: a killed mid-write worker may leave index.lock / partial worktree state — run the #195 worktree-recovery path before re-dispatching into that worktree.

Bounded nested pi launch — the ONLY sanctioned form

# Bounded nested pi launch — the ONLY sanctioned form of nested pi:
LOG="$(mktemp /tmp/nested-pi.XXXXXX)"
PI_MODE=print TASK_HEARTBEAT=1 pi -p "<prompt>" > "$LOG" 2>&1 &
launch_pid=$!
echo "launched $launch_pid → $LOG"

# Deadline watchdog — portable sleep-deadline + kill -0 liveness probe
# (no GNU timeout on macOS): SIGTERM at 1800s; SIGKILL after a 60s grace.
( sleep 1800; if kill -0 "$launch_pid" 2>/dev/null; then kill "$launch_pid"; pkill -P "$launch_pid" 2>/dev/null; sleep 60; kill -9 "$launch_pid" 2>/dev/null; fi ) &

# Liveness check + abort trigger — [task-heartbeat] markers land on stderr
# every 30s (2>&1 merged). No marker within 60s → the process is dead or
# blocked at the OS level → ABORT. Markers present → the launch is ALIVE and
# the 30-min deadline watchdog is the only remaining bound — do NOT kill here:
sleep 60
if grep -q '\[task-heartbeat\]' "$LOG"; then
  echo "ALIVE — bounded by the 30-min deadline watchdog"
else
  echo "NO MARKER — ABORTING (process dead/blocked at OS level)"
  kill "$launch_pid" 2>/dev/null
  pkill -P "$launch_pid" 2>/dev/null
  echo "ABORTED — surfacing to user"
  exit 1
fi

(docs(skills): portable bounded-launch template + drift guard for the never-unbounded-pi rule (#206))

Anti-Patterns

Anti-Pattern Why It Matters
Dispatching 20+ agents at once Context pressure on fan-in (orchestrator ingests all results) + worktree/merge contention. Cap at 16, batch the rest
No staggered launches Burst dispatch can spike provider load / hit account-level concurrency caps. 200ms stagger smooths load
Sub-agents touching same files Merge conflicts, inconsistent state. Partition file ownership before dispatch
No structured output format Can't deduplicate or sort results. Enforce format in agent prompts
Infinite review cycles Always cap at 10 cycles. Surface remaining issues to human
Waiting for background tasks synchronously Defeats the purpose. Check background results before they're needed
Unbounded nested/background pi launch (no timeout, no log redirect, no liveness marker) A gate-stalled pi can hang the fleet ~29h (#206). Always use the bounded template (PI_MODE=print TASK_HEARTBEAT=1, mktemp log, sleep 1800 + kill -0 watchdog, abort on no-marker); the terminal one-liner is the ONLY guard escape.
No fallback for subagent tool failures If the subagent tool is unreachable (network, API down), the skill hangs. Always have a fallback: retry 3× with backoff, then surface to human with options to (a) retry, (b) proceed sequentially, (c) abort.
Spawning worktrees/branches with no teardown record Aborted dispatch silently orphans them (#195). Record every artifact before dispatch (record-worktree.sh add), remove on clean completion (done), sweep with scan-orphans.sh — abort leaves the record as the teardown manifest.

| Blocking on a cut/dead sub-agent | Parent waits are bounded; a cut must return partials + reason within the watchdog bound, then the orchestrator re-dispatches instead of hanging. |


Continue following the workflow as mandated by this skill. Do not skip steps.