Workflow Audit

Purpose

Enumerate, for one orchestration, the following five findings from the records the host wrote under workspace/orchestrations/<orchestration_id>/.

1. Terminal state and failure diagnosis — every fail / fail_closed transition, and the host's own failure_analysis.json.
2. Redos — substeps attempted more than once, and runs rejected at terminal validation.
3. Sandbox enforcement violations — the records under violations/.
4. Token cost per leaf — the usage rows of agent_runs.jsonl.
5. Dangling launch — an active_child window left open by a driver that died mid-launch.

The audit reads in-repo records only. The one read outside the checkout is the leaf transcript tail that finding 5 correlates, which tools/audit_orchestration.py performs itself (## Transcripts).

Scope

• Run from an operator terminal at the checkout root. The audit is not part of a workflow run and is never launched by one.
• Read-only. No artifact under workspace/ is modified.
• Backend-neutral: every record the audit reads is written by the host, so a claude_cli, codex_cli, openai_compatible or anthropic_api leaf is audited by the same command.

Canonical sources

• Record placement and the meaning of each file: docs/WORKSPACE_LAYOUT.md.
• The tool's arguments, output formats and exit codes: python3 tools/audit_orchestration.py --help (docs/CLI_REFERENCE.md makes --help canonical for this tool).
• The dangling-launch section and the recovery it leads to: docs/RUNBOOK.md §"Incomplete launch recovery" ({#launch-incomplete-recovery}). Exit code 2 is defined there and in --help; this document does not restate it.

Procedure

Step 1 — Fix the orchestration_id

ls workspace/orchestrations/

Take the instructed orchestration_id, or the newest orch_YYYYMMDDTHHMMSSZ_* directory when none is instructed.

Step 2 — Run the tool

python3 tools/audit_orchestration.py --orchestration-id <orchestration_id> --format markdown

--format json returns the same content as one document for a script. Exit code 2 means the report cannot be trusted at face value; read the ## ⚠ sections first and follow docs/RUNBOOK.md §"Incomplete launch recovery" for the cause before concluding anything from the other sections.

Step 3 — Read the sections

Each section states one conclusion and names the record to open for the detail.

• ## Phase state failures — every fail / fail_closed transition (a set_status row) with its reason_code and reason_detail, and the fail_closed instant. The row is orchestration-level and names no node or step; which agent run failed is the next section's. Detail: the matching line of phase_state_log.jsonl.
• ## failure_analysis — the host's diagnosis: reason_code, the failed agent run (agent_run_id, node, step, substep), the failed step_result.json paths, and each sidecar with its existing_file_status. The line carries orchestration_meta.json#status so that an absent file is read against the terminal status: absent on a pass run is normal, absent on a fail / fail_closed run is a finding. Detail: launches/<agent_run_id>.reply.txt, agents/<agent_run_id>/dialogs/agent.result.json, and the named step_result.json.
• ## Dangling launch (active_child window) — read as docs/RUNBOOK.md §"Incomplete launch recovery" instructs.
• ## Sandbox enforcement violations — the record that bwrap confinement was in force, or that it failed. One writer, five reasons: sandbox_profile_build_failed at record-launch; sandbox_runtime_not_bwrap, sandbox_not_enforced, sandbox_profile_missing and sandbox_profile_not_found at record-agent-run. bwrap is the only confinement a leaf runs under, so any of the five is a finding about that leaf's launch. Three states are rendered and must be read apart: violations/ absent (nothing recorded — the directory is created only when a violation occurs), directory present with no record (pre-created by a run before issue #171 PR-2), and sandbox enforcement records present (counted per reason, then listed with agent_run_id and evaluated_at). A record of another kind is listed apart under its kind and is not a sandbox enforcement finding. Detail: violations/<agent_run_id>.sandbox_enforcement_violation.json, and the leaf's sandbox_profiles/<agent_run_id>.json.
• ## Token cost (per leaf) — the leaf total, its reasoning and prompt-cache shares, the provider-reported cost, and a per-leaf table ranked by total. not_measured rows are deterministic in-process substeps that launched no leaf and are not a gap; unavailable rows are leaves whose usage channel failed and are a gap. docs/GLOSSARY.md §per-leaf usage defines both markers.
• ## Pure-leaf A/B metrics — per node, the attempt and repair-turn counts and the summed usage of each pure leaf. A row's result= is the LOOP outcome (a schema-valid document was obtained); a reviewer row's verdict= is what that document decided, read from the projection beside the record (ir_meta.json / source_meta.json#verification_status, semantic_review.json#decision) — so a rejecting reviewer reads result=pass, verdict=fail. When a retry route has since revoked the artifact, the reviewer's own decision is still what verdict= shows (read from prior_verification_status) and the route's reason and agent run follow it — verdict=pass (revoked: validate_execute_post_execute_violation by arid …) is a reviewer that ACCEPTED and a downstream failure that reopened the phase, not a rejection. Resolve that arid against THIS run's agent_runs.jsonl before attributing the revocation to this run: the generate rows list every source directory under the node's pipeline, which the node's orchestrations share, and a later run revokes an earlier run's artifact too, by a conductor route on a cold re-run or by an operator's revoke-artifact by hand (docs/RUNBOOK.md §3-1) — an arid this run does not hold means the revocation is another run's. verdict=none means no projection was written — the reviewer loop ended with no accepted document, by budget exhaustion or a transport death, and the row's failure: says which — for generate / validate; on compile the same end reads verdict=pending, the status the producer wrote before the reviewer ran. Count rejections by verdict=, never by result= and never by the revocation. Detail (per attempt): the *_meta.json next to the node's artifact (docs/WORKSPACE_LAYOUT.md).
• ## agent_runs summary — the run count per status, the rows missing finished_at, and Repeated substeps — every (node_key, step, substep) with more than one attempt, one line per attempt with its status and agent_run_id in start order (a step row such as build has no substep and is listed by its step). Detail: launches/<agent_run_id>.reply.txt of each attempt. A run rejected at terminal validation (agent_runs_invalid.jsonl) is counted under its status, typically fail.

Step 4 — Report

Report in three groups, each item with its cause and its final result.

1. Terminal state and failure diagnosis — the terminal status, the reason_code, the failed agent run and the record that explains it.
2. Redos — each repeated substep in chronological order: how many attempts, which statuses, and what changed between them (from the launch replies of the attempts).
3. Sandbox enforcement — each violation with its reason and agent_run_id, or the statement that none was recorded (naming which of the three negative sentences the tool printed: directory absent, directory present with no record, or records of another kind only).

Close with one line on cost: the leaf total and the most expensive leaf.

Transcripts

The audit needs no transcript. A transcript is the raw stream a CLI leaf's backend kept; everything a pure-function leaf received is launches/<agent_run_id>.prompt.txt, and everything it returned is launches/<agent_run_id>.reply.txt and agents/<agent_run_id>/dialogs/. A pure leaf launches with no tool, so a transcript holds no tool call to inspect. Where one exists, per backend:

• claude_cli: ~/.claude/projects/<cwd-slug>/<agent_run_id>.jsonl, where <cwd-slug> is the checkout's absolute path with / replaced by -, and the session id is the leaf's agent_run_id. The CLI expires sessions, so a transcript is often absent for a run older than the retention window; tools/audit_orchestration.py locates it through tools/operator_private_root.py::claude_leaf_projects_roots for the dangling-launch section and reads it for nothing else.
• HTTP providers (openai_compatible / anthropic_api): no transcript exists. The raw provider response is launches/<agent_run_id>.http_response.txt.
• codex_cli: <orchestration_meta.json#codex_workflow_home>/<codex_lineage_id>/sessions/YYYY/MM/DD/rollout-*-<agent_session_id>.jsonl — the thread's own lineage home under the orchestration's durable container ~/.atmofab/homes/<orchestration_id>/codex/ (docs/RUNBOOK.md §"The operator-private root"; tools/prune_workflow_homes.py is the only thing that removes it). Both agent_session_id and codex_lineage_id are read from the leaf's session_run_index.json row, which maps agent_run_id to them (the launch response launches/<agent_run_id>.response.json carries codex_lineage_id too); every attempt of one thread shares the lineage id, so a warm repair's rollout is in the same file tree as the attempt it resumed. A run recorded before issue #245 has rows with no codex_lineage_id and its rollouts directly under <codex_workflow_home>/sessions/… (one home per orchestration). ~/.codex/sessions holds the operator's own sessions, not a leaf's.

Notes

• The conductor is a Python process with no session of its own; the row with agent_role: orchestration in agent_runs.jsonl is the conductor's own record and is excluded from the per-leaf cost.
• A transcript can be tens of thousands of lines. Extract the needed fields rather than reading from the top.