ai-worker-orchestration

Scope: app/jobs/**, app/services/**, config/sidekiq.yml, config/schedule.yml, config/initializers/sidekiq.rb (scheduler specifics: ai-worker-scheduling). Read app/models/workflow.rb, stage.rb and project.rb only for the public methods and enums this layer calls (current_stage, effective_template, skill_list, advance!, status/kind enums, project settings) — don't otherwise touch the model layer (ai-worker-domain) or the UI (ai-worker-ui). The overall design is in docs/ARCHITECTURE.md and docs/RUNBOOK.md; where they and the code disagree, the code wins.

File map (the whole scope)

Scheduling

• config/schedule.yml, config/sidekiq.yml, app/jobs/workflow_scheduler_job.rb, app/services/scheduling/** — the scheduler tick, its eligibility/capacity rules and the concurrency-1 contract: see the ai-worker-scheduling skill. The Jira, Slack-review and Bugsnag sync jobs and the hourly brainstorm job (also in schedule.yml) run on default.
• app/services/stage_executor.rb — runs one workflow's current stage and never waits on the agent: call polls a running agent when workflow.processing?, else hands off to Stages::StartHandler (per-stage start paths and start_ai_run, which detaches the agent and records run state on the workflow), poll_ai_run hands the run to AiCli::RunPoller, and call rescues every Git::CommandError / Github::CommandError / AiCli::CommandError / ArgumentError into fail_stage, which logs a failed StageRun — never let one bubble up and crash the Sidekiq job. Everything after the run lives in app/services/stages/ (below). Per-stage start/finish touch points: see ai-worker-stage-types.
• app/services/stages/start_handler.rb — start path per stage type (branch/checkout setup, PR-check decision, start_ai_run). app/services/stages/completion_handler.rb — success path per stage type (finish_*): decides the next workflow status, logs the StageRun, best-effort follow-ups. stages/failure_handler.rb — failed-stage logging, the MAX_CONSECUTIVE_FAILURES cap that parks a workflow as errored. stages/pr_check_decision.rb (pure PR-check decision), stages/local_test_policy.rb (pure local-test retry/outcome rules), stages/review_handoff.rb (best-effort reviewer Slack ping and auto-merge), stages/workflow_run_gate.rb (runbook wait_for_workflow gate).
• app/services/ai_cli/run_poller.rb — non-blocking finished/running/stale decision for a detached run (kills stale ones); ai_cli/run_logs.rb — per-workflow agent log naming (log/ai_runs/workflow-<id>-<unix>.log).

Agent CLI and prompts

• app/services/ai_cli/runner.rb — the single place that launches the agent CLI (claude, cursor agent, or codex; the app-wide agent is read from Rails.cache, falling back to the AGENT env var). It is not blocking: start detaches the process, redirects output to a log file and returns its pid; poll is a non-blocking wait that returns :running or the exit status; tail_new_lines logs new log output across ticks. It also handles per-run options (allowed/disallowed tools, extra MCP config, env vars, model/effort).
• app/services/ai_cli/browser_test_run.rb — per-run setup for the browser-testing stages (uat_test, local_test): credential checks, Playwright MCP config and output dir, credential env vars, disallowed tools.
• app/services/ai_cli/cursor_mcp_file.rb — writes the run's MCP servers to <cwd>/.cursor/mcp.json for the cursor agent and restores the original file when the run ends.
• app/services/ai_cli/detached_run.rb — state machine (pid/start/log/result file in a tmp/ dir) shared by the detached-agent sync services below.
• app/services/ai_cli/output.rb — parses the agent's final output for the marker lines stages rely on (VAR:, JIRA_RESULT, UAT_RESULT, LOCAL_TEST_RESULT, retest/feedback blocks, POSTED, fixed summary). ai_cli/command_error.rb, git/command_error.rb, github/command_error.rb — one error class per service.
• app/services/prompts/stage_prompt.rb — builds the final prompt text for a stage (stage prompt, else assigned/default template, else a built-in default; plus conflict prompt, Slack-message wrapper and review-fix additions). app/services/prompts/renderer.rb — the only place {TOKEN} placeholders and workflow variables are substituted.

Git / GitHub

• app/services/git/branch_service.rb — all git operations in the project's clone: create/resume a workflow's branch from fresh main, merge main into it (reporting conflicts), check out a remote branch, detect/abort an in-progress merge.
• app/services/github/client.rb — facade (Github::Client.new(project)) delegating to github/reader.rb (query-only: finding a PR, PR info, merged/review-decision/checks state, Actions run state) and github/writer.rb (requesting review, enabling auto-merge, checking out fork PRs). github/runner.rb builds the gh argv in the project's clone and raises Github::CommandError; Reader and Writer share it. PR creation is never done here — the agent does it during the implementation run, per its prompt.

Detached-agent sync services (same shape: a state directory under tmp/ holding pid, start time, result file and log; start spawns an agent via AiCli::Runner; running?/poll read back the result file once the process ends and raise the module's own FetchError on a bad or missing result; import creates records idempotently; blockers lists why it can't run, on the fetcher or on its sync wrapper)

• app/services/jira/agent_fetcher.rb + jira/ticket_sync.rb — fetch a project's assigned Jira tickets through the agent's Atlassian MCP and create workflows for ones not yet imported. app/jobs/jira_sync_job.rb drives it.
• app/services/slack/mention_fetcher.rb + slack/review_sync.rb — fetch Slack mentions containing PR URLs (agent with a Slack-read-only tool allowlist) and create PR-review workflows (skipping closed, own or already-imported PRs). app/jobs/slack_review_job.rb drives it and enqueues the scheduler right after creating workflows.
• app/services/agent_runs/sync_cycle.rb — one poll → import → start pass for a record, shared by the sync jobs; records expected errors so later records still run.
• app/services/slack/review_notifier.rb — pings reviewers by starting a short detached agent run (haiku, only the Slack MCP send-message tool allowed) that posts the message and prints its permalink; slack/ping_collector.rb reads the permalink back into Workflow#slack_review_thread_ts. State under tmp/slack_pings/. Optional and best-effort (Slack::NotifyError, rescued by Stages::ReviewHandoff).
• app/services/slack/web_client.rb — minimal bot-token chat.postMessage client; currently has no caller in app/ (only its spec), so don't route new Slack work through it.
• app/services/slack/review_candidate.rb — pure "should this PR from a mention become a review workflow" rules used by slack/review_sync.rb.
• app/services/bugsnag/triager.rb — agent run that files Jira Bugs for new Bugsnag errors and records them in the sync's ledger. app/jobs/bugsnag_sync_job.rb drives it (weekday, once per day for auto-sync ones).
• app/services/brainstorm/planner.rb + app/jobs/brainstorm_job.rb — goal-driven, read-only agent that proposes workflows for personal projects, only while the project has no pending workflow. brainstorm/prompt.rb builds its prompt; brainstorm/proposal_normalizer.rb cleans its result rows.
• Adding or changing one of these (fetcher on AiCli::DetachedRun, sync wrapper, job, schedule entry, specs): see the ai-worker-detached-agent-runs skill.
• Each sync job imports a finished result first, then starts a new run only if none is running and its own trigger rule holds (last fetch old enough for Jira/Slack, once per weekday after a set hour for Bugsnag, no pending workflow for brainstorm); fetch errors are logged and recorded on the sync/project as its "last result".
• app/jobs/application_job.rb — Rails default base class.

Conventions

• Never build a shell command as a cd X && ... string. Subprocesses use Open3.capture3(*command, chdir: dir) (Git/Github services) or Process.spawn(..., chdir:) with an argument array (AiCli::Runner). Never interpolate untrusted text (ticket text, Slack text, PR titles) into a shell string — pass it as an argument or in the prompt/env.
• Each service raises its own CommandError subclass on a non-zero exit instead of returning a boolean. StageExecutor is the one place that turns these into a failed StageRun; the sync jobs rescue their own FetchError / AiCli::CommandError. Don't rescue inside the Git/Github/Runner objects.
• Agent runs are detached and polled across ticks; a job must never block on one. State that lets a later tick find the run lives on the workflow (processing, pid, log path, start time, log offset) or in the sync's tmp/ directory. Everything that starts a run must record that state; everything that finishes one must clear it.
• Scheduler concurrency rules (per-clone, MAX_CONCURRENT_AGENTS, head-of-line, Sidekiq concurrency 1): see ai-worker-scheduling.
• Every code path in StageExecutor that attempts work or reaches an outcome logs a StageRun whose action is in StageRun::ACTIONS. Add the new action to that constant (domain skill) before logging it — an unlisted action fails the model validation and turns into a crash or a bogus failure.
• A failed stage stays in its current workflow status so the next tick retries it; after MAX_CONSECUTIVE_FAILURES the workflow becomes errored, which the scheduler ignores until a person resumes it.
• Agents are given the least tool access that works (tool allowlists for the Slack, Bugsnag and brainstorm runs; denied gh workflow run/gh api for UAT/local tests). Secrets (UAT/local logins) reach the agent only as environment variables, never in prompts or logs; the review ping's agent gets only the Slack send-message tool. Jira, Slack and Bugsnag access goes through the agent's own MCP connectors, so the app holds no tokens for them.
• Test with RSpec, not scratch runners. Put specs under spec/services/** or spec/jobs/** (stub Open3, AiCli::Runner and Github::Client, or use temp directories and temp git repos) and run RAILS_ENV=test bundle exec rspec. Never run scratch scripts, bin/rails runner or console code against the development DB, and never start a real agent against a real project to "try it".