jankneumann

validate-feature

Deploy locally, run security scans and behavioral tests, check CI/CD, and verify OpenSpec spec compliance

jankneumann 4 1 Actualitzat fa 1 mes

Recursos

1
GitHub

Instal·la

npx skillscat add jankneumann/agentic-coding-tools/validate-feature

Instal·la des del registre de SkillsCat.

SKILL.md

Validate Feature

Durable state artifact authority

Shared holder, writer, authority, fallback, and rehydration semantics live in docs/guides/state-artifacts.md. The procedures below retain this skill's phase-specific commands and gates.

Deploy the feature locally with DEBUG logging, run security scans and behavioral tests against live services, check CI/CD status, and verify OpenSpec spec compliance. Produces a structured validation report and posts it to the PR.

Arguments

$ARGUMENTS - OpenSpec change-id (required), optionally followed by flags:

  • --skip-e2e or --skip-playwright — skip the Playwright E2E phase
  • --skip-ci — skip the CI/CD status check
  • --skip-security — skip the Security Scan phase
  • --phase <name>[,<name>] — run only specified phases (e.g., --phase smoke,security)
  • --ephemeral — run write-capable validation steps in a disposable detached worktree
  • --include-dirty — with --ephemeral, materialize staged, unstaged, and untracked source state instead of refusing a dirty checkout

Valid phase names: deploy, smoke, gen-eval, security, e2e, architecture, spec, logs, ci

Prerequisites

  • Feature branch exists with implementation commits (default openspec/<change-id>, or the operator-mandated branch when OPENSPEC_BRANCH_OVERRIDE is set)
  • A usable container runtime (docker or podman, daemon answering info) for the Deploy phase. Detection is the same as DockerStackEnvironment._detect_runtime: PATH presence is not enough.
  • Approved OpenSpec proposal exists at openspec/changes/<change-id>/
  • Run /implement-feature first if no implementation exists

Provider-Neutral Dispatch

When validation delegates checks or evidence review, treat the provider-neutral dispatch adapter
as the canonical cross-provider path. Claude Code, Codex, Antigravity, Grok, and
Pi are first-class providers when configured; Claude-specific harness
examples are adapter internals, with inline validation as the fallback.

OpenSpec Execution Preference

Use OpenSpec-generated runtime assets first, then CLI fallback:

  • Claude: .claude/commands/opsx/*.md or .claude/skills/openspec-*/SKILL.md
  • Codex: .codex/skills/openspec-*/SKILL.md
  • Fallback: direct openspec CLI commands

Coordinator Integration (Optional)

Use docs/coordination-detection-template.md as the shared detection preamble.

  • Detect transport and capability flags at skill start
  • Execute hooks only when the matching CAN_* flag is true
  • If coordinator is unavailable, continue with standalone behavior

Local CLI Mutation Boundary

Validation writes reports, evidence, logs, and sometimes follow-up artifacts. In
local CLI execution, validation MUST run inside the feature worktree or another
managed worktree before the first write:

eval "$(python3 "<skill-base-dir>/../worktree/scripts/worktree.py" setup "$CHANGE_ID")"
cd "$WORKTREE_PATH"
python3 "<skill-base-dir>/../shared/checkout_policy.py" require-mutation

Read-only CI status checks may inspect from the shared checkout, but any report
or evidence file must be written in a worktree so it lands on the PR branch.
--ephemeral adds a disposable validation checkout inside this managed feature
worktree boundary; it never makes the shared checkout writable.

Steps

0. Detect Coordinator and Recall Memory

At skill start, run the coordination detection preamble and set:

  • COORDINATOR_AVAILABLE
  • COORDINATION_TRANSPORT (mcp|http|none)
  • CAN_LOCK, CAN_QUEUE_WORK, CAN_HANDOFF, CAN_MEMORY, CAN_GUARDRAILS

If CAN_MEMORY=true, recall relevant validation history:

  • MCP path: recall
  • HTTP path: "<skill-base-dir>/../coordination-bridge/scripts/coordination_bridge.py" try_recall(...)

On recall failure/unavailability, continue with validation and log informationally.

1. Determine Change ID and Configuration

# Parse change-id from argument or current branch
BRANCH=$(git branch --show-current)
CHANGE_ID=${ARGUMENTS%% --*}  # Everything before first flag
CHANGE_ID=${CHANGE_ID:-$(echo $BRANCH | sed 's/^openspec\///')}

# Detect worktree context and resolve OpenSpec path
# Note: detect auto-discovers context from the working directory;
# agent-id information is available via the worktree registry if needed.
eval "$(python3 "<skill-base-dir>/../worktree/scripts/worktree.py" detect)"
PROJECT_ROOT="${MAIN_REPO:-$(git rev-parse --show-toplevel)}"

Parse flags from $ARGUMENTS:

  • --skip-e2e or --skip-playwright → set SKIP_E2E=true
  • --skip-ci → set SKIP_CI=true
  • --skip-security → set SKIP_SECURITY=true
  • --phase <names> → set PHASES to comma-separated list; only run those phases
  • --ephemeral → set EPHEMERAL=true
  • --include-dirty → set INCLUDE_DIRTY=true; reject it unless EPHEMERAL=true

If --phase is provided, only the listed phases execute. If --phase includes phases other than deploy, assume services are already running (skip deploy and teardown).

2. Verify Prerequisites

# Resolve the expected feature branch — honors registry + OPENSPEC_BRANCH_OVERRIDE
eval "$(python3 "<skill-base-dir>/../worktree/scripts/worktree.py" resolve-branch "$CHANGE_ID")"
FEATURE_BRANCH="$BRANCH"

# Verify on feature branch
CURRENT_BRANCH="$(git branch --show-current)"
if [[ "$CURRENT_BRANCH" != "$FEATURE_BRANCH" ]]; then
  echo "ERROR: on '$CURRENT_BRANCH' but expected '$FEATURE_BRANCH' (source: $BRANCH_SOURCE)" >&2
  exit 1
fi

# Verify proposal exists
openspec show $CHANGE_ID

# Verify implementation commits exist
COMMIT_COUNT=$(git log --oneline main..HEAD | wc -l)
if [ "$COMMIT_COUNT" -eq 0 ]; then
  echo "ERROR: No implementation commits found on this branch."
  echo "Run /implement-feature $CHANGE_ID first."
  exit 1
fi

# Classify deployable surface (issue #432). Container-dependent phases are
# required only when this change has a running service; a skills/docs/openspec
# change records those phases as not applicable rather than skipped.
CHANGE_DIR="$(git rev-parse --show-toplevel)/openspec/changes/$CHANGE_ID"
SURFACE_JSON=$(python3 "<skill-base-dir>/scripts/gate_logic.py" --describe-surface \
  --change-dir "$CHANGE_DIR")
echo "$SURFACE_JSON"
DEPLOYABLE=$(printf '%s' "$SURFACE_JSON" | python3 -c \
  'import json,sys; print("true" if json.load(sys.stdin)["deployable"] else "false")')

# Check for a usable container runtime only when Deploy will run.
# Same predicate as DockerStackEnvironment._detect_runtime: PATH presence is
# not enough — `docker info` / `podman info` must succeed. Prefer docker when
# both work; fall through to podman when docker is installed but its daemon
# is down (issue #433).
if [ "$DEPLOYABLE" = "true" ]; then
  if RUNTIME=$(python3 "<skill-base-dir>/scripts/environments/docker_stack.py" --detect); then
    echo "Container runtime is available: $RUNTIME"
  else
    echo "ERROR: no usable container runtime (docker or podman)."
    echo "  A binary on PATH is not sufficient — its daemon must answer \`info\`."
    echo "  macOS: brew install --cask docker   OR   brew install podman"
    echo "  Linux: sudo systemctl start docker  OR   sudo systemctl start podman"
    exit 1
  fi
else
  echo "No deployable surface — Deploy/Smoke/Security/E2E are not applicable."
  echo "  Record those phases as **Status**: not applicable (not skipped)."
  DEPLOY_NOT_APPLICABLE=true
fi

If not on the feature branch, check out $FEATURE_BRANCH (which honors OPENSPEC_BRANCH_OVERRIDE). If no implementation commits exist, abort with guidance.

2.25. Enter Ephemeral Validation Scope (Optional)

When EPHEMERAL=true, enter the canonical prepare/finalize lifecycle before
Step 2.5. The source is the current feature checkout, not the previously
resolved PROJECT_ROOT (which may name the main repository from inside a
managed worktree). This is an executable shell boundary: Steps 2.5 through 12
run from VALIDATION_PATH; Step 12.5 copies the durable allowlist back and
removes the scratch checkout; Steps 13 and 14 then run from VALIDATION_SOURCE
so PR comments, the session log, and its handoff are durable.

VALIDATION_HELPER="<skill-base-dir>/scripts/validation_worktree.py"
VALIDATION_STATE_FILE=$(mktemp "${TMPDIR:-/tmp}/validate-feature-state.XXXXXX")
INCLUDE_DIRTY_FLAG=""
[ "$INCLUDE_DIRTY" = "true" ] && INCLUDE_DIRTY_FLAG="--include-dirty"

if VALIDATION_PREPARE_OUTPUT=$(python3 "$VALIDATION_HELPER" prepare \
    --source "$PWD" \
    --change-id "$CHANGE_ID" \
    --state-file "$VALIDATION_STATE_FILE" \
    $INCLUDE_DIRTY_FLAG \
    ); then
  validation_json_field() {
    printf '%s' "$VALIDATION_PREPARE_OUTPUT" | \
      python3 -c 'import json,sys; print(json.load(sys.stdin)[sys.argv[1]], end="")' "$1"
  }
  if VALIDATION_SOURCE=$(validation_json_field source) && \
      VALIDATION_PATH=$(validation_json_field path) && \
      VALIDATION_VALIDATED_COMMIT=$(validation_json_field validated_commit) && \
      VALIDATION_VALIDATED_TREE=$(validation_json_field validated_tree); then
    export VALIDATION_SOURCE VALIDATION_PATH
    export VALIDATION_VALIDATED_COMMIT VALIDATION_VALIDATED_TREE
  else
    VALIDATION_PARSE_STATUS=$?
    python3 "$VALIDATION_HELPER" finalize --state-file "$VALIDATION_STATE_FILE" || true
    rm -f -- "$VALIDATION_STATE_FILE"
    echo "ERROR: could not parse ephemeral validation state" >&2
    exit "$VALIDATION_PARSE_STATUS"
  fi
else
  VALIDATION_PREPARE_STATUS=$?
  rm -f -- "$VALIDATION_STATE_FILE"
  echo "ERROR: could not prepare ephemeral validation" >&2
  exit "$VALIDATION_PREPARE_STATUS"
fi

finalize_ephemeral_validation() {
  validation_status="${1:-$?}"
  validation_cleanup_status=0
  trap - EXIT INT TERM
  cd "$VALIDATION_SOURCE" || validation_cleanup_status=$?
  python3 "$VALIDATION_HELPER" finalize --state-file "$VALIDATION_STATE_FILE" || \
    validation_cleanup_status=$?
  if [ "$validation_status" -eq 0 ]; then
    validation_status=$validation_cleanup_status
  fi
  return "$validation_status"
}
trap 'finalize_ephemeral_validation $?' EXIT
trap 'finalize_ephemeral_validation 130; exit 130' INT
trap 'finalize_ephemeral_validation 143; exit 143' TERM

cd "$VALIDATION_PATH"
PROJECT_ROOT="$VALIDATION_PATH"
OPENSPEC_PATH="$VALIDATION_PATH/openspec"

On a dirty checkout, omit --include-dirty to fail closed or pass it explicitly
to reproduce the exact index, working-tree, and untracked state. The helper
records VALIDATION_VALIDATED_COMMIT and VALIDATION_VALIDATED_TREE, copies only
validation-report.md, validation-findings.json, and architecture-impact.md
back to the feature checkout, and removes the scratch worktree even when an
earlier step exits. Under a cloud harness whose environment profile already
provides isolation, it logs a downgrade and runs in place.

2.5. Prepare Validation Artifacts

Preferred path:

  • Use runtime-native verify/continue workflow (opsx:verify equivalent) for artifact guidance.

CLI fallback path:

openspec instructions validation-report --change "$CHANGE_ID"
openspec instructions architecture-impact --change "$CHANGE_ID"
openspec status --change "$CHANGE_ID"

Ensure validation-report.md and architecture-impact.md are updated in the change directory as part of this validation run.

3. Deploy Phase

Phase name: deploy
Criticality: Critical when the change has a deployable surface; not applicable otherwise (issue #432). Do not record skipped for a phase that could never have applied.

if [ "$DEPLOY_NOT_APPLICABLE" = true ]; then
  echo "NOT APPLICABLE: no deployable surface — Deploy is not a skipped check."
  DEPLOY_RESULT="not applicable"
else
# Find docker-compose file
COMPOSE_FILE=$(find "$PROJECT_ROOT" -maxdepth 2 -name "docker-compose.yml" | head -1)

if [ -z "$COMPOSE_FILE" ]; then
  echo "SKIP: No docker-compose.yml found. Skipping Deploy phase."
  echo "  Smoke tests will run against already-running services."
  DEPLOY_SKIPPED=true
else
  COMPOSE_DIR=$(dirname "$COMPOSE_FILE")
  LOG_FILE="/tmp/validate-feature-${CHANGE_ID}-$(date +%s).log"

  echo "Starting services with DEBUG logging..."
  echo "  Compose file: $COMPOSE_FILE"
  echo "  Log file: $LOG_FILE"

  # Start services with DEBUG logging, redirect output to log file.
  #
  # When the compose file gates the API server behind a profile (e.g. the
  # agent-coordinator's `coordinator-api` service uses `profiles: [api]` so it
  # doesn't auto-start during simple `docker compose up`), pass
  # `COMPOSE_PROFILES` so the API process IS started here — without it the
  # smoke + e2e phases get connection-refused on the API port. Multiple
  # profiles can be comma-separated (`api,langfuse`).
  AGENT_COORDINATOR_DB_PORT=${AGENT_COORDINATOR_DB_PORT:-54322} \
  AGENT_COORDINATOR_REST_PORT=${AGENT_COORDINATOR_REST_PORT:-8081} \
  AGENT_COORDINATOR_REALTIME_PORT=${AGENT_COORDINATOR_REALTIME_PORT:-4000} \
  COMPOSE_PROFILES=${COMPOSE_PROFILES:-api} \
  LOG_LEVEL=DEBUG $RUNTIME compose -f "$COMPOSE_FILE" up -d --build 2>&1 | tee "$LOG_FILE"

  # Wait for health checks
  echo "Waiting for services to be healthy..."
  $RUNTIME compose -f "$COMPOSE_FILE" ps

  # Wait for PostgreSQL health check (up to 30 seconds)
  for i in $(seq 1 30); do
    if $RUNTIME compose -f "$COMPOSE_FILE" exec -T postgres pg_isready -U postgres > /dev/null 2>&1; then
      echo "PostgreSQL is ready"
      break
    fi
    sleep 1
  done

  # Wait for REST API health endpoint (up to 30 seconds — the API container
  # may need build time on first run + warmup before /health flips to 200)
  for i in $(seq 1 30); do
    if curl -sf http://localhost:${AGENT_COORDINATOR_REST_PORT:-8081}/health > /dev/null 2>&1; then
      echo "REST API is ready"
      break
    fi
    sleep 1
  done

  # Collect running container logs in background
  $RUNTIME compose -f "$COMPOSE_FILE" logs -f >> "$LOG_FILE" 2>&1 &
  LOG_PID=$!

  DEPLOY_RESULT="pass"
fi
fi

If Deploy fails, report the failure with Docker logs and skip to Teardown.

4. Smoke Phase

Phase name: smoke
Criticality: Critical (stops validation on failure)

Run the reusable pytest smoke test suite against the live services. The suite is configurable via environment variables so it works with any deployed HTTP API.

if [ "$DEPLOY_NOT_APPLICABLE" = true ]; then
  echo "NOT APPLICABLE: no deployable surface — Smoke is not a skipped check."
  SMOKE_RESULT="not applicable"
else
# Configure for the target API (adjust per project)
export API_BASE_URL="${API_BASE_URL:-http://localhost:8000}"
export API_HEALTH_ENDPOINT="${API_HEALTH_ENDPOINT:-/health}"
export API_READY_ENDPOINT="${API_READY_ENDPOINT:-/ready}"
export API_AUTH_HEADER="${API_AUTH_HEADER:-X-Admin-Key}"
export API_AUTH_VALUE="${API_AUTH_VALUE:-$ADMIN_API_KEY}"
export API_PROTECTED_ENDPOINT="${API_PROTECTED_ENDPOINT:-/api/v1/settings/prompts}"
export API_CORS_ORIGIN="${API_CORS_ORIGIN:-http://localhost:5173}"

# Run smoke tests
SKILL_DIR="<skill-base-dir>"
pytest "$SKILL_DIR/scripts/smoke_tests/" -v --tb=short 2>&1
SMOKE_EXIT=$?

if [ $SMOKE_EXIT -eq 0 ]; then
  SMOKE_RESULT="pass"
elif [ $SMOKE_EXIT -eq 5 ]; then
  # Exit code 5 = no tests collected (services not running, all skipped)
  SMOKE_RESULT="skip"
  echo "SKIP: Services not running — smoke tests auto-skipped"
else
  SMOKE_RESULT="fail"
  SMOKE_FAILED=true
fi
fi

The smoke tests cover:

  • Health: Health and readiness endpoints respond with 2xx
  • Auth enforcement: No credentials → 401/403, valid credentials → 2xx, garbage credentials rejected
  • CORS: Preflight returns correct Access-Control-* headers (skipped if CORS not configured)
  • Error sanitization: Error responses don't leak filesystem paths, stack traces, internal IPs, or credentials
  • Security headers: Content-Type set correctly, Server header not overly detailed, no X-Powered-By

If Smoke fails (SMOKE_EXIT != 0 and != 5), stop validation and skip to Teardown.

4b. Gen-Eval Phase (Optional)

Phase name: gen-eval
Criticality: Non-critical (continues on failure)

Run generator-evaluator testing when interface descriptors exist for the project. This phase auto-detects descriptor files and selects between two modes:

  • cli-augmented mode when both an interface descriptor AND an OpenSpec change directory at openspec/changes/<change-id>/specs/ exist. Gen-eval is invoked with --mode cli-augmented --openspec-change <change-id> so the generator seeds scenarios from the change's WHEN/THEN spec blocks.
  • template-only mode (existing fallback) when descriptors exist but no OpenSpec change directory. Requires no CLI or SDK dependencies.
  • Skipped when no descriptors are found.
# Auto-detect gen-eval descriptors
GENEVAL_DESCRIPTORS=$(find "$PROJECT_ROOT" -path "*/evaluation/descriptors/*.yaml" -type f 2>/dev/null)

if [ -z "$GENEVAL_DESCRIPTORS" ]; then
  echo "SKIP: No gen-eval descriptors found. Skipping gen-eval phase."
  GENEVAL_RESULT="skip"
else
  # Mode selection: cli-augmented requires both descriptor AND OpenSpec change dir
  GENEVAL_CHANGE_DIR="$PROJECT_ROOT/openspec/changes/$CHANGE_ID/specs"
  if [ -d "$GENEVAL_CHANGE_DIR" ]; then
    GENEVAL_MODE_FLAGS="--mode cli-augmented --openspec-change $CHANGE_ID"
    GENEVAL_MODE_LABEL="mode=cli-augmented"
    echo "gen-eval: $GENEVAL_MODE_LABEL (descriptor + OpenSpec change present at $GENEVAL_CHANGE_DIR)"
  else
    GENEVAL_MODE_FLAGS="--mode template-only --no-services"
    GENEVAL_MODE_LABEL="mode=template-only"
    echo "gen-eval: $GENEVAL_MODE_LABEL (no OpenSpec change at openspec/changes/$CHANGE_ID/specs/, falling back to template-only)"
  fi

  echo "Running gen-eval testing ($GENEVAL_MODE_LABEL)..."
  GENEVAL_FAILED=false

  for DESCRIPTOR in $GENEVAL_DESCRIPTORS; do
    echo "  Descriptor: $DESCRIPTOR"
    # Resolve the module root (parent of evaluation/) and cd into it.
    # Descriptor is at <project>/evaluation/descriptors/<name>.yaml — 3 levels up.
    GENEVAL_MODULE_ROOT=$(dirname "$(dirname "$(dirname "$DESCRIPTOR")")")
    GENEVAL_PYTHON="$GENEVAL_MODULE_ROOT/.venv/bin/python"
    if [ ! -f "$GENEVAL_PYTHON" ]; then GENEVAL_PYTHON="python3"; fi
    (cd "$GENEVAL_MODULE_ROOT" && GEN_EVAL_DATA_DIR="$GENEVAL_MODULE_ROOT/evaluation" "$GENEVAL_PYTHON" -m gen_eval \
      --descriptor "$DESCRIPTOR" \
      $GENEVAL_MODE_FLAGS \
      --report-format both \
      --output-dir "$PROJECT_ROOT/openspec/changes/$CHANGE_ID" 2>&1)
    GENEVAL_EXIT=$?

    if [ $GENEVAL_EXIT -ne 0 ]; then
      GENEVAL_FAILED=true
      echo "  gen-eval: FAIL for $DESCRIPTOR (exit $GENEVAL_EXIT)"
    else
      echo "  gen-eval: PASS for $DESCRIPTOR"
    fi
  done

  if [ "$GENEVAL_FAILED" = true ]; then
    GENEVAL_RESULT="fail"
    echo "Gen-eval: FAIL — One or more descriptors had failures (non-blocking)"
  else
    GENEVAL_RESULT="pass"
    echo "Gen-eval: PASS — All descriptors passed"
  fi
fi

Gen-eval failures are non-critical and do not block validation. Results are included in the validation report for informational purposes. cli-augmented mode failures (e.g., from prompt-injection attempts caught by the parser) still degrade gracefully — the validate-feature pipeline continues to subsequent phases.

5. Security Phase

Phase name: security
Criticality: Non-critical (continues on failure)

Run security scanners (OWASP Dependency-Check and ZAP) against the live deployment using the existing security-review orchestrator.

if [ "$DEPLOY_NOT_APPLICABLE" = true ]; then
  echo "NOT APPLICABLE: no deployable surface — Security is not a skipped check."
  SECURITY_RESULT="not applicable"
elif [ "$SKIP_SECURITY" = true ]; then
  echo "SKIP: Security phase skipped (--skip-security flag)"
  SECURITY_RESULT="skip"
else
  echo "Running security scans against live deployment..."

  # Invoke the security-review orchestrator with the live API target
  python3 "<skill-base-dir>/../security-review/scripts/main.py" \
    --repo . \
    --out-dir docs/security-review \
    --zap-target "http://localhost:${AGENT_COORDINATOR_REST_PORT:-3000}" \
    --change "$CHANGE_ID" \
    --allow-degraded-pass 2>&1
  SECURITY_EXIT=$?

  if [ $SECURITY_EXIT -eq 0 ]; then
    # Exit 0 under --allow-degraded-pass can mean "no findings" OR "a scanner
    # never ran and we passed anyway". Those are different facts, so read the
    # gate reasons rather than trusting the exit code alone (D6).
    if grep -q "DEGRADED" docs/security-review/gate.json 2>/dev/null; then
      SECURITY_RESULT="DEGRADED"
      SECURITY_NOT_CHECKED=$(python3 -c "import json,sys; print('; '.join(r for r in json.load(open('docs/security-review/gate.json')).get('reasons', []) if 'DEGRADED' in r))" 2>/dev/null)
      echo "Security: DEGRADED — ${SECURITY_NOT_CHECKED:-a scanner did not run; coverage incomplete}"
    else
      SECURITY_RESULT="pass"
      echo "Security: PASS — No threshold findings"
    fi
  elif [ $SECURITY_EXIT -eq 10 ]; then
    SECURITY_RESULT="fail"
    echo "Security: FAIL — Threshold findings detected"
  elif [ $SECURITY_EXIT -eq 11 ]; then
    SECURITY_RESULT="DEGRADED"
    echo "Security: DEGRADED (INCONCLUSIVE) — Scanners could not run (check prerequisites)"
  else
    SECURITY_RESULT="fail"
    echo "Security: ERROR — Unexpected exit code $SECURITY_EXIT"
  fi
fi

The Security phase reuses the /security-review skill's scripts without requiring a separate invocation. The --allow-degraded-pass flag ensures missing prerequisites (Java, container runtime) degrade gracefully instead of blocking validation.

Write $SECURITY_RESULT into validation-report.md verbatim — including DEGRADED
— together with a one-line "what was not checked and why". DEGRADED is not a
pass: gate_logic.py blocks the pre-merge gate on a DEGRADED required phase unless the
operator passes --accept-degraded Security, and that override is echoed into the gate
summary:

# Blocks: Security could not be checked
python3 "<skill-base-dir>/scripts/gate_logic.py" openspec/changes/"$CHANGE_ID"/validation-report.md

# Proceeds, recording the override in the gate summary
python3 "<skill-base-dir>/scripts/gate_logic.py" openspec/changes/"$CHANGE_ID"/validation-report.md \
  --accept-degraded Security

6. E2E Phase

Phase name: e2e
Criticality: Non-critical (continues on failure)

if [ "$DEPLOY_NOT_APPLICABLE" = true ]; then
  echo "NOT APPLICABLE: no deployable surface — E2E is not a skipped check."
  E2E_RESULT="not applicable"
elif [ "$SKIP_E2E" = true ]; then
  echo "SKIP: E2E phase skipped (--skip-e2e flag)"
  E2E_RESULT="skip"
else
  # Check if pytest-playwright is installed
  if python3 -c "import playwright" 2>/dev/null; then
    PLAYWRIGHT_AVAILABLE=true
  else
    PLAYWRIGHT_AVAILABLE=false
  fi

  # Check if E2E tests exist
  E2E_DIR=$(find "$PROJECT_ROOT" -path "*/tests/e2e" -type d | head -1)

  if [ -z "$E2E_DIR" ]; then
    echo "SKIP: No tests/e2e/ directory found. Skipping E2E phase."
    E2E_RESULT="skip"
  elif [ "$PLAYWRIGHT_AVAILABLE" = false ]; then
    echo "DEGRADED: E2E tests were NOT CHECKED because pytest-playwright is unavailable. To install:"
    echo "  pip install pytest-playwright"
    echo "  playwright install chromium"
    E2E_RESULT="DEGRADED"
  else
    echo "Running E2E tests from $E2E_DIR..."
    pytest "$E2E_DIR" -v --tb=short 2>&1
    E2E_EXIT=$?

    if [ $E2E_EXIT -eq 0 ]; then
      E2E_RESULT="pass"
    else
      E2E_RESULT="fail"
    fi
  fi
fi

6b. Architecture Diagnostics Phase

Phase name: architecture
Criticality: Config-ratcheted via gates.architecture.mode in architecture.config.yaml
(OpenSpec introduce-fitness-function-gates, D4)

gates.architecture.mode Effect
advisory (shipped default, and the fallback when the config file is absent or unreadable) Findings are reported prominently in validation-report.md with their severities; the phase is not in REQUIRED_PHASES and never fails a run.
blocking "Architecture" joins REQUIRED_PHASES in gate_logic.py; a new dependency cycle (severity_thresholds.new_cycle: critical) fails the pre-merge gate.

The flip to blocking is a deliberate one-line config change made after
clean_runs_before_flip (3) clean advisory runs, recorded with a date and rationale —
not something a validation run decides for itself. Report the findings either way:

python3 -c "import sys; sys.path.insert(0, '<skill-base-dir>/scripts'); \
  import gate_logic; print(gate_logic.architecture_mode())"

Run the baseline architecture diff, flow validation, and structural linters
against the changed files. The diff producer runs first so new dependency cycles
exist as findings before gate_logic.architecture_status() evaluates the phase:

# Ensure architecture artifacts are current, immediately before the first read.
# `--ensure` is `--check` plus a staged refresh only when the check is not fresh,
# so on an already-fresh checkout it writes nothing. PYTHON must name the same
# interpreter this repository's architecture targets use: the check runs in-process
# and the pipeline runs in a subprocess, and if the two disagree about which
# optional grammars are importable they report permanent, unfixable drift.
ARCH_PY="${PYTHON:-python3}"
if "$ARCH_PY" "<skill-base-dir>/../refresh-architecture/scripts/run_architecture.py" --ensure --python "$ARCH_PY"; then
  ARCH_FRESHNESS="ensured"
else
  ARCH_FRESHNESS="DEGRADED"
  echo "DEGRADED: architecture artifacts could not be made current; the last known-good analysis is left intact but unverified. Report every architecture-derived finding below as unverified rather than as current." >&2
fi

# Get changed files relative to main
CHANGED_FILES=$(git diff --name-only main...HEAD | tr "\n" ",")

# --- Sub-phase 0: Baseline graph diff (new-cycle producer) ---
ARCH_BASE_SHA=$(git merge-base main HEAD)
ARCH_DIFF="docs/architecture-analysis/architecture.diff.json"
if [ -f Makefile ] && [ -f "docs/architecture-analysis/architecture.graph.json" ]; then
  echo "Running architecture baseline diff against $ARCH_BASE_SHA..."
  if make architecture-diff BASE_SHA="$ARCH_BASE_SHA"; then
    ARCH_NEW_CYCLES=$(python3 -c "import json; d=json.load(open(\"$ARCH_DIFF\")); print(d[\"summary\"][\"new_cycles\"])")
    ARCH_DIFF_RESULT=$(python3 -c "import json,sys; sys.path.insert(0, \"<skill-base-dir>/scripts\"); import gate_logic; d=json.load(open(\"$ARCH_DIFF\")); findings=[{\"category\": \"new_cycle\", \"description\": \"New dependency cycle: \" + \" -> \".join(c)} for c in d[\"details\"][\"new_cycles\"]]; print(gate_logic.architecture_status(findings))")
    echo "Architecture diff: $ARCH_NEW_CYCLES new cycle(s); gate status: $ARCH_DIFF_RESULT"
  else
    echo "DEGRADED: Architecture baseline diff was NOT CHECKED because the producer failed"
    ARCH_DIFF_RESULT="DEGRADED"
  fi
else
  echo "DEGRADED: Architecture baseline diff was NOT CHECKED because Makefile or graph artifacts are unavailable"
  ARCH_DIFF_RESULT="DEGRADED"
fi

# --- Sub-phase 1: Flow validation (validate_flows.py) ---
if [ -f "<skill-base-dir>/../validate-flows/scripts/validate_flows.py" ] && [ -f "docs/architecture-analysis/architecture.graph.json" ]; then
  echo "Running architecture flow validation on changed files..."
  # Scoped run: omit --output so the validator writes the scoped artifact
  # (architecture.diagnostics.scoped.json) rather than the committed full-scope
  # architecture.diagnostics.json, which only refresh-architecture full run
  # should produce.
  ARCH_DIAGNOSTICS="docs/architecture-analysis/architecture.diagnostics.scoped.json"
  python3 "<skill-base-dir>/../validate-flows/scripts/validate_flows.py" \
    --graph docs/architecture-analysis/architecture.graph.json \
    --files "$CHANGED_FILES" 2>&1
  ARCH_EXIT=$?

  if [ $ARCH_EXIT -eq 0 ]; then
    FLOW_RESULT="pass"
    ARCH_ERRORS=$(python3 -c "import json; d=json.load(open(\"$ARCH_DIAGNOSTICS\")); print(d[\"summary\"][\"errors\"])" 2>/dev/null || echo 0)
    ARCH_WARNINGS=$(python3 -c "import json; d=json.load(open(\"$ARCH_DIAGNOSTICS\")); print(d[\"summary\"][\"warnings\"])" 2>/dev/null || echo 0)
    if [ "$ARCH_ERRORS" -gt 0 ]; then
      FLOW_RESULT="fail"
    elif [ "$ARCH_WARNINGS" -gt 0 ]; then
      FLOW_RESULT="warn"
    fi
  else
    FLOW_RESULT="fail"
  fi
else
  echo "DEGRADED: Architecture flow validation was NOT CHECKED (missing scripts or artifacts)"
  echo "  ARCH_FRESHNESS=$ARCH_FRESHNESS — re-run the ensure call above to generate them"
  FLOW_RESULT="DEGRADED"
fi

# --- Sub-phase 2: Structural linters (dependency direction, file-size, naming) ---
echo "Running structural architecture linters..."
LINTER_OUTPUT=$(python3 "<skill-base-dir>/scripts/run_architecture_linters.py" \
  --files "$CHANGED_FILES" 2>&1)
LINTER_EXIT=$?

LINTER_FINDINGS=$(echo "$LINTER_OUTPUT" | head -1)  # JSON on stdout
if [ $LINTER_EXIT -eq 0 ]; then
  LINTER_RESULT="pass"
else
  LINTER_RESULT="fail"
fi

echo "Structural linters: $LINTER_RESULT"

# Aggregate without allowing a later passing sub-phase to erase an earlier
# failure or unavailable checker.
ARCH_RESULT="pass"
for SUBPHASE_RESULT in "$ARCH_FRESHNESS" "$ARCH_DIFF_RESULT" "$FLOW_RESULT" "$LINTER_RESULT"; do
  if [ "$SUBPHASE_RESULT" = "fail" ]; then
    ARCH_RESULT="fail"
  elif [ "$SUBPHASE_RESULT" = "DEGRADED" ] && [ "$ARCH_RESULT" != "fail" ]; then
    ARCH_RESULT="DEGRADED"
  elif [ "$SUBPHASE_RESULT" = "warn" ] && [ "$ARCH_RESULT" = "pass" ]; then
    ARCH_RESULT="warn"
  fi
done

Render architecture.diff.json in validation-report.md before running the hard
gate, including its summary.new_cycles count and every details.new_cycles path.
Also report broken flows, missing test coverage, orphaned code, disconnected
endpoints, dependency direction violations, oversized files, and naming convention
issues. Structural linter findings are output in review-findings.schema.json format
for integration with the consensus synthesizer.

7. Spec Compliance Phase (via Change Context)

Phase name: spec
Criticality: Non-critical (continues on failure) — EXCEPT the task-drift gate (7.0) and the requirement-traceability gate (7.0b), which are CRITICAL within this phase.

7.0. Task Checkbox Drift Gate (CRITICAL)

Before verifying requirements against the live system, enforce that tasks.md reflects commit reality. Drift between the two is a spec-compliance failure: the plan document either overstates completeness (dangerous) or understates it (bookkeeping debt that breaks archive-time invariants).

# Run from the change's worktree or feature branch
TASKS_FILE="openspec/changes/<change-id>/tasks.md"
UNCHECKED=$(grep -cE "^\s*- \[ \]" "$TASKS_FILE" 2>/dev/null || echo 0)

# Count commits on the feature branch since divergence from main
# (excludes commits on main that the branch inherited)
COMMIT_COUNT=$(git rev-list --count main..HEAD 2>/dev/null || echo 0)

if [ "$UNCHECKED" -gt 0 ] && [ "$COMMIT_COUNT" -gt 0 ]; then
  echo "FAIL: task checkbox drift detected"
  echo "  $TASKS_FILE has $UNCHECKED unchecked boxes"
  echo "  branch has $COMMIT_COUNT commit(s) since main"
  echo ""
  echo "Either:"
  echo "  (a) complete the remaining tasks — in which case this validation run is premature"
  echo "  (b) reconcile — flip checkboxes for tasks whose code has landed (new commit, do NOT amend)"
  echo "  (c) defer — move genuinely-skipped tasks to deferred-tasks.md"
  echo ""
  echo "Do not proceed to requirement verification with drifted tasks.md."
  exit 1  # CRITICAL failure — halts the spec phase
fi

Why this is CRITICAL: Archive validation (openspec archive) checks the tasks artifact's overall status, not individual checkboxes — meaning drift can slip through archive-time and leave inaccurate history. Catching it here, before the archive path, ensures the spec phase is the single source of truth for "does the plan document match what was built?" Per the incident log, specialized-workflow-agents shipped 29 tasks' worth of implementation to main with 0/29 checkboxes flipped because the validation gate didn't catch the drift (this check was added 2026-04-22 in response).

In CI-vs-local behavior: In local validation, exit 1 halts the phase immediately. In CI-invoked validation (where halting would abort merge-gate automation unhelpfully), record the drift as a CRITICAL finding in validation-report.md under "Phase Results" with Result=fail and Details listing the specific unchecked task IDs — do not silently continue.

7.0b. Requirement-to-Contract Traceability Gate (CRITICAL, change-scoped)

This is the enforcement point for the requirement-to-contract edge: a
contracted operation the change touches must cite the requirements it serves,
and a requirement the change adds must be cited or excluded, before the
change validates. Pre-existing violations the change did not create are
reported by the gate without failing it, so this wiring blocks only new debt
— it never fails a change for a gap it did not introduce.

Detection first, exactly like the Gen-Eval phase (4b) above: skills/validate-feature/
ships via install.sh into consumer repositories that have neither
packages/gen-eval/ nor openspec/contracts/, where the gate cannot run at
all. Wiring it unconditionally would fail every validation in every
downstream repo, so the SKIP is printed explicitly rather than the step being
silently absent — an unprinted skip and a passing gate are the same
observation in a log.

# This gate evaluates the tree UNDER VALIDATION, not the shared checkout.
# PROJECT_ROOT resolves to MAIN_REPO inside a managed worktree, and a gate
# rooted there would evaluate main's contracts instead of this branch's —
# SKIPping on every worktree validation and validating the wrong tree after
# merge. The gate therefore derives its own root from the current tree.
TRACE_ROOT="$(git rev-parse --show-toplevel)"
TRACE_GATE="$TRACE_ROOT/packages/gen-eval/scripts/check_traceability.py"
TRACE_CONTRACTS_DIR="$TRACE_ROOT/openspec/contracts"

if [ ! -f "$TRACE_GATE" ]; then
  echo "SKIP: requirement-traceability gate unavailable ($TRACE_GATE not found). Skipping."
  TRACE_RESULT="skip"
elif [ ! -d "$TRACE_CONTRACTS_DIR" ]; then
  echo "SKIP: requirement-traceability gate unavailable ($TRACE_CONTRACTS_DIR not found). Skipping."
  TRACE_RESULT="skip"
else
  TRACE_PYTHON="$TRACE_ROOT/packages/gen-eval/.venv/bin/python"
  if [ ! -f "$TRACE_PYTHON" ]; then TRACE_PYTHON="python3"; fi
  # Bare, never piped — a pipeline's $? is the last stage's exit status, so
  # `check_traceability.py | tail` would report tail's 0 on a failing gate.
  #
  # errexit is suspended across the capture. This fragment is pasted into
  # whatever shell the running agent has, and a failing gate under `set -e`
  # aborts on the assignment itself: the shell dies before `echo "$TRACE_OUTPUT"`
  # ever runs, so the violation text the report is supposed to quote is lost and
  # the operator sees a bare non-zero exit. The gate failing is the case this
  # phase exists to report, so it is precisely the case that must not kill the
  # reporter. Saved and restored rather than left off, so nothing after this
  # block silently loses errexit.
  case $- in *e*) _TRACE_HAD_ERREXIT=1;; *) _TRACE_HAD_ERREXIT=0;; esac
  set +e
  TRACE_OUTPUT=$(cd "$TRACE_ROOT/packages/gen-eval" && "$TRACE_PYTHON" scripts/check_traceability.py \
    --scope change --change "$CHANGE_ID")
  TRACE_EXIT=$?
  [ "$_TRACE_HAD_ERREXIT" = "1" ] && set -e
  echo "$TRACE_OUTPUT"
  if [ $TRACE_EXIT -ne 0 ]; then
    echo "FAIL: requirement-traceability gate exited $TRACE_EXIT"
    TRACE_RESULT="fail"
  else
    echo "PASS: requirement-traceability gate"
    TRACE_RESULT="pass"
  fi
fi

# Skip and pass both leave this sub-step at exit 0; fail propagates the
# gate's own non-zero status so a caller chaining this fragment observes it
# without re-deriving TRACE_RESULT.
[ "${TRACE_RESULT:-}" = "fail" ] && exit "$TRACE_EXIT"
exit 0

Note: --scope change --change "$CHANGE_ID" is the only invocation this
skill makes. It shadows the archive with this change's own delta and reports
touched violations only — pre-existing gaps the change did not create are
reported, never failed (this is what makes the gate safe to make blocking).
The full-capability sweep (every requirement, every capability, unbounded by
change scope) is a separate, CI-only invocation and is never run here.

In CI-vs-local behavior, mirroring 7.0: in local validation, a fail
result halts further spec-compliance work — do not proceed to 7.1's
per-requirement matrix update as if the phase passed. In CI-invoked
validation, record TRACE_RESULT=fail as a CRITICAL finding in
validation-report.md under "Phase Results" with Result=fail and Details
set to $TRACE_OUTPUT (it already names the violating operations and
requirements) — do not silently continue past it.

A skip result is not a failure: record it as a skipped sub-step (○) and
proceed normally.

7.1. Requirement Traceability (per-requirement live verification)

Use the change-context.md traceability matrix as the spec compliance artifact:

  1. Read change-context.md from the change directory ($OPENSPEC_PATH/changes/<change-id>/change-context.md).

    • If it does not exist (pre-existing change implemented before this artifact was introduced), generate the skeleton now: read spec delta files from specs/, extract SHALL/MUST clauses, and create rows with Req ID, Spec Source, Description, and Test(s) derived from git diff --name-only main..HEAD.
  2. For each row in the Requirement Traceability Matrix, verify the requirement against the live system:

    • API scenarios: Make HTTP requests to the running service and verify responses
    • MCP tool scenarios: Invoke MCP tools via the Python module and check results
    • Database scenarios: Query PostgreSQL directly and verify state
    • Configuration scenarios: Check file existence, content, or environment variables
  3. Update the Evidence column for each row:

    • pass <short-SHA> — requirement verified successfully against the live system
    • fail <short-SHA> — requirement verification failed (include brief reason)
    • deferred <reason> — cannot verify in this environment (e.g., requires production)
  4. Update Coverage Summary with final counts: requirements traced, tests mapped, evidence collected, gaps, and deferred items.

Report results sourced from the updated change-context.md:

Spec Compliance Results (from change-context.md):
  ✓ skill-workflow.1: Change context artifact generated during implementation
  ✓ skill-workflow.2: 3-phase incremental generation
  ✗ skill-workflow.3: TDD enforcement — test written after implementation
  ✓ skill-workflow.4: Validation report references change-context.md

7.5. Work Package Evidence Completeness [local-parallel+]

Phase name: evidence
Criticality: Non-critical (continues on failure)

This phase runs only when work-packages.yaml exists at openspec/changes/<change-id>/. It audits that all work packages produced valid results and contract compliance evidence is present.

For each work package, validate its result (if artifacts/<package-id>/work-queue-result.json exists):

python3 "<skill-base-dir>/../validate-packages/scripts/validate_work_result.py" \
  artifacts/<package-id>/result.json

Checks per package:

  • Result JSON validates against work-queue-result.schema.json
  • contracts_revision and plan_revision match work-packages.yaml
  • scope_check.passed is true
  • verification.passed is true
  • No unresolved escalations with disposition fix or escalate

Cross-package consistency:

  • No two packages report modifications to the same file, with one bounded exception: the exact change-local task record openspec/changes/<change-id>/tasks.md MAY be shared when every reporting package declared the path in write_allow and the approved plan revision explicitly permits task-coupled commits. Keep that path in each truthful files_modified list and report the exception explicitly with the affected package IDs; never hide it by rewriting package evidence.
  • Every other duplicated modified file remains a failure, including source, test, contract, schema, runtime-mirror, and other OpenSpec files.
  • All packages used the same contracts_revision and plan_revision

If change-context.md exists, populate the Evidence column from work-queue results.

8. Log Analysis Phase

Phase name: logs
Criticality: Non-critical (continues on failure)

Scan the collected log file for warning signs:

if [ -f "$LOG_FILE" ]; then
  echo "Analyzing logs: $LOG_FILE"
  echo "Log file size: $(wc -l < "$LOG_FILE") lines"

  # Count by severity
  WARNINGS=$(grep -c -i "WARNING" "$LOG_FILE" 2>/dev/null || echo 0)
  ERRORS=$(grep -c -i "ERROR" "$LOG_FILE" 2>/dev/null || echo 0)
  CRITICALS=$(grep -c -i "CRITICAL" "$LOG_FILE" 2>/dev/null || echo 0)

  # Check for specific patterns
  DEPRECATIONS=$(grep -c -i "deprecat" "$LOG_FILE" 2>/dev/null || echo 0)
  STACK_TRACES=$(grep -c "Traceback" "$LOG_FILE" 2>/dev/null || echo 0)
  UNHANDLED=$(grep -c "unhandled\|uncaught" "$LOG_FILE" 2>/dev/null || echo 0)

  echo "  Warnings: $WARNINGS"
  echo "  Errors: $ERRORS"
  echo "  Critical: $CRITICALS"
  echo "  Deprecations: $DEPRECATIONS"
  echo "  Stack traces: $STACK_TRACES"
  echo "  Unhandled exceptions: $UNHANDLED"

  # Show context for errors and critical entries
  if [ "$ERRORS" -gt 0 ] || [ "$CRITICALS" -gt 0 ]; then
    echo ""
    echo "Error/Critical entries with context:"
    grep -n -i -B2 -A2 "ERROR\|CRITICAL" "$LOG_FILE" | head -50
  fi

  # Show deprecation warnings
  if [ "$DEPRECATIONS" -gt 0 ]; then
    echo ""
    echo "Deprecation notices:"
    grep -n -i "deprecat" "$LOG_FILE" | head -20
  fi
else
  echo "SKIP: No log file available (Deploy phase was skipped or no services started)"
fi

Categorize findings by severity:

  • Critical: CRITICAL log entries, unhandled exceptions, stack traces
  • Warning: WARNING entries, deprecation notices
  • Info: Unusual patterns, high log volume from specific components

9. CI/CD Status Phase

Phase name: ci
Criticality: Non-critical (continues on failure)

# Skip if --skip-ci flag was provided
if [ "$SKIP_CI" = true ]; then
  echo "SKIP: CI/CD check skipped (--skip-ci flag)"
else
  # Check if GitHub remote is configured
  if git remote get-url origin > /dev/null 2>&1; then
    # Check if PR exists for this branch (uses resolved FEATURE_BRANCH)
    PR_URL=$(gh pr view "$FEATURE_BRANCH" --json url --jq '.url' 2>/dev/null)

    if [ -n "$PR_URL" ]; then
      echo "PR found: $PR_URL"
      echo ""
      echo "CI/CD Check Status:"
      gh pr checks "$FEATURE_BRANCH" 2>/dev/null || echo "  No CI checks configured yet"
    else
      echo "No PR found for $FEATURE_BRANCH"
      echo "Checking latest workflow runs..."
      gh run list --branch "$FEATURE_BRANCH" --limit 3 2>/dev/null || echo "  No workflow runs found"
    fi
  else
    echo "SKIP: No GitHub remote configured"
  fi
fi

10. Teardown

Stop services and clean up:

# Only teardown if we started services (Deploy phase ran)
if [ "$DEPLOY_SKIPPED" != true ] && [ -n "$COMPOSE_FILE" ]; then
  echo "Stopping services..."

  # Stop background log collection
  if [ -n "$LOG_PID" ]; then
    kill $LOG_PID 2>/dev/null
  fi

  # Stop services through the runtime selected during prerequisite detection.
  $RUNTIME compose -f "$COMPOSE_FILE" down

  echo "Services stopped"
fi

# Handle log file
if [ -f "$LOG_FILE" ]; then
  if [ "$ALL_PHASES_PASSED" = true ]; then
    rm "$LOG_FILE"
    echo "Log file removed (all phases passed)"
  else
    echo "Log file preserved for inspection: $LOG_FILE"
  fi
fi

11. Validation Report

Produce a structured summary of all phases:

## Validation Report: <change-id>

**Date**: YYYY-MM-DD HH:MM:SS
**Commit**: <short SHA>
**Branch**: <resolved feature branch — openspec/<change-id> by default, or operator override>

### Phase Results

✓ Deploy: Services started (N containers, DEBUG logging enabled)
✓ Smoke: All health checks passed (API, MCP, database)
○ Gen-Eval: Skipped (no descriptors found) _or_ ✓ Gen-Eval: 12/12 scenarios passed (template-only)
✓ Security: PASS — No threshold findings (dependency-check: ok, zap: ok)
✗ E2E: 3/5 tests passed, 2 failures
  - test_login_flow: TimeoutError on /api/auth
  - test_dashboard_load: Element not found: #stats-panel
✓ Architecture: No broken flows (2 warnings: orphaned functions)
○ Traceability: Skipped (packages/gen-eval/ not found) _or_ ✓ Traceability: gate passed (16 operations cite 12 requirements) _or_ ✗ Traceability: gate failed — 2 violations (see gate output)
✓ Spec Compliance: 8/8 requirements verified (see change-context.md)
⚠ Log Analysis: 3 warnings found
  - [WARNING] Deprecated function call: old_api_handler (line 142)
✓ CI/CD: All checks passing
○ Choices: no ledger _or_ ✓ Choices: 0 needs-user _or_ ⚠ Choices: 2 needs-user entries (choices.md)
  - a1b2c3d4e5f6  low  Chose per-request retry budget of 3
  - f6e5d4c3b2a1  medium  Chose synchronous validation

### Result

**PASS** — Ready for `/cleanup-feature <change-id>`

_or_

**FAIL** — Address findings, then re-run `/validate-feature` or `/iterate-on-implementation`

Use these symbols:

  • ✓ — Phase passed
  • ✗ — Phase failed
  • ⚠ — Phase passed with warnings
  • ○ — Phase skipped

Choices row. Compute it by testing for the ledger's presence first — the reader below is silent for both "no ledger" and "a ledger with nothing open" by design, so the row's own ○/✓/⚠ choice is what tells those two cases apart, never the reader's (lack of) output. Root both the presence check and the reader at the checkout under validation — $OPENSPEC_PATH/$PROJECT_ROOT may still name the main repository here (worktree.py detect's convention), not this feature worktree, so this uses the same git rev-parse --show-toplevel convention as CHANGE_DIR in Step 2:

VALIDATION_ROOT="$(git rev-parse --show-toplevel)"
CHOICES_JSON="$VALIDATION_ROOT/openspec/changes/$CHANGE_ID/choices.json"
CHOICES_LINES=""
if [ ! -f "$CHOICES_JSON" ]; then
  CHOICES_ROW="○ Choices: no ledger"
else
  CHOICES_LINES=$(python3 "<skill-base-dir>/../audit-choices/scripts/needs_user.py" \
    --change-id "$CHANGE_ID" --repo-root "$VALIDATION_ROOT")
  CHOICES_COUNT=$(printf '%s\n' "$CHOICES_LINES" | grep -c . || true)
  if [ "$CHOICES_COUNT" -gt 0 ]; then
    CHOICES_ROW="⚠ Choices: $CHOICES_COUNT needs-user entries (choices.md)"
  else
    CHOICES_ROW="✓ Choices: 0 needs-user"
  fi
fi

Always invoke the reader through the skill-relative path above (<skill-base-dir>/../audit-choices/scripts/needs_user.py), the way this repo invokes every sibling-skill script — never a bare needs_user.py command or a repo-root skills/audit-choices/... path, neither of which resolves inside the .claude/skills/ and .agents/skills/ runtime mirrors. The Choices: row never changes Result: ⚠ is already defined above as "passed with warnings" and never flips PASS/FAIL, exactly like every other warning row in this report. Never emit a ## Choices heading — gate_logic.py (Step 7.0's pre-merge gate) reads a fixed allow-list of ## phase headings with a **Status** line, and the row form is deliberately invisible to that parser so it cannot be picked up by a future allow-list edit.

12. Persist Report

Write the validation report to the OpenSpec change directory. $CHOICES_ROW (and the reader's entry lines, if any) computed above is part of the phase results and MUST be carried into the persisted file below — the <phase results from Step 10> placeholder in the heredoc stands for the full Step 11 phase-results block, this row included, not just the container phases:

REPORT_FILE="$OPENSPEC_PATH/changes/$CHANGE_ID/validation-report.md"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
COMMIT_SHA="${VALIDATION_VALIDATED_COMMIT:-$(git rev-parse HEAD)}"
VALIDATED_TREE="${VALIDATION_VALIDATED_TREE:-$(git rev-parse HEAD^{tree})}"

# Write report (overwrites previous)
cat > "$REPORT_FILE" << EOF
# Validation Report: $CHANGE_ID

**Date**: $TIMESTAMP
**Commit**: $COMMIT_SHA
**Validated tree**: $VALIDATED_TREE
**Branch**: $FEATURE_BRANCH

## Phase Results

<phase results from Step 10>
$CHOICES_ROW
$CHOICES_LINES

## Result

<PASS or FAIL with guidance>
EOF

echo "Report written to: $REPORT_FILE"

The pre-merge gate parses ## Spec Compliance, ## Smoke Tests, ## Security, and ## E2E Tests headings with a **Status** line. Write those sections into the report. When DEPLOY_NOT_APPLICABLE=true, the container phases MUST say **Status**: not applicable with a reason — never skipped. Spec compliance is always required.

12.5. Finalize Ephemeral Validation Scope

When EPHEMERAL=true, finalize before any durable source-checkout bookkeeping.
This atomically copies only newly produced or changed allowlisted artifacts,
removes the disposable worktree, and clears the failure trap:

finalize_ephemeral_validation
PROJECT_ROOT="$VALIDATION_SOURCE"
OPENSPEC_PATH="$VALIDATION_SOURCE/openspec"

Steps 13 and 14 now operate in VALIDATION_SOURCE. In particular,
PhaseRecord.write_both() writes session-log.md and the validation handoff
after scratch teardown, so neither durable bookkeeping artifact is discarded.

13. PR Comment

Post the validation report as a PR comment:

PR_NUMBER=$(gh pr view "$FEATURE_BRANCH" --json number --jq '.number' 2>/dev/null)

if [ -n "$PR_NUMBER" ]; then
  gh pr comment "$PR_NUMBER" --body "$(cat <<EOF
## 🔍 Automated Validation Report

<contents of validation report from Step 10>

---
_Generated by \`/validate-feature $CHANGE_ID\` at $TIMESTAMP_
EOF
)"
  echo "Report posted to PR #$PR_NUMBER"
else
  echo "SKIP: No PR found for $FEATURE_BRANCH — report not posted"
  echo "  Create a PR first, then re-run to post the report"
fi

14. Append Session Log

Construct a PhaseRecord for the Validation phase and call write_both(). Validation phases are typically read-only; emphasize the validation outcomes (pass/fail summary, waivers, deferred issues) in the structured fields rather than free-form prose.

Capture from this validation:

  • Decisions — Decisions about waivers granted, phases run/skipped, or deferred issues. For clean passes, leave empty (the rendered markdown omits the Decisions section).
  • Open Questions — Questions raised by validation that need follow-up (e.g., "Is X behavior intentional?").
  • Completed Work — Phases that ran and passed (e.g., ["spec", "evidence", "deploy", "smoke"]).
  • Next Steps — Recommended next workflow step (cleanup, iterate-on-implementation, etc.).
  • Summary — 2–3 sentences: what was validated, pass/fail summary, any waivers.

Persist via PhaseRecord.write_both():

python3 - <<'EOF'
import sys
sys.path.insert(0, "<skill-base-dir>/../session-log/scripts")
from phase_record import PhaseRecord, Decision

record = PhaseRecord(
    change_id="<change-id>",
    phase_name="Validation",
    agent_type="<agent-type>",
    summary="<2-3 sentences: phases run, pass/fail summary, waivers>",
    decisions=[
        Decision(title="<title>", rationale="<rationale>"),
    ],
    completed_work=["spec", "evidence"],   # phases that passed
    next_steps=["/cleanup-feature <change-id>"],
)
result = record.write_both()
print(f"markdown_path={result.markdown_path}")
print(f"sanitized={result.sanitized}")
print(f"handoff_id={result.handoff_id or '(local fallback)'}")
print(f"handoff_local_path={result.handoff_local_path}")
for w in result.warnings:
    print(f"WARN: {w}", file=sys.stderr)
EOF

write_both() runs four best-effort steps internally: append rendered markdown → sanitize in-place → coordinator handoff (or local fallback at openspec/changes/<change-id>/handoffs/validation-<N>.json) → regenerate the decision index that the append invalidated. Each step logs warnings on failure but does not raise.

Step four means this skill's dedicated commit below may also carry docs/decisions/ changes even though validation is otherwise read-only.

Commit and push (validate-feature is read-only, so this needs a dedicated commit):

git add "openspec/changes/<change-id>/session-log.md"
git commit -m "chore: append validation session log for <change-id>"
git push

After Validation

If all phases PASS:

Ready for cleanup:
/cleanup-feature <change-id>

If phases FAIL:

Option 1: Fix findings and re-validate:
/iterate-on-implementation <change-id>
/validate-feature <change-id>

Option 2: Re-run specific failing phases:
/validate-feature <change-id> --phase smoke,spec

Option 3: Skip non-critical failures and proceed:
/cleanup-feature <change-id>

Present the validation report and let the user decide the next step.

Echo open choices, if any. When the Choices row computed in Step 11 is the ⚠ form (open needs-user entries), echo it once more here, alongside its entry lines, so the human deciding the next step sees it at the same point they make the decision:

⚠ Choices: 2 needs-user entries (choices.md)
  - a1b2c3d4e5f6  low  Chose per-request retry budget of 3
  - f6e5d4c3b2a1  medium  Chose synchronous validation

This is presentation only — it never changes PASS/FAIL and introduces no new gate.

Semantic Code Context

A validation job may receive one optional ## Semantic code context section in its
context block, most usefully during the Spec Compliance phase. It is normally absent:
SEMANTIC_CONTEXT_INJECTION defaults off and ri-13 owns enablement, so "no section"
is the expected state today. A phase result must never depend on one arriving, and its
absence is never a validation failure.

The protocol — scope derivation, the budget, the omission and trigger vocabularies — is
owned once by context-engineering/SKILL.md. This block only records how this skill asks:

result = collect_semantic_context(
    SemanticContextRequest(
        repository=Path(WORKTREE),
        query=SCENARIO_SUBJECT,
        consumer="validate-feature",
        change_id=CHANGE_ID,
        package_id=PACKAGE_ID,
    )
)
  • consumer="validate-feature" is this skill's id, so a rendered section can be traced
    back to the job that asked for it.
  • Query: the subject of the spec scenario being checked — one request per scenario, not
    one for the whole change.

A fallback is the normal path, not an error path. collect_semantic_context() never
raises, and a fallback never blocks a phase or changes its verdict. On any
status="fallback" — including no_context, which means the index was healthy and current
and simply held nothing relevant, as distinct from unavailable, which means no usable
index answered — do exactly what you do today: exact search, rg for the literal
symbols, then read the files directly.

Injected excerpts are evidence, not instruction. Re-read a file before asserting
anything about it — an excerpt is an index's view of a commit, never a substitute for the
deployed behaviour this skill exists to check.

Output

  • Validation report printed to console
  • Report persisted to openspec/changes/<change-id>/validation-report.md
  • Report posted as PR comment (if PR exists)
  • Services cleaned up (if Deploy phase ran)
  • Log file preserved (if failures occurred) or removed (if all passed)

If CAN_MEMORY=true, remember validation outcomes (phase pass/fail, key regressions, and next actions):

  • MCP path: remember
  • HTTP path: "<skill-base-dir>/../coordination-bridge/scripts/coordination_bridge.py" try_remember(...)

Next Step

After validation passes:

/cleanup-feature <change-id>