Playwright Validator

Behavioral validator for frontend surfaces. Reads OpenSpec scenarios
(openspec/changes/<id>/specs/**/*.md) and a frontend descriptor
(evaluation/descriptors/<id>.yaml), generates Playwright
TypeScript test files, executes them with npx playwright test --reporter=json,
and emits findings-playwright.json conforming to
openspec/schemas/review-findings.schema.json.

This is the peer skill chosen in design D2 of
factory-missions-architecture-alignment — packaged separately from
packages/gen-eval/ so its Node/browser-binary
dependencies don't bleed into the gen-eval Python runtime.

When to use

• You have a frontend at a URL (or a static fixture) and an OpenSpec change
  with #### Scenario: blocks describing click-and-assert flows.
• You want behavioral evidence (deployed-system behavior) to feed into
  consensus_synthesizer alongside scrutiny reviewer findings.
• validate-feature --phase gen-eval auto-detects a frontend-descriptor
  and dispatches here.

For HTTP/MCP API surfaces, use /gen-eval instead — that path remains the
non-frontend dispatch target.

Arguments

$ARGUMENTS — at minimum, the change-id:

• <change-id> (required, ^[a-zA-Z0-9_-]+$) — OpenSpec change identifier.
• --descriptor PATH — path to frontend-descriptor YAML (default:
  evaluation/descriptors/<change-id>.yaml).
• --specs-dir PATH — override OpenSpec specs directory.
• --output-dir PATH — where findings-playwright.json is written
  (default: openspec/changes/<change-id>/).
• --test-dir PATH — where the generated .spec.ts files land
  (default: skills/playwright-validator/.generated/<change-id>/). Must not
  be a directory named test-results — that is Playwright's own outputDir,
  which it clears before each run, deleting the generated specs.
• --browsers chromium [firefox webkit] — override descriptor's matrix.
• --dry-run — emit the .spec.ts but do not invoke npx playwright test.

Invocation

Direct (Python):

python3 "<skill-base-dir>/scripts/cli.py" <change-id>

Module form (after the skill is on PYTHONPATH):

python -m playwright_validator <change-id> [--descriptor PATH] [--output-dir PATH]

Dispatch shim (used by validate-feature --phase gen-eval):

bash "<skill-base-dir>/scripts/dispatch.sh" <change-id>

Exit codes

Code	Meaning
0	All scenarios passed
1	One or more Playwright tests failed; findings-playwright.json emitted
2	Pipeline error: missing descriptor, malformed YAML, missing env var
64	Invalid change-id (failed ^[a-zA-Z0-9_-]+$)
127	Playwright CLI not on PATH; no findings file emitted

Auto-detection in validate-feature

validate-feature --phase gen-eval walks
evaluation/descriptors/*.yaml and routes each descriptor:

1. If the YAML validates against
   contracts/frontend-descriptor.schema.json → dispatch to this skill via
   dispatch.sh.
2. Otherwise → existing HTTP/MCP gen-eval path
   (packages/gen-eval/).

The detection predicate is descriptor.is_frontend_descriptor(path) from
<skill-base-dir>/scripts/descriptor.py.

Findings output

Per contracts/findings-vendor-source.md and design D8, this skill
always writes to findings-playwright.json (never findings-gen-eval.json).
The two files coexist when both validators run on the same change.

Each finding:

• type: "behavioral_failure" (per design D3).
• criticality: "high" (default for failed Playwright assertions).
• file_path + line_range reference the OpenSpec spec.md, not the
  generated .spec.ts file (per the "Playwright findings trace to OpenSpec
  scenarios" spec scenario).
• metadata.browser — chromium / firefox / webkit (per the "Browser matrix"
  scenario).
• metadata.scenario_id — the OpenSpec scenario's name.

Sample frontend (smoke test)

A minimal static HTML page is available at
packages/gen-eval/tests/fixtures/sample-descriptor.yaml (per design D7 —
package-shipped data). To exercise the full pipeline (requires Node + Playwright):

python3 "<skill-base-dir>/scripts/cli.py" \
    sample-frontend-demo \
    --descriptor packages/gen-eval/tests/fixtures/sample-descriptor.yaml \
    --specs-dir openspec/changes/sample-frontend-demo/specs

Localhost-bind invariant (D7)

The descriptor schema's lifecycle.bind_address defaults to 127.0.0.1.
The runner additionally inspects lifecycle.startup_command and refuses to
launch when 0.0.0.0 appears unless the operator explicitly set
bind_address to a non-localhost value.

Auth-flow env vars

auth_flow[].value may reference ${VAR_NAME} (regex ^[A-Z_][A-Z0-9_]*$).
Substitution uses Python's string.Template — never shell expansion. A
missing env var causes the pipeline to abort before any browser launches
with the exact error:

auth_flow: required env var <VAR_NAME> not set

Operators can preflight by setting the optional env_vars_required list in
the descriptor; the validator checks it before the auth_flow walk.

Testing

Unit + integration tests at skills/tests/playwright-validator/. Run:

# Source-contribution-only validation from the agentic-coding-tools checkout
# (not an installed runtime command):
python3 -m pytest skills/tests/playwright-validator/ -v

Tests that would invoke real npx playwright skip gracefully when the CLI
is unavailable (pytest.skip("requires npx playwright")).