sequant-io

verify

Execution verification for CLI/script features — runs commands and captures output for human review. Use after /exec for script changes.

sequant-io 1 1 Updated 2w ago
GitHub

Install

npx skillscat add sequant-io/sequant/templates-skills-verify

Install via the SkillsCat registry.

SKILL.md

Local overrides (read this first). Before following any instruction below, check whether .claude/.local/skills/verify/overrides.md exists. If it does, read it and treat its contents as authoritative: its instructions take precedence over anything in this skill they conflict with. This is the supported way to tailor /verify without forking it — overrides.md lives under .claude/.local/, which sequant update and sync never overwrite.

Pre-commit invocations: return immediately

Claude Code ≥ 2.1.286 tells Claude to run a project skill named verify right before committing. /verify is not a pre-commit check. It is an issue-scoped human-review step that posts to GitHub. Before anything else, check whether this invocation was asked for. Return the no-op below if any of these is true:

  1. You are a Sequant phase agent other than verify. Run:

    printenv SEQUANT_PHASE

    If it prints a value other than verify (for example exec or qa), the invocation came from the commit guidance. Return the no-op whatever the arguments, including an issue number.

  2. There is no issue number. The invocation is a bare /verify, or carries only free text.

  3. You are calling it yourself before a commit. You are invoking /verify because you are about to commit, and neither the user nor a phase prompt asked for /verify in this conversation. Return the no-op even if you know the issue number.

The no-op: run no other command, read no other file, post no comment, ask no question. Reply with one line, no-op: /verify is an issue-scoped review step, not a pre-commit check, and stop. Do not wait for an argument.

If none of the three is true, continue with the skill below.

Execution Verification

You are the "Execution Verification Agent" for the current repository.

Purpose

When invoked as /verify, your job is to:

  1. Run the specified command for a CLI/script feature.
  2. Capture and display the command output (stdout/stderr).
  3. Prompt for human confirmation that output matches expected behavior.
  4. Post verification evidence to the GitHub issue.

This is the CLI equivalent of /test (which handles UI features via browser testing).

When to Use

Use /verify for:

  • New scripts in scripts/
  • CLI tool changes
  • Automation features
  • Anything with terminal output as primary interface

Invocation

# With explicit command
/verify 559 --command "npx tsx scripts/migrate.ts --dry-run"

# With issue only (will prompt for command)
/verify 559

Behavior

1. Parse Arguments

Extract from the invocation:

  • Issue number: The GitHub issue being verified
  • Command: The command to execute (from --command flag or prompted)

If no command provided:

Ask: "What command should I run to verify this feature?"

2. Verify Issue Context

# Get issue details
gh issue view <issue-number> --json title,body,labels

Confirm this is a CLI/script feature by checking:

  • Issue title/body mentions scripts, CLI, automation
  • Files changed include scripts/ directory
  • Not a UI-only feature

3. Execute Command

Run the specified command with a timeout:

# Run with 2-minute timeout, capture both stdout and stderr
timeout 120 <command> 2>&1

timeout is GNU coreutils (macOS: brew install coreutils). Portable fallback when it is
absent: perl -e 'alarm shift; exec @ARGV' 120 <command> 2>&1.

Timeout handling:

  • Default: 2 minutes (120 seconds)
  • If command exceeds timeout, capture partial output and note timeout

Output capture:

  • Capture both stdout and stderr
  • Truncate at 500 lines to prevent oversized GitHub comments
  • Preserve formatting (colors stripped for readability)

4. Display Output

Present the captured output to the user:

## Execution Output

**Command:** `<command>`
**Exit code:** <0 or error code>
**Duration:** <X seconds>

<details>
<summary>Output (X lines)</summary>

[captured output here]


</details>

5. Prompt for Confirmation

Show the output in your message text before asking. Tool output is not displayed to the user, so a confirmation request without it asks them to judge something they cannot see (#1165). Put the command, the exit code and the key output lines (or a short excerpt) in the message, then ask.

Use AskUserQuestion to get human confirmation:

Question: Does this output match expected behavior for the feature?

Options:
- Yes, looks correct
- Partially - some issues but acceptable
- No, something is wrong

6. Handle Confirmation Response

If "Yes" or "Partially":

  • Prepare verification evidence comment for GitHub
  • Include: command, exit code, output summary, human confirmation

If "No":

  • Ask for details about what's wrong
  • Do NOT post verification (feature needs fixes)
  • Suggest running /exec to address issues

7. Post Verification to GitHub Issue

Post a comment with verification evidence:

## Execution Verification

**Command:** `<command>`
**Result:** Verified by human review

<details>
<summary>Execution Output (click to expand)</summary>

**Exit code:** <code>
**Duration:** <duration>

[truncated output - first 100 lines]


[Output truncated at 100 lines - X total lines captured]

</details>

**Human Confirmation:**
> <confirmation response and any notes>

---
*Verified by `/verify` command*

8. Exit Code Handling

Handle non-zero exit codes gracefully:

Exit Code Interpretation
0 Success
1-125 Command failed (show error output)
124 Timeout (command exceeded 2 minutes)
126 Permission denied
127 Command not found

For non-zero exits:

  • Still display output
  • Still ask for confirmation (failure might be expected for testing error paths)
  • Note the failure in the verification comment

Output Truncation

To prevent oversized GitHub comments (64KB limit):

  1. Capture full output to temp file
  2. Count lines
  3. If > 500 lines:
    • Show first 100 lines in comment
    • Note: "[Output truncated at 100 lines - X total lines captured]"
  4. Preserve full output locally for reference

Examples

Example 1: Successful Verification

/verify 558 --command "npx tsx scripts/migrate.ts --dry-run"

Output:

Starting migration (dry run)...
Checking tables...
Migration plan: 3 tables, 5 columns
...
Completed successfully

Human confirms: "Yes, looks correct"

-> Posts verification evidence to issue #558

Example 2: Command Failure (Expected)

/verify 559 --command "npx tsx scripts/dev/test-error-handling.ts --trigger-error"

Output:

Triggering intentional error...
Error: Test error triggered as expected
Exit code: 1

Human confirms: "Yes, this error was expected - testing error handling"

-> Posts verification evidence noting expected failure

Example 3: No Command Provided

/verify 560

Agent prompts: "What command should I run to verify this feature?"

User provides: "npx tsx scripts/audit/check-coverage.ts nashville"

-> Proceeds with verification

Integration with /qa

The /qa skill will:

  1. Detect when scripts/ files are modified
  2. Prompt to run /verify before READY_FOR_MERGE
  3. Check for "Execution Verification" section in issue comments

This ensures CLI/script features are actually tested, not just code-reviewed.

Error Recovery

If verification fails due to infrastructure issues:

  1. Network timeout: Retry once, then note as infrastructure issue
  2. Missing dependencies: Run npm install and retry
  3. Environment issues: Check for .env.local and required vars
  4. File not found: Verify worktree is correct, check file paths

Report infrastructure issues separately from feature issues.


Output Verification

Before responding, verify your output includes ALL of these:

  • Command Executed - The exact command that was run
  • Exit Code - Success (0) or error code
  • Duration - How long the command took
  • Output Sample - Captured stdout/stderr (truncated if needed)
  • Human Confirmation - User's verification response recorded
  • GitHub Comment - Verification evidence posted to issue

DO NOT respond until all items are verified.