A Scenario File is a markdown UI test with Verify markers and a declared screenshot count — the count is the PASS condition — so the evidence on disk decides, not the agent's self-report.

Primitive

Scenario File

The artifact

Numbered steps. Verify markers. Declared shot count.

.claude/commands/e2e/test_query.md

screenshots: 2

# User StoryAs an analyst, I want to run a natural-language queryand see tabular results, so I can validate the dataset.# Test Steps1. Navigate to Application URL2. Take screenshot of initial state [SHOT 01]3. Verify page title is "NL SQL Interface"4. Enter query: "list top 5 products"5. Click `#run` button6. Verify results table has > 0 rows7. Take screenshot of results [SHOT 02]# Pass Criteria- All Verify steps succeeded- Screenshot count on disk = 2

The non-negotiable

Screenshot count is a disk check, not a claim.

Agent says

"PASSED"

all verifies OK

vs

Disk shows

review_img/ has

2 .png files

If claim ≠ disk, the scenario failed. Doesn't matter what the agent said. Screenshot count collapses self-report into evidence — the file either exists or it doesn't. No interpretation.

What Verify means

The difference between a demo and a test.

Demo

Narrates the happy path

"I clicked Run and the results

appeared, so everything is working."

No assertion. The agent watched something happen and described it. If the results were empty, the narration might still say "working." Demos can silently pass on broken output.

Verify

Runtime-checks an assertion

Verify results table has > 0 rows

→ agent MUST check, short-circuit if fails

Binary pass/fail. Agent must run a check. If the check fails, the scenario stops and reports which Verify failed — not "I think it worked."

What it produces

Structured JSON. Downstream tooling parses, not scrapes.

/test_e2e → review_report.json

{"test_name": "test_query","status": "passed","screenshots": ["agents/cc73faf1/e2e/review_img/01.png","agents/cc73faf1/e2e/review_img/02.png" ],"verifies_passed": 2,"error": null}

Structured so review phases, documentation phases, and failure triagers consume it without parsing prose. The contract extends past the test run.

What breaks

Three failure modes, all silent until too late.

No screenshot count declared

"The agent said it worked" becomes the PASS signal. Phantom test passes slip through undetected for weeks.

Verify markers without enforcement

Agent narrates "I verified the title" without actually checking. Verify must be runtime-checked, not just mentioned.

Scenarios used as documentation

They drift from the actual UI. Use them as tests — the review loop forces them to stay current; docs have no such forcing function.

The discipline

The evidence on disk is the PASS.

A scenario without a screenshot count is a demo. A scenario with narration instead of Verify markers is a wish list. The scenario earns its test status by being runtime-checkable and disk-verifiable — the agent can say "it worked," but the ls review_img/ decides.