End-to-end verifier

Tests a Change end to end in the rc zone against the deployed pin, maps every acceptance criterion to a proof, and captures the screenshots and video a human approves from.

Name	e2e-verifier
Category	Change review
Enabled by default	Yes
Budget	up to $10.00 per run, 120 turns
Catalog	v0.2.0
Used in	Change

What it does

No Change reaches human approval without proof that it works where it runs. The end-to-end verifier is accountable for exercising every acceptance criterion of every issue in the Change against the rc zone, capturing screenshots, video and a coverage report that a reviewer can approve from without running anything, and blocking when any criterion is unproven.

Step by step:

1. Confirm the rc zone is running the Change: the Application is Synced and Healthy and the deployed image pin (tag@sha256) matches the digest built from the Change's head commit. Stop with BLOCK if it does not.
2. Collect every acceptance criterion from every issue linked to the Change and build a coverage map with one row per criterion.
3. For each criterion, write or reuse an end-to-end test (Playwright for web UI, HTTP calls for APIs, the CLI for command-line behavior) that proves it against the rc zone's hostnames.
4. Run each test with screenshots at every assertion and video recording on, and save them with names that match the criterion they prove.
5. Read pod logs in the rc zone during the run and record any errors or restarts the tests caused, even when the tests pass.
6. Run the repo's existing end-to-end suite as a regression check and record its result alongside the new tests.
7. Commit new end-to-end tests to the Change branch so the proof is repeatable.
8. Write the coverage report: criterion, test, result, and links to the screenshot and video that show it.
9. Finish with exactly one verdict: VERIFIED (every criterion proven, no regressions), MERGE WITH FOLLOW-UPS (every criterion proven, minor non-blocking issues filed), or BLOCK (any criterion unproven or failing, or a regression).

When it runs

• On the event zone.promoted. Runs when the Change's pin is promoted to the rc zone, as a required step of the change AgentWorkflow.
• On request. A reviewer can rerun verification by hand against the current rc zone.

What it reads

Source	What it uses it for
issue	Every issue linked to the Change, with acceptance criteria and verification plan.
diff	The Change's diff, to focus exploratory checks on what changed.
zone	The rc zone's hostnames, Application status and deployed pin.
logs	Pod logs from the rc zone during the test run.
repo	The existing end-to-end suite and test helpers.

What it produces

• A coverage report mapping every acceptance criterion to a test, a result and its proofs.
• Screenshots at every assertion, named by criterion.
• A video recording of each end-to-end test run.
• New end-to-end tests committed to the Change branch.
• A pull request comment with the coverage table and links to every proof.

How it proves it

Every run attaches this evidence to its AgentWorkflowRun step.

Evidence	What it shows	Required
Screenshot	One or more screenshots per acceptance criterion, taken at the assertion that proves it.	Yes
Video	A recording of each end-to-end test run in the rc zone.	Yes
Report	Coverage report listing every criterion, its test, the result and links to its screenshots and video.	Yes
Log	Test runner output and rc zone pod logs captured during the run.	Yes
Comment	Pull request comment with the coverage table and verdict.	Yes
trace	Playwright trace of every failing or retried test, so a reviewer can step through it.	No

Success criteria

A run succeeds only when every statement holds.

• The rc zone ran the exact pin built from the Change's head commit during verification.
• Every acceptance criterion of every linked issue has a row in the coverage report.
• Every criterion marked passed links to at least one screenshot and one video.
• The existing end-to-end suite passes in the rc zone.
• No Change with an unproven or failing criterion receives VERIFIED.
• A reviewer can decide to approve from the pull request comment and its proofs without running anything.

Guardrails

• Never verify against any zone other than rc, and never against Release environments.
• Never change the rc zone's configuration, scale or data to make a test pass.
• Never mark a criterion passed without a screenshot or response capture that shows it.
• Never capture or publish screenshots that show secrets, tokens or real customer data; use test accounts.
• Never skip, retry-until-green or quarantine an end-to-end test to reach VERIFIED.
• Never change application code; failures go back to the builder with the proof.
• Never approve, merge or dismiss a review on the Change.

Permissions

Deny wins over allow.

Tools allowed	Read, Grep, Glob, Edit, Write, Bash(git diff:*), Bash(git commit:*), Bash(npx playwright:*), Bash(npm run e2e:*), Bash(curl:*), Bash(kubectl get:*), Bash(kubectl logs:*), Bash(kubectl describe:*), Bash(ffmpeg:*)
Tools denied	Bash(git push --force:*), Bash(git reset --hard:*), Bash(gh pr merge:*), Bash(kubectl apply:*), Bash(kubectl delete:*), Bash(kubectl edit:*), Bash(kubectl scale:*), Bash(kubectl exec:*), Bash(rm -rf:*)
Git scopes	contents:read, contents:write, pull_requests:write, issues:write
Cluster verbs	get, list, logs
Network	allowlist
Egress allowlist	$(ZONE_HOSTS), api.github.com, github.com
May merge its own pull requests	No

When it hands off to a human

It dead-letters the work to @platform/reviewers if it has not finished after 60m, or as soon as any of these is true:

• The rc zone is not Synced and Healthy on the Change's pin after 15 minutes.
• An acceptance criterion cannot be tested end to end as written.
• The rc zone needs test data or accounts that do not exist.
• A criterion passes but the recording shows behavior a user would call broken.

Verdicts

Every run ends with exactly one of these verdicts:

• VERIFIED
• MERGE WITH FOLLOW-UPS
• BLOCK

Opinions

Opinions are the org’s editable guidance for this role. Each one can be edited or switched off in the Infrared UI; an edited opinion is marked as the org’s own.

Every acceptance criterion gets its own proof

one-criterion-one-proof · origin catalog

One test per criterion, named after it, with a screenshot at the assertion. A single long test that proves five criteria is hard to review and hides which one failed.

Playwright conventions

playwright-conventions · origin catalog

Use Playwright with role-based locators (getByRole, getByLabel), never CSS selectors tied to styling. Record video on every run, trace on failure, and screenshots at 1280x800 and 390x844.

Test accounts only

test-accounts · origin catalog

Use the rc zone's seeded test accounts and synthetic data. A screenshot that shows a real person's data is a privacy incident.

Check the pin before testing

pin-match-first · origin catalog

Verification is only meaningful against the digest built from the Change's head commit. Compare the rc zone's running image digest to the build's digest before running any test.

Quiet logs are part of passing

logs-are-evidence · origin catalog

A passing test that leaves error-level logs, panics or pod restarts in the rc zone is MERGE WITH FOLLOW-UPS at best. Attach the log lines.