Give your AI agents a repeatable way to prove that a change preserves the business workflows that matter. Build the Playwright suite in your own repository, run it in CI, and keep the test contract and enforcement settings under accountable human ownership. The agent can propose a fix and use failure evidence to improve it; its normal permissions should not let it waive the checks, approve its own changes, or rewrite the merge rules.
Turn critical workflows into an independent contract
Name the customer actions a change must preserve before asking an agent to implement it. For each workflow, record the role, starting data, expected outcome, and consequence of failure. A human who understands the business approves that contract.
- Cover a successful path, invalid input, and a denied action; verify that rejected requests do not change protected data.
- Choose the relevant browsers, roles, tenants, and integration boundaries. Document known defects separately from intended behavior.
- Give each critical scenario a stable identifier and an owner so reviewers can see exactly which behavior the suite protects.
Build a suite the agent can run and humans can trust
Use isolated fixtures and a controlled test environment. Prefer role or label locators and assertions that wait for an observable result. A passing page load is not enough for a workflow that saves data, changes permissions, or triggers a business action.
- Keep tests independent, use synthetic or approved sanitized data, and isolate payments, email, and other external side effects.
- Check persisted outcomes and relevant API boundaries as well as the UI. Demonstrate that a deliberate wrong result makes the relevant assertion fail.
- Enable Playwright's forbidOnly in CI. Separately reject unapproved skips, narrowed test selection, weakened assertions, and snapshot changes; forbidOnly does not cover those cases.
- Document installation, fixture reset, local execution, and failure reproduction so the client can maintain the harness without the original implementer.
Protect the harness and the merge path
Assign human code owners to the tests, fixtures, Playwright configuration, test-runner scripts, CI workflows, and CODEOWNERS itself. Enable required owner review; a CODEOWNERS file alone does not enforce approval. Protect the target branch with an active ruleset requiring pull requests, review of the latest changes, and the named CI check.
- Select the expected status-check source where supported, and verify that another identity cannot satisfy the check with an unrelated success status.
- Give agents only the repository and task permissions they need. Keep rule administration, bypass rights, protected-branch writes, production credentials, and approval authority outside the agent identity.
- Review bypass lists, including administrator, team, and app entries. A bypass-capable identity can undermine the intended gate.
- Limit the CI token's permissions and use isolated runners and test credentials. A check source restriction does not make an editable workflow trustworthy.
Require evidence and test the controls
Run the agreed suite on the change being reviewed and retain the commit, environment, executed scenario IDs, results, and failure evidence. Configure Playwright traces on a failed run or retry and keep artifacts access-controlled. A stable required job should fail when expected scenarios did not run or their result is missing.
- Preserve failed-attempt evidence and track flaky tests. A passing retry must not silently remove an agreed critical scenario from review.
- Keep approval explicit for test quarantine, updated snapshots, reduced coverage, and CI changes. Do not use success-on-error settings for the required gate.
- Rehearse with a disposable pull request: a broken workflow fails CI, a deleted or skipped critical test requires intervention, and a modified harness cannot merge without its owner.
- Use the same rehearsal to confirm that the agent's real credentials cannot bypass the active rules. Record any exception and who controls it.
Illustrative agent brief: change contact validation
Update contact validation within the agreed application files. Run the approved Playwright scenarios for authorized edits, invalid values, and read-only users. Use failure traces to diagnose problems. Do not remove checks, change fixtures to hide a failure, update snapshots, or alter CI to get a pass. If the contract needs to change, explain why and request owner review.
Acceptance criteria
- An authorized edit survives reload; invalid input and a read-only user's update leave the stored record unchanged.
- The CI report identifies the exact commit and shows every agreed critical scenario ran.
- A seeded defect makes the appropriate test fail and produces reviewable diagnostic evidence.
- Changes to tests, fixtures, runner configuration, workflows, or ownership rules receive the required human approval.
- The agent's configured identity cannot merge through a failing required check or waive the review.
The artifacts to keep
- Critical-workflow map and approved acceptance criteria
- Client-owned Playwright tests, fixtures, and run instructions
- CI reports and traces with retention and access rules
- Protected test and workflow ownership with verified required checks
- Agent permission boundary, control rehearsal, and maintenance handover
Limits to keep visible
- No harness is unbreakable. Administrators or other authorized identities can change repository settings, and an identity with applicable bypass rights can circumvent rules. Verify permissions and enforcement in the client's actual repository.
- Passing tests only establish the behavior exercised by those tests. Missing scenarios, incorrect expectations, or compromised execution can still let a defect through; retain human review and other appropriate checks.
- GitHub protection features depend on plan and repository visibility. Confirm availability before promising a particular enforcement setup.
- The workflow and examples are illustrative delivery guidance, not claims about a completed client engagement.
Start with one critical workflow, then verify both the Playwright checks and the repository controls before widening agent access.
Primary sources
- Playwright: Best practicesIndependent tests, user-visible outcomes, resilient locators, and web-first assertions.
- Playwright: Test configurationforbidOnly, test selection, reporters, retries, and trace configuration.
- Playwright: Continuous integrationRunning the suite and collecting reports in CI.
- Playwright: Trace viewerFailure and retry trace options and diagnostic artifacts.
- GitHub: About rulesetsActive enforcement, rule-editing authority, bypass lists, and plan availability.
- GitHub: Available rules for rulesetsRequired reviews, latest-change review controls, and required status-check sources.
- GitHub: About code ownersOwner-review enforcement, base-branch ownership, and protecting CODEOWNERS itself.
- GitHub: GITHUB_TOKEN permissionsRestricting workflow token access to the permissions a job requires.
Written with AI and not reviewed by a human. It may contain mistakes. Examples are illustrative unless an article explicitly provides executed results. These are working methods, not customer case studies or guaranteed outcomes.