Wiring an LLM into Your Playwright Test Runner

An end-to-end test fails at 02:13. The screenshot shows the checkout form. The trace contains the DOM snapshot. The error says that page.locator('.submit-btn') matched nothing. By morning, an engineer has opened the trace, inspected the markup, replaced the selector, and rerun the pipeline.

That workflow is useful—but most of it is pattern recognition. A language model can accelerate it. The dangerous leap is allowing the model to edit the test suite and commit its own “repair.” A locator change can make a test green while silently weakening what the test proves.

The safer pattern is simpler: let the model read a deliberately restricted failure bundle, return a structured diagnosis and a proposed patch, then require deterministic validation and human approval before any code changes.

The boundary: recommendation, not mutation

Treat the LLM as an untrusted diagnostic service. Its job is to answer three questions:

1. 01What failure class is most likely?
2. 02What evidence supports that classification?
3. 03What minimal change should a reviewer consider?

It should not receive a repository token, a writable checkout, shell access, production credentials, or permission to update snapshots. The test runner exports evidence to a separate triage process. That process calls the model and stores its response as an artifact or pull-request comment.

This boundary matters because a failed locator is not always a broken test. It may expose a product regression, missing accessibility semantics, an authentication failure, stale test data, or a page that never finished loading. “Fixing” the selector before identifying the failure class can hide the defect.

Build a small, sanitized failure bundle

Do not send the entire repository or raw trace by default. Create a bounded JSON payload containing only what the diagnosis needs:

{
"test": "guest can place an order",
"project": "chromium",
"error": "Timed out waiting for getByRole('button', { name: 'Place order' })",
"currentLocator": "getByRole('button', { name: 'Place order' })",
"step": "submit checkout",
"domExcerpt": "<button data-testid='place-order'>Confirm order</button>",
"consoleErrors": [],
"networkFailures": [],
"retryOutcome": "passed-on-retry",
"allowedFiles": ["tests/checkout/guest-order.spec.ts"]
}

Redact access tokens, cookies, customer data, email addresses, request bodies, and secrets before the payload leaves the runner. Cap every field. A 200-line DOM excerpt around the intended element is usually more useful than a megabyte of markup.

Playwright already supplies the right evidence sources: test errors, attachments, screenshots, traces, and reporter hooks. Its locator guidance also gives the model a sound hierarchy: prefer role, label, text, and explicit test-id contracts over brittle CSS or XPath. Locators provide auto-waiting and retryability, but that does not make every locator equally maintainable. See the official Playwright locator guidance and best practices.

Require a machine-checkable response

Never accept a free-form answer as the integration contract. Validate the model output against a schema:

type TriageResult = {
classification:
| 'product-defect'
| 'locator-drift'
| 'test-data'
| 'environment'
| 'timing-or-flake'
| 'unknown';
confidence: number;
evidence: string[];
proposedLocator?: string;
rationale: string;
validationCommands: string[];
needsHumanReview: true;
};

Reject unknown fields, oversized strings, invalid commands, confidence outside 0..1, and output that references files outside the allowlist. The needsHumanReview field is deliberately fixed to true: the model cannot approve itself.

Your prompt should explicitly prohibit weakening assertions, adding sleeps, increasing timeouts, using .first() to suppress strictness errors, deleting coverage, or changing application code. Ask for “insufficient evidence” when the bundle does not support a conclusion.

Make the model prove the locator is better

A plausible replacement is not enough. The proposed locator must survive deterministic checks:

• —It resolves to exactly one element in the captured state
• —It reflects user-visible behavior or an explicit testing contract
• —It does not cross an iframe or shadow boundary incorrectly
• —It remains stable across the browsers and projects relevant to the test
• —The original assertion still verifies the same business outcome
• —The test passes repeatedly without relying on retries

Playwright’s strict locators are helpful here: operations that imply one target fail when multiple elements match. Keep that signal. Replacing a strict locator with .first() often converts an honest failure into a false pass.

Run the candidate in an isolated temporary checkout owned by your validation service—not by the model. Apply the proposed diff there, execute the single failing test several times, then run the surrounding feature pack. Preserve the before-and-after trace for review. The model proposes text; deterministic automation decides whether that text even qualifies for human consideration.

Separate triage from merge authority

A production-ready workflow has four identities:

1. 01The test runner can write artifacts, not source.
2. 02The triage service can read sanitized artifacts and call the model.
3. 03The validator can create an ephemeral patch and execute tests.
4. 04A human or tightly governed bot can open a pull request.

Do not combine those identities into one overpowered CI token. Restrict network egress, pin dependencies, log model and prompt versions, and retain the evidence hash with the response. If the model output becomes a PR comment, escape Markdown and never execute commands copied from it without an allowlist.

Measure diagnostic value, not green builds

Track the workflow like a QA product. Useful metrics include classification precision, accepted suggestion rate, false-fix rate, median time to diagnosis, repeat failure within 30 days, and reviewer time saved. Slice the data by failure class. A model may be strong at renamed accessible labels but poor at distinguishing backend latency from UI timing.

Create an offline evaluation set from historical failures. Remove the final fixes, ask the system to diagnose each case, and compare its response with the reviewed resolution. Add every harmful or misleading suggestion to a regression set. This is how the workflow improves without experimenting on the main branch.

“Self-healing tests” are attractive because they promise less maintenance. The better goal is evidence-assisted maintenance: faster diagnosis, smaller patches, visible uncertainty, and unchanged accountability. Give the model enough context to help—but never enough authority to redefine what passing means.