AI Playwright Tests: Write Stable End-to-End Tests

AI Playwright tests are quick to produce and easy to get wrong. An assistant can turn a user story into a working end-to-end test in a minute, but the first draft often relies on brittle selectors, fixed sleeps and data that exists only on your machine. Those tests pass today and fail at random next week. The skill is not asking an assistant for tests. It is asking for the right kind of test, and then running every one of them before you trust it.

This guide shows how to prompt for stable Playwright tests in TypeScript or JavaScript, which patterns to demand, which mistakes to look for in generated code and how to fit the result into a pipeline. It assumes you know what an end-to-end test is and have Playwright installed.

What end-to-end tests are for

An end-to-end test drives a real browser through a real flow: sign in, add an item to the basket, pay, see the confirmation. It is slower than a unit test and it fails for more reasons, so you write fewer of them and aim them at the paths that matter most. A sensible suite for a small product has a handful of tests for sign-up and sign-in, the main purchase or save flow, the main settings and one or two admin actions. Keep them few and valuable, and cover everything else with faster unit and integration tests.

Start from a clear description of behaviour, written the way a user would say it. If you have user stories with acceptance criteria, paste them. The guide to AI user stories shows how to write criteria that translate directly into test steps. An assistant that receives vague instructions such as “test the checkout” will invent steps, and the test will check what it imagined.

Prompting for AI Playwright tests

A good prompt contains the framework and language, the page or flow, the exact steps, the expected results, the test data rules and the style rules. Here is a pattern you can adapt:

Write a Playwright test in TypeScript for this flow.
Flow: a signed-in user adds a product to the basket and sees it in the basket page.
Rules:
- use getByRole, getByLabel or getByText, not CSS or XPath
- use web-first assertions, never waitForTimeout
- no hard-coded data from my database; create data through the API fixture
- one behaviour per test, clear test name
- list every assumption you made about the page

Paste the HTML of the relevant part of the page, or a description of the visible text and roles. Without it, the assistant guesses button names. The last line, asking for assumptions, is the most useful: it shows you where the draft depends on something it could not see. Then run the test and fix the assumptions against the real page.

Locators that survive change

Most flaky tests fail because of locators. Generated code often uses long CSS chains such as div.container > div:nth-child(3) button, which break when the layout changes. The Playwright best practices recommend locators that reflect what users and assistive technology see: roles, labels and visible text, with test ids for elements that have no good accessible name.

Locator style	Example	Stability	When to use it
Role	getByRole(‘button’, { name: ‘Add to basket’ })	High	Buttons, links, headings, inputs
Label	getByLabel(‘Email’)	High	Form fields
Text	getByText(‘Order confirmed’)	Medium	Messages and static copy
Test id	getByTestId(‘basket-count’)	High, if agreed	Elements without a clear role or name
CSS or XPath	div > ul li:nth-child(2)	Low	Last resort

Tell the assistant which style to use and ask it to flag any locator it could not make role-based. Role locators also nudge you towards accessible markup, because a button that has no accessible name is hard to find in the test and hard to use with a screen reader. That is a free accessibility check.

Waiting without sleeping

The second source of flakiness is timing. Playwright waits automatically for elements to be ready before it clicks or types, and its assertions retry until they pass or time out. Generated code often ignores this and adds await page.waitForTimeout(3000), which wastes time when the page is fast and fails when it is slow. State in the prompt that fixed sleeps are forbidden.

Ask for web-first assertions such as await expect(page.getByRole('heading', { name: 'Basket' })).toBeVisible() instead of reading a value and comparing it once. When a test must wait for a network response, wait for that specific response or for a visible result, not for the network to be “idle”. If a step still fails intermittently, run it with the trace viewer and look at what happened, instead of adding retries. Retries in configuration are a safety net for rare infrastructure noise, and they should not hide a real race.

Test data and isolation

A reliable test creates what it needs and does not depend on leftovers. Each Playwright test runs in a fresh browser context by default, so cookies and storage do not leak between tests, but your database does. Ask the assistant to create data through an API call or a seed helper in a fixture, to use unique values such as a timestamp in an email address and to clean up afterwards if the data persists. Our guide to an AI test data generator explains how to produce realistic fake data without using real people.

Signing in through the user interface in every test is slow. Playwright can save the signed-in state of a browser context once and reuse it, using a setup project and a stored state file. Ask for that pattern for any suite with more than a few authenticated tests, and keep the credentials in environment variables, never in the test file or the prompt. Never paste real passwords or tokens into an assistant.

Structure: fixtures, page objects and naming

Generated tests tend to repeat the same steps. Ask for a small page object or a helper function when three tests share a flow, and for a fixture when they share setup. Keep the abstraction light: one class per page with methods that describe user actions, such as addToBasket(name), and no hidden assertions inside the methods that make failures hard to read.

Name tests as behaviours: “user sees an error when the card is declined”. Group related tests with test.describe. Keep each test short, ideally under fifteen lines, and ask the assistant to explain any test that is longer. One behaviour per test makes failures easy to understand and avoids tests that fail for two unrelated reasons. For visual checks, use screenshot comparison sparingly and only for stable components, since small rendering differences between machines produce noise.

Mobile views, browsers and accessibility checks

Playwright can run the same test in several browser engines and with phone-sized viewports, which is useful when your product has a different layout on a small screen. Ask the assistant to keep one test for the behaviour and to configure projects for each browser and device, rather than writing a separate copy of the test for each. Run the full set of browsers on a schedule and a smaller set on every change, so that feedback stays fast.

The same role-based locators also help you find accessibility problems. If a button cannot be found by its name, a screen reader user cannot find it either. You can add automated accessibility scans to a few key pages, but a scan finds only some problems, so treat it as a smoke alarm and not as a certificate.

Reviewing generated tests

Treat a generated test like a pull request from a new colleague. Read it, run it, break it and run it again. Change the application on purpose, for example by renaming the button, and confirm that the test fails. A test that cannot fail proves nothing. The checklist below covers the most common problems.

• Does the test assert something meaningful, or only that the page loaded?
• Are all locators role, label, text or test id based?
• Is there any fixed sleep, or any assertion that runs once without retrying?
• Does the test depend on data that may not exist on another machine?
• Does it clean up, or pollute the environment?
• Does it pass ten times in a row, and in parallel with the others?

Our guides on AI-generated tests and when not to trust AI-generated code explain why a test that was written by the same assistant that wrote the code can share its blind spots. Whenever you can, write the expected behaviour first from the requirement, not from the code.

Running tests in CI

A test suite helps only when it runs on every change. Run the tests in a clean environment with the same browsers, the same data and the same versions each time. Playwright can record a trace, screenshots and a video for failed tests, and uploading them as artifacts turns “it failed in CI” into something you can examine. Keep the suite fast by running tests in parallel and by splitting slow flows into a separate job.

An assistant can draft the workflow file for you; see our guide to an AI GitHub Actions workflow. Review it for secrets, caching and timeouts. When a test fails, paste the error, the test and the trace summary into the assistant and ask for the most likely cause and a fix, as described in how to debug code from an error message using AI. Do not accept a fix that only adds a retry or a longer timeout until you know why the test failed.

How Ask Mio helps with Playwright tests

Ask Mio is one assistant with five modes, and Mio routes each request to a suitable model. On the free plan, Chat and Write can draft a test from a user story and explain a failing one. The Coding plan adds Code mode with a code execution sandbox, which suits snippets and helper functions, and Research mode with web search and cited sources helps you check the current Playwright documentation instead of relying on memory. Playwright changes between releases, so always compare generated code with the official documentation.

Upload a page’s HTML or a failing log on a paid plan, and keep your style rules in memory or in a project so that every new chat starts with them. Mio cannot see your private application or run your browser, so you still run the tests yourself. Chats and files stay yours, can be exported or deleted and are not used to train models. Check the live pricing page for current points, file sizes and plan features.

Frequently Asked Questions

Can AI write Playwright tests that do not flake?

It can if you demand the right patterns: role and label locators, web-first assertions, no fixed sleeps and data created inside the test. Without those rules, generated tests often contain brittle selectors and timeouts. Run each new test many times, in parallel, and change the page on purpose to see it fail.

Should I paste my page HTML into the assistant?

Paste only the part of the markup that the test needs, and remove secrets, real user data and private URLs. Without the markup, the assistant guesses names and roles. With it, the locators match your page. For private code, check your provider’s data policy before you paste anything.

How many end-to-end tests do I need?

Fewer than you think. Cover the flows where failure costs money or trust, such as sign-in, checkout and the main save path, and test the rest with faster unit and integration tests. A small, stable end-to-end suite is worth more than a large one that people ignore.

Why does a generated test pass locally but fail in CI?

Common causes are missing test data, different timing on slower machines, fixed sleeps, a different browser version or reliance on state left by earlier tests. Use the trace from the failed run, reproduce the CI conditions locally and fix the cause. Adding retries only hides the problem.

Which Ask Mio plan helps with Playwright tests?

The free plan can draft and explain tests in Chat and Write. The Coding plan adds Code mode and a code execution sandbox, and paid plans add Research with cited sources, file uploads and memory. Check the live pricing page for current points and limits before choosing.

The Bottom Line

AI Playwright tests save time when you give the assistant a clear behaviour, strict style rules and real page markup, then run, break and review every result. Insist on role-based locators, web-first assertions and self-contained data, and keep the suite small and focused. You can begin on the free plan and compare the options on the Ask Mio pricing page.