Test Automation That Survives a Redesign

Most abandoned test suites were not abandoned because automation failed. They were abandoned because a redesign shipped, four hundred tests went red in a way that had nothing to do with behaviour, and fixing them was a fortnight of work nobody had budgeted.

After that happens once, the suite is run less. After it happens twice, it is deleted. The technical cause is almost always the same, and it is decided on the first day of writing.

Anchor to intent, not to appearance

A test that finds a button by its position in the DOM, or by a generated class name, or by an XPath expression describing a route through the markup, is not testing the application. It is testing the markup, and the markup is the thing most likely to change for reasons unrelated to behaviour.

The alternative is to anchor on things that only change when meaning changes. In order of preference: an explicit test identifier added for the purpose, the element's accessible role and name, its visible label.

The accessible-role approach is worth singling out because it pays twice. A test that locates a control the way a screen reader would fails when the control stops being reachable — so the same assertion that protects the test from a redesign also catches an accessibility regression. Those are not usually the same work, and here they are.

Explicit test identifiers attract an objection: that they put test concerns into production markup. In practice the trade is heavily favourable. A stable attribute is a contract between the interface and its tests, and the alternative contract — "the third div inside the second section" — is one the design team never agreed to and will break without knowing.

Assert on outcomes, not on the path taken

The second cause of mass breakage is tests that encode a journey rather than a result.

A test that clicks through six screens in a fixed order is asserting that the flow has six screens in that order. Merge two of them and the test fails even though the user can still do the thing. That is a false negative, and false negatives are what destroy confidence in a suite: once people expect red, red stops being information.

Write the assertion against the outcome. The order was created. The record shows the new state. The email is queued. Get to that outcome by the shortest available route, and use the interface only where the interface is what you are actually testing.

Set up through the back door

Much of what makes end-to-end tests slow and fragile is setup performed through the UI: creating an account by filling the registration form, adding items by clicking through the catalogue.

None of that is under test in a checkout scenario, and all of it can break the checkout scenario. Create the state directly — API, fixture, seeded database — and drive the interface only for the behaviour the test exists to verify. Suites restructured this way typically get several times faster and dramatically less flaky, because the number of UI interactions that can go wrong has dropped by an order of magnitude.

Treat flakiness as a defect with an owner

A test that fails one run in twenty is worse than no test. It trains the team to re-run rather than investigate, and that habit applies equally to the real failure when it comes.

The rule that works is unsentimental: a test that flakes is quarantined immediately and fixed or deleted within a fixed window. Keep the flake rate visible next to the pass rate. Most flakiness resolves to a small number of causes — waiting on a fixed timer instead of a condition, tests sharing mutable data, animations, a race between navigation and assertion — and each has a known fix.

Decide what does not get automated

Automation has a maintenance cost per test, forever, and some tests do not repay it.

Areas under active design churn are usually better covered manually until they settle. Purely visual judgement is better served by screenshot comparison than by assertions about pixels. And a scenario that runs once a quarter may be cheaper to do by hand than to keep working.

A smaller suite that is trusted and green is worth far more than a comprehensive one that everybody ignores. The measure to watch is not coverage. It is whether anyone believes the result when it goes red.