Pointing an Agent at XCUITest: The Seven Things That Decide Signal From Noise

Most teams meet agentic QA on the web first. The agent gets a screenshot and a DOM, it reasons about what to click, and a tool layer translates that into a Playwright or Selenium call. The DOM matters here: even when the agent works visually, there is a complete, queryable description of the page underneath it as a fallback and a ground truth.

iOS takes that away. There is no DOM. XCUITest sees the app through the accessibility tree — the same tree VoiceOver uses — surfaced across a process boundary by a proxy that talks to your app under test. Elements resolve by accessibility identifier, label, and type, not by CSS selector. The app runs in a simulator (or on a device) that carries state between launches. And the framework relaunches your app, from a cold process, on essentially every test.

None of this makes agentic QA on iOS impossible. We run it. But it moves where the work is. On the web, most of the engineering is in the harness around the agent. On iOS, a surprising amount of it is in the app and the environment — decided before the agent ever makes its first tap. There are seven of these, and they are the difference between an agent that produces signal and one that produces a wall of non-reproducible red.

Here is the trap. XCUITest launches a fresh process for each test, so engineers assume each test starts clean. It doesn't. The process is new; the simulator isn't. UserDefaults, the Keychain, Core Data or SQLite stores, the URL cache, and downloaded files all survive the relaunch. The first test logs in and writes a token to the Keychain. The second test launches, finds the token, skips the login screen it was written to exercise, and fails on a screen it never expected to see.

For an agent this is worse than for a scripted test, because the agent doesn't know what state it inherited. It sees a home screen where it expected a login screen, reasons that it must already be authenticated, and confidently does the wrong thing. You log that as a reasoning error. It isn't. It's a state-leak error wearing a reasoning error's clothes, and you'll chase it in the wrong place for a day.

Reset deterministically, and do it from outside the test logic. The levers, in order of how much we reach for them:

You want the agent to exercise the app the way a user does — tap through the real screens, real navigation, real view models — without the real backend underneath. A live backend introduces latency, rate limits, auth expiry, and data that changes between runs. Every one of those turns into a test failure that has nothing to do with the app, and on an agentic suite it's poison: you can no longer tell a perception or reasoning error from a backend hiccup, because they all look like red.

The clean seam on iOS is the HTTP layer. Register a URLProtocol subclass at launch — again, gated by a launch flag so it only loads under test — and serve canned responses from fixtures. The app makes its real network calls, the view models parse real-shaped payloads, the screens render exactly as they would in production. Only the bytes on the wire are deterministic.

The discipline that makes this pay off is keying fixtures to scenarios rather than to endpoints. "Empty inbox", "inbox with one unread", "payment declined", "server returns 500" — each is a named bundle of responses the agent selects via launch environment. Now the agent isn't just clicking around; it's driving a specific, reproducible journey, and when it fails you know precisely which backend state it was standing on.

On the web the agent can fall back to the DOM. On iOS the accessibility tree is all it has, and accessibility identifiers are how XCUITest names things in that tree. They are the contract between the app and the test. When they're missing, the agent is reduced to matching on visible labels — which are localised, which change for copy reasons, and which collide (three buttons that all say "Done"). When they're duplicated or unstable, element resolution becomes ambiguous and the agent's taps land on the wrong control.

This is the most common root cause we find behind "the agent is flaky on iOS". It usually isn't the agent. It's that nobody owns the identifiers, so they rot — a refactor renames a view, a new screen ships with none, a designer's relabel quietly changes what the agent was matching on.

Two things keep them honest. First, identifiers are set deliberately in the app — accessibilityIdentifier on UIKit views, the .accessibilityIdentifier() modifier in SwiftUI — and reviewed like any other interface, because they are one. Second, the agent audits them. Before a run, it walks the accessibility tree of each screen it knows about and flags identifiers that are missing, duplicated, or no longer resolve. That audit is cheap and it catches the rot before it becomes a 2am misclick that you waste a morning calling a reasoning error.

Above the raw identifiers sits a layer of screen actions — the iOS equivalent of page objects. "Log in as a returning user", "open the third item in the list", "pull to refresh and wait for the spinner to clear". These encapsulate the sequence of taps, waits, and assertions that make up a meaningful unit of behaviour, so the agent reasons in user intent instead of re-deriving a tap sequence from pixels every single run.

The reason to write them down rather than let the agent improvise each time is determinism. An agent that re-plans how to log in on every run will eventually plan it wrong; an agent that calls a named, stable login action does the same thing every time, and the variation is saved for where it earns its keep — exploring the screen under test. This is the same instinct as seeding the tool-call order on the web: pin the parts that shouldn't vary.

But screens change, and that's where the agent earns a second job. When the UI shifts — a field moves, a step is added to a flow, a button is renamed — the old screen action breaks. The agent detects the break, walks the new screen, and proposes an updated action: the new tap sequence, the new identifiers, the new waits. Crucially it proposes; it doesn't silently rewrite. A human reviews the diff and a new golden trace is recorded before the change ships. The agent does the tedious part — re-deriving the path through a changed screen — and a person keeps the judgement.

A pass/fail result tells you whether the assertion held. It tells you nothing about how the run got there. On iOS especially, the interesting problems live in the gap between green and clean — a run can pass while doing something it shouldn't, and that's the regression you ship.

So we watch the execution as a stream, not just grade the result at the end, and we flag anomalies independently of pass/fail:

Everything the agent knows about your app — the screens, their identifiers, the named actions, the mock scenarios, the bounded step counts — lives in a declarative config. Call it agent.yml. The single most important decision you make about that file is to treat it as code: it lives in the repo, it's reviewed, and every change to it is a versioned, traceable diff.

This closes the loop with the anomaly watching. When the run surfaces an anomaly — an identifier that no longer resolves, a screen action that broke, a new interruption that needs handling — the response is not to hand-edit the config in a panic. The agent proposes a version bump: here is the screen that changed, here is the new identifier, here is the updated action, here is the golden trace that now passes. A human reviews it exactly like a code change, because it is one.

Versioning the config is what lets you answer the question every agentic suite eventually faces: did the suite change, or did the app? When agent.yml is versioned alongside the app, the diff tells you. When it's an opaque blob someone tweaks between runs, you're back to arguing about a red dashboard with no way to settle it. Pin the model version, pin the config version, and a regression has exactly one place left to hide — the app — which is the whole point.

When xcodebuild runs your tests it produces an .xcresult bundle, and most teams glance at the pass/fail summary and bin it. That bundle is the single best triage artifact Xcode gives you, and for an agentic suite it's close to essential. Inside it: per-step screenshots, the full test activity timeline, console and system logs, performance metrics, and any attachments you chose to capture — including, if you wire it up, the screenshot the agent saw and the natural-language plan it emitted at each step.

Capture it deliberately with xcodebuild test -resultBundlePath, and read it programmatically with xcrun xcresulttool, which emits JSON you can parse. Then feed it back into triage. This is where the iOS suite finally gets the thing the web suite had for free: a ground truth to check the agent's perception against. The screenshot from the bundle plus the plan the agent wrote is exactly the pair you need to split a perception error (the plan describes a screen the screenshot doesn't show) from a reasoning error (the plan reads the screen right but picks the wrong next move).

Export it, attach it to the run, and keep it long enough to investigate. The bundle is also how you reconstruct an anomaly after the fact — the slow transition, the unexpected dialog, the double-tap — without re-running and hoping it reproduces. On a non-deterministic actor, the run you can't reproduce is the run you're glad you recorded.

Read the seven back and a pattern shows up. Only two of them — watching the run and reading the bundle — are about the agent at runtime. The other five are about the environment the agent runs in: clean state, mocked network, honest identifiers, maintained screen actions, versioned config. That ratio is the real lesson of doing agentic QA on iOS. The browser lets you put most of the engineering in the harness; iOS makes you put it in the app and the environment first.

It's the same discipline a good iOS test shop already applied to launch arguments, mock servers, and accessibility a decade ago — transposed to a non-deterministic actor and made a bit more demanding because the actor can't read a DOM as a safety net. The framework is mature. The agent is new. The engineering between them is what decides whether you get signal or a wall of red, and almost all of it is decided before the first tap.