Espresso, UI Automator, and an Agent: Taming the Android Device Matrix

If you've read our piece on agentic iOS testing, the spine of this one will feel familiar: reset state, mock the boundary, keep the locators honest, maintain the screen actions, watch the run, version the config, export the artifacts. The framework changes — Espresso for in-process interaction, UI Automator for cross-app and system UI — but the discipline is the same.

What changes is the shape of the problem. iOS gives you a small, well-behaved set of devices and one vendor's idea of how the UI should work. Android gives you fragmentation: a long tail of OEM skins, a spread of API levels you still have to support, screen densities and aspect ratios that reflow your layout, and manufacturer customisations that change how a permission dialog looks or whether a back gesture does what you expect.

That fragmentation is the whole story. A scripted Android suite already spends most of its flake budget on 'works on the emulator, fails on the device' problems. Point an agent at that matrix without preparing for it and you don't get insight — you get the same noise, now narrated in fluent English. The work is in making the matrix legible, and almost all of it happens before the agent's first tap.

Android instrumentation runs your tests in an app process, and engineers assume each test starts clean because the test runner looks like it isolates them. It doesn't isolate the thing that matters. SharedPreferences, the app's SQLite or Room database, the internal files directory, the WebView cache, and account state all survive between tests on the same device. The first test signs in; the second launches into an authenticated home screen it was written to reach from a login flow, and fails on a screen it never planned for.

An agent handles this worse than a script, because it doesn't know what it inherited. It sees a home screen, reasons it must already be signed in, and confidently does the wrong next thing. You log that as a reasoning error and go hunting in the agent. The bug is upstream — a state leak wearing a reasoning error's clothes.

Reset deterministically, from outside the test body. In rough order of how often we reach for each:

You want the agent to drive the real app — real screens, real view models, real navigation — against a backend that never wavers. A live or staging backend brings latency, auth expiry, rate limits, and data that drifts run to run, and every one of those becomes a red test that has nothing to do with the app. On an agentic suite that's corrosive: you can no longer tell a perception or reasoning failure from a backend hiccup, because all three render as red.

On Android the clean seam is the HTTP client. Inject a MockWebServer instance, or register an OkHttp interceptor in debug builds, and serve canned responses from fixtures. The app makes real calls, parses real-shaped payloads, and renders exactly as it would in production. Only the bytes on the wire are deterministic.

The detail that makes this pay off is keying fixtures to named scenarios rather than to endpoints: 'empty feed', 'one unread message', 'payment declined', 'server returns 503'. The agent selects a scenario through an instrumentation argument, so it isn't just poking around — it's driving a specific, reproducible journey. When it fails, you know exactly which backend state it was standing on, on which device.

Espresso can match on view properties and UI Automator matches on the accessibility-exposed tree, but in practice the stable handle an agent relies on is the view's resource-id, backed by content-description for the things screen readers need. That pair is the contract between the app and the test. When it's missing — and it's routinely missing on custom views, RecyclerView rows, and anything drawn with Canvas — the agent falls back to matching on visible text, which is localised, which changes for copy reasons, and which collides (three buttons reading 'OK').

This is the single most common root cause we find behind 'the agent is flaky on Android'. Usually it isn't the agent. It's that nobody owns the identifiers, so they rot: a refactor renames a view, a Compose migration drops the testTag, a new screen ships with nothing stable to grab.

Two habits keep them honest. First, identifiers are set deliberately — android:id and contentDescription on View-based UI, Modifier.testTag and semantics on Compose — and reviewed like any other interface, because that's what they are. Second, the agent audits them: before a run it walks the tree of each screen it knows and flags identifiers that are missing, duplicated, or no longer resolve. That audit is cheap and it catches the rot before it becomes a 2am misclick you waste a morning blaming on the model.

Above the raw identifiers sits a layer of screen actions — Android's version of the page object. 'Sign in as a returning user', 'open the third item in the list', 'pull to refresh and wait for the spinner to clear'. These wrap the taps, waits, and assertions that make up a meaningful unit of behaviour, so the agent reasons in user intent instead of re-deriving a gesture sequence from the screen on every run.

Writing them down rather than improvising each time is what buys determinism. An agent that re-plans how to sign in every run will eventually plan it wrong; an agent that calls a named, stable sign-in action does the same thing every time, and saves its judgement for the screen actually under test. It's the same instinct as pinning tool-call order: fix the parts that shouldn't vary.

But Android screens move — a field shifts, a step joins a flow, a Material component swaps for its Material 3 successor with different semantics — and the old action breaks. The agent detects the break, walks the new screen, and proposes an updated action: new gestures, new identifiers, new waits. It proposes; it does not silently rewrite. A human reviews the diff and a new golden trace is recorded before it ships. The agent does the tedious re-derivation; a person keeps the judgement.

A pass/fail result tells you the assertion held. It says nothing about how the agent got there, and on a fragmented device matrix the most expensive problems are the ones that still pass. We watch the run as a stream and flag anomalies independently of the result:

Everything the agent knows about your app — the screens, their identifiers, the named actions, the mock scenarios, the device matrix it runs against, the bounded step counts — lives in a declarative config. Treat that file as code. It lives in the repo, it's reviewed, and every change is a versioned, traceable diff.

This closes the loop with the anomaly watching. When a run surfaces a broken identifier, a changed screen, or a new OEM interruption that needs handling, the response is not a panicked hand-edit. The agent proposes a version bump — here's the screen that changed, here's the new identifier, here's the device it showed up on, here's the golden trace that now passes — and a human reviews it like any code change.

Versioning the config answers the question every agentic suite eventually faces: did the suite change, or did the app? When the config is versioned alongside the app, the diff tells you. Pin the model version, pin the config version, pin the device matrix, and a regression has exactly one place left to hide — which is the entire point.

Android won't hand you a single tidy bundle the way Xcode's xcresult does, so you have to assemble the ground truth yourself — and on a device matrix you cannot afford not to. The set we always capture: the instrumentation/Gradle test report, full logcat per test, per-step screenshots, and, where the runner supports it, failure video. Add the two artifacts that make agentic triage possible: the screenshot the agent saw and the natural-language plan it emitted at each step.

Capture screenshots and logs through the runner (the AndroidX test APIs write them into the run, and managed device or cloud-device runs collect them per device), then pull them off with adb and attach them to the run. The reason this matters more on Android than anywhere else: the failure you most need to understand is the one that only happened on one device, and you may not have that device on your desk. The recording is how you investigate without re-running and praying it reproduces.

The screenshot-plus-plan pair is also how you split a perception error (the plan describes a screen the screenshot doesn't show) from a reasoning error (the plan reads the screen correctly but picks the wrong move). On a non-deterministic actor running across a non-uniform fleet, the run you can't reproduce is the run you're grateful you recorded.

Read the seven back and the Android-specific weight is obvious. The locator contract is heavier because there's no DOM and custom views routinely ship without identifiers. The artifact set is heavier because there's no native bundle and the failing device may be one you don't own. And the anomaly watching carries an extra job no other platform demands as sharply — separating a device problem from an app problem, every single run.

None of it is exotic. It's the discipline a good Android shop already applied to idling resources, device farms, and TalkBack a decade ago, transposed onto a non-deterministic actor and made more demanding by fragmentation. The framework is mature, the agent is new, and the engineering between them is what decides whether the device matrix becomes signal or just louder noise.