The CRM That's Different in Every Org: Agentic QA for Configurable Enterprise Apps

The first three posts in this series tested apps you build and ship whole. A CRM is different in kind. It ships as a platform, and every customer reshapes it: custom objects and fields, record types, page layouts that change by profile, validation rules, approval processes, and automations that run on save. Two orgs running the same CRM version can present completely different screens to their users. The vendor's QA tested the platform; your QA has to test your org's configuration of it, which no two are alike.

This breaks the assumption underneath traditional UI testing. Selector-based tests assume the DOM is stable enough to address by structure — this button, that field, in this position. A CRM's DOM is generated from configuration, often through a heavy component framework (think dynamically-rendered Lightning-style components, shadow DOM, iframes), and it shifts when an admin changes a layout, adds a field, or a managed package updates. The selectors that passed last sprint resolve to nothing this sprint, and nobody changed a line of application code.

This is exactly the terrain where an agent earns its place. The seven-lever spine from the platform posts still applies — state, boundary, locators, maintained actions, anomaly watching, versioned config, artifacts — but the locator problem dominates, and two new ones appear that mobile apps never forced: per-role rendering and test-data lifecycle. An agent that reasons in intent ('create an opportunity for this account and move it to the proposal stage') instead of brittle selectors is the only thing that survives a surface that reconfigures itself.

On a CRM, the locator contract you wish you had — stable identifiers on every field — is partly out of your hands, because much of the UI is generated from configuration you don't control at the markup level. You can and should add stable identifiers where the platform allows it (many give you API names, component attributes, or automation-id hooks), and the agent audits those the way it audits any locator. But you cannot rely on them everywhere, and that changes the strategy.

The agent compensates by reasoning at the level of intent and meaning rather than structure. It reads the screen the way a user does — this is the 'Stage' picklist because it's labelled 'Stage' and sits in the opportunity layout, regardless of where the admin dragged it this quarter. It finds the 'Save' action by what it does, not by a generated DOM path. This is precisely the capability that makes an agent more robust than a scripted suite on a configurable platform: a layout change that would shatter a hundred selectors is, to an intent-driven agent, just a slightly different arrangement of the same meaningful controls.

The contract you do enforce hard is the API-name layer. Custom fields and objects have stable API names even when their labels and positions change. Anchoring the agent's understanding of the org to those API names — captured in its config — gives you a stable spine under a shifting surface. The label can move; the API name is the identity.

This is the bug class CRM teams underestimate most. A CRM renders by permission: profiles, roles, permission sets, sharing rules, and field-level security all decide what a given user sees and can do. The opportunity screen a sales rep sees is not the one a sales manager sees, which is not the one a read-only finance user sees. A field is editable for one, read-only for another, invisible to a third. A button exists for one role and is absent for the rest.

A suite that tests as one user — usually a system administrator, because that's whose credentials were handy — tests the one role that sees everything and can do anything. It is structurally blind to the bugs that matter: the rep who can suddenly edit a field they shouldn't, the manager who lost access to an approval they need, the finance user who can see salary data through a misconfigured sharing rule. Those are the failures that become compliance incidents.

So persona is a first-class axis of the suite. The agent runs the same intent across a defined set of personas and checks that each sees and can do exactly what their role permits — no more, no less. 'Create an opportunity' should succeed for the rep, and the read-only user attempting the same should hit a wall, gracefully. The agent asserts both the positive and the negative: the absence of a control for a restricted role is as important a result as its presence for an authorized one. Permission regressions are silent by nature — nothing errors, the wrong person just quietly gains or loses access — and replaying journeys across personas is how you make them loud.

On a mobile app, 'state between tests' mostly meant local storage and caches. On a CRM, state is the database of records, and it's both the thing under test and the thing that poisons tests when it leaks. A test that creates an account, an opportunity, and a quote leaves all three behind. The next test finds a world with more records than it expected, search returns extra hits, a uniqueness rule trips, a rollup field carries a number from last run. The agent, reasoning about what it sees, inherits a polluted org and reasons wrongly about it.

Reset has to happen at the data layer, deterministically, and from outside the test body. The levers, in rough order:

A CRM is a hub: it sends email, places and logs calls through telephony, charges cards, syncs with ERP and marketing platforms, and calls external services from its automations. For testing, every one of those outbound dependencies is a source of non-determinism and, worse, a source of real-world side-effects you do not want a test to trigger. A test that actually sends the email, actually charges the card, actually fires the webhook to a partner system is a test that will eventually cause an incident outside your walls.

So mock at the integration boundary. Stub the email service, the telephony provider, the payment gateway, and the outbound callouts with deterministic responses, and assert against the mock: did the CRM attempt to send the right email to the right contact, rather than did an email actually arrive. Keep the fixtures keyed to named scenarios — 'payment approved', 'payment declined', 'ERP sync conflict', 'partner webhook times out' — so the agent drives specific, reproducible business situations.

This is the same 'mock the boundary' principle from the platform posts, with higher stakes. On a mobile app, an unmocked network call mostly just flakes. On a CRM, an unmocked integration can email a real customer, charge a real card, or push bad data into a real downstream system. Mocking the boundary here is not only about determinism; it's about containment.

Here is what makes a CRM genuinely hard to test: the most important things that happen are invisible. You save a record and, behind the screen, automations fire — a workflow updates a field, a trigger creates a follow-up task, an approval process routes for sign-off, a rollup recalculates, an outbound message queues. None of that is on the screen the user just looked at. A test that checks only the visible result of a save misses most of what the save actually did.

So watching, on a CRM, means watching for side-effects, not just grading the visible screen. The agent flags as anomalies the things that happen out of view:

The agent's config for a CRM is richer than for an app, because it has to capture the org's configuration: the custom objects and fields by API name, the personas and their expected permissions, the business processes as named multi-step journeys, the integration mocks, the automations it expects to fire on each action. Treat that map as code — versioned in the repo, reviewed, every change a diff.

This matters more on a CRM than anywhere else in the series, because the app under test changes without any developer touching code. An admin adds a field, changes a layout, edits a validation rule, installs a managed package update. To everything downstream, that's a silent change to the application. When the agent's org-schema map is versioned, you can see it: the diff between what the agent expected and what the org now presents is the change record the admin didn't write. The agent proposes the version bump — here's the new field, here's the layout that moved, here's the automation that now fires — and a human reviews it.

Pin the model version, version the org-schema map, and you can finally answer the CRM team's perpetual mystery: did the test break because the code changed, because an admin changed a setting, or because a package updated? With the map versioned, the diff names the culprit instead of leaving you to guess across three teams who all swear they changed nothing.

CRMs hold regulated data and sit inside processes that auditors care about — who can see what, who approved what, whether controls actually work. That raises the value of the artifact set beyond debugging: the run is also evidence. Capture the full trace — per-step screenshots, the persona under test, the data state before and after, the integration calls attempted, the automations observed to fire — and keep it.

For triage it does the usual work: the agent's screenshot plus the plan it emitted separates a perception error (the plan describes a screen the screenshot doesn't show) from a reasoning error (the plan reads the screen right but picks the wrong move). For a CRM it does more. A run that demonstrates a read-only finance user could not access salary data, captured and dated, is a control test you can hand an auditor. A run that shows the approval process routed correctly for each persona is evidence the process works as designed.

Export it in a form the business can read, not just the engineers — which persona did what, what the system did in response, what stayed hidden. On a platform where the most important behaviour is invisible and the stakes are compliance, the recorded run is both how you debug the failure and how you prove the controls hold.

A CRM stresses the parts of the framework that mobile apps let you off lightly. The locator contract inverts — you stop relying on the DOM and lean on intent and API names, because the surface is configured, not coded. Two axes appear that the app posts never needed: persona, because the screen renders by permission, and test data, because state is a database of records. And anomaly watching turns into side-effect watching, because on a CRM the consequential behaviour happens out of view, on save.

What carries over is the discipline. Reset state deterministically (here, data). Mock the boundary (here, integrations, for containment as much as determinism). Version the config (here, the org schema, because admins change the app without code). Keep the artifacts (here, as audit evidence). The technology is the same agent; the enterprise platform just demands you take every lever more seriously, because the failures cost more and hide better.