Ten Thousand Article Templates and Three Ad Networks: Agentic QA for Publishers

The platform and CRM posts tested applications with a knowable set of screens. A publisher breaks that assumption from the other direction. There is no fixed set of pages. There is a small set of templates — article, liveblog, gallery, section front, author page — and a content pipeline that pours thousands of pieces of content through them, including content that doesn't exist yet at the moment you write your tests. The page a reader hits tomorrow is rendered from today's template and tomorrow's article.

So the thing under test is the template, and the failures live in the gap between the template and the content it has to survive. The headline that's three words on the mock and 200 characters in reality, overflowing the hero. The article with no lead image, where the layout assumed one. The embedded tweet, TikTok, or interactive that pushes everything below it off the grid. The pull quote inside a numbered list inside a table. Editorial reality generates content permutations no designer drew and no hand-written test anticipated.

This is where an agent's ability to generate and explore beats an enumerated suite. Rather than hand-writing a case per article — impossible, the content is unbounded — the agent generates representative content permutations and drives the template against them, looking for the arrangement that breaks. The seven-lever spine from the rest of the series still holds; what changes is that the input space is content, and the agent's job is to sample it intelligently instead of pretending it's finite.

Hand-writing test content for a publisher is a losing game: you write ten articles, the suite passes, and the eleventh real article — the one with the absurdly long headline and no image and four embeds — breaks the hero in production. The agent flips this. Instead of a fixed corpus, it generates content permutations designed to stress the template's assumptions, and drives the page against each.

The permutations that matter are the ones editorial actually produces at the extremes:

A publisher renders the same article differently depending on who's asking. An anonymous reader hits a metering counter ('3 free articles this month'); a registered-but-free reader sees a different gate; a subscriber sees the full piece; a reader who's hit their meter sees the wall. Entitlement, metering count, region, and referrer (the 'arrived from Google' grace some sites grant) all change what renders. The article is one piece of content with several presentations, decided by reader state.

That makes the reader tier a first-class persona axis, much like role was on the CRM. The agent replays the same article across personas — anonymous under the meter, anonymous over the meter, free registered, active subscriber, lapsed subscriber — and asserts each sees what they should: the subscriber gets the whole article, the over-meter anonymous reader gets the wall and not a leaked full body in the page source, the free reader gets exactly the preview length the business intends.

Metering is itself state that has to be reset deterministically, the way every state in this series does. The agent controls the meter count — via the cookie, local storage, or a test entitlement hook — so 'reader has read two of three free articles' is a reproducible starting condition rather than an accident of which tests ran before. And one assertion earns its keep more than any other: that the paywalled body is genuinely absent from the response for a gated reader, not merely hidden with CSS. A paywall that ships the full text and hides it visually is a paywall that a determined reader — or a scraper — walks straight through.

Third-party scripts are to a publisher what integrations were to the CRM: the boundary you don't own and can't trust. Ad networks, analytics, consent-management platforms, recommendation widgets, and social embeds load asynchronously, render unpredictably, inject content of variable size, and fail in ways you can't reproduce. They are the single largest source of both layout instability and performance regression on a publisher site, and they behave differently on every load.

This needs a two-sided approach, and it's worth being explicit about both. To test the template's own correctness, mock the third-party boundary — serve deterministic, fixed-size ad slots and stubbed embed responses — so the layout you're asserting against doesn't shift because an ad network felt like serving a different creative this second. The agent drives the page with the boundary mocked and checks the template behaves.

But you cannot only mock, because the third-party scripts are themselves a primary cause of the failures readers experience. So the agent also runs with the real (or realistically-sized) ad payloads and measures the damage: the cumulative layout shift as a late-loading ad shoves the article down just as the reader goes to tap a link, the largest-contentful-paint delay while a render-blocking script resolves, the interaction latency under a heavy tag load. Mock to test your layout in isolation; measure with the ads to catch what they do to real readers. Both are the suite; neither alone is enough.

Publishers run experiments and personalization constantly — A/B tests on paywall copy and placement, personalized homepages and recommendation rails, regional editions, breaking-news takeovers. That means the page is non-deterministic by design: two readers, two different pages, on purpose. For a test, undisciplined personalization is just noise — the agent can't tell a real regression from a variant it happened to be served.

The fix is the same instinct as pinning temperature or tool-call order in the determinism post: pin the variant under test. The agent forces a specific experiment assignment and personalization context — via the assignment cookie, a query override, or a test hook — so it knows which variant it's looking at and can assert against that variant's intended behaviour. Then it can test each variant deterministically in turn, rather than being surprised by whichever one the bucketing served.

This also lets the agent do something a human QA can't do at scale: systematically walk every active variant of a critical surface — every paywall test cell, every homepage personalization segment — and confirm each one renders and converts as designed. An experiment that's quietly broken in one cell, serving a paywall that never appears or a layout that collapses, is a revenue leak that hides precisely because only a fraction of readers see it. Pinning lets the agent find it.

For a publisher, structured data is not a nice-to-have — it's how the content surfaces in search, in news aggregators, in AI answers, and how the site earns the rich results and Top Stories placement that drive a meaningful share of traffic. Article schema, author and publisher markup, breadcrumb, liveblog and video structured data, and the metadata that decides how a link unfurls on social are all generated by the same templates that render the visible page, and they break the same way: silently, on the content permutations nobody tested.

So the agent validates structured data as part of the same run that tests the visible template, across article types. It checks that each template emits valid, complete markup for its content type — that the article has a headline, a date, an author, and an image in its structured data; that a video article carries video markup; that the markup matches what's actually on the page rather than a stale default. And it does this across the permutations from lever one, because the structured data breaks on the same edge cases the layout does: the missing image that leaves the schema's image field empty, the missing author, the headline that got truncated in one place but not the other.

This is invisible-but-consequential in the same way CRM automations were. Nothing on the visible page looks wrong when the Article schema is missing its image field; the page renders fine. But the rich result doesn't appear, the click-through drops, and the traffic quietly erodes. The agent watching structured data alongside layout is how you catch the break that costs you search visibility weeks before anyone connects the dip to a template change.

The agent's config for a publisher captures the templates, the content types and the components each allows, the reader personas, the experiment variants, the third-party boundaries it mocks, and the Core Web Vitals budgets it holds the templates to. Version it like code. When a template changes — a new component, a reflowed hero, a different ad placement — the agent proposes the versioned update and a human reviews it, exactly as in the rest of the series.

The artifacts that matter for a publisher are visual and performance proof. Capture per-permutation screenshots (the rendered template against each stress-test content case), the layout-shift and Core Web Vitals measurements under real ad load, the structured-data validation results, and — as everywhere — the agent's-eye screenshot plus the plan it emitted, to split perception errors from reasoning errors. The visual artifact is what turns 'the embed broke the layout' into a screenshot you can show an editor, and the performance artifact is what turns 'the page feels slow' into a CLS number you can attribute to a specific ad slot.

Pin the model version, version the templates-and-content-types map, and you can answer the publisher's version of the recurring question: did the layout break because the template changed, because editorial published a content shape we never tested, or because an ad network changed what it serves? The versioned config plus the captured permutations name the cause instead of leaving the newsroom and engineering blaming each other.

A publisher inverts the input problem. The platform and CRM posts had a knowable set of screens and the challenge was state, permission, and side-effects. Here the screens are effectively infinite — templates times content permutations times reader tiers times experiment variants — and the challenge is sampling that space intelligently rather than pretending it's finite. The agent's generative reach, which was a convenience elsewhere, becomes the core capability: it manufactures the edge-case content the template has to survive.

The rest of the framework carries straight over, re-pointed at content. State is the metering and entitlement you reset per persona. The boundary is the ad tech you mock to test layout and measure to catch the damage. The invisible-but-consequential failure is the structured data that costs you search visibility. Version the templates, keep the visual and performance proof. The agent is the same; the publisher just demands you treat content itself as the thing that varies, because it is.