Your Untouched ASINs Lifted Too: The Incrementality Problem in AI Visibility Testing

Analytics & Measurement | 8 | Published:

By Stephen Honight, Founder of The Lmo7 Agency

You reworked a handful of Amazon listings for AI visibility and the tracker shows them lifting. The ASINs nobody touched lifted too. That is a finding, and it changes what the test proved. Here is why it happens and what a credible AI visibility test looks like.

Your Untouched ASINs Lifted Too: The Incrementality Problem in AI Visibility Testing

By Stephen Honight, Founder of Lmo7

A brand reworks a handful of Amazon listings for AI visibility. Titles, bullets, A+ content, the usual. A few weeks later the tracker shows the reworked ASINs (the individual product listings, in Amazon’s terms) appearing more often in Rufus answers. Then someone checks the ASINs nobody touched, and they have lifted too.

The natural move is to report that as a win. Everything is up, so the content worked, so roll it out. My sense is that this reads the result the wrong way round. It is a finding, and it changes what the test proved. The stakes are not academic either. Rollout budgets get funded off these numbers, and a result that flatters the work now costs you credibility with your finance team later.

The short version

If you only have two minutes, these are the five points:

  • AI answers are composed fresh for each query. A brand holds a probability of being named, not a fixed position, so single readings are noisy.
  • Untouched products move on their own, for reasons that have nothing to do with your edits.
  • In the public Publicis study, the control group moved seven points while nobody touched it. That is most of the headline result.
  • A credible test fixes its query pack and its control set before it starts, reads repeatedly over a long window, checks the things that move sales anyway, and agrees a decision rule up front.
  • The right claim is the change versus control, with ranges. A raw lift figure takes no account of what the untouched products did.

Testing is still the right move. The argument here is for testing in a way that survives a finance director asking “how do you know?”. The rest of this post covers the mechanism and the practical design.

Why the control moves on its own

Start with how these surfaces work. An AI answer surface such as Rufus does not look up a ranked list. It composes an answer for each query, and each answer is a draw from a distribution. Ask the same question twice and you can get two different answers with two different lead brands.

That means a brand does not hold a position. It holds a probability of being named for a given prompt. Your tracker is sampling that probability, and a sample is a sample.

Volatility compounds this. The draw is re-made every night, so a single before and after comparison is a coin toss with a screenshot attached. If you read once before the change and once after, you have measured one draw against another.

Then there is everything else that moves while your test runs. Four things matter most:

  • Halo. Products share brand signals such as reviews, brand story and the wider catalogue. Improve some listings and the brand’s other listings can benefit because the model reads the brand, not only the ASIN.
  • Competitor activity. Rivals rework their own content, change price or push media. Your untouched products gain or lose share without you doing anything.
  • Model and surface changes. The underlying model gets updated, or the answer surface changes how it composes responses. Every product moves at once.
  • Seasonality. Demand for the category shifts through the year, and the questions shoppers ask shift with it.

Any of these can lift a control ASIN. None of them is your content. Until you have ruled them out, a lift in the treated group tells you very little.

What the Publicis study shows at scale

The public Publicis “Decoding Rufus” study has the same problem at study scale, and the numbers are instructive. I covered the detail in our teardown, Decoding Rufus: read the appendix, so here is the short version as Lmo7 read it.

The headline was an 11 percentage point delta between test and control on SOAR, the study’s share-of-answer visibility measure. The study’s chart then shows where that delta came from. The test group moved from 39% to 43%, a gain of 4 points. The control group moved from 48% to 41%, a fall of 7 points (source: Publicis, “Decoding Rufus”, chart, as read by Lmo7).

So most of the headline delta is the control decaying untouched, not the test group climbing. The test group did improve. It improved by less than the control fell.

The active-pairs statistics table in the same study tells a similar story. It reports a test-group mean delta of 0.001 (median 0.009) against a control mean of -0.072, with a t-statistic of 2.185 and a p-value of 0.030, on 79 test and 85 control active pairs (source: Publicis, “Decoding Rufus”, active-pairs statistics table, as read by Lmo7).

My honest reading is this. Content optimisation on eligible products may protect visibility that would otherwise erode. Any lift beyond that protection is small and hard to separate from noise at this sample size. The study’s p-value of 0.030 clears the usual bar, but that significance is carried substantially by the control falling rather than by the test group climbing. Protection against erosion is still a real, defensible result, and commercially it is easier to stand behind than “visibility rose 11 points”.

The control group moved 7 points while nobody touched it. A test design that cannot account for that movement will misread its own result, which is why the control set deserves as much design effort as the content changes.

We have seen the same pattern in live work

We have seen the same pattern on live pilot work this autumn: reworked listings lifted, and listings nobody touched lifted alongside them. The only version of that result worth reporting was the careful one, claiming uplift solely where the Buy Box (the default seller on the listing) held and media spend was stable through the window.

Retail media has been here before

None of this is new to anyone who has run retail media. Incrementality discipline grew up because last-click reporting tended to flatter spend: platforms graded their own homework, and brands eventually asked what would have happened anyway.

The answer was a set of methods that are now standard practice: holdout groups, geo splits, matched controls and media mix modelling. Each one exists to answer the same question. What would have happened if we had done nothing?

My view is that AI visibility testing is at the stage retail media was before that discipline arrived. Trackers show movement and teams attribute that movement to their own work, without a counterfactual to check it against. My sense is the brands that borrow the retail media playbook early will be the ones whose numbers their finance teams trust. I have written before about why control versus test is non-negotiable, and the argument holds here with more force, because the noise is higher.

What a credible AI visibility test looks like

Here is the design I would recommend. Most of it is agreed before a single listing is edited.

Fix the query pack before the test. Agree the exact prompts you will track, grouped by category and shopper intent, and lock the list. If prompts are added or dropped after you see results, you are choosing the story.

Repeat the reads. Do not take snapshots. Our own tracker ran 70,764 queries through Amazon Rufus over 30 days, covering 1,082 distinct prompts across 392 nightly runs on amazon.co.uk, amazon.de and amazon.es, from 25 July to 23 August 2026 (source: Lmo7 first-party tracker data). The lead brand rotated constantly. The full data is in our post on the nightly re-draw.

Respect the arithmetic of small samples. At roughly 14 reads a week per prompt, one answer moves a single prompt’s weekly rate by about 7 percentage points, and two answers move it by about 14 (source: Lmo7 first-party tracker data, arithmetic on 14 weekly reads per prompt). When we compared our tracker against a second independent tracker on a matched 76-prompt pack in October 2026, weekly prompt-level moves correlated at 0.23 (source: Lmo7 first-party head-to-head measurement work). That is noise. Two trackers watching the same prompts should agree far more if the weekly movement were real signal.

Report at category or intent level, over a rolling window. Use a 28-night rolling window with confidence ranges, long enough for the nightly re-draw to average out. Claim a change only when the ranges separate. Read 28 nights on each side of the change, so each side is built from a full set of draws. Our post on measuring Amazon Rufus honestly covers the method in more detail, and the IAB measurement standard is worth reading alongside it.

Choose a matched control set before the test starts, not after. Match on category, price band, review depth and prior visibility. If you pick the control after seeing the results, you will pick products that make the result look good, whether you mean to or not.

Run stability checks on the things that move sales anyway. Check the Buy Box, stock, price and media through the window. If the Buy Box was lost for a fortnight on a treated ASIN, or media spend doubled on a control, the read is contaminated and the report should say so.

Agree a decision rule up front. Write down what result funds the rollout and what result kills it. Do this before the data arrives. Otherwise every outcome becomes an argument for carrying on.

How to report it honestly

Back to the opening scene. “Untouched lifted too” has four candidate explanations: halo from shared brand signals, competitor decay, a model or surface change, and seasonality. A good report names all four and says which of them the data can rule out and which it cannot.

Sometimes the data lets you rule one out. If competitor share fell across the whole category, competitor decay is a live candidate. Often you cannot separate them at this sample size, and the report should say that plainly.

The claim shape matters as much as the data. “Delta versus control, with ranges” is a claim a sceptical reader can check. “Visibility rose across the reworked listings” is not, because it takes no account of what the untouched products did.

I will be straight about the trade-off. A test reported this way reads as less impressive. The numbers are smaller and the language is more careful. It is also worth a great deal more, because it is the version that survives scrutiny when you ask for budget to roll the work out across the catalogue.

So test, and test with a control. The content work and the measurement discipline fund each other: the rework gives the test something to measure, and honest measurement is what gets the rework rolled out. Keep the longer game in view too. Content decides whether you are retrievable this quarter, while reviews and authority decide how often you get picked, so a good test also tells you where the next pound of effort should go.

What to do next

  1. Audit any test you have already run. Find the control group, or confirm there was not one. If untouched products moved, list the four explanations above and mark which you can rule out.
  2. Lock a query pack and a matched control set before your next content change. Write them down, date them and keep them out of reach of later edits.
  3. Set up repeated reads. Move from one-off checks to nightly reads, and report on a 28-night rolling window with ranges at category or intent level.
  4. Write the decision rule and the stability checklist. Agree with your finance lead what result funds the rollout and what kills it, and check Buy Box, stock, price and media each week of the test.

If you would rather someone ran this measurement discipline for you, that is what we do at Lmo7 (lmo7.com).

Stephen Honight is the founder of Lmo7, a UK agency helping consumer brands win visibility and sales in AI-powered search and agentic commerce, across Amazon and D2C.

Explore More

AI Search Optimisation Services | LLM Visibility Framework | Free AI Search Audit | Search Lab | Alexa Shopping Radar

Related Articles

Explore Share of Model partnership and tracking.