Why does a single AI visibility score say so little?

AI answers are not stable

A study by SparkToro and Gumshoe (January 2026) examined this with around 600 volunteers and just under 3,000 runs across ChatGPT, Claude and Google. All of them asked the same twelve questions seeking recommendations. The probability of getting the same list of brands twice was below 1 in 100, and in the same order close to 1 in 1,000. Even the length of the lists varied considerably.

The more important part of the result: what was stable was not the list but the share. Some brands appeared in 60 to 90% of the answers to a question, even when their position varied. The authors conclude: repeated presence is meaningful, an exact rank is not.

Profound arrived at a related result from a different direction in July 2026. Across 753 prompts on seven platforms over 14 days, visibility was 78.7% with one run per day and 80.4% with ten runs per day. From day to day, too, the gap was only around 2 percentage points. In this study, the run frequency therefore made little difference, as long as measurement continued over many days. A single query does not replace that.

An example from our own work, including a mistake of our own

In an early test run in summer 2026, we measured a Swiss online shop with 14 purchase queries across four AI systems, one run each. One system named the shop in 6 of 14 answers, another in 1 of 14. Internally, we recorded this as “a factor of six between the models”.

Once the uncertainty is factored in, the picture looks different:

System	Mention	95% interval
System A	6 of 14 · 43%	21–67%
System B	1 of 14 · 7%	1–31%

The intervals overlap. With 14 answers per system, it is not possible to say whether the two systems actually treat the shop differently or whether the difference is down to chance. An exact test gives p ≈ 0.08. The “factor of six” was a snapshot, not a measurement.

More robust differences only come from a baseline with considerably more answers: 465 answers across six access paths over five days, for a Swiss online shop:

Access path	Mention	95% interval
ChatGPT, chat interface	22 of 108 · 20%	14–29%
OpenAI model, API	7 of 108 · 6%	3–13%
Gemini	6 of 48 · 13%	6–25%
Google AI Overviews	1 of 46 · 2%	0–11%
Claude, API	0 of 48 · 0%	0–7%
Perplexity	0 of 107 · 0%	0–3%

Two things are robust here. First: the same provider delivers different results via two access paths. Via the chat interface that a person uses, the shop was named around three times as often as via the programming interface that a self-built shopping agent might use (Fisher's exact test, p < 0.01). We had planned this comparison between interface and API in the measurement design from the start. It is not a general rule: in a second pilot measurement, for a different provider, the API was slightly ahead of the chat interface, though there without a robust difference. Second: in one system with more than a hundred answers, the shop did not appear a single time. If you expect your customers to be mainly there, you have a different problem from someone whose customers use ChatGPT.

An average across all six paths would hide both.

What a confidence interval tells you, in one sentence

A 95% interval shows how precise a measured share is: if the measurement were repeated very many times, around 95% of the intervals calculated in this way would contain the true value. The fewer the answers, the wider the interval. With 14 answers, a measured value of 43% is compatible with a true value anywhere between 21 and 67%. With 108 answers, a measured value of 20% narrows to 14 to 29%.

That is why we work with a deliberately cautious rule: we only report a difference between two numbers as a change if their intervals do not overlap. Overlapping intervals do not rule out a difference, but they are not enough to claim one. This applies to comparisons with competitors just as much as to comparisons before and after an intervention.

What even many runs do not fix

Repeated measurement reduces chance. It does not correct systematic errors. Four of them remain in every measurement, including ours:

• The choice of questions. A precise number for questions your customers do not ask remains irrelevant.
• The conditions of the query. We query with a Swiss location and without logging in. Logged-in users with a history may see different answers.
• Changes to the systems. If a provider changes its model or its search during the measurement, two states are mixed in one number. We document model and access per run; we cannot prevent it.
• The evaluation itself. Whether a brand is mentioned or recommended is detected by an automated evaluation. It makes mistakes too, which is why we disclose the method.

How to recognise a meaningful number

Before you trust an AI visibility metric, five questions are worth asking:

• How many answers lie behind it? Without this information, the number cannot be put into context.
• Is uncertainty reported? An interval, or at least the sample per value.
• Are systems shown separately? An average across ChatGPT, Gemini and Perplexity does not describe any system that anyone actually uses.
• Has it been pooled over time? A single query, even if it was made today, says little about tomorrow.
• Which questions were asked? The most precise number is of little help if it was measured on questions with which your customers do not start a purchase decision. More on this: Which prompts actually count?

When a one-off check is enough

For technical questions, a one-off check is often sufficient: whether AI crawlers are allowed to fetch your pages, whether product data is in the HTML without JavaScript, whether structured data is free of errors. As long as the pages stay the same, these checks return the same result. For the question of how often AI systems mention or recommend you, by contrast, a single run is not enough.