The most important measurement decision happens before the first run: which prompts actually count?

Every measurement starts with an assumption

Anyone who wants to measure how often AI systems recommend a company needs a set of questions first. This is exactly where one of the biggest methodological risks arises.

The obvious solution is to ask the company itself:

“Which questions do you want to be found for?”

That is useful, but it is not the same as:

“Which questions do your customers start a decision with?”

Companies think in products, categories and internal terms. Customers think more often in situations, problems and decision criteria.

A retailer may think of “office chairs Switzerland”. A customer may ask an AI system:

“Which office chair is suitable for long days working from home, and where can I try it out in Zurich first?”

A dentist may want to be visible for “Invisalign Zurich”. A prospective patient, by contrast, might ask:

“Which practice in Zurich offers invisible orthodontics for adults and also has evening appointments?”

Both kinds of question are relevant. But they measure different parts of the same demand.

On terminology: a decision intent is the underlying buying or selection motivation. A prompt is the concrete wording in which that question is put to an AI system. Several prompts can cover the same decision intent. Together, the measured prompts form the measurement set.

A measurement can be precise and still irrelevant

If you only measure the questions a company would like to win, you get exact numbers for the wrong question. The win rate is then statistically clean and commercially meaningless.

The problem cannot be solved by having a language model generate as many prompts as possible. A language model does not know the real distribution of customer questions either. It produces plausible wording, not demand.

Many measurement approaches start with a prompt list from the client or with automatically suggested prompts. What gets measured is then whatever is on the list at the start.

That is why we treat the choice of decision intents as a step of its own: the measurement design. The goal is not the largest possible measurement set, but the most relevant possible picture of a category's decision intents that stays comparable over time.

From keywords to use cases

Classic search long began with relatively short keywords. AI systems increasingly allow context-rich, use-case-based questions: product, situation, price, location, availability and personal preferences in one sentence.

Instead of:

“running shoes Switzerland”

it becomes, for example:

“Which running shoe is suitable for long trail runs with wide feet, and where can I order it online in Switzerland?”

It is precisely these additional conditions that can change which providers AI systems recommend. How much the results can differ depending on the type of question is shown by a case from a pilot measurement: When does a well-known shop drop out of AI recommendations? For the measurement, it is therefore not enough to turn classic keywords into full sentences.

What we know about real prompting behaviour, and what we don't

Representative data on how people actually prompt commercial AI systems is limited today. Query data per category and market is not public, and independent behavioural studies are only beginning.

We do not know the exact prompts or the search volumes of your customers in ChatGPT either.

This gap is not a reason to skip measurement. It is the reason why the choice of questions has to be evidence-based, disclosed and iterative. A set that is justified can be checked and corrected. A wish list cannot.

What we derive the decision intents from

No single source is the truth. Together, several signals give a much better picture than the list of a single company.

Demand signals

Observable demand from classic search: search demand, related queries, frequently asked questions about the category and, where access exists, Search Console data. They show which topics and criteria already generate demonstrable demand.

Category and use cases

How the category is structured: use cases, buying criteria, typical constraints such as budget, delivery time or location, and the decision stages from orientation to selection.

Competitive field

Who is recommended in the category today and with which arguments. The open competitive set of the first runs shows which questions actually lead to selection decisions.

Client and agency input

Knowledge of range, margin and target groups, plus the agency's demand and keyword research. Not every frequent question is equally important commercially. Internal site search, support requests and sales conversations add to the picture where available.

Market signals

Reviews, forums and public discussions show which criteria people actually care about when comparing. Supplementary, where available for the category.

Behavioural data on how people actually prompt AI systems is something we run as a research track of its own. It is not a standard source of the measurement design today and is only included where suitable data is available.

Measurement design, not market research

Deriving the decision intents is not a market study. It serves one purpose: being able to measure relevantly and comparably over time.

That is why the set is created together. The client knows their range and their customers. The agency knows demand, keywords and content. Aions contributes structure, derivation and measurement. Nobody in this triangle knows the market better than the others. The measurement design makes the available knowledge and the observable signals comparably measurable.

Core questions and exploratory questions

A set that keeps changing destroys the comparability it is supposed to deliver. A set that never changes misses new questions. That is why we work with two parts:

Core questions

Stay stable. Define the denominator of the win rate and carry the comparison over time across baseline, re-test and ongoing measurement.

Exploratory questions

Take in new hypotheses, use cases and competitors. Reported separately and never mixed into the trend values.

Promotion to the core questions

Only at a documented cut-over, with the break in the series reported. That keeps it traceable which change comes from the measurement and which from the set.

Decision intents cover different stages

A sensible measurement set distinguishes, for example, between:

Orientation

“Which running-sports online shops in Switzerland are worth recommending?”

Use case

“Which running shoe is suitable for long trail runs?”

Comparison

“Which shop offers the best value for money on trail running shoes?”

Constraint

“Which shop delivers trail running shoes to Bern within two days?”

Selection

“Which of these shops would you choose?”

Only together do they show whether a brand is merely known or actually wins a concrete decision.

The choice is not a one-off step

Even the right questions change. New products appear, customers learn new terms, AI systems change how they present information and ask follow-up questions.

That is why a measurement set is not defined once and then used unchanged for years. Ongoing measurement needs continuous review of what should be measured in the first place. The exploratory questions are where that happens.

Our position

We do not believe that anyone today knows the exact distribution of real decision intents put to AI systems. The market is too young for that and the data too thin.

That is why we treat the choice not as a solved problem but as measurement design: derived, disclosed, defined together and reviewed iteratively.

“A precise measurement of the wrong questions is still a wrong measurement.”