Methodology

We ask AI assistants the questions a buyer would actually type, many times over, and record which products each assistant names. Nothing on this site is an opinion about which product is best. It is a record of what the machines said, on a date, under conditions we publish.

What we ask, and whom

The questions. Each question is written the way a buyer would type it into an assistant — “best email marketing platform for Shopify”, “cheaper alternatives to Klaviyo”, “is Klaviyo worth the price for a small store”. Questions are grouped into clusters by category. A cluster is between 30 and 50 questions and launches as a whole or not at all. The full list of questions behind any category is on that category's page, and each question's own page shows the question verbatim as its title. Questions are sent to each assistant exactly as written — no instruction, no role, no format request, nothing that steers the answer towards or away from any brand. Brands cannot submit questions.

The assistants. Today: ChatGPT, with web search enabled, and Perplexity. Both are queried through their programmatic interfaces. For ChatGPT, every call is required to perform a web search: left to decide for itself, the model answered about half of our test questions from memory without searching, and an answer from memory is a different measurement wearing the same label. So what we record is what a user whose assistant searched would see, not what a model remembers from training. The exact model identifier is recorded with every answer and shown in the round table below. Further assistants are added only once we can query them the same way, round after round.

The cadence. One round per week. Every enabled question is asked again in every round; nothing is carried over from a previous round.

1. AI Pick Rate

AI Pick Rate is the share of samples in which an assistant recommended a given product, within one cluster of questions, on one engine. A product an answer names only to warn against is not a pick: it is recorded, but it does not count here.

Pick Rate = samples recommending the entity ÷ total samples for that cluster and engine

It is calculated and displayed per engine. ChatGPT and Perplexity read different sources and return different brands; averaging them produces a number that describes neither. Where we show a cross-engine figure it sits in its own column, labelled cross-engine, and is never mixed into a single-engine table.

Below n = 10, a question page shows a count — “2 of 3” — rather than a percentage. Three samples do not support a decimal place, and “66.7%” would imply a precision the sample size cannot carry.

Where two products have the same count, they are ordered by count, then by slug, so ties render in the same order every round (since 7 October 2026; earlier rounds' tie order was not fixed).

2. Answer repeatability

Ask an assistant the same question three times in one round and it will not give the same three answers. Repeatability is the average overlap between those answer sets — the Jaccard similarity of the entities recommended, averaged over every pair of samples in the round.

It is a property of the category, not of any brand. A category where assistants agree with themselves is one where opinion has settled; a low score means the category is still open, and a brand can still move into it. Pick Rate and repeatability are read together or not at all.

Repeatability is computed per engine and never merged across engines. Two engines disagreeing with each other is a different phenomenon from one engine disagreeing with itself, and combining them would hide both.

3. N = 3 sampling and the publishing threshold

Each enabled question is asked three times per engine per round. During the discovery phase, a delayed scheduled job ran the full question set twice on the same day, 5.6 hours apart, one sample per question. The two runs were not designed as a comparison, but they are one: same questions, same engines, same day. Across all eight clusters the two answers to a question shared only 67.8% (Perplexity, 142 question pairs) and 67.5% (ChatGPT, 149 pairs) of the products they recommended. Roughly a third of any single answer is noise, so one ask is not a measurement. These figures are computed under the current extractor with merged names folded; see Corrections. The three samples are staggered within the round — every question is asked once, then all again, then all a third time — and each sample records its own timestamp and model version.

A Pick Rate is only published at n ≥ 3. Below that it may appear only as provisional . Pages that cannot clear the threshold are not generated at all rather than published thin:

• A question page requires at least one engine with n ≥ 3 that surfaced at least three distinct products.
• A product page requires the product to have been recommended at least three times within its cluster.

Page counts are an outcome of those gates, never a target. We do not loosen a threshold to reach a number.

Checking for cached answers (rounds 6–7, 29 September 2026, extractor v2)

If three samples in one round agree unusually well, the first suspect is the API returning cached results — in which case we would be measuring cache consistency, not category consensus. We ruled this out with a controlled re-run.

Round 6 spaced its three samples a median of 74 seconds apart, inside any plausible cache window. Round 7 re-ran the same 34 questions on the same engines the same day, changing only the spacing: a median of 11.5 minutes between samples, measured from the timestamps rather than the setting. If agreement came from caching, longer spacing should lower it. Instead ChatGPT rose from 87.5% to 89.4% and Perplexity fell from 84.2% to 76.6% — the two engines moved in opposite directions, which a cache effect cannot produce.

The 89.4% figure also only looks anomalous against the site-wide baseline. Measured at the time, under extractor v2, that baseline was 64.9% for ChatGPT, and this category ran 11–14 points more stable than the site average; its own cross-round baseline, 34 questions 2.3 hours apart, was 82.6%. Against that, round 7 sat 6.8 points higher — below the 15-point threshold at which we flag a round for suspicion.

This does not prove there is no caching at all. It shows that a ninefold increase in spacing did not move agreement the way caching predicts, and that the level we see matches the category's own baseline.

Every figure in this section is a record of that day's measurement, with the same extractor on both sides. Round 7 has since been re-extracted under version 5 with merged names folded, and round 6 has not, so the comparison cannot be recomputed from current data; to re-test the caching hypothesis we would run the same design again. Under the current extractor, round 7 repeatability is 81.8% (Perplexity) and 88.9% (ChatGPT) against a category baseline of 75.4% / 84.3% from rounds 4–5: +6.4 and +4.6 points, within threshold.

Re-reading the same answers (one observation, 5 October 2026)

Our extractor runs at temperature 0, and in a same-batch test it returned identical judgments on repeated runs (recorded below, under How products are identified). On 5 October the same 204 answers were read twice about ten hours apart, with the same extractor version and settings. Three of the 204 answers (1.5%) received a different judgment the second time, which moved a single product's Pick Rate by up to 2 points. This is one observation, not a distribution. We now treat a change of this size between rounds as within re-reading noise unless the answers themselves changed, and we do not re-read stored answers automatically.

Engine version breaks

Every scan records the model version behind each answer. When a round shows synchronised movement across a whole cluster beyond 15 points, or the model identifier changes, that round is flagged an engine update. Its change figures are shown as an engine change and excluded from market drift. An assistant being upgraded looks exactly like every brand moving at once, and mistaking one for the other is the easiest error a measurement site can make.

How products are identified

Each answer arrives as a paragraph of prose. Turning “which products did this answer recommend” into rows of data is itself done by a language model, not by keyword matching or a fixed parser. It is the one step in the chain where a model exercises judgement, so we publish all of its settings.

The extractor. Anthropic's Claude Haiku 4.5 (claude-haiku-4-5) reads each answer in full and returns the products it names, the passage that names each one, and whether the answer recommends it or warns against it. A product the answer tells the buyer to avoid is recorded too: being named as one to avoid is a real signal, and not the same as never being mentioned. It is recorded separately and never counted toward Pick Rate, which counts recommendations only.

Temperature 0, since 1 October 2026. Before that date the extractor ran at the default temperature of 1.0. We measured what that cost. The same prompt run twice over the same 100 answers found 420 mentions one time and 409 the next, a 2.6% difference; 15% of product names were counted differently between the two passes, by up to 4 for a single product — Brevo was counted 25 times in one pass and 21 in the other. At temperature 0, the same check on 50 answers found 215 mentions both times, with no difference for any of the 68 product names.

That matters because this site publishes change from one round to the next. Extraction jitter lands on both ends of a comparison and looks like an assistant changing its mind — a layer of noise on top of the assistants' own sampling variation. Zero difference between runs in the same batch is what we measured on 50 answers, not a guarantee that it is always zero: a language-model service can still be nondeterministic at temperature 0 because of how requests are batched on the server. But the difference in scale between the two settings is decisive.

Every row carries its extractor version. Each change to the extraction rules raises a version number, and every extracted row records the version that produced it, so any figure on this site can be traced to the rules that produced it. The current version is 5.

The model does not decide what a product is called, or whether two names are one product. The same product can be written several ways — “Kit (formerly ConvertKit)”, “Brevo (formerly Sendinblue)”. The extractor records the spelling it saw, along with any other spellings in the same answer. Which spelling is canonical, and which names belong to the same product, is decided by an alias table that people maintain. That is a judgement about the world: a model can get it wrong, and its mistakes cannot be audited. A table lookup is deterministic — when an entry is wrong, it can be found, and every figure it touched recomputed in one pass. Such a fix is an extraction correction, dated on every page it changes; see Corrections.

Splitting works the same way. Whether “Shopify Email / Shopify Messaging” is one product or two depends on whether a buyer can purchase them separately. That is a judgement about the world too, kept as a rule maintained by people; the extractor does not make it.

4. The four provenance labels

Every figure we publish carries one of four states:

• measured — observed in a scan, n ≥ 3.
• provisional — observed, but n < 3. Not publishable as a rate.
• estimated — modelled, not observed. Used for costs, never for Pick Rate.
• unverified — not checked. Shown instead of a value, never next to a guessed one. An empty field is a fact; a filled one that nobody checked is a liability.

An overstated label is more expensive than a missing number. What this site sells is that its figures are what they claim to be.

5. Disclosure policy

• No paid placement, ever. No affiliate links, sponsorships or paid rankings appear in any data table on this site. Ranking position cannot be bought.
• Our own product is marked. AppearAI is built by AUKSON INC., a sister company under the same founder as MIVO INC. Wherever it appears in a ranking it is labelled our product and ranked by identical rules. It is not in any current ranking — its category has not launched — and this rule stands before it does, not after.
• Every data block states its conditions: scan date, n, engine, and a link back to this page.
• We do not sell conclusions. What is for sale elsewhere in our business is coverage and monitoring, never a favourable finding.

What the current round covers

Round 9, scanned 12 October 2026. 2 clusters — Ecommerce Email & Retention (34 enabled questions) and AI Customer Support for Ecommerce (45 enabled questions) — across 2 engines.

A call that errors or returns nothing parseable is retried, not recorded as a non-mention. Only answers actually received enter a Pick Rate, so a retried attempt never changes a denominator — a failure is an absence of evidence, not evidence of absence. Where a retry cannot fill the gap, the sample count for that engine falls and the n shown on every affected block falls with it.

What this does not measure

• It is not a quality ranking. A high Pick Rate means an assistant recommends a product often. It says nothing about whether the product is good.
• It is one account, one region, one moment. Assistants personalise and vary by geography. These figures describe our scan conditions, not every user's experience.
• Change figures appear only when both rounds in the comparison are measured. Round 7 was the first round scanned under this methodology, so no change figure compares against anything earlier. Where an earlier round exists but was itself provisional, no change figure is shown rather than an annotated one.

Using and citing this data

Every figure on this site may be quoted with a link to the page it came from. Cite the scan date and the round, because a Pick Rate is a measurement at a moment, not a standing fact. A suggested form:

What AI Buys — AI Pick Rate, Ecommerce Email & Retention, round 9, scanned 12 October 2026. whataibuys.com/categories/email-retention/

Figures are published on their pages, not as a bulk download. Brands that want their own figures in a form they can reuse, or a product page corrected, can write to hello@whataibuys.com.

Corrections

Measured numbers are never revised after publication. The only exception is an extraction error: a product name matched to the wrong entity, two names for one product counted as two, or a wrong judgment of whether a product was recommended in a stored answer. Every such correction is dated and marked † on the page. The methodology version does not change. If you believe a product on this site is mis-attributed, write to hello@whataibuys.com.