Fundamentals · 5 November 2026

By Eddie Eastwood, Founder

Can You A/B Test Content for AI Visibility? What You Can Run Instead

Can you A/B test content for AI visibility?

No, not in the sense the term has on a website. Google describes website testing as trying out different versions of a website, or a part of it, and collecting data on how users react to each version. It describes A/B testing as testing two or more variations of a change.^[1] That definition fits a site owner who decides which visitor sees which version.

Several features of AI answers remove that control. First, the engine chooses what each person sees from pages across the web. No switch on your side serves version A to one person and version B to another. Second, an edit can reach an answer only after the engine has read the new version, and you may not be able to see when that happens. Third, the same prompt can return different answers from the language models covered below. No single answer stands for the page as it stood when the answer was given.

None of this makes experiments on AI systems impossible in general. A lab that runs its own experiment can set the parameters of each run itself. A site owner can edit the page but has no hand in the system that answers. The repeated comparison substitutes a fixed record, kept before and after, for random assignment.

Can the same prompt give a different answer twice?

Identical requests to the language models covered below can return different text, even where the settings are meant to remove randomness.

An arXiv preprint by Atil and colleagues (version 5, April 2025) tested five LLMs configured to be deterministic on eight common tasks across 10 runs.^[2] The authors report task accuracy varying by up to 15% across naturally occurring runs. They also report that none of the tested LLMs consistently delivered repeatable accuracy across all tasks.^[2] The 15% describes accuracy on common language tasks and is no measure of how far AI visibility moves.

Thinking Machines Lab, a vendor research lab, ran its own inference experiment on an open model. It reports sampling 1000 completions from Qwen/Qwen3-235B-A22B-Instruct-2507 at temperature 0 with the prompt “Tell me about Richard Feynman”. It reports 80 unique completions, and the most common appeared 78 times.^[3] The lab describes temperature 0 as theoretically deterministic. The experiment used an open model on the lab’s own setup. It tested neither ChatGPT nor business recommendations.

OpenAI’s API documentation points the same way. OpenAI’s Chat Completions API reference lists temperature as an optional request parameter with a default of 1 and a range of 0 to 2.^[4] For its seed parameter, the reference says the system will make a best effort to sample deterministically, and adds: “Determinism is not guaranteed”.^[4] These are API request parameters, and the reference says nothing about consumer apps.

One sibling guide explains why a business can show up in ChatGPT one week and be absent the next. The guide to what AI visibility analytics can and cannot measure reports a third-party repeated-prompt study.

A single answer is therefore one sample of what a system can say. A before and after built from one answer each compares two samples, and any difference between them is a difference between samples before it is a difference between pages.

What can you test, and what can’t you?

You can test whether one edit to a page you control is followed by a different pattern of answers on a fixed set of prompts. You cannot separate the edit from model updates, competitor changes or retrieval changes (a different set of pages fetched for the same prompt). You also cannot test pages you do not control.

An edit may reach an answer by an indirect route. A guide to how ChatGPT may split one prompt into several searches and merge the results describes that mechanism. It treats this route as the best current read, drawn from one vendor’s inspection of client-side code. If a prompt reaches your page through searches you never typed, the prompt measures the whole route, so it cannot isolate the page’s wording. Compare a page with itself over time and with a control, and leave competitors’ pages out of the comparison.

How do you design a repeated comparison?

Each choice below narrows the rival explanations for a difference. Make each choice before the baseline round and write it down. A choice made after seeing results bends the comparison toward what you hoped to find.

Choose one edit

Pick one element on one page and write the hypothesis in a sentence before you touch the page. Name the element that changes, the prompts it is meant to matter for, and the difference in answers that would count as consistent with it. Leave the title, headings, internal links and unrelated text alone, and publish nothing else on the same topic while the comparison runs. Keep the edit’s wording out of your prompts.

Fix the prompt set

Write prompts the way a customer would put the question, and leave your business name out. For a bookkeeping firm in York, a customer-style prompt is “who can help me with my self-assessment tax return in York”. A prompt that contains the name tells the engine which business to discuss, so it cannot show a change in whether you are recommended. Phrasing customer-style prompts and setting up a tracking sheet are covered in the guide to checking whether AI recommends your business.

Split the set into test prompts, which cover the topic of the page you will edit, and control prompts, which cover topics you will leave alone. Freeze the wording. A reworded prompt is a new prompt and starts a new series.

Vary the order of the prompts from round to round, so that no prompt is always asked first.

Fix the engines and settings

Treat each engine as its own series: its answers to the same prompts, round after round. ChatGPT, Perplexity and each Google surface are read separately and never blended into one figure. Keep each engine’s settings identical in every round.

Where you call an API, record the temperature and any seed you send. In OpenAI’s API these are request parameters, so a run can state the values it sent.^[4] For a manual run, record the model name if the interface shows one, and the date. Do not change a setting partway through a series. If you learn that a model has changed, mark the date in the log and treat later rounds as a new series.

Choose a control

A control is a page or a prompt set that you leave alone while you change the test page. Choose controls that resemble the test page: a similar service page on the same site, or a prompt on a neighbouring topic.

Controls have flaws that you should write down in advance. A control on the same site can be reached by a sitewide change, such as a new template or a new page that links to both pages. A control prompt that is too far from the test prompts may not move together with them even when something shared has changed. A competitor’s new page may alter answers for test and control prompts alike, which a control can reveal and cannot explain.

Set the windows and rounds

A round is one pass through every prompt on every engine. Run several rounds in a baseline window before the edit and the same number after it. Decide the count in advance from what you can run and record, and keep it equal before and after.

After the edit, allow time for the engines to re-read the page. Record when you can see that they did, for example when an answer repeats the new wording. Start the after window on the date of that record.

Record every answer in full

Save the whole answer each time. A log that keeps only a named-or-not tick cannot be checked again when a question comes up later. Each row needs seven fields.

• the prompt identifier and its exact text;
• the engine, model name if shown, country, language, login state and web-search setting;
• the date and time of the run;
• the full answer text;
• every business named, in the order named;
• every link cited;
• any change made to the site on or just before that date.

The guide to setting a baseline and re-measuring an AI visibility score shows how to take a dated baseline. Use that routine for the dated record, and add the fields above for the comparison.

How many repeats do you need?

No cited source sets a number. Repetition shows how far answers move when you have changed nothing, and that spread is the yardstick an after-edit difference has to be read against. A difference that sits inside the baseline spread says little about the edit.

More rounds describe the spread better, but they do not make each round independent. Answers to one prompt on one engine on one day may be correlated, so repeated answers are not automatically independent evidence. Log identical answers as identical and do not count them as extra evidence.

In QBiz’s probe of two businesses on 2 November 2026, whether each business was named did not change between the two runs of each prompt. The other businesses named in the customer-style answers did change between runs.^[5] The probe sent its prompts to ChatGPT, Perplexity and Google AI Mode through the DataForSEO API, so the results are API responses.^[5]

None of the customer-style answers named either business, so what stayed the same there was an absence.^[5] The answers to prompts that contained a business name all named it.^[5] The score page states the limit: “The probe covers 2 businesses, 1 date and English-language prompts, so it illustrates two cases and gives no rate.”^[5]

In your own log, record whether you are named as one series and the other businesses named as another.

What repeats did when nothing changed

On 4 November 2026 we sent five customer-style prompts about electricians in York, UK, five times each. They went to ChatGPT, Perplexity and Google AI Mode through the DataForSEO API, with nothing on any website changed.^[6] We call this set of runs the panel.

In this panel, every successful run named the same business first in 5 of the 15 prompt and engine combinations. In the other 10, at least two different businesses were named first. Named first means first in the order the names appear in the answer text. It is not a rank the engine states. The list below reads each engine in this panel on its own, since the engines ran with different settings:

• ChatGPT: the same business was named first in all five runs for one prompt and in two of five runs for another.
• Perplexity: the same business was named first in all five runs for three of the five prompts. For one of the other two prompts, the most common first-named business appeared in four of five runs, and for the other in three of five.
• Google AI Mode: the most common first-named business appeared in two of five runs for two prompts. It appeared in three of five for one prompt and four of five for another. For the prompt with a failed call, it appeared in four of four runs.

In this panel, across five runs of one ChatGPT prompt, the answers named at least 25 different business names between them. For two of the five prompts, all five Perplexity answers named the same set of business names. None of the 15 prompt and engine combinations returned two identical answers here. The five runs of each prompt were sent minutes apart on one account, so they are not independent draws.

Method: the runs took place on 4 November 2026, between 22:54 and 23:08 UTC, through the DataForSEO API. Engines: ChatGPT (gpt-5.5, web search on, provider-default sampling), Perplexity (sonar-pro, temperature 0.77) and Google AI Mode (desktop, United Kingdom, English). Settings were the same in every round, no website changed, and output length was left at the 2048-token default on ChatGPT and Perplexity, not the 1024 first planned. One Google AI Mode call returned a server error and was not repeated, so that prompt has four runs. These are API results, not screenshots of the consumer apps. Business names were extracted by script and corrected by hand against the answer text. The panel covers one date, one trade and one town, so it describes these prompts only and gives no rate.

How do you read the result?

Read the test series and the control series side by side, one engine at a time, and ask which explanations each pattern leaves open.

When a number you track has moved, report it with its baseline spread and the engine it came from. The guide to measuring brand share of voice in AI answers covers keeping a tracked number comparable with your own earlier runs.

Repetition measures sampled variation, and it does not remove changes in the model or the web between rounds. You chose the prompts, the dates and the pages, so the result is a comparison without random assignment.

Two conclusions are off the table whatever the pattern. A higher mention count after an edit does not show that the edit worked. A flat series does not show that the edit had no effect. No cited source links answer movement to rankings or leads.

When is a repeated comparison not worth running?

A repeated comparison is not worth running in these situations:

• The baseline never names you. Check readiness first: what an AI visibility checker actually measures separates readiness signals from being named in an answer.
• The prompt set is too small to read. If one prompt changing its answer moves the whole picture, a single answer decides the result.
• The edit adds no fact or answer. A reworded sentence that says what the page already said adds no new fact for an answer to repeat.
• The edit arrives bundled with others. A redesign changes everything at once, so treat it as a before and after of the whole site and log it as such.

Where does a tracker fit?

A tracker fits the recording step. Peec AI’s documentation says it runs a customer’s prompts across AI platforms daily and analyses patterns over time because, in its words, AI responses naturally vary day to day.^[7] That is one tracker describing its own method. A schedule does not choose the edit, the prompt set, the controls or the date the after window starts. Those stay with you. A buyer guide to AI visibility trackers for small and local businesses compares what several vendors say they measure.

The QBiz audit tests its buyer prompts at baseline and again at retest, and keeps a record of the sources each platform cites.

See what the QBiz audit records →

Frequently asked questions

Can I run a split test on one page for ChatGPT?

Not in the way you split-test visitors. You choose which visitor sees which version only on your own site, and an engine that you do not control decides what each person is shown. You can change one page, keep a log of fixed prompts before and after, and watch similar pages you left alone. A log and controls cannot establish that the edit caused a change.

Can I set ChatGPT or Gemini to give the same answer every time?

Nothing cited here shows a way to fix the answers in the ChatGPT app or in Gemini. OpenAI’s API lets a developer send temperature and seed as request parameters. For seed, the API reference adds: “Determinism is not guaranteed”. Where answers cannot be fixed, repeat the prompt and read the spread.

Do I need a control page, and what should it be?

Yes. Leave one comparable page, or one set of prompts, untouched while you change the test page, and log every change made to the site during the comparison.

How long should the before and after windows be?

No cited source sets a length. Set both windows in advance and run the same number of rounds in each. Start the after window on the date you can record that the engines have re-read the edited page.

Should I test on one engine or several?

Run each engine as its own series and read the series separately. One engine keeps the comparison small. Several engines show whether a pattern appears on more than one, but never merge them into one score.

What if my business is never named in the baseline?

Then the comparison can only show a move from absent to present, and your baseline has no spread to read that move against. Before you edit the page, run a readiness check on the site. Keep the baseline anyway, because it records where you started.

Does a higher mention count after a change prove the change worked?

No. A higher count is consistent with the edit, and it is also consistent with shared movement or run-to-run variation. Compare it with the control series and with the spread in your baseline rounds.

Can a tracker run the comparison for me?

Partly. One tracker, Peec AI, states that it runs a customer’s prompts daily. The edit, the prompts and the controls stay with you.

What a finished comparison gives you

A finished comparison gives you a dated log of full answers to a fixed prompt set and a record of the one edit you made. It also gives you the list of explanations the pattern leaves open.