Tested by Sidekick vs a Live A/B Test: What Can Simulated Shoppers Tell You?
A change that Sidekick tests with simulated shoppers has been tried against AI personas, not against your real customers. It is a useful way to screen ideas quickly, but it is not a live A/B test. Only a live test splits real traffic between versions and measures orders. Use simulation to pick what to test, then run a live test to prove it.
TL;DR
- Simulated shoppers are AI personas, not your customers.
- They are good for screening ideas fast and spotting obvious problems.
- They cannot show how real shoppers on your traffic will behave.
- Treat a simulated win as a hypothesis, then confirm it with a live test.
What does Tested by Sidekick actually test?
Shopify's Fall '26 Edition describes recommendations that are identified and tested by Sidekick using SimGym, with test statuses shown as complete, running or draft, and an option to set a shopping goal for the next simulation. SimGym uses AI shoppers to try a store, so the result reflects how simulated personas respond to a change. Independent write-ups describe the output as directional rather than statistically validated.
How do simulated shoppers differ from a live A/B test?
Simulation vs live test
SIMULATED SHOPPERS
Who: AI personas.
Speed: fast, no traffic needed.
Best for: screening ideas and catching obvious problems.
Proves: nothing about real orders.
LIVE A/B TEST
Who: your real shoppers.
Speed: needs enough traffic and time.
Best for: proving a change moves orders.
Proves: the effect on revenue.
The distinction is who is on the other side of the test. A simulation answers whether a change looks better to a model of a shopper. A live test answers whether it earns more orders from the people who actually visit.
How should you use both?
1. Use simulation to shortlist
Run several ideas through simulation and keep the ones that look promising. This is cheaper than testing everything live.
2. Read results as hypotheses
A simulated win tells you what to test next. It does not tell you how much revenue the change will add.
3. Prove the winner live
Run the shortlisted change as a live A/B test on real traffic and measure orders and revenue, not clicks.
4. Watch for errors in the variant
A variant can lose because it is slower or throws an error on one browser, not because the idea was weak. Check error rates and speed for each version before you call a result.
How do Noibu's agents fit in?
Noibu's A/B Testing Agent sets up tests on real changes and reads them on revenue, so the proof comes from real shoppers. The Bug Resolution Agent finds the errors shoppers hit, so a broken variant does not get mistaken for a weak idea. Every agent runs the same loop: find, build, your approval, ship, measure. Nothing reaches the live store until a person approves it.
Noibu is AI for ecommerce operations, the always-on optimization system that keeps online stores running at their best. A team of supervised AI agents finds what's broken, slow, or leaking revenue, does the work, and hands results back for human approval. Trusted by retailers like Free People, Guess, Liquid I.V., and Ariat.
Frequently asked questions
What does Tested by Sidekick mean?
Shopify's Fall '26 Edition describes recommendations that are identified and tested by Sidekick using SimGym, which uses AI shoppers to try a store. The test reflects simulated personas, not your real customers.
Is a simulated test the same as an A/B test?
No. A live A/B test splits real traffic between versions and measures orders. A simulation uses AI personas, so it can screen ideas but cannot prove an effect on revenue.
Can I trust a simulated result enough to ship a change?
Treat it as a hypothesis. For low-risk changes you may ship on a simulated result, but for anything that touches product, cart or checkout, confirm it with a live test first.
When is simulation most useful?
When you have more ideas than traffic to test them. Simulation helps you shortlist, so your limited live tests go to the ideas most likely to matter.
Why might a good idea lose a live test?
The variant may be slower or throw an error on one browser or device. Over 90% of website errors are never reported, so check error and speed data for each version before judging the idea.
Related reading
- Client-side vs code-level A/B tests
- Shopify AI tools in 2026: what Sidekick does and where AI agents fit
- Why CRO programs plateau after six months
Find what is holding your store back: run a free website audit →
