How Long Should an Ecommerce A/B Test Run?

Every ecommerce team that runs experiments eventually asks the same question: how long should an ecommerce A/B test run before the result can be trusted? End a test too early and a random streak picks the winner. Run it too long and the team loses weeks it could spend on the next idea.

Most ecommerce A/B tests should run for at least one to two full weeks, and until each variant reaches the sample size you calculated before launch. Stop on the planned date, not when a result first looks significant, and keep a single test from straddling a major sale like Black Friday unless the sale is what you are testing.

How long should an ecommerce A/B test run?

An ecommerce A/B test should run until two conditions are both true: each variant has reached the sample size calculated before launch, and the test has covered at least one full weekly cycle. For most online stores that means a minimum of one week and a typical run of two to four weeks. A test that hits its sample size on day three still needs to run to the end of the week.

The weekly cycle matters because shoppers behave differently across the week. Weekday lunch-break browsers, weekend buyers and payday shoppers convert at different rates and respond to changes differently. A test that only sees Monday to Wednesday traffic measures a slice of the store's customers, not the whole audience.

What decides how long an A/B test needs?

Three inputs set the length of an ecommerce A/B test: the page's traffic, its current conversion rate, and the smallest lift the team wants to detect. A high-traffic product page with a healthy add-to-cart rate can reach a reliable answer in about a week. A low-traffic landing page chasing a small lift can need many weeks, or may never reach a confident result.

The minimum detectable effect is the input teams most often skip. Detecting a small lift, such as a fraction of a percentage point on checkout completion, takes far more sessions than detecting a large one. Before launch, plug traffic, baseline conversion rate and the lift worth acting on into a sample size calculator, then divide the result by daily sessions per variant to get the planned duration.

Why shouldn't you stop an A/B test as soon as it looks significant?

Stopping an A/B test the first time a dashboard shows a significant result is the most common way ecommerce teams ship false winners. Early in a test, small samples swing widely, and a variant can look like a clear winner for a few days before the gap closes. Checking repeatedly and stopping at the first good-looking moment raises the chance of declaring a winner that does not exist.

The fix is a rule set before launch: a planned end date and a planned sample size. Teams can watch a running test for problems, such as a broken variant or a tracking gap, but the decision about the winner waits for the end date. A test that ends without a clear winner is still a result: it says the change did not matter enough to detect.

Should you run A/B tests during Black Friday and Cyber Monday?

Running an A/B test across Black Friday and Cyber Monday mixes two very different audiences into one result. Peak-season shoppers arrive with high intent and deep discounts, and they often convert at rates the store never sees the rest of the year. A variant that wins during that week may not win in January, and a test that starts before the sale and ends after it blends both behaviors together.

Most stores are better off finishing tests before peak traffic begins, pausing new launches during the sale, and starting fresh once traffic settles. The exception is a test built for the sale itself, such as two versions of a promotional banner or a holiday shipping message, where the peak audience is exactly the audience the team wants to learn from.

What if your store doesn't have enough traffic to finish a test?

A store that cannot reach its sample size within about four to six weeks should change the test, not stretch it. Long tests collect noise from seasonality, cookie deletion and returning shoppers who see both variants. Testing a bolder change, moving the test to a higher-traffic page, or measuring a step closer to the change, such as add-to-cart instead of completed orders, all shorten the time to a confident answer.

The guide on what a low-traffic store should fix before running A/B tests covers the sample-size math in more depth. The comparison of multivariate testing vs A/B testing explains why tests with many variants need even longer.

How does Noibu's A/B Testing agent handle test length?

Noibu's A/B Testing agent builds and runs multivariate experiments on an online store, measures the lift, and reports back with the winning variant to ship. The agent takes the build work off the team, so more ideas reach a test and each test can run its full planned length without tying up developers.

The A/B Testing agent follows the same loop as every Noibu agent: find the opportunity, build the experiment, wait for your approval, ship it, measure the result, and compound what it learns into the next test. Nothing goes live without your approval. The agent works in the Noibu console and in the AI tools teams already use, such as Claude, ChatGPT and Microsoft Copilot, through Noibu MCP. Teams deciding what to test next can read what to A/B test next on an ecommerce site.

The short version: how to set an A/B test's length

1. Calculate the sample size per variant from traffic, baseline conversion rate and the smallest lift worth acting on.
2. Turn that into days, then round up to full weeks. Plan for at least one week, and usually two.
3. Check the calendar and keep the test from straddling Black Friday, Cyber Monday or a major promotion.
4. Read the result once, on the planned end date.
5. If the page cannot finish a test in about four to six weeks, test a bolder change or a busier page instead.

To see which experiments Noibu's agents would run on your own store, start with the free website audit.

Noibu is AI for ecommerce operations, the always-on optimization system that keeps online stores running at their best. A team of supervised AI agents finds what's broken, slow, or leaking revenue, does the work, and hands results back for human approval. Trusted by retailers like Free People, Guess, Liquid I.V., and Ariat.

Frequently asked questions