You launched an A/B test, after three days variant B is ahead by 15% and you are tempted to declare it the winner. But with only 200 visitors per variant, that 15% could be pure chance. To know whether a difference is real or noise you need a sufficient sample, and the surprise is how large it must be: to reliably detect a 20% improvement on a 5% conversion rate, you need over 8,000 visitors per variant. This A/B test sample size calculator tells you, before launching, how many visitors you need and how long the test will take, so you avoid the most common testing trap: concluding too early on insufficient data.

What an A/B test is

An A/B test (or split test) is a controlled experiment in which two versions of something — a page, a button, an email, a price — are compared to understand which works better. Visitors are randomly split between version A (the control, the current one) and version B (the variant), and a metric of interest is measured, typically the conversion rate. If the variant converts better in a statistically reliable way, it is adopted. The A/B test is the scientific method applied to optimization: instead of deciding by opinion, you decide by evidence. It is used everywhere — marketing, product, e-commerce, email, apps — in every country and sector. But a test is only as good as its statistical rigor: without an adequate sample, results deceive.

Why a minimum sample is needed

The central problem of the A/B test is distinguishing the signal from the noise. Every conversion measure fluctuates by chance: if you flip a coin 20 times you do not always get 10 heads. With few visitors, a variant can seem ahead just by luck. The minimum sample is the number of observations needed for an observed difference to be probably real and not the result of chance. The smaller the effect you want to detect, the larger the sample must be, because distinguishing a 2% improvement from noise requires many more trials than distinguishing a 50% improvement. Calculating the sample before launching the test — and committing to collect it all before reading the result — is the golden rule of serious testing. The calculator gives you this number.

The four parameters of the calculation

The sample size depends on four quantities. The baseline conversion rate: how much the current version converts today (for example 5%). The minimum detectable effect (MDE): the smallest improvement you want to be able to detect (for example a +20% relative, that is from 5% to 6%). The confidence level: how sure you want to be that a positive result is not a false alarm (typically 95%). The statistical power: the probability of detecting a real effect when there really is one (typically 80%). These four numbers determine how many observations are needed. The calculator combines them in the standard formula for comparing two proportions and returns the sample per variant.

The baseline conversion rate

The baseline conversion rate is the starting point: the percentage of visitors who take the desired action (purchase, sign-up, click) on the current version. It is fundamental because it heavily influences the needed sample: lower baseline rates require much larger samples. Detecting an improvement on a 20% rate is much easier than the same relative improvement on a 1% rate, because at low rates there are very few conversions to compare and the relative noise is enormous. This is why tests on rare actions (expensive purchases, niche conversions) require a lot of traffic or long times. Use your real rate, measured over a representative period. The calculator uses it as the basis of the comparison.

The minimum detectable effect (MDE)

The minimum detectable effect (MDE) is the most powerful and least understood lever. It is the smallest improvement the test must be able to detect: if you set an MDE of 20%, the test is sized to notice a 20% improvement or more, but it might not "see" a 5% improvement. Here is the crucial point: the smaller the effect to detect, the larger — and in a non-linear way — the needed sample. Halving the MDE roughly quadruples the sample. The table shows it, with a baseline rate of 5%:

Detecting a 5% improvement requires over 120,000 visitors per variant; detecting a 50% one, fewer than 1,500. This is why, with limited traffic, it is better to test big, bold changes rather than micro-optimizations: small effects, even if real, are often impossible to detect in reasonable times. The calculator immediately shows you this trade-off.

Confidence level and statistical power

Two parameters control the rigor of the test. The confidence level (typically 95%) governs false positives: with 95%, you accept a 5% probability of declaring a variant the winner when it actually is not. The statistical power (typically 80%) governs false negatives: with 80%, you have an 80% probability of detecting a real effect when there is one. Raising both makes the test more reliable but requires larger samples. The table shows the typical values and the related statistical scores (z):

95% confidence and 80% power are the de facto standard in most marketing and product tests: a good balance between rigor and practicality. For high-stakes decisions (large investments, irreversible changes) it can be worth raising to 99% and 90%, accepting larger samples. The calculator lets you set both.

The sample size formula

The calculation is based on the standard formula for comparing two proportions. The sample per variant is proportional to the square of the sum of the two z scores (confidence and power) and to the variability of the two rates, and inversely proportional to the square of the difference between the rates. It is this square in the denominator that explains the explosive growth: if the difference to detect halves, the sample quadruples. The z scores are derived from the confidence and power levels through the normal distribution. You do not need to know the math in detail: the calculator applies the correct formula and returns the number. The important thing is to understand the logic — small effects and low rates cost dearly in terms of sample — to design realistic tests.

How much sample is needed as the baseline rate varies

Besides the effect, the baseline rate also changes the needed sample a lot. The table shows the sample per variant to detect a 20% relative improvement, as the baseline rate varies:

For the same relative improvement to detect (20%), going from a 20% baseline rate to a 1% one multiplies the sample by twenty-five. This is why optimizing high-rate actions (for example add-to-cart) is faster than optimizing rare actions (for example the purchase of an expensive product). If your baseline rate is very low, consider testing higher up the funnel, where rates are higher and tests faster. The calculator makes this effect evident.

Test duration and traffic

Knowing how many observations are needed is half the work; the other half is how long it takes to collect them. The duration depends on traffic: total sample divided by daily visitors gives the days of testing. With 16,000 total visitors needed and 1,000 a day, the test lasts about sixteen days. But there is a practical rule beyond the math: a test should last at least one or two full weeks, to cover complete weekly cycles (Monday behaviors differ from weekend ones) and absorb variations. A test too short, even if it reaches the sample, risks capturing an atypical period. The calculator estimates the duration from your traffic: use it to plan, remembering the one-to-two week minimum.

The "peeking" mistake

The most common and most insidious mistake in A/B testing is peeking: looking at the results while the test is running and stopping it as soon as one variant seems to win. It seems harmless, but it is statistically devastating. Since the data fluctuates, if you look often enough sooner or later you will see a "significant" difference by pure chance, and if you stop there you declare a variant the winner when it is not. It is like rolling the dice until the number you wanted comes up and then stopping: the result proves nothing. The defense is simple but requires discipline: decide the sample and duration in advance (with this calculator), then read the result only at the end, without stopping earlier. If you really must monitor along the way, there are dedicated statistical methods (sequential tests), but naive peeking must be avoided.

False positives and false negatives

Every test can err in two ways. A false positive (type I error) is declaring a variant the winner when it actually is not better: the confidence level controls this risk (with 95%, a 5% probability). A false negative (type II error) is not noticing a real improvement: power controls this risk (with 80%, a 20% probability of missing it). There is a trade-off: reducing one error increases the other, unless you increase the sample. Understanding these two errors is essential to interpret results honestly: a "non-significant" test does not prove that the two versions are equal, only that you do not have enough data to distinguish them. The calculator lets you set confidence and power to balance the two risks according to what is at stake.

Practical tips for reliable tests

Some rules for tests you can really believe. Size first: calculate the sample and duration before launching, not after. Do not peek: read the result only when the sample is reached. Cover whole cycles: at least one or two weeks. Test one thing at a time, or use dedicated factorial designs. Verify randomization and that the two variants receive comparable traffic. Beware of many variants: testing A/B/C/D multiplies the comparisons and you need to correct significance. Distrust effects too good: a sudden +80% is often a tracking error, not a success. The calculator covers the first and most important of these tips, sizing; the others are method discipline.

Cautions and warnings

Some warnings. First: this tool calculates the sample size with the standard formula for two proportions and a normal approximation; it is designed for conversion rates (binary yes/no metrics), not for continuous metrics like average order value, which require different formulas. Second: it assumes a two-variant test with split traffic; for more variants adjustments are needed. Third: the duration is an estimate based on the constant traffic you enter; real traffic varies. Fourth: still respect a minimum of one to two weeks to cover the cycles. Fifth: the calculation guarantees the statistical rigor but not the quality of the experiment (randomization, tracking, absence of interference), which remain the responsibility of who runs it. Use the calculator to design honest tests, then run them with method.

How the calculator works

Enter the current conversion rate (baseline, as a percentage), the minimum detectable effect (the minimum relative improvement you want to catch, as a percentage), the confidence level (typically 95%), the statistical power (typically 80%) and the total daily visitors who will enter the test. The calculator returns the sample needed per variant, the total sample, the estimated test duration and the target conversion rate you would be able to detect, with an assessment. Try changing the minimum detectable effect or the baseline rate and watch how the sample explodes as the effect shrinks: you will immediately understand why designing a test well matters more than running it.

The data you need

You need five pieces of information. The baseline conversion rate: how much the current version converts today (measured over a representative period). The minimum detectable effect: the minimum relative improvement you want to be able to detect (the smaller it is, the larger the sample). The confidence level and the power: leave 95% and 80% if you have no particular needs. The daily visitors who will enter the test, to estimate the duration. The result is the sample per variant, the total, the duration and the target rate. Remember it is an estimate for binary conversion metrics: use it to size the test before launching it, the move that separates reliable tests from deceptive ones.

A worked example

A SaaS has a landing page that converts visitors into free-trial signups at 5%. The team has designed a new page and hopes for a 20% improvement (from 5% to 6%). The page receives 1,000 visitors a day. Before launching, they calculate the sample with 95% confidence and 80% power. The result surprises everyone: they need 8,155 visitors per variant, that is 16,310 in total, which with 1,000 visitors a day means about 16 days of testing. The product manager would have liked to "take a look" after three days and decide: but at that point there would be only about 1,500 visitors per variant, less than a fifth of what is needed, and any observed difference would be largely noise. Worse: if they looked every day and stopped as soon as B seemed ahead, they would declare a winner that often is not (peeking inflates false positives). The right decision is to launch the test, not watch it, and read the result only on the sixteenth day, having collected the whole sample and covered two full weekly cycles. Only then will the difference between 5% and 6% (if real) emerge from the noise reliably. If sixteen days were too many, the alternative is not to cheat on the data, but to test a more ambitious change: aiming at a +50% would require only 1,468 visitors per variant, less than three days.

Frequently asked questions

What is an A/B test?

It is a controlled experiment comparing two versions of something (page, button, email, price) by randomly splitting visitors between version A (current) and B (variant) and measuring a metric, typically the conversion rate. It serves to decide by evidence instead of opinion. But it is only as good as its statistical rigor.

How many visitors do you need for an A/B test?

It depends on four factors: the baseline conversion rate, the minimum effect you want to detect, the confidence level and the power. To detect a 20% improvement on a 5% rate with 95% confidence and 80% power you need over 8,000 visitors per variant. The calculator computes it from your values.

What is the minimum detectable effect (MDE)?

It is the smallest improvement the test must be able to detect. It is the most powerful lever: the smaller the effect, the larger the sample, non-linearly. Halving the MDE roughly quadruples the sample. Detecting a +5% on a 5% rate requires over 120,000 visitors per variant, a +50% fewer than 1,500.

What are confidence and power?

Confidence (typically 95%) controls false positives: the probability of declaring a variant a winner when it is not. Power (typically 80%) controls false negatives: the probability of detecting a real effect when there is one. Raising them makes the test more reliable but requires larger samples.

Why can I not stop the test as soon as a variant wins?

Because it is "peeking", the most insidious mistake. The data fluctuates: if you look often, sooner or later you will see a significant difference by chance, and stopping there declares a variant a winner when it is not. Decide sample and duration in advance and read the result only at the end.

Why do low conversion rates require more sample?

Because at low rates there are very few conversions to compare and the relative noise is enormous. For the same 20% improvement to detect, a 1% baseline rate requires about 25 times the sample of a 20% rate. This is why optimizing rare actions is slow.

How long should an A/B test last?

At least the time to collect the calculated sample, but also a minimum of one to two full weeks to cover complete weekly cycles (behaviors vary between weekdays and weekends). A test too short, even if it reaches the sample, risks capturing an atypical period.

What are false positives and negatives?

A false positive (type I error) is declaring a variant a winner when it is not better; confidence controls it. A false negative (type II error) is not noticing a real improvement; power controls it. Reducing one error increases the other, unless you increase the sample.

Does a "non-significant" test prove the versions are equal?

No. It only proves you do not have enough data to distinguish them. The absence of significance is not proof of equivalence: a real difference might exist that your sample is too small to detect. To conclude equivalence you need dedicated tests.

Does the calculator work for any country and sector?

Yes: A/B test statistics are universal, they do not depend on jurisdiction or currency. It applies to marketing, product, e-commerce, email, apps, in every country. Enter baseline rate, effect, confidence, power and traffic of your case and the calculator works.

Does it work for metrics other than conversion?

This calculator is designed for conversion rates, that is binary metrics (yes/no: converted or not). For continuous metrics like average order value you need different formulas (based on means and standard deviations). Use it for conversions, click-throughs, sign-ups and similar.

How do I use the calculator with more than two variants?

The calculation assumes two variants (A and B). With more variants (A/B/C/D) the multiple comparisons increase the risk of false positives and you need to correct significance (for example with Bonferroni), which increases the sample. As a starting estimate, size each pair and use the largest value with a margin.

Is the result a guarantee?

No. It guarantees the statistical rigor of the sizing, but not the quality of the experiment: correct randomization, reliable tracking, absence of interference and discipline (no peeking) remain the responsibility of who runs the test. Good sizing is necessary but not sufficient.

Updated July 2026