A/B Test Significance Calculator

Visitors, variant A

Conversions A

Visitors, variant B

Conversions B

Confidence level, %

Result

Is the difference significantДа, различие значимо при 95%
p-value0.0273
Conversion A, %3%
Conversion B, %3.8%
Uplift, %26.67%
z statistic2.207

The uplift on paper and the uplift in reality

Applies a two-proportion z-test to decide whether the difference between variants is real or noise.

Variant B converted at 3.8 percent against A's 3.0 — an apparent win. But on small samples a gap that size appears by chance constantly, and a decision made on it costs more than no decision at all.

The test compares the observed difference against the spread chance alone would produce. The output is a p-value: the probability of seeing a gap this large if the variants are in truth identical. Below five percent is the conventional threshold for calling it real.

What this calculator shows

Each variant's conversion rate and the relative uplift
The p-value and the z statistic
A direct verdict at the confidence level you choose

What to keep in mind

A two-tailed z-test for two proportions, the standard tool for comparing conversion rates.
It assumes traffic was split at random and both groups ran simultaneously; comparing consecutive weeks will mislead.

FAQs

When can I stop a test?

When the pre-calculated sample has been collected. Peeking and stopping the moment p first dips below 0.05 is a methodological error that sharply raises the false-positive rate.

What does a p-value of 0.03 mean?

That with identical variants we would see a gap this large or larger three times in a hundred. It is not the probability that B is better — a common and damaging misreading.

Why is my test never significant?

Usually too little data. No significance does not mean no difference; it means the difference is invisible at this sample size. Work out the required sample in advance.

How many variants can I test at once?

Two. With three or more the chance of finding a spurious winner rises and a multiple-comparison correction is needed. Testing in pairs is simpler and safer.

Worked example

Two weeks of testing

Input: A: 150 of 5,000, B: 190 of 5,000

Output: 3 percent against 3.8, uplift 26.67 percent, p = 0.0345 — significant

Note: At the same percentages on a sample five times smaller, p would be around 0.3 and the conclusion would reverse. Sample size decides as much as the gap itself.

Useful links

All tools in Marketing

A/B Test Significance Calculator

Result

Related tools

A/B Test Sample Size Calculator

Conversion Rate Calculator

P-value Calculator

Useful links