Bayesian A/B testing: a better way to reason about experiments

The underlying idea

You run an A/B test. After two weeks, variant B is converting at 5.2% versus 4.8% for the control. Your analyst tells you the result “isn’t significant yet” and you need to keep waiting. Another week passes. Still not significant. Leadership wants a decision. You’re stuck, because the framework you’re using was never designed to answer the question you actually have.

The question you have is simple: what’s the probability that B is better than A? And how much better?

Standard frequentist A/B testing cannot answer that question. It’s a strange fact that surprises most people when they first encounter it. A p-value doesn’t tell you the probability that B beats A. A confidence interval doesn’t tell you there’s a 95% chance the true lift falls inside it. Both are statements about the long-run behavior of your testing procedure, not about the specific experiment in front of you.

Bayesian A/B testing answers the question you actually asked. It gives you a direct statement: “there’s an 89% probability that B is better than A, and the most likely lift is 1.3 percentage points.” That sentence is something a product manager can act on. It maps cleanly onto a business decision in a way that “we failed to reject the null at alpha 0.05” never will.

The shift is not cosmetic. It changes what you can conclude, when you can stop, and how you handle the constant pressure to peek at results early.

Historical root

The two approaches to probability have been in tension for more than 250 years.

The Bayesian approach traces to Thomas Bayes, an English Presbyterian minister whose theorem was published posthumously in 1763. Pierre-Simon Laplace independently developed and substantially extended the same ideas in the following decades, applying them to problems in astronomy and demography. For roughly 150 years, this was simply how probability was done. Probability represented a degree of belief, updated as evidence arrived.

In the early 20th century, the frequentist school led by Ronald Fisher, Jerzy Neyman, and Egon Pearson largely displaced the Bayesian approach in applied statistics. Their motivation was partly philosophical: they distrusted the subjectivity of the prior, the belief you hold before seeing data. They built a framework that avoided priors entirely, defining probability strictly as long-run frequency. This became the dominant paradigm taught in nearly every statistics course for the rest of the century, and it’s why p-values and confidence intervals are the default in most A/B testing tools today.

The Bayesian approach returned to prominence for a practical reason: computation. Bayesian inference often requires evaluating difficult integrals that have no closed-form solution. The development of Markov Chain Monte Carlo methods in the late 20th century, combined with cheap computing power, made these calculations tractable. By the 2010s, companies running large-scale experimentation, including several major technology platforms, had adopted Bayesian methods specifically because they answer business questions more directly and handle early stopping more gracefully.

Key assumptions

You can specify a prior. Bayesian inference begins with a prior distribution, your belief about the conversion rate before the experiment. For A/B testing, a common and defensible choice is a weakly informative prior, one that expresses “conversion is probably somewhere between 0 and 20%, but I’m not committing to a value.” The Beta distribution is the natural choice for conversion rates because it’s bounded between 0 and 1 and pairs cleanly with binary outcomes. If you have genuine historical data, the prior can encode it. If you don’t, a weak prior lets the data dominate quickly.

The outcome is well-defined and binary (for the standard setup). The classic Bayesian A/B test models conversions as a binomial process: each visitor either converts or doesn’t. This maps to a Beta-Binomial model, which has a clean closed-form solution. Revenue, time-on-page, and other continuous outcomes require different likelihood models and usually simulation, though the logic is identical.

Observations are independent. Same requirement as frequentist testing. Each visitor’s outcome should be independent of the others. This breaks under network effects, repeat visitors counted twice, or interference between variants.

The prior is honest. The one genuine risk in Bayesian testing is choosing a prior that biases the result toward what you want to see. The discipline is to select the prior before seeing the data and to prefer weak priors unless you have real justification for a strong one. A well-chosen weak prior has negligible influence once you’ve collected a few hundred observations.

The math

The engine is Bayes’ theorem:

$P(\theta \mid \text{data}) = \frac{P(\text{data} \mid \theta) \cdot P(\theta)}{P(\text{data})}$

In words: the posterior belief about the conversion rate $\theta$, after seeing data, is proportional to the likelihood of the data given $\theta$, times the prior belief about $\theta$.

For conversion rates, the Beta-Binomial model makes this concrete and clean. Suppose you model each variant’s conversion rate with a Beta prior:

$\theta \sim \text{Beta}(\alpha, \beta)$

The Beta distribution has a useful interpretation: $\alpha$ acts like a count of prior successes (conversions) and $\beta$ like a count of prior failures (non-conversions). A $\text{Beta}(1, 1)$ prior is flat, expressing no preference across the 0 to 1 range.

When you observe $s$ conversions out of $n$ visitors, the posterior is simply:

$\theta \mid \text{data} \sim \text{Beta}(\alpha + s, \beta + n - s)$

This is the value of conjugate priors. The posterior is the same family as the prior, updated by adding your observed successes and failures to the parameters. No integration required. You start with $\text{Beta}(1, 1)$, observe 48 conversions out of 1000 visitors, and your posterior is $\text{Beta}(49, 953)$.

To compare two variants, you compute:

$P(\theta_B > \theta_A)$

the probability that B’s true conversion rate exceeds A’s. This is the number you actually wanted all along. There’s no simple closed form for it in general, but it’s trivial to estimate: draw many samples from each variant’s posterior and count the fraction where B’s sample exceeds A’s.

You can also compute expected loss, the average amount you’d give up by picking the wrong variant. This is what lets you stop a test responsibly: when the expected loss of choosing B drops below a small threshold you set in advance, you’ve learned enough to decide.

The code

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)

# Observed experiment data
# Control (A): 48 conversions out of 1000 visitors
# Variant (B): 61 conversions out of 1000 visitors
conversions_a, visitors_a = 48, 1000
conversions_b, visitors_b = 61, 1000

# Weak prior: Beta(1, 1) is flat over [0, 1]
prior_alpha, prior_beta = 1, 1

# Posterior parameters (conjugate update: add successes and failures)
post_a = (prior_alpha + conversions_a, prior_beta + visitors_a - conversions_a)
post_b = (prior_alpha + conversions_b, prior_beta + visitors_b - conversions_b)

# Draw samples from each posterior
n_samples = 200_000
samples_a = rng.beta(post_a[0], post_a[1], n_samples)
samples_b = rng.beta(post_b[0], post_b[1], n_samples)

# The question you actually care about:
# What is the probability that B is better than A?
prob_b_better = np.mean(samples_b > samples_a)
print(f"P(B > A) = {prob_b_better:.1%}")

# How much better? Distribution of the lift.
lift = samples_b - samples_a
print(f"Expected lift: {lift.mean():.3%}")
print(f"95% credible interval for lift: "
      f"[{np.percentile(lift, 2.5):.3%}, {np.percentile(lift, 97.5):.3%}]")

# Expected loss from choosing B (how much you'd lose if B is actually worse)
loss_choosing_b = np.mean(np.maximum(samples_a - samples_b, 0))
print(f"Expected loss if you pick B: {loss_choosing_b:.4%}")

# Relative probability each variant is the winner
print(f"\nP(B is best): {prob_b_better:.1%}")
print(f"P(A is best): {1 - prob_b_better:.1%}")

Running this produces:

P(B > A) = 89.8%
Expected lift: 1.297%
95% credible interval for lift: [-0.705%, 3.304%]
Expected loss if you pick B: 0.0497%

P(B is best): 89.8%
P(A is best): 10.2%

Read what this gives you. There’s an 89.8% probability that B genuinely beats A. The most likely lift is 1.3 percentage points. The expected loss from shipping B, even if B turns out to be the worse choice, is tiny: about 0.05 percentage points of conversion. A product manager can look at that and make a defensible call: nearly 90% confident, small downside, ship it. No one has to explain what “failing to reject the null” means.

Notice also that the 95% credible interval for the lift still includes zero. In a frequentist framework, this experiment would be “not significant” and you’d be told to keep waiting. The Bayesian framing lets you see that while there’s genuine uncertainty, the weight of evidence and the small downside already support a decision. The middle panel of the chart shows this directly: most of the lift distribution sits above zero, but a thin tail crosses into negative territory.

Business application

Stopping tests responsibly. The single biggest practical advantage. In frequentist testing, peeking at results and stopping when you see significance inflates your false positive rate badly, because every look is another chance to cross the threshold by luck. Bayesian testing with an expected-loss stopping rule handles continuous monitoring gracefully. You decide in advance on a loss threshold, watch the experiment, and stop when expected loss drops below it. This aligns with how teams actually want to run experiments: watch, learn, decide when you know enough.

Communicating to stakeholders. “There’s a 90% probability B is better and the downside is negligible” is a sentence any executive can act on. “We achieved p = 0.04” invites misinterpretation every single time. The Bayesian output speaks the language of decisions, not the language of hypothesis-rejection procedures.

Small-sample and low-traffic settings. When you don’t have millions of visitors, frequentist tests are often hopelessly underpowered and never reach significance. Bayesian methods still give you a coherent, honest picture: a wide posterior that says “we’re genuinely uncertain, here’s the current best estimate and how uncertain it is.” That’s more useful than a binary “not significant.”

Multi-variant testing. Comparing five variants at once is awkward in the frequentist world because of multiple-comparison corrections that sap your power. In the Bayesian framing, you simply compute the probability each variant is the best, and those probabilities sum to one. It extends naturally.

Where Bayesian testing can mislead. The prior is a real responsibility. Choose a strong prior that happens to favor your preferred variant and you can bias the outcome, which is why the discipline of pre-committing to a weak prior matters. Bayesian methods also don’t rescue you from bad experimental design: confounded variants, interference between test groups, and non-independent observations break Bayesian analysis exactly as they break frequentist analysis. And “90% probability B is better” is still a probabilistic statement, not a guarantee. One test in ten framed that way will point the wrong direction, which is entirely consistent with the method working correctly. The framework improves how you reason about evidence. It doesn’t remove uncertainty, and it can’t manufacture signal that the experiment didn’t capture.