A/B Testing: Statistics

Zolvio supports both traditional “confidence” stats and Bayesian analysis for easier decisions. Both live under Technical details → Advanced stats on the test page.

Advanced Stats, Frequentist tab
Technical details → Advanced Stats, Frequentist: confidence, observed lift, confidence intervals and the sample needed per effect size.

Start with the decision engine

You don’t have to interpret the raw statistics yourself. Zolvio’s results view leads with a plain-language verdict from a single decision engine that reads every arm at once and reports each variant’s chance to win. The stats below are the depth behind that verdict—open them when you want to see the underlying numbers.

  • Chance to win — the probability a variant is the best of allarms (control included). Across variants these add up to ~100%, so it works for A/B/n, not just two-way tests.
  • The control can win. If no variant beats the original, the verdict says so.
  • Data-quality honesty. Zolvio won’t call a decision-ready winner while a test is too new, under-sampled, has a skewed traffic split (SRM), or a variant isn’t applying reliably. You can still publish early if you choose.

Traditional (Frequentist)

Traditional stats are useful when you want a familiar significance-style view.

  • Confidence: how likely the result is not random
  • Lift: percent change vs control
  • Sample size: visitors per variant

Bayesian (Easy to Understand)

Bayesian stats answer questions that map more directly to decisions:

  • Probability to beat control: chance the variant is better than control
  • Credible intervals: plausible range for conversion rate and lift
  • Expected loss (regret): what you risk by choosing the wrong variant
Advanced Stats, Bayesian tab
The Bayesian tab: the chance each version is better, expected loss, credible intervals and the lift distribution.

Tip

If your goal is “should we ship this?”, Bayesian probability and expected loss are often easier to act on than a single confidence number.

How to Use This in Practice

  • Run tests long enough to cover weekly cycles (avoid early stopping)
  • Segment by device when behavior differs (mobile vs desktop)
  • Validate “why” with recordings and heatmaps before deciding

Related