A/B Testing: Statistics

Zolvio supports both traditional “confidence” stats and Bayesian analysis for easier decisions.

Stats toggle: Traditional vs Bayesian
Screenshot placeholder
1200×650px • /docs-images/ab-testing/stats-toggle-1200x650.png
Add a file at this path to replace the placeholder.
Statistics view with Traditional vs Easy-to-understand (Bayesian) toggle.

Start with the decision engine

You don’t have to interpret the raw statistics yourself. Zolvio’s results view leads with a plain-language verdict from a single decision engine that reads every arm at once and reports each variant’s chance to win. The stats below are the depth behind that verdict—open them when you want to see the underlying numbers.

  • Chance to win — the probability a variant is the best of allarms (control included). Across variants these add up to ~100%, so it works for A/B/n, not just two-way tests.
  • The control can win. If no variant beats the original, the verdict says so.
  • Data-quality honesty. Zolvio won’t call a decision-ready winner while a test is too new, under-sampled, has a skewed traffic split (SRM), or a variant isn’t applying reliably. You can still publish early if you choose.

Traditional (Frequentist)

Traditional stats are useful when you want a familiar significance-style view.

  • Confidence: how likely the result is not random
  • Lift: percent change vs control
  • Sample size: visitors per variant

Bayesian (Easy to Understand)

Bayesian stats answer questions that map more directly to decisions:

  • Probability to beat control: chance the variant is better than control
  • Credible intervals: plausible range for conversion rate and lift
  • Expected loss (regret): what you risk by choosing the wrong variant

Tip

If your goal is “should we ship this?”, Bayesian probability and expected loss are often easier to act on than a single confidence number.

How to Use This in Practice

  • Run tests long enough to cover weekly cycles (avoid early stopping)
  • Segment by device when behavior differs (mobile vs desktop)
  • Validate “why” with recordings and heatmaps before deciding

Related