A/B Testing: Statistics

Zolvio supports both traditional “confidence” stats and Bayesian analysis for easier decisions.

Stats toggle: Traditional vs Bayesian
Screenshot placeholder
1200×650px • /docs-images/ab-testing/stats-toggle-1200x650.png
Add a file at this path to replace the placeholder.
Statistics view with Traditional vs Easy-to-understand (Bayesian) toggle.

Traditional (Frequentist)

Traditional stats are useful when you want a familiar significance-style view.

  • Confidence: how likely the result is not random
  • Lift: percent change vs control
  • Sample size: visitors per variant

Bayesian (Easy to Understand)

Bayesian stats answer questions that map more directly to decisions:

  • Probability to beat control: chance the variant is better than control
  • Credible intervals: plausible range for conversion rate and lift
  • Expected loss (regret): what you risk by choosing the wrong variant

Tip

If your goal is “should we ship this?”, Bayesian probability and expected loss are often easier to act on than a single confidence number.

How to Use This in Practice

  • Run tests long enough to cover weekly cycles (avoid early stopping)
  • Segment by device when behavior differs (mobile vs desktop)
  • Validate “why” with recordings and heatmaps before deciding

Related