A/B testing framework and statistical guardrails

Project Description

We run experiments and call them when the numbers look good, which means we are shipping noise. We want a proper framework: sample size calculation before launch, fixed horizons or sequential testing done correctly, guardrail metrics, and automated readouts that state significance honestly. We would like the tooling to make peeking hard rather than merely discouraged, because discouragement has not worked. Please include how you would handle multiple comparisons and low-traffic surfaces.

Skills Required