Sorry if this sounds condescending. I'm asking out of total naivety. What is the background of people who typically run A/B tests? Are they often unfamiliar with things like multiple comparisons, stopping problems, and the like?
I can answer from personal experience. I'm a dev at a tiny startup. I took statistics at college 20 years ago but haven't used it since the final exam, and remember little other than sample size is critical. I've figured out through recent experience that you need to let a test run a while - results seem to flip flop for the first few thousand visitors. I use tools like VWO because it makes it simple to run a simple one variable test, and I'm happy with the results (though ignorance is bliss, so maybe others would not be so content).
I think that background varies - some people are fluent statisticians and others come from different disciplines / careers. Anecdotally, my experience from browsing A/B testing advice online is similar; some content is very statistically savvy and some is a bit oversimplified.
