Always Valid Inference: Bringing Sequential Analysis to A/B Testing — HN submission

Wed, 17 Aug 2016 20:11:59 GMT

Sequential A/B testing at Optimizely

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

Always Valid Inference: Bringing Sequential Analysis to A/B Testing — mentioned by leo_pekelis

Fri, 08 Apr 2016 01:02:23 GMT

Hi, this is Leo, Optimizely's statistician. If you're looking for a more scientific paper, maybe take a look at this one we wrote recently: http://arxiv.org/abs/1512.04922 Should have everything you would ever want to know about the method. I agree with you that the problem of inference and interpretation between A/B data, algorithms, and the people who make decisions from them is a hard one and worth working on. That said, I do think the two sources of error our stats engine addresses - repeatedly checking results, and cherry picking from many metrics and variations - did make progress in…

Paper and discussion history. A mention is not an endorsement.

Always Valid Inference: Bringing Sequential Analysis to A/B Testing · HackerPapers discussions

Always Valid Inference: Bringing Sequential Analysis to A/B Testing — HN submission

Always Valid Inference: Bringing Sequential Analysis to A/B Testing — mentioned by leo_pekelis