Just committed UI for specifying/calculating experiment duration:

The idea is that, as I mentioned last week, there are statistical problems with not specifying a sample size (i.e. an experiment duration) ahead of time. This UI simply asks the user what the “detection level” for this metric is (here, 0.01, in other words, 1% differences in conversion rate) and computes an experiment duration based on that, automatically.

Doing this in the general case is hard, as we need to know the variance of the metric we are using. However, ShrimpTest can take advantage of the fact that our goal metrics are organized around “metric types”, “conversion” being one of them. Because conversion events are Bernoulli trials, their maximum variance is known, enabling this calculation. For custom (manual) metrics, an input box is shown to ask the user to specify a variance, if they would like to take advantage of the experiment duration calculation.

This experiment duration value will be used in calculating when an experiment can be completed, enabling notifications when experiments are over without just waiting on a significant result.