Notes from lunch with Alex King:

  • Focus on scale: cache/store summary data, not just granular data. Also, look into updating some of the aggregate stats (avg, standard deviation) incrementally so that the summary computations are more lightweight.
  • The advantages of recording metrics in JS over PHP. Consider allowing the AJAX call to record metrics to be a different machine.
  • Again, scale: allow granular/raw data to be on another db/another service?
  • Perhaps we don’t even need to record variant assignment and metric data on the server at all, if the cookie we give people includes information on what variants they’re in? This is kind of a radical idea, but could make ShrimpTest much less db-intensive.