How Microsoft uses their web sites to experiment with web performance tuning

Eric Schurman manages performance for Microsoft's web sites. In this interview, he talked about how Microsoft uses small sets of their web site visitors to see how different changes affect user behavior.

---

Like any of the big sites--Google, Yahoo, Amazon--we have an experimentation platform. And what this lets us do is take some fraction of our audience, some bucket of our audience and give them a different experience than other buckets of our audience. And so what we tend to do is have users in an experiment and users in a control. And we can give the experimental group pretty much any kind of experience that we want. And then we have an analysis system that runs after the fact and compares end-user methods for those two groups. And so we can track everything from how much money did we make from each organization, from each of the groups, to what was the click-through rate, to what was the perceived relevance. There's all sorts of different things that we can track.

We make different business decisions based off of this data, and so one of the things that we've always wondered is how much of an impact does performance make on our key business metrics. What we've tried to do is come up with many different ways of measuring a performance impact. So in some cases, we've been able to take some upcoming improvement that we plan on releasing and, before we release it to everybody, we'll release it to a small number of people first and see what the metrics show.

But most of the time, when we have something that we know is going to improve the experience for the users, we just want to get it out there to everybody as quickly as we can. In these cases, we ran several different classes of experiments. In some cases, we tried adding more content to the page, just in the form of HTML comments. So we made the page fatter. We made the page fatter in a bunch of different locations. We also tried adding different kinds of actual latencies to the page where we're actually putting a pause on the server for a certain period of time. We were never routing more than a very tiny fraction of our users to this experience, and what we generally found was, depending on how much of a latency we were adding, or how much page weight we were adding, well, as you would expect, it impacted users different amounts depending on how much we did.

In cases where we were giving what was a significantly degraded experience, the data moved to significance extremely quickly. We were able to tell when we delayed people's pages by more than half a second, and it was very obvious that this had a significant impact on users very quickly. And so we were able to turn off that experiment. So the reasoning that we did it was it helps us make a strong argument for how we can prioritize work on performance against work on other aspects of the site. I mean, I'm the performance guy; I don't like hurting end-users in terms of performance. But if it helps me make a good, strong business argument to make other changes that will improve the experience for all of my users, for all time to come, and it means that a small segment of users for a small period of time will experience what we think will likely be a negative thing but we're not sure, it was a test worth running.