The Sweet Relief of A/B Testing

Sometimes, when someone asks you a question, and there's a lot riding on the answer, it's a relief when you can say, "I don't know" and really mean it.

I've worked for a lot of small companies that set up shop online, and management always ends up asking questions like "What should the text be in the header on the homepage?" or "Where should the 'Add to Cart' button be on the product page?" or "How many steps should the checkout product have?" They ask themselves, and ask their cohorts, but don't come up with any definitive answers, so they start asking the technical people who are busy coding up their site what they think.

I'm one of those people coding up the HTML for your site. And I'm telling you flat-out: I don't know.

Part of this is based on the fact that I'm dealing with incomplete information. If you ask the question more specifically, I might be able to give you my opinion. For example, what change are you hoping to affect in the results? In the land of e-commerce, the answer to this question is almost always related to the number of customers that place orders. You want to know if changing some part of your site will increase the total number of conversions.

Even given the intended goal, I have very little relevant experience on which to base my opinion. Marketers tend to ask developers these kinds of questions because they're hopeful that web programmers understand the domain better than they do. Unfortunately, no matter how experienced we are in e-commerce programming, we really don't know. There are lots of reasons for this. The requirements of your site might be different than other sites, even those of your competitors. Most importantly, it's because we are not your customers; we are developers.

Your customers are the highest authority to which you should appeal for answers about how you should design your site, what the copy should be, and what the contents of each page should be. If you ask me, decisions about the wording of value propositions and calls to action are too important to be left up to anyone other than your customers.

So what is the solution? A/B split test. Use a tool like the Google Website Optimizer which allows you to present two different versions of the same page, randomly, to different customers. After enough time has passed and you've gotten enough samples of customers viewing the page you're testing, the Website Optimizer will test both of them, and then give you a report about which version of the page led to the highest conversion rate.

This takes the guesswork out of it. I've been in several meetings where two managers are arguing with one another about some little point about how the site should look, and one of them turns to me, the developer, and says, "What do you like better, my solutions or his?" And then they both turn and look at me, waiting for me to answer, each one hopeful that I'll agree with their opinion so they can rub it in the other's face. Eagerly looking to me for a vote, as if I were some kind of "trump card".

You might be tempted to just agree with the one of the two people who is in a better position to approve your pay increases. Or you might actually be egocentric enough to think you know the correct answer. Don't risk it. Even if you offer an opinion, they might ask you to explain why, and then you're just going to become a fountain spewing forth made-up reasons in an attempt to justify yourself.

As a developer, you should practice saying it a few times, until it becomes automatic: "I don't know. We should split test it." Remember it the next time a couple of managers are engaged in a pissing contest and ask you what you think. You'll feel the relief of never actually having the burden of the decision rest on you. It's a great way to stop arguments before they even start.