Creating a Hypothesis for Site Optimization

Creating a hypothesis should be one of the first things that you do when you start running A/B and multivariate tests on your Web site. Just because you have the keys to an optimization tool (even a free one like Google Web Site Optimizer), you should NOT be starting out saying, “hey, let’s see if changing this button from ‘Add To Cart’ to ‘Buy Now’ works better!” It’s vital to understand that you need to start with a hypothesis and then set clear goals before you start testing.

Setting a hypothesis is not a difficult thing to do, and it will help you stay clear on exactly what you are trying to accomplish in running a test. Here are a few examples of what might be appropriate hypotheses:

  • By changing the button on our product details page, we expect that we will be able to increase the rate at which visitors add products to their carts.
  • If we can decrease our shopping cart by one complete step, we can make it easier for customers to complete their purchase, thereby increasing conversion rate.
  • If we can provide more targeted information on our most popular landing pages, we can decrease bounce rates.
  • Maybe if we make it easier for visitors to use our internal search, visitors will more easily find products of interest, increasing conversion rate.
The common theme among all of these ideas/hypotheses is that none of them address, specifically, what will be done. This is the best way to start, because:
Creating and starting with a hypothesis, frees you from simply testing graphics and content, enabling you to test your business ideas and site effectiveness (i.e. conversion).

It is the hypothesis that you should be taking to the rest of your team when asking for the best user experience and design ideas to prove your hypothesis. You should not let a designer alone be the one that starts the process of site optimization.

Creating a hypothesis also makes it easier to measure the results of site optimization. If you start with just a design that is going to simply be “better than the last,” there’s no clear way to measure that. For example, if you were to change how you present your internal search results, is your success measure conversion rate, add to cart rate, product views or maybe average order value? There’s no real answer here, and starting a test without a hypothesis will result in a lot of debate over what success is when it comes time to evaluate the test.

Your hypothesis should make it clear what you are trying to improve, so that everyone can agree upon the success measure in advance of the test.

So if you start your testing and site optimization with an appropriate hypothesis, your goals and the eventual evaluation of your success should more easily fall into place.

3 thoughts on “Creating a Hypothesis for Site Optimization

  1. Hi Jason,

    Good point, most times hypothesis get discussed as theories where the audience( decision makers) debate about what to test first and what later. What are your thoughts on prioritizing a list of hypothesis statements.

    Do you present them as Mini Projects with a budget estimate, complexity and impact or take some other approach.

    Most times if testing projects are left to decision makers to discuss they end up being HIPPO’ed. What should be the format so the audience says Oh YES!/ NOPE.


  2. Aman, prioritizing a list of hypothesis is a great idea. I also think that it would be an excellent idea to open up the creation of hypotheses to your entire online business team, getting everyone involved in the idea of optimization (democratization of optimization?).

    I don’t know that I’d refer to each test as a Mini Project in so many words, but different tests can easily have varying degrees of complexity and impact. Instead of stand alone Mini Projects, site optimization should become a standard part of your every day business, where you focus on site conversion just as much as you do on your external marketing campaigns. It can be tough to get that level of commitment from the HIPPOs sometimes.

    Depending upon how advanced your company is in terms of testing, the decision of what to test getting “HIPPO’ed” might not be the worst thing ever. At least you are testing! Ideally though, you can become agile enough so that testing is occurring without intervention from the HIPPOs.

  3. Just like any argument, Aman, it helps to back up your points with data.

    It may help to start with some analytics work to find the important pages to test and do some informal user testing to come up with some initial hypotheses.

    Of course, if you don’t test all the ideas, you’ll never know which ones are best. Hippos can sometimes have good ideas too.


Leave a Reply

Your email address will not be published. Required fields are marked *