13-03-2014 10-48-08

Split testing is the ‘Holy Grail’ for website optimisation, why spend efforts getting more traffic each week when you could just make more out of the traffic you already have?

The art of split testing is to simply randomly show two or more different versions of something to a large number of people and then measure which version gives the best results.  Luckily split testing is now really simple to setup online, you’ll need no technical or mathematical knowledge to make it work or to draw valuable conclusions from it.


What can split testing be used to improve on?

Split testing can be used to improve any aspect of a website which you can imagine; examples of these ‘conversion rates’ to improve are:

  • The number of sales you make on average per 100 visitors
  • The number of email signups you receive on average per 100 visitors
  • The number of pages viewed on average per 100 visitors
  • The number of phone calls generated on average per 100 visitors
  • The average visit duration
  • The average revenue generated (requires ecommerce tracking)

The effectiveness of split testing can be used on metrics such as reducing the bounce rate of a website (people who never visited a second page) or to improve goals such as the number of people who filled out an important form to name two examples.

See my previous post on all the different goals you can setup on your website

When a visitor performs an action they have then converted from a typical visitor into a valuable visitor, such as a new customer, an engaged user or a new potential lead.  These conversions are what we aim to improve on by split testing the website, they are the main measure of success.


What is a ‘split test’ exactly?

The simplest type of split testing is called A/B split testing which involves a test between two webpages online.  At random, 50% of visitors will be shown Webpage A and the other 50% of visitors will be shown Webpage B:

17-03-2014 11-01-29

We then measure the number of conversions created from Webpage A and Webpage B and compare the results.  Stopping the split test afterwards the winning page can then be used 100% of the time to increase the number of conversions on average per 100 visitors:

17-03-2014 11-33-54

In the example above Page B converts over 5% more people than Page A (an increase of 27.70% comparatively).  This makes Page B the better performing page overall and the one to keep for further testing.

More than two pages can be split tested to try out several variations at once, this is called an A/B/N split test:

17-03-2014 11-01-29

A/B/N split tests have traffic divided between them evenly so an A/B/N split test with 4 pages would each get a quarter of the traffic.


Example Google Content Experiment split test

One of our wonderful clients The Storage Bed Company has recently started a bold split test on their homepage.  The aim of the split test is to retain more visitors on the website as around 37% of visitors are landing on the homepage and then are not visiting a second page (a 37% bounce rate).

The top carousel banner on the website has 6 high quality images showcasing the products but there is only time for visitors who are quickly visiting the website (the ones that might bounce) to only see the first image shown.

We have decided to see which of these 6 images has the most positive impact on visitors by creating 6 versions of the homepage each starting at a different image within the banner carousel:

13-03-2014 10-59-16

The 6 versions of the homepage are clearly labelled with unique URLs:

  1. (original)
  2. (variation 1)
  3. (variation 2)
  4. (variation 3)
  5. (variation 4)
  6. (variation 5)

If you would like to perform a split test and have your variation pages setup then firstly visit the content experiment section in Google Analytics: Behaviour > Experiments

Create a new experiment and start by giving it a clear name, an objective (the type of conversion you wish to track) and a confidence threshold which I recommend a value of 99% to give only a 1% chance of statistical error.  Note that the objective can be a pre-set Analytics Goal within Analytics or a metric such as the number of bounces that occur:

13-03-2014 10-53-51

Next add in all the split test page URLs into the experiment remembering to use the original page as a control for the experiment:

13-03-2014 10-55-45

Once you see all the pages appear properly Google will provide you with a unique code which is unique to each individual experiment.  This code needs to placed immediately after the <head> tag within the HTML code making it the first piece of code a browser will come across.  This code randomly redirects people to the different variation page URLs or lets them stay on the original page and needs to be ONLY placed on the original split test page (in this example experiment it’s just the homepage):

13-03-2014 10-59-57

Luckily Google checks all the pages involved to see if the split testing code is found correctly on the original page and the Google Analytics tracking code is present on every page to track the visitors.  A common mistake is to place the split testing code on all the pages (not just on the original page) or not having the Google Analytics tracking code on the variant split test pages:

13-03-2014 11-05-20

Once setup and confirmed by Google you should be able to see different split test pages by visiting the original page URL (you can test it out on different browsers or computers to randomly get different results if you wish):

13-03-2014 11-06-27

Notice that Google has added some parameters after the URL, this is perfectly normal to track to page view.  Split testing will not have an impact on your SEO but you may wish to 301 redirect the unused split test page URLs back to the original URL once the experiment is over, just in case people start linking to the split test page URL from external websites.  Visiting the split test pages whilst the experiment is live can obviously skew the results so please bare that in mind if conversions are scarce.

Now all you have to do is to sit back and let Google handle the experiment.  Google will run the experiment in a way to get results the quickest and do all the complex maths behind the scenes for you.  After a minimum of two weeks Google may pick a clear winner.  If there is no clear winner then you may have to wait longer for the results to become clear.  If the variations are very similar performing it might be a good time to reconsider what is being tested and the variables you are testing against (see below for tips):

13-03-2014 11-32-11

My experiment here hasn’t quite finished yet but as you can see there is over a 95% certainty that Variation 4 will out perform the original with a whopping reduction of  31% of the bounce rate.  I need to wait until there is over a 99% certainty that Variation 4 out performs the original before ending the experiment.  Things are looking very good so far getting over 30% more people to visit two or more pages on the website than before with Variation 4 (shown below):

Winning split testing page design

Winning split testing page design


Why bother with split testing?

The biggest ecommerce websites in the world all split test, it’s a method of extracting more money per visitor on average.  There’s many examples of websites reporting that a split test of a button graphic, main heading or a different layout resulted in a significant percentage increase in sales.  Even a 1% or 2% increase in sales would bring in significant extra revenue for a website in the long term and it may only take a couple of weeks of testing to run the test.

Google is one of the most famous split testers with the design of the search results taking different shapes over the years and recent controversy over the new way ads are shown and titles not being underlined.  Amazon and Ebay are common ecommerce examples of heavy split testers, most of the changes are subtle but the constant refinement of the websites makes them convert more and more over the years.  Some of the best practices in web design can be found by looking at Ebay’s or Amazon’s key areas such as the sign up process or shopping checkouts where hundreds of tests have shaped the design and usability.

There are no right or wrong avenues to explore when split testing and no proven best practices for web design.  I have seen a budget service website change its font from Arial to Comic Sans and surprisingly see more sales result from it, also I have seen simple changes such as underlining links result in a 2-3% increase in overall newsletter sign ups.  The only prove that the split test works is to try it out, there is no reason ever to stop split testing if it keeps improving the bottom line factoring the small costs to get it running.



Best practices for A/B/N split testing

Without delving too far into the mathematics behind split testing, it’s important to test and measure the right elements on a website or you could either spend months waiting for a result or never even get any clear result at all.

Below are the main things to consider when setting up a split test to ensure results are found in the shortest time possible:

  • Test something big – When you start split testing test out something major which will have an impact on your visitors, this could be a main header image, a huge heading or the layout of the web page for example.  Testing something small like two different shades of blue may require many thousands of visitors to see which shade converts better and you may well find no difference at all, after months of testing.
  • Test the most visited pages firstly – For most websites the homepage will be the most visited page on the website (see if you are not sure).  The more visitors you have, the quicker you can determine which variation is performing the best for example flipping a coin 8 times could end up getting 5 heads and 3 tails but flipping it 8,000 times will be close to 4,000 heads and 4,000 tails, a much more accurate test and result.
  • Use a common conversion metric – Pick a goal (see or a metric that has at least one conversion per 100 visitors.  You will need many conversions for each variant to determine a winner, the more you have again the more accurate the results will be.  If you tested the number of visitors who say spent more than twenty minutes on the website, then you may find only one or two conversions a month and will have to wait several months to get enough conversions to make a conclusion.

As mentioned before there is no reason to ever stop split testing.  Even the most perfect web page can be tweaked or improved slightly to make it convert more, with the rise of mobile devices accessing websites it is certainly worth split testing your mobile designs for the 25%+ of mobile only visitors.



Advanced split testing methods

Google-Website-OptimizerGoogle’s retired free Website Optimiser program used to allow multivariate split tests, a powerful method of split testing different elements on a single web page.  For the past couple of years it hasn’t been available but hopefully it’ll make a comeback later in 2014 as it’s a great free tool.

Multivariate split testing allows you to test different combination of elements on a web page at the same time, let’s say you wanted to test three main images and two main headings this would give you six (3 x 2) variations of a web page:

  • Variation 1 = Main Image 1 + Main Heading 1
  • Variation 2 = Main Image 1 + Main Heading 2
  • Variation 3 = Main Image 2 + Main Heading 1
  • Variation 4 = Main Image 2 + Main Heading 2
  • Variation 5 = Main Image 3 + Main Heading 1
  • Variation 6 = Main Image 3 + Main Heading 2

After time the test results can show if the main image has an effect on the conversion rate, if the main heading has an effect on the conversion rate and if a combination of one of the main images and one of the main headings had an even bigger effect on the conversion rate.

Current paid split testing software such as Visual Website Optimizer can be used across an entire website instead of just on a single web page.  Multivariate split testing across every page on a website can have very powerful results and also keeps the design consistent for visitors.

Examples of site-wide split tests would be an “add to cart” button across a whole ecommerce website, the common background design for a website or even a structural change to the whole website such as removing a side bar or changing a top navigation bar.  I have seen examples of a “buy now” button making over 10% more sales when it was moved to a different location and a very dark background website gaining over 4% more sales by simply switching to a white background!



Compounded split testing method

exponential-growthWhen split testing, why stop at just one page or one element on the website when you could split test multiple things at the same time?  There are no technical issues holding you back from performing two or more concurrent split tests so it’s possible to achieve compounded results.

Let’s say you have an ecommerce website; there is a clear path from a visitor entering the website all the way to making a purchase that looks something like this:

  • Step 1 – Entering the website, possibly on the homepage
  • Step 2 – Looking at a product category page
  • Step 3 – Looking at a product page and adding it to the basket/cart
  • Step 4 – Looking at the shopping basket/cart
  • Step 5 – Looking at the checkout page
  • Step 6 – Going through the different stages of the checkout (possible multiple steps)
  • Step 7 – Going through the payment gateway
  • Step 8 – Confirmation of the payment

Steps 1 to 7 can all be split tested to improve the number of people who make a sale on average per 100 visitors.  If all steps from 1 to 7 were split tested at the same time then there will be a cumulative positive effect on the overall conversion rate if one or more tests result in an improvement.  If say the “step 1” split test brought in 20% more visitors to “step 2” then “step 2” would have 20% more possible visitors to convert even before it’s started split testing itself.

Let’s say each step in the ecommerce example has a measly 1% improvement overall, from start to finish that’s around a 7.2% improvement overall from entering the website to confirmation of the payment (1.01 x 1.01 x 1.01 x 1.01 x 1.01 x 1.01 x 1.01 = 1.0721 = 7.21%).  A 7.21% increase in sales could mean an extra sale per week to some businesses or perhaps an extra sale per hour to other businesses, either way it’s a huge gain getting more revenue without paying for any extra web traffic or marketing.

Leave a Reply

Your email address will not be published. Required fields are marked *