Tel: (020) 7617-7848
or Multivariate vs. A/B Testing
In this post about Multivariate vs. A/B testing, you will find out when to use A/B and multivariate testing & the pros and cons of each method.
1. A/B Split Testing
Overview:
A/B testing compares the performance of two or more alternatives of a single page element, page or funnel.
When to use:
To test radically different ideas, e.g. layouts, different conversion paths (checkout, sign up & lead generation process) etc.
NOTE: when doing a test on a site you always start with A/B testing first, then after you find the winning page, you move on to multivariate testing to tune up the page (buttons, messages, hero shot).
2. Multivariate Testing MVT
Overview:
With multivariate testing MVT you test different content simultaneously determining the most effective combination. Democratic as your site visitors “vote” through their actions.
When to use:
For more sophisticated comparisons of multiple elements and to develop broader/deeper insight.
NOTE: Google Website Optimizer is using full factorial method, i.e. tests all combinations.
There are testing tools which provide fractional factorial method, however I recommend to make yourself aware of potential downsides of using fractional factorial method by reading this white paper by Tim Ash, an author of the book “Always be Testing”.
Watch the full webinar
Will open in new window. Registration prior viewing might be required.
About the author
- More at
Questions or comments?
For queries regarding conversion optimization of your site, or for more information on this article please contact Jan Petrovic, founder of proimpact7.com and master certified in conversion optimization and web analytics.
jan@proimpact7.com
@Jan
It always surprises me when I hear from WhichTestWon readers who not only don’t know the difference between A/B testing and multivariate testing, but also don’t know the difference between a controlled experiment (A/B Test) and before and after testing.
Just last week, at least two of our Top 7 Lead Gen Form A/B Tests webinar attendees asked us to explain the difference between an A/B test and a Before and After test. Just goes to show, more marketers need the kind of info you’ve provided here.
It’s a great way to see the very clear difference between two types of testing … love the full factorial vs. fractional factorial explanation as well.
Natalie Myers
Senior Reporter
WhichTestWon.com
Natalie, I know exactly what you mean. We had a client who approached us. He said that he did some A/B testing. When I asked him how he did it and what was the result he said: “1 week we run version A and then the next week version B”, which as we both know isn’t the way to do, because your test results are basically invalid. So there is still a lot of confusion out there about how to run A/B tests. Jan
Thanx for this article explaining the different MVT tests. we at http://www.reedge.com use full fractional but are indeed thinking of adding others because not everyone has lots of traffic. Giving the clients an option is good.