Proimpact7.com E-commerce Blog » A/B Testing and Optimization

Consequences of Ending Your Test Too Soon

Jan Petrovic — Thu, 26 Jan 2012 16:07:33 +0000

In this post we will talk about the consequences of ending your A/B test too soon. The idea came to me when I saw some A/B and MTV tests case studies, which after running only a few days were declared as winning by the testing tools. Then I got ‘lucky’ and came across this very good example, as one of our tests we were running turned out to be a winning one on the following day. Mind you, this was a site with a high volume of traffic.

Let’s have a look at this actual example below. The following day after we launched the test, our testing tool declared a winner. According to our testing tool, we improved our conversion by a respectable 87.25% with 100% confidence level. Great! Well, not really. What’s the issue?

Technically, if you input the data (conversions & visits) into any statistical tool, it would show that this test was statistically valid. So seems like no issue here. However the issue is that the test didn’t run for long enough.

If we stopped the test then and pat each other on the shoulder about how great we were, then we would probably make a very big mistake. The reason for that is simple: we didn’t test our variation on Friday or Monday traffic, or on weekend traffic. But, because we didn’t stop the test (because we knew it was too early), our actual result looked very different.

The actual test result after 4 weeks of running was 10.49% improvement with 99% confidence level. The actual results differ from the initial ‘winning’ result by -731.74%. How is this possible? The reason is, every day you receive different traffic to your website and each day’s traffic behaves differently too.

Now, back to the consequences if we stopped this test then. Let’s say you were running this test in checkout, and on the following day you say to your boss something like “hey boss, we just increased our site revenue by 87.25%”. If I was your boss, you would make me extremely happy and probably would increase your salary too. So we start celebrating, but at the end of the month, instead of having 87% more money in our bank account, we see the same money we had last month.

To avoid this type of blunder, always be patient and run your tests for a minimum of 2 weeks with recommended maximum of 6 weeks and confidence level no less than 95%. Also, once your testing tool declares a wining variation, don’t stop your test immediately. Run it for another week to see if the result is solid. A solid winning variation should, during this ‘control’ week, hold its winning status. If it doesn’t, then you haven’t found your winning version.

If you test like this, you will keep bringing sustainable, solid improvements to the site and results you can rely on.

How a Simple Change Can Improve Your Conversion Rate

Jan Petrovic — Wed, 29 Sep 2010 13:44:18 +0000

When launching a campaign, relevancy and the continuity of an offer are very important factors in order to increase conversion rate. To demonstrate it, lets have a look at this email campaign from chefscatalog.com.

Below is the email I received, telling me that I could win something if I enter. It shows me products I could win, plus the message is very compelling, as is the call to action. Even the subject line was well crafted – “Win a top rated product from CHEFS catalog!”

1. subject line: Win a top rated product from CHEFS catalog!

2. email with the offer

3. and this is actual landing page

So what’s the problem here. Subject line and the email talks about winning the top rated product. There is not a single word about writing a review in order to win! And suddenly the landing page is asking me to write a review in order to win.

Clearly the message between the offer (email) and landing page is broken. What does it mean? In this case high email click-through rates, but very likely low conversion rate (in this case number of reviews). Plus, this email campaign could create a mistrust of customers to all future offers from ChefsCatalog.

How would I approach this email campaign to make this offer relevant?

Below I’ve drafted the email and subject line relevant to the landing page.

1. My subject line – Write a review and win a top rated product from CHEFS catalog!

2. My email – I added “Write a Review and Win“

3. ChefsCatalog actual landing page

What I have done is simply telling people, that in order to win, they need to submit a review. Clearly, this email should be segmented and sent only to visitors who purchased a product from their site before, rather than send it to every person on their mailing list (as it happened in my case).

By making a little change I have created clear and transparent offer. It’s possible, that this approach will have lower click-through than the original email, but due to its transparency and consistency very likely higher conversion rate (more reviews), which I believe was a primary purpose of this campaign. Obviously, to get the best results you would like to conduct email and landing page optimization (A/B test) with different calls to action, messaging etc. to find your winning combination.

How Using the Home Page as a Channel Led to a 59% Increase in Conversion – Webinar Recoding

Jan Petrovic — Mon, 13 Sep 2010 08:00:47 +0000

In this webinar “How using the home page as a channel led to a 59% increase in conversion” by MarketingExperiments, you will learn how to approach home page testing, plus a real home page landing page optimization example, which led to a 59% increase in conversion.

Process for optimizing home pages:

Step1: Identify all home page objective

Step2: Prioritize the objectives into three categories: primary objective, major objective, minor objectives

Step 3: Connect key success metrics to each objective category

Step 4: Design the home page to weight objectives strategically according to priority

Step 5: Test the effect of competing objectives on your key performance indicators KPI

Here are the screenshots from the home page landing page optimization which led to a 59% increase in conversion.

Watch the webinar recording:

Presentation will open in new window

Which Test Won: Can You Tell Which Product Tour Page Increased Free Trial Downloads by 85%?

Jan Petrovic — Mon, 30 Aug 2010 08:00:15 +0000

Can you tell which product tour page increased free trial downloads by 85%?

To vote and see the result go to whichtestwon.com website.

Would you like to increase you conversion rate? Try our landing page optimization service with money back guarantee.

Which Test Won: Can You Tell Which Landing Page Convinced 17% More Visitors to Fill Out a Lead Generation Form

Jan Petrovic — Thu, 19 Aug 2010 09:00:25 +0000

Can you tell which landing page convinced 17% more visitors to fill out a lead generation form?

To vote and see the result go to whichtestwon.com website.