A/B Split Testing will GET YA

A conversation about Internet Marketing:

“So, which version do you like?“

“I LOVE them both!  Which one do you think will convert better?”

“Honestly, I have NO idea…”

[Long Pause]

“Let’s test them both – that way, we DO NOT HAVE TO RELY ON OUR OPINION“

That was more or less the Skype conversation that I had with Dan Thies, co-founder of SEO BrainTrust and Link Liberation.

I had just created two versions of a Squeeze Page Video for Dan and Leslie Rohde – and we had NO idea which one would convert the most viewers to Opt-ins.

Dan, in a moment of complete (and obvious) clarity, suggested that while we were “capable marketers” and that what I had created “would perform to expectations” (He has a tendency to get very formal when being logical…), instead of guessing, or going with gut instinct – we should probably spend the 11 minutes required to A/B Split Test the two versions.

We did.  <sarcasm>That’s an 11 minutes of my life that I’ll never get back. </sarcasm>

Here’s the Skype Screen Shot of what happened when I asked what the Opt-in Rate was for the test winner:

When Leslie says “Audible”, that was a little inside joke.  At StomperNet, whenever I made a last minute change to a plan, I would type “Audible” and call a new play.  Everyone on staff had the word “Audible” set to alert them in Skype. Leslie has a memory like a ColecoVision.

Anyway…

Despite the aforementioned 226 words, I STILL have not made my point…  heh.

I want you to notice that there are TWO conversations there.

The first is with Dan – where he’s reporting an opt-in rate of between 60 and 70 percent.

Now, notice that Leslie’s Skype is actually at 7:45 AM the next morning (Good LORD Leslie!).  And he’s specifically talking about AFFILIATE TRAFFIC.

Dan was talking about Opt-in rates from their house list, Leslie was talking about Opt-in rates from Joint Venture traffic.

(NOPE!  Point STILL not made!)

Clearly, there is a difference in Opt-in Rates between house subscribers and Joint Venture Traffic…

OR… did it have something to do with the 18 or so hours difference between the reading of the stats…

(Getting Warmer. )

CUT TO:  PRESENT DAY (With more pictures)

Last week, I sorta lost my mind.

I don’t know if it’s because of all of the Garlic, Flax-Seed, and Fish Oil supplements I’m taking while doing the 4-Hour Body diet… Or, if it’s the constant, concussive impact-trama of my face hitting the ground during my snow boarding lessons.

But I forgot something – something that I know I had tested a few times about email subject lines.

So, I decided to set up a test:

Here’s the deal:

Subject Line Version A: (Download) “Oh Hai, I mind-mapped Ur Biznezz”

Subject Line Version B: “Oh Hai, I mind-mapped Ur Biznezz” (Download)

The ONLY difference between the two subject lines was the position of the “(Download)” text.

I sent each version to 10% of my subscriber list, hit send, and went off to blend a whey protein shake (Or some equivalent drink that left a dust-like aftertaste)

Now, it takes about 9 minutes for iContact to send a message to my list.  It took about 9 minutes for me to consume that delicious (ummm) foamy, sudsy beverage.

At exactly 10 minutes, I grabbed a screen shot of my test results:

DAYUM!

Version B was CRUSHING Version A in not only Open Rate, but also Click Rate!  I’m talking about an almost 2 to 1 ratio.

As my hand hovered over the big green “Send This Message” button on Version B, my Google Calendar literally saved the day.

Quick Tangent: I’ve been skiing for 25 years. My favorite run is 3.2 miles long  - it’s at Squaw Valley in Lake Tahoe. The problem is, by the time I get about 2 miles into the run, my feet are visiting the 9th circle of hell.  I’ve even had custom ski boots made to try to deal with the torture – to no avail.

While agonizing over my feet, I’ve observed that the boots that snow boarders wear looked MGHTY comfy — it was time to give snow boarding a try.

As it turns out, there is Snow Boarding School in Encinitas that uses a “Revolving Carpet” to teach people how to ski and snow board.  The carpet is 20 feet wide by about 35 feet long and it’s at a 15 degree down-hill angle.  And the whole thing MOVES – kinda like a belt-sander, but at about 5 miles per hour. I immediately booked 10 lessons with them.

Anyway, The Adventure Ski and Snowboard School is about 20 minutes away, and my Google Calendar rudely took over my browser (Consequently hiding the iContact page with the Green “Send This Message” Button) to let me know that my lesson was in 25 minutes.

POOF – I bolted out the door to take my lesson.

CUT TO: LATER THAT DAY (After 1 hour of lessons)

I spasmodically climbed the stairs to the 2nd floor of my house, and collapsed on the floor.  Ever Skied? Ever Snowboarded?  When you’re on the mountain, it works something like this:

Stand in Chair Lift Line for 5-10 minutes

Ride Chair Lift for 5-10 minutes

Ski or Ride down slope 3-5 minutes

Stand in Chair Lift line for 5-10 minutes

Ride Chair Lift for 5-10 minutes

Eat, Drink, Look at Landscape at top of run for…however long.

Ski or Ride down slope for 3-5 minutes.

Repeat.

So, for every 20 minutes, you’re actually skiing for about 3.

Now, on the Revolving carpet, you’re skiing or snowboarding the ENTIRE 60 minutes.

It…is…exhausting.

By the time I got home, I had completely spaced on the test, LOL!  I’m just going to file this moment under “Happy Accident”, because when I woke up the next morning, here’s what the stats read:

And VERSION A comes from behind for the WIN!!!

Couple important points here (I know…FINALLY he gets to the point…)

With less contacts (868 vs 874), Version A had 32 MORE Opens than Version B.

Version A also sent 13 more clicks than Version B

So, less people got it, yet more clicks were generated by Version A.

IMPORTANT:  This is NOT a Slam Dunk (Thank You George Tenant for immortalizing that phrase) – but I want you to consider the following:

This was to 10% of the list.  That means multiply the results TIMES 10 once I sent to the entire list.

Now, we’re talking about at least 130 more clicks to a destination.

That reminds me of the last webinar I did with Frank Kern where an offer we made closed 11% of the Webinar Attendees.

At a 11% close rate, those 130 clicks might have resulted in 13 more sales.  Lets say that the commission for each sale was $100…  Or, what if it was $1,000…?

You’re starting to see my point?

Second IMPORTANT POINT:

Here’s the deal – Your email list contains 2 types of people:

• Hyper Responders that take immediate action (A fairly small percentage of any list).
• Everyone else.

Tests take TIME to be conclusive! In fact, the longer this test ran, the MORE conclusive it became.

When Dan and Leslie sent me the two sets of opt-in results for the Squeeze Video, that was my first clue.  When I was Google Calendar-ed into patience on this test, that was my second.

No matter WHAT you test, you need to be prepared to allow THREE things to happen before you can render a judgement:

1. TIME

Especially with Blog Posts, Email Blasts, Squeeze Pages and Sales Videos/Letters – you have to allow for your entire list to respond to the Call to Action and NOT base your judgement on Immediate Test Results.

2. Statistical Significance

To judge the effectiveness, your Promotions or Calls to Action require different measurement perspectives.  There are times where a 1% difference between two test versions can create a SIGNIFICANT Impact (As in the Webinar Example).

There are also times where the difference between a profitable Ad and a non-profitable campaign will require far more than a 1% difference – it may take a 5% difference.  The point is, you need a decent Data Set (Which translates into a significant QUANTITY of test results – i.e. it might take hundreds or even thousands of visitors to render an accurate judgement).

3. Test SMALL Changes

In my Email Subject line test, the differences in Versions were TINY.  Thus, the differences in RESULTS were TINY.  And that’s NOT a bad thing!  The way I look at it is:

The changes were so small that even the LOOSING test would still have yielded good results – so I’m NOT placing my campaigns in DANGER by testing vastly different creatives.

Yes, it takes patience.  Yes, it takes time.  Yes — if something that you’re doing is already working, then don’t mess with it TOO MUCH.  You should “ALWAYS TEST WHEN YOU MESS” with stuff, but keep the tests slanted towards the creative that’s already working so you’re not losing total effectiveness.

Even if you want to TEST BIG, then do it to SMALL portions of your traffic.  For example, send 90% of your traffic to the proven creative, and sacrifice 10% to the experimental creative.

4. (BONUS TIP)  “Be Bold, and Mighty Forces Will Come To Your Aid”

I LOVE that quote.  And the fact is, every now and then, you’ve got to pull down your pants.

Uh… What I mean is, let it all hang out.

Uh…  Every now and then, you’ve just got to GO FOR IT. When you get a funky idea – DO IT!  Just protect it with a test.

I’ll close with a quote that my mother taught me when I was Five.  Meaning, if I’ve ever offended you — blame her:

“For fear of taking risks in life I’ve missed a lot of fun.  Now the only things that I regret are things I HAVE NOT DONE”

Testing.  It lets you do EVERYTHING (Pull Down Your Pants) without the risk of losing ANYTHING.

So Go Ahead!  Drop Your Drawers!

Andy

P.S.  Help me with my next test!

I’m putting Facebook Viral Marketing to the test.  Do me a favor?

If you liked this post, hit the LIKE button for Facebook.

Better yet, if you smiled one time while reading it – leave a Facebook Comment and tell me so.  I’ll tell you the results of using Facebook for Blog Comments after I’ve got some solid data.

Tagged as: A/B Split Testing, Analytics, Andy Jenkins, Dan Thies, Leslie Rohde, opt-in, split testing, testing