Long versus Short Copy Split Testing Results

Presume.  Assume. Presuppose. Surmise, Guess…

…Arrogance.

The results of this A/B Split test SUGGEST a lot of things.  But they PROVE something even more profound:

Conventional Wisdom is for Suckers.

Before I drew, or in this post draw any conclusions, let me give you the background, show the stats, and then together we can walk down the deliciously bewildering path of stupefication.

The GOAL:

From email messages, find a way to INCREASE Click Throughs to Blog Posts, Offers, Squeeze Pages, etc.

The TEST:

An A/B Split test to 10% of my email list, where The CONTROL Email used LONG Copy with a Call to Action and the B-Test used SHORT Copy with a Call to Action.

All other variables were identical (Email Template, Font Size, Treatment of Hyperlinked Call to Action) except ONE very minor detail in the subject line.  More on that in a moment…

The OFFER:

The Call to Action was to visit a blog post that offered Tips on Maximizing Landing Page Conversion (You can view that post here).

The TEST CANDIDATES:

Below is a Screen Shot of the CONTROL.  The reason that I assigned this version the CONTROL designation is because, as a rule, my email copy is usually on the longer side. Besides potentially boring my subscribers to death with my long, drawn out musings on whatever topic I’m talking about, the fact is (After MUCH reflection), I’ve drawn some conclusions about my ever-present Long-Copy “Style” that definitely impacts the results of this test (More on that below).

Here’s the Long Copy:

Long Copy - Notice the Bullet Points

Notice the “You’ll Discover” sub-head.  After it are 7 bullet points designed to give the reader a preview of the content of the post.

Now, here’s the SHORT COPY:

Short Copy - NO BULLET POINTS

The ONLY difference is NO BULLET POINTS.

The test was sent to 10% of my “general” subscriber list – meaning, pretty much people with all types of interests.  So, there is a mix of demographics in terms of recipients.

The TEST RESULTS:

BEFORE I show you screen caps and bust out the calculator, please heed the following:

Do NOT draw any conclusions on the Analytics alone.  I know that it’s a labor and chore to read my soap-box quality summaries and thoughts, but in this case, please DO READ what follows.  My conclusions might not be 100% accurate, but I swear to you that they will impact how you think about this little jaw-dropper.

Having equivocated responsibly, lets show you what happened…

Split Test Results as Tracked

First, let me show you a blow up of the most important stats:

Magnified Version of Important Stats

The FIRST set of numbers, going vertically, represent  The SHORT COPY.

The SECOND set of numbers represent the CONTROL – The LONG COPY.

(I know that’s a little backwards, but that’s just the way that iContact displays the data.)

STAT Results:

The LONG COPY (Second Set of Numbers) got 1,295 OPENS and 369 CLICKS.  That’s a Click Through Rate of  28.4%

The SHORT COPY (First Set of Numbers) got 1,235 OPENS and 445 CLICKS.  That’s a Click Through Rate of 36.0%

That’s a 21.8% INCREASE in Click Through Rate.

(Cue Playback of Led Zeppelin’s “Immigrant Song”)

Let’s get some perspective (i.e “connect the dots”) – here’s what that could (and did) mean to this email effectiveness:

To 10% of my subscribers, the SHORT COPY got 76 MORE clicks.  Multiply that times 10 to get the results of sending it to my full list and you get…

…760 MORE Clicks.

Let’s blue-sky for a second:

Assume a 1% close rate as being typical to a Money Offer, and that could result in 7 MORE sales.  Go ahead and insert any number you want for a profit calculation – lets be conservative and say $50.  $50 times 7 = $350.00 MORE profit…

…for writing LESS COPY.

But wait… There’s MORE… (And then, less.)

First, I gleaned some potentially inconsequential information for the SUBJECT Line of this test.

Let me explain:  iContact’s A/B Split Test features are relatively new, and in MY Humble (Heh) Opinion, have a little bit of usability development left before they’re 100% awesome.

For example, you can’t RE-USE emails from a previous test.

Also (And this is where I discovered the Subject Line weirdness), it’s VERY hard to differentiate tests when the subject lines are the SAME. Meaning, if you want to test BODY COPY or the content of an email without testing the subject line, it’s almost impossible to tell which version is which.

So, this is what I did – I added some punctuation to the end of the SHORT COPY version so I could tell the difference.

The POWER(?) of Punctuation

Notice that Test A – The SHORT COPY has a period at the end of the subject line.  I put a period to help me remember that the SHORT COPY was “to the POINT, PERIOD“.  I know, I have to do weird little memory exercises to keep myself organized.  But the point (har) is actually this:

…The Subject line with the PERIOD at the end of it earned 1235 OPENS from 5,508 contacts delivered.  That’s an Open Rate of 22.4%

Now, take a look at this:

Opens from LONG COPY

The Subject line WITHOUT the PERIOD (The LONG COPY) earned 1296 OPENS from 5,470 contacts delivered.  That’s an Open Rate of  23.7%

That’s about a 5.5% increase in Open Rate.

NO – it’s NOT huge.  But, it’s interesting.  I’ll discuss why below, because I think it indicative of an overall concept call “Transderivational Search”. MmmmHmmm.  So SEXY!

Test Thoughts and “Conclusions” (I.e. How the HELL did THIS happen?)

In this test, the Short Copy WON for Click Throughs (CTR), hands-DOWN.  Personally, I think any result that improves your effectiveness more than 10% is a winner.  This test improved results by OVER 20%.

But now, the real reason is WHY?

I mentioned that “Conventional Wisdom is for Suckers”, and I think you’ll agree that MOST people think that Long Copy, especially the kind of copy where you list out the BENEFITS of what a reader is about to recieve, is generally thought to be MORE effective in getting a “Conversion”.

But – is it more effective in getting a “CLICK”? Let’s think about that:

1. The AVERAGE Google Search Engine Result is less than 30 WORDS Long.

2. Google Ad Words Results are about the SAME length.

3. The Average Banner Ad is less than 10 WORDS Long.

4. Amazon Product LIST Results are an average of 32 WORDS Long.

And the ONLY thing you can do with any of the above is “CLICK” to get more information.

Do they know something that I don’t know?  Almost assuredly YES! But still, I have to ask myself (As I look disappointedly into the mirror) – What ELSE could be causing these results to defy convention wisdom?

Here are some guesses (heh):

1. My copy sucks Rocks?

Don’t cry for me friend – there are absolutely days where the words that flow out of my finger-tips onto the LCD screen leave me smiling like a drunk lottery winner.

But in this case, the Bullets that I used in the LONG COPY were not exactly “Kern or Carlton-esque”.  No, they were not hideous, but perhaps they could have been a smidgen better.

So, maybe the reader just decided that there was NOTHING appealing in the Longer Copy, and decided to surf back over to TMZ.com.

2. My Subscribers LOATHE my long-winded-ness (Getting Warmer)?

You know those little remote controls that come with an Apple Product?  They’re about the size of a thin pack of gum, and they have, like FOUR buttons.

Apple Remote Control

Yeah, well – on any given day, I could do a full-length webinar about one.

It’s a curse.  My mouth “Runneth ON and OVER”

So, maybe a loyal subscriber is actually CRINGING at the thought of reading yet another one of my LONG winded, over-wrought emails.  But they can’t help themselves, because if they just open it up, they might get to see one of the many possible train-wreaks that illustrates my communication style – and people love a good self-implosion.

Heh. Too much?  Well, more to the point, people are more starved for time than they have EVER been, so maybe the extra 18 seconds it took them to read the bullets made the difference between clicking through vs. marking as SPAM?

Yeah, there’s a lot of reasons – but ONE in particular makes the most sense out of them all. And it might also explain the 5.5% increase in the subject line results.

Introducing “Transderivational Search”

Transderivational Search is essentially about OPENING a LOOP of Uncertainty. And if you do it right, the loop can be SO strong that the person subjected to a Transderivational Search feels STRONGLY compelled to CLOSE the loop and end the Uncertainty.

So, they look for a way to end the Uncertainty – and in the case of this test, they OPENED and/or CLICKED.

With the SHORT COPY example, there were NO indications about WHAT actually maximizes a Landing Page.  Just reading that copy, the reader was left with a very uncertain and possibly curious feeling about how to maximize a landing page.

Maybe their thinking:

• There are actually ways to increase the conversion of a landing page?
• This blog post will actually give me some TIPS that will help?

…and most importantly:

• What if I’m doing something WRONG on my landing pages?

Versus the LONG Copy.  The subscriber reads that and maybe they see a list of bullet points that they recognize…

Maybe the ones that decided NOT to click were thinking:

• Yeah, I know about Headlines…
• Yeah, I know about Bullets…
• Yeah, I know about Calls to Action…
• Okay – nothing here that I haven’t heard of… Back to TMZ.com

The LONG Copy creates LESS Uncertainty.  And Less Uncertainty means LESS Urgency.

Less Urgency means… Less Clicks.

Now, what about the Subject line?

The period means END OF STATEMENT.  That’s it – and in a subject line, it’s represents the END of the thought.

NO punctuation IS AN OPEN LOOP – when we’re taught to write, we’re taught that the end of a thought gets punctuation – and the presence of punctuation means END OF IDEA.

Maybe, just maybe the absence of punctuation in the subject line vibrated some latent 4th grade psychology in our learned experience that COMPELS us to COMPLETE the thought by opening the email and seeing what’s next.

Or maybe, it’s just one of those statistical aberrations that has no place in high-brow conversation.  And that’s a good thing, because I am just fine being the repository of all things ridiculous.

Final Thoughts (And some more important cautions)

I want you to imagine a turbulent ocean.

Multitudes of Angry Waves, with piercing cold wind blasting their tops into frothy white crests.  Chaos.

Then, all of the sudden, you spot a place of calm. Mirror-like flat seas of solitude and comfort.

With those two images in your brain, consider this:

Constant Communication, like the kind that I have with my email subscribers, creates turbulence.  I (arrogantly) think that every email I send to my subscribers is important enough that I unleash all of the little tricks and tools that I’ve learned over the past 11 years to persuade them to OPEN and CLICK.

Now, all of the sudden, I send them a communication where the subject line is STRAIGHT UP.  No guessing as to what’s inside that email  - it’s to the point “Landing Page Tips to Maximize Conversions”.

“Wow!”, they’re thinking – “Pretty straight-forward…”  And there is their first glimpse of calm seas.

Then, they read one of the shortest emails I’ve ever sent.

Again, in the absence of my typical “persuading” copy – they see CLARITY.  ”Here’s what I’ve got  (“Tips”) – Here’s what I want you to do (Click)”

Calm.  Smooth.  Flat Seas.  Set a course for OVER THERE.

Here’s my point:

The SHORT COPY worked so well because it was VERY different than from what I usually send my subscribers.  So different and economical that it’s sheer existence was a breath of “calm and safe”.

Which leads me to a final warning.

If my subscribers were USED to getting SHORT COPY all the time, there’s a greater than ZERO percent chance that the short copy would become the turbulent oceans, and the long copy would represent the smooth and safe seas.

What’s also at play here is an emergent property called “Pattern Interrupt”.

But that’s a topic for another post…

See what I did right there?

Thanks for reading this.  It was a lot of fun to write, and I hope you enjoyed the ride.

Do me a favor? Tell me what you thought in the comments (Facebook Comments Prefereed), and if you want to share it, Tweet, Like, Stumble, Digg It, etc… I’ll keep publishing more.

“I’m looking for the unexpected. I’m looking for things I’ve never seen before.”

~ Robert Mapplethorpe

Thanks for your time today!

Ole Cuzin Andy

P.S.  In reading some of the comments, I want to shout-out to and thank the people that say that they open just because it’s me that’s sending the email.  Seriously, that’s awfully nice to hear.

Having said that – remember the point of the test – GET CLICK THROUGHS.  And while I am just humbled that people click without reading when they get a email from me – lots of folks carefully consider something called “Satisfice”, which is a calculation of value vs. time spent that everyone does before they click a link .  Like I said, there is a LOT that you can read into any test – and YES, no single test is enough to draw a rule from.  I’m just trying to demonstrate some of the potential influencing factors that contribute to user behavior.

And wow, that last paragraph sounded like I had Thesaurus.com open when I wrote it.  Sorry.  :)