The Complete Direct Mail Split Test Framework

How do you know when your direct mail is truly optimized and firing on all cylinders?

How do you know an offer on a postcard is better than one sent in an envelope? That a neutral offer outperforms a ranged offer? And so on.

How do you really know?

Is it something concrete like response rates or more of a feeling like intuition?

It’s probably a bit of both, no? We all want to see a response rate that meets our expectations and like to think we intuitively know what makes a buyer or seller respond.

In reality though, even a good response rate can be improved and going with your gut is not an ideal business strategy.

Fortunately, there’s a straightforward way to truly know what works and helps improve your direct mail efforts:

Split testing.

By the end of this guide, you’ll understand the entire process and have a little cheat sheet of where to start your own split tests.

What is split testing and how does it work?

Split testing is running an experiment where you test slightly different versions of your mail against each other. The key to success is making sure those differences are slight though, as in changing a single variable, so you can be sure you’re only seeing the effects of that adjustment.

Also known as A/B testing, the idea is to split your audience and send version A to one half and version B to the other half.

This sort of testing is routinely used in digital marketing for websites, apps and the like but it’s actually something you can do with direct mail too.

Whether you’re just getting started with mailing or you’ve been doing it for years, split testing gives you real data to make decisions with.

In other words, it’s the only effective way to improve your results.

Why bother split testing?

Small changes can create outsized responses from your audience.

Take a look:

• Csek Creative, a marketing agency in Kelowna, British Columbia, tested its website’s headline text and increased its clickthrough rate by 8.2%.
• Humana, a health insurance company, tested a variation on its homepage banner and boosted clickthroughs by 433%.

Yes, these examples are from the digital world but the point is that it’s the details that make a difference no matter the industry or type of business. Split testing is what gives you actionable insights into those details and a way to enhance your direct mail efforts.

10 steps to split test like a champ

The concept is pretty simple – test this vs that – and the steps to set up a test are no more difficult.

The only requirements to run a split test are a mailing list, some ideas and time. Rookie investor or seasoned vet, everyone has those.

1. Know your goal

Split testing without a goal is like taking a road trip without a map. Sure, you’re moving but is it in the right direction? With mailers being our core outreach and us being in the land investing space, the only goal will be to boost the response rate.

The starting point for working towards that goal is getting baseline numbers; knowing where your response rate currently is.

No matter if it’s your first mail campaign or your 51st, split testing is the one way to compile the actionable data that allows you to optimize mailers and improve how effective they are.

2. Decide on one variable to test

With a goal locked in – improving response rates – now it’s time to identify what to test.

Picking those variables is a matter of looking at what’s editable on your mailers, which is to say that everything is a variable.

From the design (bright colors vs muted tones) to offer strategy (ranged vs neutral vs blind) to mail type (postcard vs envelope) and beyond. Anything you can change, no matter the size of the detail, has the potential to produce a different response when in the hands of the person receiving the mail.

You can come up with as many variables as you’d like but you can only test one at a time for your results to be meaningful. Otherwise, you’re introducing noise into the data which makes it incredibly difficult to identify what actually produced the better (or worse) numbers.

If you’re not sure where to start, here’s a “what to test” cheat sheet:

• Offer strategy – Neutral vs blind vs ranged
• Format – Postcard vs envelope letter
• Location map – Map vs no map
• Photos – Photo with faces/family vs a photo with no faces/no family
• Colors – Bright vs dark

3. Choose a metric for success and build your hypothesis

Keeping with the response rate goal, our metric for success is the percentage improvement in response rates based on the individual independent variable we change.

“Hypothesis” sounds fancy but it’s really just a way to make sure you have all the elements of the test lined up. The issue to solve, how you intend to solve it and the measurement.

For example, my response rate is low and I think using a postcard mailer with a photo of a family enjoying a picnic will increase that rate because photos of people tend to increase trust and encourage feelings of positivity. I’ll know it’s successful if I see a 1% increase in response in 2 months.

The reason we want to have a specific percentage in mind is that we need to be sure it’s the variable we changed that’s really moving the needle, not random chance. If the percent change is too low, we just can’t be too sure the change is what made the difference.

4. Make the “challenger” mailer

Think of this vs that in split testing as challenger vs control.

The “control” mailer is your existing, original mailer. The one you’d like to improve.

The “challenger” is the mailer that has the single variable difference that you’ll be testing.

In our example, the control mailer would be the postcard with a photo of a piece of raw land, the challenger would be the postcard with a photo of a family having a picnic.

5. Decide the sample size

You don’t have to test this on your entire mailing list but it does have to be a large enough sample size that the results are meaningful.

Sending one mailer to a group of four and another mailer to a different group of four won’t exactly yield results you can trust, ya know?

Given that, we’d recommend a sample size of no less than 1,000 addresses.

6. Split your mailing list evenly

Finally, the “split” in the split testing!

This is where the magic happens so to speak.

Once you’ve determined the size of the audience you’ll run the test on, it’s critical to split them evenly.

Look at it this way, sending one mailer to 75% of people and the other to 25% makes it all but guaranteed that the larger group will have a higher response rate regardless of which mailer you send.

The test is only valid when split 50/50.

7. Determine how long you’ll run the test

Not everyone responds to a mailer the moment they receive it so getting data that’s useful and takes time.

Sending mailers and only allowing a week for responses before drawing conclusions isn’t necessarily going to yield conclusive results or give you much to work off of.

You need to give your audience enough time to receive the mail and respond. PostGrid notes that the average lifespan of direct mail is 17 to 20 days so running a test for 20-30 days is a safe bet.

It’s equally important to keep your test length consistent.

If you assess the gathered results for test group A after 30 days and test group B after 75 days, you’re now introducing the variable of time. Put differently, you won’t be able to tell if the move in response rate is due to the individual variable you changed or the difference in time. You can of course go longer than 20 to 30 days but whatever timeframe you decide on, you must be consistent with it.

8. Send the mailers and measure results

With everything in place, now you’re ready to pull the trigger and send out both versions of your mailer to the respective audiences. At this point, you go into tracking and data collection mode for the duration of the test.

But all tracking isn’t created equal.

Typically, land investors track the property, rather than the mailers, by assigning each piece of land a code.

If you run multiple campaigns that include the same property though, or if someone replies half a year later, how do you know which particular mailer and which specific campaign they’re responding to?

How can you improve the effectiveness of your mailers without that knowledge?

For split testing results to mean anything, it’s the mail that needs to be tracked, not the property.

To that end, Pebble assigns and prints a unique code onto each and every mailer so when a lead gets in touch, you know exactly what sent them to you.

Tracking the mail itself is the key to improving your marketing’s response rate.

9. Analyze the results against your baseline

So, how did the challenger do versus the control?

Did your response rate go up when you changed the photo? Did it surpass the testing metric you set (a 1% increase in response rates in this case)?

A boost of 6% is nice but if it’s not meeting your hypothesis or the bar you set, the change isn’t particularly important. On the other hand, if the mailer you sent with a photo of a family on a picnic gets 16% more responses, that’s a big deal.

What do you do with that though?

10. Optimize and run another test

The split test has conclusively shown that having a postcard that features a photo of a family on a picnic gets a better response than the same exact postcard but with a picture of raw land.

That means, for all future mailers you’ll replace the photo.

Annnnd done…?

Not so fast my friend.

Go back to our cheat sheet above and check out all those variables. You’ve tested one of them. The winning result becomes your new benchmark and you can run another test to further improve your response rate.

Now you can test your CTA or the colors or whether postcards work better than letters in an envelope and so on.

After that, when you choose a new county as your target, you start the process again there. Continuous improvement is possible and split testing creates those incremental gains.

It’s a never-ending process.

Set up split tests with just a few clicks in Pebble

Good news, split testing is already built right into Pebble’s workflow.

Once you’re to the point of sending out mailers, setting up the test itself will only take you a minute or two. Pebble does all the dirty work, automatically splitting the campaign in half and then assigning a unique code to every piece of mail so you can track results.

Here’s how to do it:

1. Click on the campaign you want to test on the Campaign page

2. Head to the drop down menu in the top right corner and click Split Campaign

3. Go Back to the Campaign page where you’ll now see the campaign has evenly been split two groups, one labeled A and the other B

4. Go into both A and B and click Activate Campaign

5. Start tracking responses!

It’s such a quick and painless process that even our tutorial video is less than 2 minutes long.

Start split testing your mail with Pebble today

Optimization is an ongoing process that requires constant tweaking. Staying static is essentially like moving backward because what works today may not work tomorrow and the only way to know for sure is to test.

With split testing you can skip the guesswork and stop going with your gut.

Pebble gives you the ability to finally getting real insights into what people actually respond to – and then adjust accordingly.

And it’s simple as pitting one version of a mailer against another and seeing which performs better.

Start testing, start improving.