The Dirty Word No Email Marketer Wants to Hear: Testing

email testing

How to make Email Testing easy, do it right, and get measurable results

The word “test” often conjures up bad memories … maybe it takes you back to high school biology or the college history classroom when what seemed like endless studying delivered a less than impressive result. Try to put those images out of your mind. There is a better, more grown up test experience that actually delivers the results you need — especially if you are an email marketer.

Some of you may be thinking that email marketing testing isn’t much better than a college history exam, but we are here today to prove you wrong. First the facts.

Here’s how we know that testing is a dirty word in the marketing industry:

  • Only 23% of marketers are using it
  • Only 18% said it was an area they focused on in 2017
  • 51% said they do not proactively test
  • Marketers report they don’t have enough time to configure or plan test
  • They are not sure what to test
  • They don’t know how to read and roll out based on results

If you have five minutes to spend with us, you will learn how good ol’ fashioned testing can become a critical component of your email marketing. We can help you truly understand what is working, and what is not, in your email marketing program and outline the steps you need to take to improve your program — with ease!

Making Tests Easy

Focus on Quick Wins. This time of year it is especially helpful to find out what subject lines, content, day/time, and CTAs work best to help garner the highest engagement.

  1. Start Small: Test the subject line, time of day, day of week, or simple personalization.
  2. Make Testing a Habit: Aim for 1-2 tests per month.

Additional Planning. If you’d like to take your testing to the next level, you can add in additional elements to test.

  1. Creative/template
  2. Send time optimization
  3. Offer/CTA testing
  4. Responsive vs. mobile optimized

Maximizing Your Tests. Working with a strategic partner will enable you to optimize your email marketing test program. Taking a deeper dive into the elements of your email marketing program enable you to test the following:

  1. Frequency
  2. Recommendations
  3. Segmentation

Here’s an interesting comparison of subject line testing that will reveal what a good vs. not so good test structure might look like. Always begin by including a CTA in your subject line.

It’s important to remember to be able to isolate the variable. In the example above you may want to play with using a question instead of a statement or making the content shorter vs. longer. Other elements that can be tested include an offer vs. no offer, emoji vs. no emoji, using hashtags or not. Now that you’ve determine what to test it’s time to talk about how.

How to Test

Just as you learned in school, there is a right and wrong way to take a test. This five-step process is a great start:

  1. Outline your objectives. Is a new program being launched? Is a program being refreshed? Is the program seeing a dip in performance that you are trying to improve?
  2. Isolate the programs that are the variable for testing.
  3. Design your test leveraging analytics. To do that, you will need to determine your test basis. What KPIs will validate the test?
  4. Determine the size of your test sample. You may need sizable populations for valid results.
  5. Develop the appropriate timing. Do you need a quick test or an ongoing test and learn program?

Creating the Test

There are a few different ways to test splits.

  1. 50/50 test: this is an easy half and half comparison
  2. 25/25/50: Version A goes to 25%, version B to another 25% and after a predetermined amount of time, the winner is sent to the remaining 50% of your list.
  3. Holdout Test: Set aside 10% of your list. Send version A to 45%, version B to 45% and then examine the conversion rate. Was A, B or no email the conversion winner?

Getting the Results

Here are some tips we apply when working with our clients to measure results:

  1. Wait 24-48 hours to read the results.
  2. For subject line, day of the week, or time of day testing base your test on open rates. Get a pulse on CTOR and CTC. If strong opens don’t translate into interest, the content may not be paying off in the subject line.
  3. For creative, offer or CTA testing, look at click rates (unique, total, CTO) and/or conversion rates.
  4. To calculate lift: New variable – Old variable/Old variable.
  5. Inconclusive results tell you one thing…Test and Test again!
  6. Document your learnings in a test library. It will be a great reference as you plan out future tests and roll out a winner. Don’t forget to include what was tested, why it was tested, results, plan of action based on results and revenue from the test.

Happy Testing!


Never miss a growth opportunity