Fallacy of the Test Plan – Why test plans for email marketing are nonsense
Published 19th April, 2012 by Sean Duffy
Why? It is only after you start testing that you identify what is and what is not worth testing, so trying to identify the things to test 6 months in advance is useless. In my decade of email marketing I have yet to get past the first month of a test plan before it is ripped up and a more flexible ongoing approach is taken.
Here is an example of what can happen. You start testing subject lines on your test plan, then within a few weeks you think you have some interesting results, but when you put them into action, repeating the success of the winning versions does not happen. The outcome? We need to test subject lines every single time to be sure of finding a winner and loser.
Another example is in testing pre-headers. You realise the email works better without it. Adding a pre-header just adds clutter and distraction from the main call-to-action. So if this is the case what other clutter can we remove? Immediately we might decide to change our test plan to see what else we can take out to reduce clutter.
One test not only gives you some answers, but it tells you what is next to test. You don’t need to go into the effort of writing a test plan.
So why do we do it still? Perhaps because we think it is the right thing to do. Google email marketing best practice and an ‘email guru’ will tell you a testing plan is one of the top 10 things to do, and Google is never wrong! Also, we feel we need to show management that we have a plan.
The best solution is to maintain a wishlist of things that might make a difference to your email programme and are worth testing, but don’t put a single date or order of priority on it. That way we still have something to show we are thinking about it, but we let the outcomes of our testing tell us what we should do next.
Now that MVT rather than just A/B testing is becoming available it makes it quicker to pursue your test ideas – no longer are you restricted to one test a month to keep results reliable, now you can go full throttle and test as much as you can create. As far as MVT is concerned, a testing plan simply adds bureaucracy and a hurdle to getting stuff done.
So next time someone asks you to write a test plan, you now know why you should tell them it is a pointless activity.
