Email Testing..Testing..Testing..

At a time that Retailers are falling around us I found myself surprised to see that the (apparently) very successful Canadian Tire Company has announced that due to its website not converting browsers (or ‘researchers’ as Canadian Tire called them) the online channel will close.

I am sure that they have looked very long and hard at their business model, including looking which channels bring in the most revenue, but I am absolutely astonished that the channel they have chosen to close is the one that most other businesses find the cheapest to manage for both acquisition & retention.

In addition to this announcement, I learn in the same week that Lycos is shutting down its web-hosting & email service, as they couldn’t find a buyer.

When you hear stories like this you realise how important it is to use the tools at your disposal to their best (most effective & efficient) advantage (whilst of course thanking your luck stars that you are still in paid work).

It is so important to keep re-addressing what you are doing:

Is it still working?

Should I be doing something different?

Am I getting the best value for my money? Etc, etc.

The answer of course (to almost everything) is to test, then test some more, then continue testing…Take marketing emails for example; when sending an email, regardless of whether it is a ‘batch’ email or a targeted email – of course we all know that targeted is the only way forward; you not only have to test varying content but also such things as the subject & from line.

Your email database can be segmented into various profiles such as frequency & recency of purchase, value of previous purchases, demographic & geographic data, clickstream data (from emails already received & from website behaviour) etc. Based on this segmentation you can use varying content according to preferences & profiles. You can then test these varying content on a ‘small’ group per segment (the size of this ‘small’ group is dependent on the size of your database) before sending the email containing the most frequently clicked (& converted) content to the remainder.

Prior to send however it is important to test the subject line (if you do this as part of your initial test you will have impacted the results, which will more or less render your initial test useless), each test needs to be done in isolation from all other tests. You can test any number of subject lines of varying lengths – do not be tempted to believe the articles that state that ‘the shorter the subject line the better’, as other articles state the opposite. The right subject line is the best one at that time for the recipients (which must be relevant to the content to remain a trusted company), which means that you need to test subject lines for every email send.

You must also decide what personalisation you wish to use – should you use the customers’ name in the subject line? Should you use it in the copy? Should you use First Name or should you use Title – Surname? You will only know the answer to this if you test. This test however will not need to be repeated per email send, but it will need to be repeated once per email type e.g. Newsletter / Confirmation of subscription / Thank you for your purchase / etc. You cannot however personalise a subscription confirmation this week & not next, but you may find that customers prefer a Thank you for your purchase email personalised & not the newsletter.

You can also test length of email, especially for emails containing mainly products. Is the right length 5 products, 15 products or 40 products? This test does not need to be done per send, but will need to be re-tested periodically.

Other possible tests include frequency of send, making the ‘from’ line personal (the line that details who the email is from in your inbox e.g. ‘from Sally at’), the volume of images versus text – text is readable in all browsers – images however must be manually downloaded in most browsers & promotional message formats (should you state ‘25% off’ or ‘£60 off’ or even ‘save £60’) etc.

Testing is not done in isolation however; you do not simply test then send. You need to keep testing to ensure that the first result was not a one-off. The sequence is: TEST – ANALYSE – REFINE – ANALYSE – CONTROL. Only then can you be confident that you are maximising the efficiency & effectiveness of your email programme.


2 responses to “Email Testing..Testing..Testing..

  1. Nice timely post Su. Can you tell me how much time (effort) you put into each test? & how many times do you test the same thing (to prove that the first one was not just a fluke)?

  2. Faye, when running a subject line test I probably spend approx 10 mins setting up the workflow in the email software – most of the segmentation (recipient data) can simply be copied from the last campaign workflow, so all I need to do is load 4 separate copies (each of the same email creative) & assign each with a different subject line (I don’t track the links separately from one email to the next, I just track the email opens separately).

    I then copy these campaigns to a later part of the workflow, so that 8 hours after each subject line has been sent out the workflow calculates the ‘winner’ & sends the remainder of the database the winning subject line.

    I don’t have do do any of the manual checking to see which subject line won, the software does it for me, & at the end of the week I simply gather these results as part of my Monday morning reporting.

    For creative (or other (non-subject line) testing) I’m afraid that the time spent will really depend on the test. As an example however, if I were to A/B test 2 different creatives, I would probably find that I spend no more that 30 mins in total in briefing the creatives to the Designer, loading the 2 creatives when designed, labelling links & setting up tracking on the links. I would then spend a further 10 mins segmenting my database in order to run the test. Again I can then simply watch & wait, & am able pull the results when carrying out my normal Monday morning reporting. I must stress that you need to know exactly what hypothesis you are looking to test – in being very clear about the hypothesis you will be able to determine the SINGLE measure that you will perform in order to check the performance of the test (knowing this before you begin the test will save you from feeling that you have to measure everything about the email when it comes to reporting your findings.

    Finally, I always test each thing twice (except subject lines – which i test on every send), if both results show the same results & accept that it was not a fluke, if however the 2 tests show different results I will test a 3rd time to see which was the result I should consider in future campaigns.

    let me know if you have any further questions Faye, I am happy to talk about email any time.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s