I have yet to meet a veteran marketer (or to read an article by one) who does not espouse testing for addressable communications as a blanket concept. I have yet to hear a marketer who’s employed testing for email, on-site messages or catalogs tell me that he or she didn’t learn anything. Yet, even among some of the most sophisticated clients, testing remains terra incognita.
There seem to be two main reasons for not testing. Thankfully, simple reasoning can undo either of them.
Reason one: Cost
In many online communications, the low cost of deploying a message translates into a perceived high cost of testing. I call this attitude “too cheap to meter,” after a promise that nuclear power companies made back in the day that they could produce electricity at a cost so low it would actually cost them more to meter it. If you’ve ever paid an electric bill, you know that this promise did not pan out.
Similarly, when e-marketing deployment costs (email sends, banner placements, etc.) cost very little, it may cost a multiple of a deployment to create multiple versions for testing. For instance, if a marketer plans to spend $5,000 to create and deploy a single banner ad, and creating a second banner ad for testing purposes costs $1,000, then extra cost may turn the return-on-investment (ROI) equation upside-down.
Reason two: Fear of apparent failure
The other reason involves human nature rather than cold calculation. Successful testing involves failing. Always. Anytime a marketer compares one communication to another, one will always win, thus requiring the other to lose.
This win/lose dichotomy makes learning possible, but it also scares the daylights out of good corporate citizens. Very few corporate cultures embrace failure. Yes, “fail faster” has become a motto, but not at many Fortune 500 companies with whom I’ve had the privilege of working. In most corporate cultures, failure of any kind looks bad.
Thus, the idea that a test would intentionally create a loser, or that a hold-out (control) cell might mean that a winning offer might not reach a small subset of users, sounds like a career-limiting move to many marketing folks.
As a result, testing sounds good to these marketers, but the reality means pain.
The answer: Time is on your side
One good chart can overcome both of these predictions.
Both the “too expensive” and “too much failure” arguments against testing generally focus on the campaign as a one-time event. Yes, it will cost 20% more (or what-have-you) to create additional creative. Yes, a small but statistically-valid group will not get the winning offer, resulting in potentially smaller total revenues.
However, most marketers run more than one campaign, don’t they?
Before talking to clients with a disposition against testing, create a chart to set expectations. Measurements matter here, so focus on the measurement that the client treasures most, whether it be revenue, conversion of another sort or clickthrough. Create a chart showing how that measurement will accumulate over time based on the historic rate and campaign frequency. So, for instance, the chart might have total revenue based on a conversion rate of 0.5% and an average order size of $54 with weekly campaigns.
The chart needs only two lines, one showing the key measurement over time at the historic rate and another showing the key measurement over time at projected growth through testing. Yes, you need to show that in campaign 1, the total measure will sink below the current projection because at least some of the audience will not receive the winning offer. However, providing that you can predict improvements via testing, you can show how the revenue will accumulate in subsequent campaigns.
Of course, if your client still objects to this simple, elegant logic...well, maybe it’s not too late to apply to law school.