By Brendan O’Kane

Back in the mid-1700s, the British Royal Navy tasked ship surgeon James Lind with finding a way to ward off scurvy, a debilitating disease leading to open sores, tooth loss and jaundice, among sailors.

In what may be the first A/B split test, Lind gave experimental groups – as opposed to his control groups – several possible remedies, including citrus fruits, vinegar and cider. Eventually Lind proved that citrus was the answer, and it was incorporated into all sailors’ rations.

In science, this test evolved into the modern clinical trial.

Test of time
More than two centuries after Lind’s discovery, former Google exec David Siroker famously harnessed the power of A/B testing to boost the number of subscribers to then-Senator Barack Obama’s presidential campaign Web site by 40 percent – about 4 million people.

A/B split testing – deploying different versions of a single message to see which yields higher value – has been used in the world of print, television and Web marketing for decades. That is because it produces concrete, actionable evidence of what works, so marketers do not have to rely on educated guesses that may or may not produce desired results.

However, it is only recently that mobile marketers have begun thinking of applying this analytics technique for gaining insights into the preferences of mobile consumers when it comes to messaging such as SMS, mobile email and push notifications.

These marketers are just in time: around the world, there are currently 1.2 billion-plus mobile data plans in use, and they are expected to grow to 9.3 billion by 2018.

As mobile adoption booms, marketers – including retailers – are rushing to follow smartphone- and tablet-toting consumers’ lead. Most marketers are planning to increase their mobile budgets in 2013 and those budgets are expected to reach a yearly $4.4 billion in the United States alone.

But if marketers are not measuring the effectiveness of their messaging, that extra spend – and any ROI – is likely to fly right out the window.

A/B split testing of mobile messages can do for marketers large and small what it did for Mr. Obama’s first presidential campaign. It can be one of the most powerful tools of engagement in any mobile marketer’s kit.

However, the metrics taken into account when measuring performance are not the same for desktop and mobile.

Where desktop Web marketers will look at page views, banner clicks and unique monthly visitors, mobile marketers should be watching open rates, organic versus prompted opens, redemption rates and other mobile-specific conversions focused on engagement and ROI.

Lickety-split
Your average split test of a push notification campaign by, say, a big-box home improvement retailer might resemble this hypothetical example:

This fictional DIY giant wants to encourage users of its mobile application to make purchases on their smartphones or tablets for later in-store pickup, hoping the store visits will lead to extra purchases. So, the company deploys two versions of one message to different customer groups.

A. 25% off on select items when you buy now and pick up in store thru Monday!

Message A had a 40 percent open rate and a 20 percent conversion rate. For every 100,000 messages sent, 8,000 led to an in-store sale.

B. Now thru Monday! Save 25% on select items when you buy now and pick up in store!

Message B had 30 percent open rate and a 30 percent conversion rate. For every 100,000 messages sent, 9,000 led to an in-store sale.

Though it had a lower open rate, message B yields better ROI because it had a higher redemption (9 percent versus 8 percent). So the store disseminates that message to a wider audience.

Here is a real-world example:

Kidspot, Australia’s top parenting site, recently fine-tuned push notifications for its mobile app. A/B tests showed that moms preferred messages describing pregnancy and parenthood in language that was more clinical and less casual. Within two weeks, user engagement with the app was up by 87 percent.

Marketers who split-test their mobile messages often follow up by using collected response data to re-target customers who either ignored the message altogether or opened it without taking further action.

Retargeting helps marketers maximize the value of their audiences, boosting customer retention and sales in the process.

OF COURSE, A/B testing has come a long way from the days of widespread scurvy.

Today, it is a critical tool for mobile marketers of all stripes, driving engagement and conversions with thoughtful, intelligent and judiciously-timed messages that can be adjusted in real time.

Remember, no matter how well a push, SMS or mobile email campaign is doing, there is always room for improvement.

A/B split testing for mobile messaging campaigns ensures that every message a marketer sends can be more effective than the last.

Brendan O’Kane is CEO of OtherLevels, San Francisco. Reach him at [email protected].