Answering the Age-Old Question: Did It Work?

 

Retailer F recently completed a pilot test of a new customer loyalty program in which consumers were rewarded with points for purchases, redeemable for store certificates. The trial was run in 10 markets around the country, while 30 other markets were identified as hold-out controls. At the end of the test period, each of the test and control markets was analyzed for change in sales from the year-ago period. The results are below:

The question now is, did it work? Is the average difference between test and control markets sufficiently big that we can safely conclude that the loyalty incentive should be rolled out to the remaining markets?

Rather than debate the statistical significance of the average 3.42% difference and all the possible variables that might have made it higher or lower, we can perform a simple test that asks, "If we randomly assigned each market a sales change percentage from the same relative range as those reported in the test and control markets, would we come up with a difference of averages greater than the 3.42%?" In other words, are we seeing an effect of random results across markets, or is there really a pattern?

To test this hypothesis, we could manually replace the sales result from each market with a number randomly selected from one of the other 39 individual market reports, but we might be skeptical that the difference of averages we come up with was just a fluke. The only way we could get more comfortable with the result would be to do it thousands of times and record the differences between the average results for test vs. control markets. Presumably, the more we did this, the higher our confidence would be that the results we saw in the test were either random or actually directional.

We used a simple Excel plug-in tool called XLSim®. It took us two minutes to download and about an hour to learn how to use. It comes with easy instructions and examples that let you build and run simulations with hundreds, thousands, or even tens of thousands of trials. It outputs the data in clearly understandable graphs.

We began by creating a matching chart to the sales results above, only this time we assigned each market a sales result randomly selected from one of the original 40 market results (for the curious, it was "resampling with replacement"). We then measured the difference between the average of the 10 markets on the left and the average of the 20 markets on the right. If the difference between averages was greater than or equal to the 3.42%, we could conclude that randomly assigning numbers was as likely as loyalty to produce the same result, and loyalty didn't drive sales. However, if the difference was less than the 3.42, we could conclude that the results were not random and that loyalty incentive did in fact lift sales. We ran this test 1,000 times, in less than 30 seconds, and examined the results.

Conclusion
With only 10 minutes work, we can conclude that the loyalty stimulus did in fact generate incremental sales with a high degree of confidence (96%). Now, if we could only use these simple tools to predict how the store managers will manage the rollout.