Using Multivariable Testing to Find Hidden "Solutions"
The Opportunity
A very successful big-box retailer had hit the wall. It had more than $10 billion in annual sales and more than 600 stores, but Wal-Mart was attacking aggressively the category; comp sales (comparable sales same stores last year to this year) — the major measure that retailers use — had been negative each quarter for more than a year; and margins were declining sharply. Shareholders were not happy. Over a year, the company had launched efforts to reduce operating costs while optimizing product assortment and updating the store environment, but the slide in results continued. The management team had hundreds of ideas to re-establish its competitive edge, but there was no clear path forward that focused strategy on the right combination of actions.
A very successful big-box retailer had hit the wall. It had more than $10 billion in annual sales and more than 600 stores, but Wal-Mart was attacking aggressively the category; comp sales (comparable sales same stores last year to this year) — the major measure that retailers use — had been negative each quarter for more than a year; and margins were declining sharply. Shareholders were not happy. Over a year, the company had launched efforts to reduce operating costs while optimizing product assortment and updating the store environment, but the slide in results continued. The management team had hundreds of ideas to re-establish its competitive edge, but there was no clear path forward that focused strategy on the right combination of actions.
The CEO was very attracted to the idea of quickly testing many potential actions and getting objective information about the impact of each one, individually and in combination. He embraced QualPro's 20 years of data that showed that only one-fourth of a company's improvement ideas actually help while almost as large a portion hurts results. He felt sure that the majority of the actions the company was taking were wasted, but he did not know which actions were the bad ones. His company had a long history of testing, but he said there were so many sacred cows that one-at-a-time testing could not possibly provide enough answers before shareholders lost confidence in the management team.
 |
The Approach
Within a few weeks, a dozen different brainstorming sessions brought to the surface more than 300 improvement ideas. The front end of the multivariable testing (MVT) process narrowed the ideas to only those that were easy to test, quick to implement, and did not raise costs. At the same time, the retailer's database was analyzed to identify other high-potential ideas.
Within a few weeks, a dozen different brainstorming sessions brought to the surface more than 300 improvement ideas. The front end of the multivariable testing (MVT) process narrowed the ideas to only those that were easy to test, quick to implement, and did not raise costs. At the same time, the retailer's database was analyzed to identify other high-potential ideas.
Each of 32 representative stores executed a unique combination of about half of the test ideas. Logistics planning delivered communications, training, and materials to each store so that the MVT "recipe" could be executed properly.
The test conditions held for eight weeks. Telephone calls and mystery shopper visits ensured that each store complied with its recipe. Each store's sales and margins were measured for the 8-week period, and results were analyzed to calculate the precise numerical impact of each potential action. After a second MVT was run in order to confirm the findings and fully explore where ideas worked differently in combination than they did individually, the optimum combination of actions was implemented.
The Results
Within four months of implementation, the comp sales trend improved by 4 percentage points, boosting sales by more than $400 million per year. Cost savings that increased margins also were identified.
Within four months of implementation, the comp sales trend improved by 4 percentage points, boosting sales by more than $400 million per year. Cost savings that increased margins also were identified.
What worked? Supervisor compensation was adjusted; television advertising content was modified; shelf detailing procedures for store operations were revised; investments in the direct mail program were reallocated; expenditures on the uniform program were suspended. Also, very importantly, three actions that probably would have been taken — new end cap displays, use of part-time associates, and revised associate compensation — were found to significantly hurt sales and aborted.
