The Relevance of Rigor

By Don Lehmann
Presenting Partner: Marketing Science Institute (MSI)

 

 

Don Lehmann represents one of marketing's most pragmatic academic influences. In a Point of View article from the 2003 MSI Working Paper Series, "The Relevance of Rigor," he offers some perspective on why the hallmarks of academic research — rigor and empirical generalization — are needed to succeed in marketing today. Here we excerpt this classic text.

In increasingly challenging times, the pressure to make quick decisions increases. Consequently, reflection and careful consideration are too often viewed as luxuries. ... Busy managers may be tempted to rely on anecdotal evidence, intuition, and quickly executed studies and analyses to answer specific questions.

(But) intuition is untrustworthy; best practices studies are not a best practice; academic research should and can be both relevant and comprehensible; and empirical generalizations are a useful way to accumulate knowledge and address current issues.

Why Not Trust Intuition/ Experience/"the Force"?
Behavioral decision theory has catalogued the tendency to over-rely on readily available information and the preference for explanations that allow for control of the outcome. ... Managers tend to consider as casual variables those which are "obvious" (e.g., have data available on them) or are easy to recall and think about or are controllable by managers. They also prefer causes that are consistent with their particular point of view or self-interest (e.g., those in advertising want to believe advertising causes sales) — generally a very incomplete set of potential causes.

As an example, consider the issue of whether one action (promotion) causes another (increased sales). When thinking about this, most people recall instances when both the cause and effect occurred, in other words, when increased promotion caused increased sales (cell A, below).

 

Unfortunately, this is a singularly bad way to assess causality. One study found that even when given the number of occurrences in all four cells in the table, people placed most weight on cell A, then B, then C, and finally D (which they essentially ignored). Yet cell D, where the lack of a sales increase is accompanied by a lack of increase in promotion, is critical for establishing causality.

The Questionable Practice of "Best Practices" Studies
Terms like "practice sharing" and "best practices" have become ubiquitous in management circles. Indeed, these now substitute for more traditional research in many cases. Without question, they can uncover interesting ideas and actions to consider. However, they are not very useful for establishing causality or projection for a number of reasons. What do best practices studies do? They look for a common practice or practices followed by successful companies (cell A in the chart below) and imply that these practices cause success. ... For example, if 80% of the companies that are successful follow a particular practice, one might label it a best practice. Would you still do so if 90% of unsuccessful firms followed the practice (cell B)?

 

Humans, sensibly, like simple explanations. Unfortunately, we carry it to extremes. Rather than asking if some other cause may be involved, we tend to seize on the first cause/practice that "works." This means we ignore the likelihood that another practice might work better or that a given practice only works in certain conditions. ... While some find such complexity irritating, its existence is important, and, for some of us, beneficial.

Copying practices often just adds cost (you never lead by following). What happens if all companies search for "best practices," find the same ones, and employ them? The result is that no one has a competitive advantage, much less a sustainable one. ... Looking for best practices can generate ineffective me-toos, as well as antiquated solutions to problems.

Relevant Academic Research: An Oxymoron?
Academic research is often couched in specialized and arcane language and uses incomprehensible (and unnecessarily complex) methods and language. Yet in spite of this, much academic research has relevance to real decisions. ... Consider work in understanding customers and brands.

Understanding Customers
Assuming a customer generates a constant level of net revenue to a firm (i.e., there is no growth/customer expansion), the long-term (discounted) value of that customer to a firm depends on the discount rate and customer retention.

 

Two main insights emerge from this straightforward but "rigorous" calculation. First, the value of such a customer is two to five times current net revenue. This provides a useful standard against which to weigh acquisition and retention costs.

Second, the multiple is much more sensitive to the retention than the discount rate. This suggests that work on product quality and marketing will add more to customer, and hence firm, value than exotic financial engineering. Put simply, marketing matters.

Understanding Brands
One study examined the impact of P&G's move toward everyday low pricing in the early 1990s. ... P&G increased advertising and price and cut deals and coupons. Unilever tended to go along. However, Colgate and Gillette did not: They increased deals and coupons. The result: P&G lost share, although its profits may have increased. This reinforces the need to consider competitor reactions when analyzing strategic options, a complication that is often overlooked.

 

Empirical Generalizations
Companies have, but often ignore, considerable information in their files. By analyzing what happened in the past (a.k.a. meta-analysis), one can obtain a prediction of what will happen in the future. This approach helps average out the infinite number of idiosyncrasies associated with a single study and allows for "triangulating" a finding. Considerable evidence suggests that even when averages differ (e.g., between large and small regions or specific products), the impact of specific variables (e.g., advertising) can be similar. Meta-analysis also has the advantage of being available yesterday, a notable advantage in a time-compressed world.

 

Another potential use of meta-analysis is for purposes of evaluating proposed budgets. Consider the budget for a slowly growing market. See if the implied budget is sensible in terms of the implied effect of advertising on sales.

Based on research, we know the impact of doubling advertising spending ("advertising elasticity") tends to increase sales, from about 1% for a mature product to about 30% for a new product. The budget here proposes a 30% increase in sales for a 60% increase in advertising, an elasticity of 50%. Because this is beyond the range of past experiences, it seems prudent to question the budget. While the campaign, copy, etc., may be "special," it is probably not that special. While sometimes it makes sense to ignore the odds, it is still useful to know you are doing so.

Rigor vs. Rigor Mortis

For a complete text copy of this article, go to www.MSI.org .

To read an interview with Don Lehmann, click here.