Metrics: You Are What You Measure

By John R. Hauser, MIT Sloan School of Management and Gerald M. Katz, Applied Marketing Science, Inc.
Reprinted from European Management Journal, Vol. 16, 1998, pp. 517-528, Hauser et al, "Metrics: You Are What You Measure!" Copyright 1998-2003, Elsevier. Used with Permission.

Firms use metrics for a variety of laudable purposes. Metrics such as market share, sales increases, margins, and customer satisfaction surveys enable firms to take stock of where they are and to plan for the future. Metrics such as projected revenue, contingent sales forecasts, the net present value of an investment, and the option value of an R&D program provide indicators of future performance. Managers use these metrics to allocate assets and select strategies. Metrics such as an R&D effectiveness index, reductions in the operating cost of a telephone service center, and reduced absenteeism provide the basis for bonuses and promotions for managers and their employees.

Every metric, whether it is used explicitly to influence behavior, to evaluate future strategies, or simply to take stock, will affect actions and decisions. If a brand manager knows that, in his or her company's culture, a "good brand is a high share brand," he or she will make decisions to maximize market share even if those decisions inadvertently sacrifice long-term profit or adversely affect other brands in the company's portfolio. If an R&D manager knows that projects are chosen based on projected net present value (NPV), he or she will encourage research scientists and engineers to work on programs and make forecasts that make NPV look good, even if somewhat misleading.

The link is simple. If a firm measures A, B, and C, but not X, Y, and Z, then managers begin to pay more attention to A, B, and C. Soon those managers who do well on A, B, and C are promoted or are given more responsibilities. Increased pay and bonuses follow. Motivated by these rewards, managers start asking their employees to make decisions and take actions that further improve the metrics. (Often, they don't even need to ask!) Soon the entire organization is focused on ways to improve the metrics. The firm gains core strengths in producing A, B, and C. The firm becomes what it measures.

If maximizing A, B, and C leads to long-term profit, the metrics are effective. If A, B, and C lead to counterproductive decisions and actions, then the metrics have failed. But it is even worse! Once the enterprise is committed to these metrics, the metrics gain tremendous inertia. Those who know how to maximize A, B, and C fear to change course and it becomes extremely hard to refocus the enterprise on new goals.

Choosing the right metrics is critical to success, but the road to good metrics is fraught with pitfalls. We recently worked with a credit card company to improve its products and services. This company had a long-standing tradition of using metrics, displaying more than 100 different measures in their lobby for all to see. One critical metric focused on the quality of the plastic used in their credit cards — no bubbles or blemishes were tolerated. But after talking to customers, we found that customers never noticed the blemishes as long as the magnetic strip on the back worked.

Unfortunately, there is no magic bullet. But there are some clear traps to avoid.

7 Pitfalls That Lead to Counterproductive Metrics

A good metric is precise, tied to overall profit, applicable to all employees, and designed to encourage extranormal effort — or is it?

Pitfall 1 — Delaying Rewards

It is rational for any employee or manager to be more short-term-oriented than the firm. Managers and employees change jobs or are promoted. They may not be around to collect future rewards. A manager or employee might do his or her job well and it might maximize long-term profit, but it might be hard to tie that profit back to that manager or employee. Change happens. Promises of future rewards might never be fulfilled. People have mortgages, face college tuition, and have other pressing needs. All of these phenomena imply that any rewards which depend upon future outcomes will be discounted more by managers and employees than by the firm. In the "present," when the actions and decisions are being made, rewards will be undervalued if they occur too far in the future. Metrics such as 5-year revenue, reduced lifetime product development costs, or defects discovered over the life of the product, all delay rewards and bias employees toward decisions and actions that have shorter-term payoffs.

Pitfall 2 — Using Risky Rewards
It is much more difficult for individual managers and employees to diversify risk than it is for the firm. While few managers and employees bear all the risk resulting from their actions and decisions, they do bear some. A business unit manager expects praise, promotions, and, perhaps, bonuses, if the business unit does well on sales, revenue, or profit targets. An employee expects praise, promotions, and bonuses if his or her product development team develops a successful new product.

Any metric that depends on an uncertain outcome from influences that are beyond their control imposes risk on the manager or employee. Managers and employees who cannot diversify this risk are likely to be risk averse. They will value guaranteed outcomes more than risky outcomes even if the risky outcomes have the same expected value to the firm. For example, consider R∓D effectiveness — a metric that measures the net revenue that R&D contributes to the firm. R&D is one of the most risky and long-term investments that the firm can make. If managers and employees perceive that they are rewarded based on long-term net revenue, they will prioritize projects that are less risky (and more short-term-oriented). The effect on the product development pipeline can be very negative.

Pitfall 3 — Making Metrics Hard to Control
Consider the engineering team charged with the design of the door for a new automobile. This is an extremely important job. Car doors have complex electrical and mechanical subsystems. They interact with ride and handling, with passenger comfort, with noise control, with styling, with power (through wind resistance and weight), and with many other components of the automobile. A good design clearly influences sales at some level.

Nonetheless, the car door is only one of many components in the automobile design. Sales also depend upon the actions of many suppliers, dealers, and competitors, and upon decisions the firm has yet to make about advertising and promotions. The macro-economy affects sales, as do interactions with other automobiles in the firm's product line. The car-door engineering team may affect revenues, but any change in revenue directly attributable to the team is probably small. From the team's perspective, the effect is tenuous at best.

Pitfall 4 — Losing Sight of the Goal
Misaligned metrics can be subtle and difficult to catch. For example, many companies driven by product innovation (such as 3M or HP) seek to identify the best scientists and engineers by tallying the new ideas, new concepts, new technologies, or new science that these employees discover. As a result, internally discovered ideas, concepts, technologies, and science ("ideas," for short) weigh heavily in the incentive system. Soon the organization values internal "ideas" to the exclusion of all others, perhaps even to the extent of reinventing the wheel. Quickly a "not-invented-here" culture takes hold. This, in turn, could lead to internal research "empires" that may be larger than necessary, generating more internal "ideas," but fewer profitable "ideas" overall.

Notice that we did not recommend rewarding managers and researchers for the profit impact of their "ideas." This would have fallen prey to the traps of pitfalls 1, 2, and 3. The key concept is not that the metrics themselves must have a direct causal effect on eventual outcomes (or other macro goals). The key concept is that the metrics are chosen so that actions and decisions which move the metrics in the desired direction also move the firm's desired outcomes in the same direction.

Pitfall 5 — Choosing Metrics That Are Precisely Wrong
There is a natural desire on the part of many managers to seek precise metrics, things that can be measured with great accuracy. But precision can be misleading. Metrics can be precisely wrong.

We worked with a variety of services firms: financial services, health-care providers, insurance providers, and public utilities. In many of these cases a services firm sought to enhance its customers' satisfaction with a telephone service center. In an effort to create incentives for the telephone representatives, the firm began to measure a variety of metrics, including the number of rings until the phone was answered, the time spent in the queue until a live representative could come on the line, the number of calls answered per hour by each representative, the number of times the customer was put on hold, and the percent of each hour that each representative spent connected to a customer. All of these metrics could be measured easily, accurately, and automatically by the telephone equipment.

Soon the firm improved dramatically on all of these measures, but the customers were still dissatisfied. In candid conversations with the telephone service center employees, we came to understand how these metrics impacted their behavior. To increase the number of calls per hour and be ready to answer the next call immediately, telephone representatives (reps) rushed customers and gave them the most convenient answer. Some reps even "gamed" the system by hanging up on a customer or two immediately after answering (without saying anything) in order to improve their metrics, hoping that no one would be the wiser! To decrease the number of times the customer was put on hold, reps were reluctant to transfer a call, even if they themselves did not know the answer. The service center became precisely what it measured — a place to process lots of calls quickly.

However, customer research showed that customers did not just want quick answers — they wanted accurate answers. Most customers would not mind waiting an extra ring or staying on the telephone a bit longer if they were then connected or transferred to a knowledgeable person who could answer their question accurately. Accuracy was much harder to measure than speed, but that was the true goal. "Accuracy" and "customer satisfaction" measures were less precise, but far more relevant to the real goals of the telephone service center.

How did the firm respond? First, they altered their information system to include a data field in which the rep could enter the customer's problem or question. After six months, the firm calculated the percent of calls that involved a problem or question that had been previously reported or asked by the same customer. An astounding 23% of all calls fit this description! By focusing on reducing this metric, customer satisfaction scores finally started to improve. An interesting side benefit was that, by doing so, call volume decreased, further reducing pressure on the reps to handle calls quickly.

The lesson: measure what is truly important, not just what is easy to measure. Vaguely right is better than precisely wrong!

Pitfall 6 — The Metrics Become the Job
Most managers and employees work hard. The goal of a metrics system should be to make them work smarter. The better the people, the better they are at redirecting their decisions and actions to maximize metrics. The best people probably already working very hard. If the imposition of a metrics system requires they work even harder, just to measure their hard work, you will likely lose them.

Consider a manager who is already working 60 hours per week. If you institute a new measurement system which requires the manager to put in an extra 6 hours per week filling out forms, he or she may do it for a while, but only long enough to polish his or her resume. And you can safely assume that the best people will leave first, leaving only those who are not in demand by your competitors.

There's no such thing as a free lunch. If the metrics system requires more effort and that effort is perceived as costly to the manager, then soon you will have to pay for the increased effort. If the metrics system imposes greater risk or time delays, then soon you will have to pay for that increased risk or time delay. It may be worth it but it does not come free.

We recommend that you consider only "lean" metrics, i.e., those that do not require a great deal of additional cost or effort to measure. We have seen organizations where managers and employees spend a significant fraction of their time just collecting the data on "fat" metrics.

Pitfall 7 — Thinking Narrowly
The last pitfall sits squarely on the shoulders of top management. Do not be paradigm bound.

In Pitfall 5 we talked about the "right" kind of metrics for a telephone service center. But why do we need a telephone service center in the first place? What if we designed our products so that they were so easy to use that customers never needed to call our service center? Impossible? Perhaps. But this was exactly the goal pursued by Intuit Software when the telephone service reps were asked to record customer questions and problems and were rewarded for communicating these issues to software designers so that the next version of the software could anticipate problems and solve them before they became problems. Market researchers literally followed customers home, watched them learn and use the software, and listened to their needs. The entire organization was oriented toward discovering and correcting problems before they occurred. Furthermore, Intuit took ownership of every problem. Even if the technical cause of the problem was a printer driver designed by a hardware manufacturer, Intuit took responsibility for the solution. In contrast to the services firms which focused on answering customer questions and solving problems, Intuit focused on anticipating customer questions and preventing problems from ever occurring.

6 Steps Toward Good Metrics: Designing a Metrics System Is Hard Work
As you can see from the pitfalls above, it is easy to select a metric; it is hard to select a good metric. Fortunately good metrics can be identified. Here's how

Step 1 — Start by Listening to the Customer
This first suggestion sounds obvious, but it is remarkable how often it is overlooked. In practice, most metrics tend to be centered around internal corporate needs such as asset utilization, staff productivity, cost reduction, and cycle time. While these needs are all honorable and very real, they usually have little direct impact on the customers' needs. Profits require sales and sales require customers. Customers purchase products and services that fulfill their needs better (and at a lower price) than current options. In order to fulfill those needs profitably, the firm has to understand those needs.

The first step must be to actually go out and talk to the customer.

There are many methods for listening to the Voice of the Customer. We have worked with firms in nearly three dozen industries. Although the specific method must be adapted to the unique characteristics of each industry, the basic concepts remain constant. The Voice of the Customer consists of a set of words and phrases that describe, in the customers' vocabulary, what the customer wants or desires in a product or service.

Consider the following six phrases as examples of the 17 needs that we identified for a specialized imaging product. Notice that they are sufficiently broad to allow a variety of solutions, but sufficiently specific to describe the product category.

"Produces a sharp image"

"Anyone can use"

"Can switch image types without waste"

"Compact and portable"

"Equipment and supplies always available"

"Equipment looks serious and professional"

Customer needs are best determined by direct interviews with customers. We normally tape-record these interviews, transcribe the recordings, identify the words and phrases, and "winnow" these words and phrases to identify a unique set. We then ask customers to sort the needs and provide prioritization of the needs.

These needs are not yet metrics. They are indicators of the outcomes that, ultimately, lead to long-term profit. If the Voice of the Customer is sufficiently detailed, these customers' needs can be tied to the decisions and actions of the imaging system. We then search for metrics which align those decisions and actions with the long-term profitability of the firm. These metrics become the "knobs" we turn to focus managers and employees on fulfilling customer needs profitably.

Step 2 — Understand the Job
After we understand the customer, we must understand the managers and employees. We need answers to the following questions: What do managers and employees value? And how do their decisions and actions affect the metrics and the desired outcomes? The methods are analogous to the Voice of the Customer, but we now focus on the firm's own managers and employees. We call this step the "Voice of the Employee (VOE)."

What managers and employees value. Managers and employees value more than salary and bonuses. They value respect, the ability to learn new skills, an interesting and exciting challenge, the chance to choose the assignment upon which they will work, the perception that their rewards are fair relative to their peers, a good work environment, and a feeling that they themselves provide value. These are but a few of the phrases we have heard from interviews with employees all the way from the factory floor to the highest professional ranks. More importantly, these "softer" rewards have high perceived value.

How decisions and actions affect metrics and desired outcomes. We worked with a company that wanted to enhance its performance and results. But before we could propose metrics, we had to understand how the scientists, engineers, managers, and support personnel produced results. We interviewed a representative sample of all employees in order to identify and classify their daily tasks. Some activities related directly to the company's core output, while other activities enhanced its capability to do so. Still other activities enabled communication among critical employees, "sold" research to internal customers, and kept costs down. Other activities maintained an attractive working environment and enhanced collegiality.

Sometimes a detailed understanding of the work process provides simple solutions to critical bottlenecks. By removing these bottlenecks we make it easier for employees to change their behavior in response to a metrics system. In one instance, we discovered that employees liked to work late in order to complete their tasks, but could not do so because the parking lots were poorly lighted and perceived as unsafe. In another instance, a high technology company had installed blackboards in the halls to encourage the informal exchange of information among scientists and engineers. The concept was good in theory, but in practice the halls were too narrow. In both cases no metric system would improve outputs without a redesign of the workspace to enable employees to make the decisions and take the actions that they knew were necessary to achieve their metrics-based goals.

Step 3 — Understand the Interrelationships
Let's return to the car-door design team. Their internal customer might be the manufacturing division. Suppose we ask the manufacturing division to rate their satisfaction with the design team's design. In turn, we might evaluate the manufacturing division on manufacturing costs and on manufactured quality (few defects). It should not surprise us that these metrics cause manufacturing to reward simple, low-cost car-door designs. Unless we balance these demands with demands to fulfill external customer needs, we risk misaligning the pressures on the car-door team. They may eschew highly-valued styling or new features for reduced manufacturing costs.

Thus, at a minimum, we must include in any Voice of the Customer analysis the demands of all internal customers and internal suppliers. Perhaps more importantly, we must consider these interrelationships in the design of a metrics system.

Step 4 — Test Manager and Employee Reaction
The best firms hire bright, creative managers and employees. No matter how carefully we design a metrics system, these managers and employees have the motives and capabilities to maximize their own well-being under the system. We hope that those decisions and actions are the decisions and actions that are in the firm's best interests, but we cannot always be sure. We recommend testing manager and employee reaction before implementation.

In the best of all worlds, try to compare the before/after reaction of a test group, which is subject to the metrics, with a control group, which is not. However, this is not always feasible. It is important, however, that we "instrument" the implementation. For example, in one pilot application we used a methodology to track the work processes identified by the Voice of the Employee. Each month all managers and employees, who were subject to the new metrics system, completed short surveys in which they indicated how they allocated their time among the work processes. By tracking changes in the patterns of work processes over time we trace the impact of the metrics system. We supplement this tracking with brief surveys to determine job satisfaction, morale, commitment to the organization, attitudes, and the judged impact of the metrics system. We also track internal measures such as attrition and other outcome measures.

Step 5 — Involve Managers and Employees
Our philosophy has always been that those who are subject to the metrics systems should be part of the team that is responsible for developing them. "Town meetings" and other feedback systems are extremely valuable in the implementation of any metrics system. Managers and employees want to do well for the firm — they are frustrated when they perceive that a metrics system is counterproductive. They are vocal and their recommendations are valuable. Any metrics system that is simply imposed from above without participation from those it impacts is likely to encounter resistance and even sabotage! The reps in our telephone service center example were, in fact, a unionized work force. Only by including some of the most senior and highly respected union members on the team was it possible to design a system that the union could accept and embrace.

We always involve managers and employees in the development of their metrics. R&D scientists will tell you when their evaluation system is forcing them to sacrifice the long-term capabilities of the firm; telephone service reps will tell you how the evaluation system is forcing them to engage in counterproductive activities; and production employees will tell you when their managers are not giving them the advice and guidance that they need. These employees are the front line of your organization. They live with the metrics system every day and it can affect their lives in big ways. They have the knowledge. You need only listen and react

Step 6 — Seek New Paradigms
Our sixth step is a caution. Use the information of Steps 1-5 creatively. Metrics enable you to get the most out of your managers, employees, and current work processes, but they should not limit you. Just as Intuit looked at their telephone service center and determined that the real goal was to design software that did not need support, so should you look at any metrics system and ask whether there is a new paradigm by which you can achieve your goals. Metrics need not reinforce current processes. By considering your goals creatively you can develop new processes. By focusing on the outputs rather than the inputs you can identify new paradigms. Good metrics free your managers and employees to discover new, more efficient ways to accomplish your goals.

A final example should illustrate this. Following industry practice, one of our electric-utility clients routinely charged a hefty deposit for any new account in order to protect itself against bad credit risks and non-payment. This was sound financial practice for residential accounts. However, many of its commercial accounts were large real-estate developers, retailers, or other businesses who constantly needed to establish service on a vacant space for a short period until the space was leased to a new tenant. To the electric company, these looked like and were treated as "new accounts." But to the customers, who were large, reliable, well-known businesses, this was an incredible nuisance and irritant. Digging deeper we found that, if the customer called to complain, the deposit requirement was routinely waived. The firm began its analysis of this customer-articulated problem by establishing three metrics: (1) the percent of commercial deposit requests that were waived, (2) the percent of customers who did not pay, and (3) the monetary value of these uncollected accounts. After analyzing several months of data, the firm discovered that an astonishing 40% of all commercial deposit requests were waived, that few commercial customers defaulted, and that the monetary value of uncollected accounts was small!

A team met to discuss this data. After several hours discussion about how to make the deposit collection process smoother and more palatable, one team member asked the key paradigm-shifting question: Does the protection against bad debts that we are obtaining from deposit requirements justify the ill will created among our customers? Does it even justify the administrative time it takes to process the waivers? It didn't take long to reach the obvious consensus or to alter the process. The policy was changed. A deposit is now required only if the customer is a known credit risk. As a default, the firm trusts its customers until there is evidence to the contrary — a new paradigm, and a better solution!

Good Metrics Empower the Organization
Metrics empower managers and employees to make the decisions and take the actions that they believe are the best decisions and actions to achieve their metrics. If the metrics are chosen carefully, then managers and employees will make the right decisions and take the right actions that enable the firm to maximize its long-term profit. But remember that the firm becomes exactly what it seeks to measure. By following these steps you can assure that your metrics system, like your company, is productive, "lean," and efficient.

John R. Hauser is the Kirin Professor of Marketing, Massachusetts Institute of Technology, Sloan School of Management.

Gerald M. Katz is vice president of Applied Marketing Science, Inc.

Applied Marketing Science (AMS) is an innovative "Voice of the Customer" market research and consulting firm that helps companies translate customer needs into new product and service designs.