Yves here. I’m waiting for VoxEU to run the generalized version of the argument it makes here, “The Dangers of Metrics.” A bizarre vogue for metrics started in the late 1990s, and it became even more acute in the dot bomb era. You’d see resumes suddenly claiming that the individual increased sales by x% or lowered complaints by y%, when nearly all the time, if you looked at their position, it would be impossible for them to have achieved that on their own. This was shades of resume fabulist Charles Asubonten, who did a brief turn as CalPERS CFO, claiming he was responsible for the increase in stock price as one of his earlier employers. Not only did we describe why that was bunk, Los Angeles Times columnist Mike Hiltzik supplied more evidence.
Another practice we’ve observed is consultants taking credit for improvements in clients’ businesses, quantifying those claims, when it’s impossible for them to take credit unless they sent in a large team and actually ran the operation.
From our 2006 Conference Board Review article, Management’s Great Addiction:
Corporate America is obsessed with numbers. Analyst meetings focus on earnings expectations, revenue growth, and margins rather than business fundamentals. PowerPoint presentations look naked if they lack charts and graphs to buttress their three-point message. Lofty corporate mission statements are often trumped by pressure to “hit the targets.”….
This love affair with figures increasingly looks like an addiction. Numbers serve to analyze, justify, and communicate. But they are, fundamentally, abstractions. When numbers begin to assume a reality of their own, independent of the reality they are meant to represent, it’s time for a reality check. Some are already frustrated with the trend:In a recent McKinsey survey of more than a thousand public-company directors, most said they wanted to hear less about financial results and more about things not so readily quantified, such as strategy, risks, leadership development, organizational issues, and markets.
Metrics presuppose that situations are orderly, predictable, and rational. When that tenet collides with situations that are chaotic, non-linear, and subject to the force of personalities, that faith—the belief in the sanctity of numbers—often trumps seemingly irrefutable facts. At that point, the addiction begins to have real-world consequences.
Here, Professor Edmans describes a specific danger of heavy reliance on metrics: promoting short-termism.
By Alex Edmans, Professor of Finance, London Business School and CEPR Research Fellow. Originally published at VoxEU
See original post for references
Policymakers, investors, and stakeholders are demanding that companies report sustainability metrics so that they can be held accountable for delivering social performance. Doing so increases the total amount of information in the market and reduces the cost of capital. However, real decisions depend on not the total amount of information, but the balance between ‘hard’ (quantitative) and ‘soft’ (qualitative) information. Since sustainability metrics only contain the former, they distort this balance – skewing managers’ sustainability investments to ones with short-term payoffs.
Sustainability is the corporate issue of the day. It was the theme of the 2020 World Economic Forum in Davos, and the call for companies to serve the wider society – not just shareholders – has only intensified in the COVID-19 pandemic.
A key challenge, however, is to measure the sustainability of a company. Accordingly, global consortiums are devising an ever-increasing set of sustainability metrics for companies to report. One example is the World Economic Forum’s framework, released in September 2020 in collaboration with the Big Four accounting firms.
The value of these initiatives is backed up by decades of academic research showing that financial efficiency – the amount of information in financial markets – boosts real efficiency. Focusing on primary financial markets, Bernanke and Gertler (1989) and Kiyotaki and Moore (1997) show that transparency reduces the cost of capital and thus increases investment.
Turning to secondary financial markets, the survey of Bond et al. (2012) discusses how financial efficiency allows managers to glean more information from prices and also increases their incentives to improve fundamental value since their compensation is tied to prices.
However, in Edmans et al. (2016), we highlight that financial efficiency – and thus information disclosure – can sometimes harm real efficiency. Central to our argument is the observation that not all information can be credibly disclosed. ‘Hard’ (quantitative and verifiable) information can be, such as the number of jobs created, but ‘soft’ information, such as the quality of those jobs, cannot.
It may seem that this distinction doesn’t matter. Even if companies can’t disclose soft information, frameworks can at least force them to disclose hard information. Our model shows that such disclosure indeed increases the total amount of information into prices and reduces the cost of capital, consistent with common wisdom. However, we also show that real efficiency (the quality of a firm’s decisions) depends not on financial efficiency (the total amount of information in prices), but the balance between hard and soft information.
The idea is as follows. If no information is disclosed, the manager will take the investment decision that improves long-term value. However, if companies are forced to disclose hard information, then it will be fully reflected in stock prices, on which the manager’s compensation and reputation are based. While investors can still gather soft information through analysing a company’s fundamentals, doing so is costly, so it will only be partially reflected in prices. Knowing that prices will depend more on hard information than soft, managers will take investment decisions that boost hard information, even if they don’t maximise long-term value.
When we originally wrote the paper, our main interpretation of hard information was ‘quarterly earnings’, and soft information referred to ‘intangible assets’. We used our model to highlight the danger of quarterly reporting, a key topic at the time: the EU Transparency Directive Amending Directive was passed in 2013, allowing companies to stop quarterly reporting, but many companies continued due to it, potentially due to the cost of capital impact.
However, hard information can equally be interpreted as sustainability metrics. Going back to the previous example, consider a company that can invest in either creating more jobs or improving the quality of existing jobs, and that the latter creates more long-term value. Unconstrained, the company will take the latter investment. But if it’s forced to disclose new job creation, it may be skewed towards the former.
Other examples are as follows:
- Disclosure of worker wages may lead to companies providing eye-catching salary raises, rather than meaningful work and skills development.
- Diversity metrics reward adding a minority to the board to tick the box, rather than developing a culture that actively encourages dissent.
- Highlighting emissions may punish products like semiconductors, which release perfluorocarbons when manufactured, yet may power the solutions to climate change.
We already recognise the incompleteness of numbers when it comes to financial information – hence the arguments against quarterly reporting. We also recognise the incompleteness of non-financial numbers in non-business settings: school league tables based on exam results may turn them into exam factories. But the same problem is overlooked in business settings.
Stakeholder value is often assumed to be ‘long-term’, in contrast to shareholder value which is labelled ‘short-term’. But stakeholder metrics can be equally as short-term as quarterly earnings, and an excessive focus on them can lead to myopic decisions.
What’s the solution? Not to throw the baby out with the bathwater and allow companies to report nothing; else, they can’t be held accountable. Instead, it’s to recognise the dangers of these metrics and interpret them with caution.
Reporting frameworks should not be prescriptive, but allow companies to opt out of reporting certain metrics if doing so may distort behaviour. Investors shouldn’t take sustainability metrics at face value, but conduct their own analysis to understand the context behind them – for example, what a company has done to promote diversity of thinking, beyond improving diversity statistics. This requires investors to take large stakes to begin with so that they have the incentive to do their own research, as studied by an extensive academic literature on blockholders (see Edmans 2014 and Edmans and Holderness 2017 for surveys).
A common question is ‘how do you measure sustainability?’ As explained in Edmans (2020), this is the wrong question. Sustainability isn’t something that you can measure; it’s something you assess. This involves starting with quantitative metrics but then supplementing them with qualitative information, just as the impact of a researcher is more than her Google scholar count and number of publications in top journals.
Critics argue that such an assessment is subjective and unreliable. They bemoan the inconsistency of ESG ratings, documented by Berg et al. (2020), and argue that greater disclosure will allow sustainability to be objectively assessed. But sustainability, just like a researcher’s impact, is inherently subjective – there’s no getting around the fact that it depends on soft information.
Even the financial value of a company is subjective because it depends not only on current financials but also management quality, competitive position, and strategic outlook. As a result, equity analysts disagree on whether a stock is a ‘buy’ or a ‘sell’ and what its price target should be, but no one bemoans this inconsistency. Moreover, the importance of soft information is an attraction, not a drawback.
If sustainability could be measured with a single, unambiguous set of quantitative metrics, there would be no need for human investors, as it could be assessed by machines and priced into the market. For investors to add value, they need to look beyond the metrics and get into the weeds of a company, rather than trying to assess it using an Excel spreadsheet.
20 comments
Comments are closed.
Since the Supreme Court provided business with the basic metrics to be pursued, that shareholders earnings returns are most critical, it still means to change this business behavior we need a new paradigm of ranked social metrics that corporations we charter or businesses that are licensed work under. That can only work if physical presence is geographically constrained . Gone would be the corporate paper mills like the State of Delaware. Each State Legislature gets back it’s responsibility to only hand out the umbrella of limited liability to corporations in strictly limited circumstances and if you are an investor you also are responsible for impacts of externalities tracked from the date you invested to the day you withdrew. Otherwise we have to be transparent and responsibly involved in the enterprise. Like a crew on the Starship Enterprise.
To get this paradigm shift requires we the People to consider either the Constitutional Amendment or Constitutional Convention process. Here is a link to the former.Www.movetoamend.org
As an afterthought to me it also means we should be cutting the salaries of Congresspeople and Senators and reining them in in other ways. Maybe then they would spend more time loving this world instead of making war on it and against it.
>we should be cutting the salaries of Congresspeople and Senators
Um, most Senators are worth 8 figures, so we should at a first pass be paying them more to discourage the industry-hopping we see so much in the Military-Industrial-Complex.
But people who become Senators are driven in ways us normals are not, so that wouldn’t make, well, a dime’s worth of difference.
Interested in hearing what “other ways” that actually are available. Just abolish the Senate would be my choice.
We have too few Congresspeople and they are underpaid, however.
@Stephen Verchinski
February 11, 2021 at 2:53 am
——-
I question your first assertion that “the Supreme Court provided business with the basic metrics to be pursued…”
The concept that shareholder returns are the singular, most important purpose of the business, to the point of disregarding all other stakeholder claims, comes from Milton Friedman in the 1970’s.
AFAIK, there was no Supreme Court decision, law, or regulation that privileged shareholder returns over all other considerations.
If you have a link or reference to such a Supreme Court decision, please post it.
——-
Separately: If you make an investor responsible for externalities, that contradicts the idea of limited liability that is the main reason for the creation of the corporate structure. Of course, we can discuss whether or not such limited liability organizations should even exist, rather than structures such as co-ops or partnerships, but they do exist and we do need to do something to rein them in, as in make them more responsible to all stakeholders, not just the shareholders.
Thank. I am so tired of this bunk.
Me, too. Always glad to help when I can. Shareholder primacy is a pernicious idea that just won’t die.
Thanks to you and Lambert I’ve learned a lot over the past several years about analyzing and structuring arguments.
If you cut Senators and Representatives pay deeply enough, you will get strictly and only rich and super-rich people becoming Sens and Reps because they can afford to be essentially unpaid political hobbyists or secret agents for their own class interests even more than now.
Well . . . you might also get purely cynical strivers like Clinton and Obama who will pass laws “now” strictly in order to get rich rewards “later” for the people they legislate to further enrich.
One could point out that we already get that. I would respond by asking how cutting Reps and Sens pay would bring us any less of that?
or you’ll get people who are in someone’s pocket, or want to go into someone’s pocket.
“What cannot be measured, cannot be controlled”. Rarely in the history of manking was so much damage done by so few words.
I keep saying that people who study management should turn to military (as in the actual fighting of a war), but they in general misunderstand it.
My point is that military is the longest running management shop in the world, ever since you had more than a warbands of a few people. Since then it found things that worked, often as the last thing tried and at a cost of countless lives. It also found that when you had it figured out things changed more often than not but the management didn’t (good solution pending).
Relevant to this, it found that there are things that can be mesaured, and more importantly are useful to be measured, but it also figured out that there are things that are pointless to measure, for example small things like morale or war progress.
Trying to measure the first may find the commanding officer fragged, and a number of politicians declaring the war won or 90% won or whatever only for the war to drag on another X years would fill a phone book.
Famously, of course, they tried to run the Vietnam war on metrics. Well, the US did. It didn’t go well, not least for all the dead civilians who were counted as combatants.
Previous militaries did use metrics, but they used sensible ones. All sides in WWII attempted as best they could to find out the effectiveness of weapons, so they often had quite precise figures on, for example, how many anti-aircraft rounds you need to use to shoot down one bomber (something like 1,500, if I recall). But those, of course, were focused and precise and could be used to sensibly allocate resources.
But with the military, more often than not, they sensibly listened to experienced commanders rather than number crunchers in most cases. For example, low flying anti-tank aircraft proved useless in terms of how many tanks they destroyed, but all sides found that they were very effective at scaring the hell out of solders on the ground, and making their own soldiers feel good, so they kept building them. Thats a lesson most MBA trained managers seem reluctant to learn.
The insistence of metrics was due to McNamara being secretary of Defence at the time who demanded a constant daily stream of metrics to know how the war in Vietnam was going and this drove strategy. The idea was to kill so many Vietnamese fighters to the point that it was more than were coming of age each year in Vietnam. That is why the obsession with body counts and its weird incentives. Kill enough VC and you get yourself a pass to go to the Bob Hope USO Show. If your unit did not kill enough VC your unit would be punished.
Before long the reports from the field became total BS as commanders either fed HQ high body counts or be relieved of command. McNamara and his “Whiz Kids” thought the war in Vietnam was going well and told the American public so. But when the Tet offensive blew up, it took the Pentagon, the US leadership and the American public by shock. It did not matter that the US won the Tet Offensive as reality had blown up MCNamara’s success model and with that, any chance of convincing people to see the war though to the end.
The military is a good example. It’s key teaching is that being well organized can greatly magnify your power. It’s history also teaches that being wildly overly optimistic and out of touch will often lead to catastrophic failure(s).
What is important about PK’s note on some of WWII’s measurements were that they really were used to learn things so that other operational things could be improved. But as you pointed out in the case of Vietnam you need to be measuring things that have a meaningful and desirable impact on your objective. Obsessing over VC body counts leads to moral corruption in the middle ranks as lying about body count and committing atrocities to actually increase body counts becomes an easy way to get noticed/promoted (or at least out of trouble for not meeting kill objectives) and it likely increased the rate of creating new VC fighters.
In Spanish we say: rule made, cheat made (“hecha la ley, hecha la trampa”), meaning you cannot make a rule that does not have a way to cheat it. Unintended consequences. Where I work, “headcount” is a critical measure, so even if I have the budget to hire someone, i cannot because Señor Management wants to keep headcount down…
Ultimately we have to use metrics for this sort of thing because there’s not much alternative and they can be done well. IT Writer Bob Lewis has been writing about the use and abuses of metrics for years. Here’s a good summary.
https://www.infoworld.com/article/2618745/the-four-fallacies-of-it-metrics.html?page=2
They’re a very powerful tool but they need to be implemented correctly. He argues four different ways to do metrics wrong.
Measure the right things badly.
Measure the wrong things, either well or badly.
Neglect to measure something important.
Extend measures to individual employees.
The problem with metrics is fundamentally twofold.
The first is related to the above, which is that you have to measure “sensible things”, but it’s actually hard to define sensible at times. It’s easy in some places (product tolerances, etc. etc.), but hard in others (staff/customer satisfaction).
The second is related to how you measure it – because, as a commenter below says, all metrics that can be gamed will be gamed. The only good way to prevent gaming is to have countervailing forces, and it’s a question whether you can set it up like that at all times.
Even if it’s not an individual metric, a set of people can often be smart enough to figure out how to game it.
second point: sounds like it all leads to juking the stats
We see this in operation whenever we buy a car. At closing the salesperson literally begs us to answer the manufacturer’s sruvey and tick off only the highest scores. Example of measuring the wrong things:
https://tap.fremontmotors.com/trusted-auto-professionals/dealer-surveys#.YCWck2hKiM8
As for extending measures to indivicual employees, well, there was was the Wells Fargo accounts scandal of the mid-2010s.
I worked at a fund management house where morale was thought to be low. This was quickly solved by: 1) tying a proportion of an individual’s annual bonus to the morale score registered by each team in a firm-wide survey and 2) making the tie between bonus levels and morale scores public pre-survey. Creative metrics 101.
“…if management sets quantitative targets and makes people’s job depend on meeting them, they will likely meet the targets – even if they have to destroy the enterprise to do it.” — Deming
I remember when they taught us how to make resumes in college (in 1994), we were supposed to put actual dollar figures or percentages on our achievements.
May I offer some synonyms for “trump” and/or “trumped”: outdo, top, cap, surpass, excel, create, cook up, concoct, makeup, manufacture, fake, contrive, fabricate, etc. As with many surnames in literature, they often accurately describe those who inhabit the name.
Management obsession with metrics is commonplace in business. I tend to believe that obsession gained strength as Finance wrested management and control of Business from Engineering and Production. Engineering and Production are far from immune to an unhealthy infatuation with numbers and metrics but I believe Finance tends to choose especially unhelpful kinds of metrics and goals. I regard General Electric’s Neutron Jack as a poster-boy for “Financecentric” metric management. Finance understands money and profits but seldom has any understanding of the Businesses they count. Though not a money or profits number crafting a sustainability label offers a particularly warm and squishy goal for metrification.
I blame NASA for some of the mania for metrics management techniques. The PERT technique developed by the military helped NASA managers identify problem areas and schedule bottlenecks ahead of time. A key difference is that most NASA managers knew intimate details about what they were working on and their teams were committed and dedicated to reaching the target completion dates which were driven by launch windows and a well-stoked sense of competition with the USSR space program. Cost was not unimportant but very much of secondary or tertiary concern to NASA’s efforts. NASA worked on one-of or a-few-of systems, and unlike General Motors Cadillac Team no 14th floor would deny engineering recommendations to add a protection capacitor to a circuit based on the costs associated with spreading the 15 mils capacitor cost over 20 million copies.
For me, my most appalling experience with management by-the-numbers came from the last parent-teacher conferences I attended at my children’s high school. Each teacher rattled off a litany of metrics that were supposed to tell me everything about how my child was doing in class. I remember stopping each teacher and asking them, “But how is my child doing?” In each case the teacher gave me a confused and then searching look and continued with their litany of numbers. If I persisted, they looked at me with a certain horror and stated that they had told me how my child was doing in their class, end of conference.