Our Uneconomic Methods of Measuring Economic Research

By Stan Liebowitz,Ashbel Smith Professor of Economics, University of Texas at Dallas; Head of the Center for the Analysis of Property Rights and Innovation (CAPRI). Originally published at VoxEU.

Lambert here: Screamingly funny, in a deadpan sort of way, and reminiscent of Outis Philalithopoulos’s seminal work on academic choice theory.

* * *

Academic economists – especially in the US – are continuously evaluated, with salaries and promotions hanging on outcomes. This column argues that the methods – identified from a survey of economics department chairs – are likely to reduce the amount of research created, perpetuate inefficiently sized research teams, promote false authorship, and penalise honest researchers. They also provide departments with excessive leeway to engage in potentially capricious behaviour.

In the movie Moneyball, a nerdy Ivy League economics major, working for a general manager played by Brad Pitt, found undervalued baseball players by applying clear-headed logic and statistical techniques.¹ Many economists watching this movie probably felt a tinge of pride in seeing our tools portrayed as rigorously objective. After all, economists have long been proponents of using logic to eliminate inefficiencies and rent-seeking in the economy (e.g. Tullock 1967). Given this, it is surprising how infrequently that penetrating gaze has been focused on our own profession.

But some attention is warranted. Our methods for measuring and rewarding research – the key component for promotions and salaries – create inefficiencies and are inconsistent with what we teach our students about efficient production. Further, this inefficiency might be caused by economists’ own rent-seeking through the vehicle of departmental politics.

The manner in which we credit coauthorship and evaluate articles induces overly large research teams, encourages false authorship, enhances subjectivity, and penalises honest researchers.

Evaluating Coauthorship Credit

It is easy to create a division-of-credit system that gives researchers the correct incentives to choose efficiently-sized authoring teams. Simply put, a rule where the coauthors’ credit shares sum to one (i.e. full proration of credit) provides the correct incentives for choosing team sizes, and any other division rule does not. Yet, as Figure 1 shows, full proration of credit is almost never used in economics departments, according to my recent survey (Liebowitz forthcoming) of department chairs.² More than a third of departments (16 out of 45) give each coauthor full credit for the entire article. Only one department completely prorates credit.

Figure 1. Discounting of authorship

Zero proration is a flagrant violation of economic logic. For two identical quality articles, one written by a single author and the other written by four authors, should the credit to each of the four coauthors really be the same as the reward to the sole author? Do we normally say that efficient production requires that inputs get paid their marginal revenue product multiplied by the number of coworkers?

If the four-authored paper is not written with each author providing one-fourth or less effort compared to each author working alone, then that size of team is inefficient. But if each coauthor is given full credit, they have an incentive to coauthor even when the number of papers written by the four-author team is much lower than the number of equal quality papers they could write working alone or with smaller teams.

Departments that fail to discount by the number of coauthors should be embarrassed to use a measurement process that incorporates a logical error that would not be allowed in a micro principles course.

Are research-oriented departments more likely to award research credit rationally? The answer, found in Table 1, is yes and no. Higher-quality departments are more likely to prorate coauthorship than are lower-quality departments. Strangely, the amount of proration for those departments that prorate is somewhat greater for low-quality departments, leading to about the same overall degree of proration regardless of departmental quality.³

Table 1. Prorating author credit by departmental quality

Note: Based on authors’ survey of department chairs, 45 observations.

Because the failure to fully prorate will lead rational researchers to use more than the optimal number of authors, too few papers will be produced relative to the population of authors – a result consistent with a finding by Hollis (2001) that coauthorship decreased total output. If, as we like to think, our research has a net positive impact on society, then inefficient research practices would imply social harm. The failure to fully prorate may also have helped cause the doubling in coauthorship that has occurred in economics (and other fields) over the last 50 years.

The failure to prorate is also likely to lead to ‘false authorship’, where an individual not involved with the research is added to a paper’s list of authors. The ‘real’ authors benefit from a personal gain in friendship (or a quid pro quo) and suffer, if at all, only from potential guilt about being dishonest or the potential punishment if this ‘fraud’ is found out. I suspect that the expected punishment cost approaches zero, since it is unclear that the profession even acknowledges this as a problem. Because false authorship does not change the size of the actual research team, however, there is no direct negative impact on the creation of research. Nevertheless, important social costs may well occur when honest researchers are under-rewarded and possibly displaced by less qualified researchers engaging in false authorship.

In light of these inefficiencies, why do we use these reward structures? For self-interest to be the answer, it must be that the more powerful members of a department are the more intensive practitioners of coauthorship. This possibility is supported by evidence that faculty seniority is positively associated with a greater incidence of coauthorship (Conley et al. 2011, McDowell and Melvin 1983). In addition, given that this reward structure has been in place for decades (Liebowitz and Palmer 1988), faculty engaging in greater-than-average coauthorship would command an inappropriately high status among the senior faculty. Self-interest among senior, high-coauthoring faculty members would imply a push for less-than-full proration – even if these faculty members understood that it would reduce the (quality-adjusted) number of publications emanating from their department.

Judging the Quality of Papers

The self-interest of senior faculty in the measurement of productivity might also explain what otherwise appear to be irrational choices among departments in judging the quality of research papers.

There are generally three sources of information used to judge the quality of a paper: the quality of the journal in which it was published, the number of citations garnered by the paper, and opinions formed from reading the paper. Because a journal’s decision depends on the opinion of only an editor and a few referees chosen by the editor, there is a great deal of latitude for gratuitous decisions. By way of contrast, the number of citations that a paper receives is determined by the entire academy, likely reducing the influence of gratuitous behaviour. The value of reading a paper depends on who is doing the reading, their qualifications for making a judgment, and their objectivity. Consequently, it is also the method of evaluation most easily abused by the senior faculty.

Table 2 presents the relative importance of these three information sources for senior promotions, according to department chairs. The journal of publication is the leading measure of a paper’s quality (although top departments have less reliance on this measure). The main problem with this measure – other than the possibility of gratuitous acceptances – is that article quality can vary widely within any given journal, making the attribution of average journal quality to a publication potentially misleading. Further, although journal-quality measurements can be reasonably objective, it is possible for departments to use their own journal rankings based on their own biases, unmoored from any consistent ranking system.

Table 2. Relative importance of article characteristics in senior promotions

Note: Based on authors’ survey of department chairs, 46 observations.

Although the numbers in Table 2 might seem to imply that citations are given fairly high prominence in measuring faculty research, I would suggest that citations are given far too small a role in these senior promotions. For example, there seems little justification in giving any role to the journal of publication as a measure of paper quality if citation information is otherwise available. Citations tell us whether a paper is influencing the literature or not, whereas the journal of publication – even assuming the journal’s acceptance decision was fully impartial – was merely a predictor of whether a paper was thought likely to have such an impact. It seems probable that the reason that citations are not given more influence in measuring article quality is because the number of citations is the measure that can be least manipulated to fit the department’s tastes (biases).

Naturally, citations should be prorated for the same reason that article credit should be prorated. But according to the survey, citations are hardly ever prorated. Although prorating citations used to be time consuming, the advent of programs like Harzing’s ‘Publish or Perish’ has made it much easier to perform the proration.

Table 2 shows that lower-ranked departments are less likely to rely on their own opinions of papers. This is one of the few findings consistent with efficiency, since these departments are presumably, on average, less qualified to judge a paper’s quality than are the members of more highly ranked departments.

Conclusion

Are academic economists little different than the old-school baseball scouts in Moneyball, clinging to unexamined rules of evaluation? Or is our profession dominated by inefficient rent-seeking? Either way, our measurement systems do not appear suited to promote productivity. Maybe Brad Pitt should play a Provost imposing some ‘scientific’ logic on our university sanctuaries.

References

Conley, J P, M J Crucini, R A Driskill, and A S Onder (2013), “Incentives and the Effects of Publication Lags on Life Cycle Research Productivity in Economics”, Economic Inquiry, 51(2): 1251–1276.

Hollis, A (2001), “Co-authorship and the output of academic economists”, Labour Economics, 8(4): 503–530.

Liebowitz, Stan J (forthcoming) “Willful Blindness: The Inefficient Reward Structure in Academic Research”, Economic Inquiry.

Liebowitz, S J and J P Palmer (1984), “Assessing the relative impacts of economics journals”, Journal of Economic Literature, 22(1): 77–88.

Liebowitz, S J and J P Palmer (1988), “Assessing Assessments of the Quality of Economics Departments”, Quarterly Review of Economics and Business, 28(2): 88–113.

McDowell, J M and M Melvin (1983), “The determinants of co-authorship: an analysis of the economics literature”, Review of Economics and Statistics, 65(1): 155–160.

Tullock, G (1967), “The Welfare Costs of Tariffs, Monopolies and Theft”, Western Economic Journal, 5(3): 224–232.

---

1 Pitt was playing a real-life baseball executive, Billy Beane, who used ‘sabermetrics’ to help pick the players on his team who would provide the highest return per dollar. The movie was based on the book of the same name by Michael Lewis.

2 The survey consists of a questionnaire given to department chairs at the top 100 or so mainly American universities, almost half of which were answered.

3 Full proration (100% in Table 1 and Figure 1) implies that each of n (equal) authors receives 1/n credit for the article. Zero proration implies that each author receives credit for the entire article. Other percentages are a linear combination of these two.