Yves here. Even thought the mainstream media, as usual, duly applauded the winners of this year’s “Nobel” prize in economics, there’s been less attention paid to the recipients than usual and far more criticism, some very measured, others more critical. The winners helped develop and promote an approach to development economics they called “randomized control trials” seeking to associate their approach with the highly respected random clinical trials in medicine. The reason their work has been received well is development economics has been a backwater full of failures, and here came an approach that appeared to work. On top of that, it has a ready clientele: NGOs who focus on poor countries and want to know if they should spend their money on, say, feeding children in school lunch versus having more teachers.
One indicator that there is something not right here is that the findings of a study can’t be replicated, which suggests that despite all the appearance of rigor, this is scientism to make wealthy donors feel better about how they are spending their money. Another set of critiques is that the approaches that these economists study are all small bore, fail to address fundamental causes, and therefore will not help people get out of poverty as much as (at best) somewhat ameliorate it. See, for instance, Peter Dorman, Lars Syll and this article from the Journal of Development Studies
Oh, and the title above is a play on the title of the layperson-oriented book, Poor Economics, written by two of the three winners, Abhijit V. Banerjee and Esther Duflo. The third winner, Michael Kremer, has the much more serious body of work than the other two.
By Ingrid Harvold Kvangraven is a Lecturer in International Development at the University of York. She tweets at @ingridharvold. Originally published at openDemocracy
This week it was announced that Abhijit Banerjee, Esther Duflo and Michael Kremer won the Economics Nobel Prize (or more accurately: the ‘Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel’). The trio of economists were awarded the prize for “their experimental approach to alleviating global poverty”.
On social media and in mainstream newspapers, there was an exceptional level of praise for the laureates, reflecting their existing rockstar status within development economics. The Financial Times even claimed that the Nobel “will help restore [the] profession’s relevance”. However, the widespread calls for celebration need to be considered with a cautionary counterweight.
The experimental approach to poverty alleviation relies on so-called Randomized Control Trials (RCTs). Inspired by studies in medicine, the approach targets specific interventions to a randomly selected group (schools, classes, mothers, etc), and then compares how specific outcomes change in the recipient group versus those who did not receive the treatment. As the groups are assumed to be otherwise similar, the difference in outcomes can be causally attributed to the intervention.
While the laureates were first pioneering this work in the 1990s in Kenyan schools, the approach is now widely considered the new “gold standard” in development economics, also sometimes simply called “New Economics”. The approach has become enormously influential among governments, international agencies and NGOs. The body of work pioneered by the laureates, or the randomistas as they are sometimes called, is meant to alleviate poverty through simple interventions such as combating teacher absenteeism, through cash transfers, and through stimulating positive thinking among the people living in poverty. Sound good so far?
While the laureates’ approach to poverty research and policy may seem harmless, if not laudable, there are many reasons for concern. Both heterodox and mainstream economists as well as other social scientists have long provided thorough critique of the turn towards RCTs in economics, on philosophical, epistemological, political and methodological grounds. The concerns with the approach can be roughly grouped into questions of focus, theory, and methodology.
Focus: Tackling Symptoms and Thinking Small
The approach that is being promoted is concerned with poverty, not development, and is thus a part of the larger trend in development economics that is moving away from development as structural transformation to development as poverty alleviation. This movement towards “thinking small” is a part of a broader trend, which has squeezed out questions related to global economic institutions, trade, agricultural, industrial and fiscal policy, and the role of political dynamics, in favor of the best ways to make smaller technical interventions.
The interventions considered by the Nobel laureates tend to be removed from analyses of power and wider social change. In fact, the Nobel committee specifically gave it to Banerjee, Duflo and Kremer for addressing “smaller, more manageable questions,” rather than big ideas. While such small interventions might generate positive results at the micro-level, they do little to challenge the systems that produce the problems.
For example, rather than challenging the cuts to the school systems that are forced by austerity, the focus of the randomistas directs our attention to absenteeism of teachers, the effects of school meals and the number of teachers in the classroom on learning. Meanwhile, their lack of challenge to the existing economic order is perhaps also precisely one of the secrets to media and donor appeal, and ultimately also their success.
The lack of engagement with the conditions that create poverty has led many critics to question to what extent RCTs will actually be able to significantly reduce global poverty. A further consequence of this impoverished economics is that it limits the types of questions we can ask, and it leads us “to imagine too few ways to change the world”.
Theory: Methodological Individualism Lives On
In a 2017 speech, Duflo famously likened economists to plumbers. In her view the role of an economist is to solve real world problems in specific situations. This is a dangerous assertion, as it suggests that the “plumbing” the randomistas are doing is purely technical, and not guided by theory or values. However, the randomistas’ approach to economics is not objective, value-neutral, nor pragmatic, but rather, rooted in a particular theoretical framework and world view – neoclassical microeconomic theory and methodological individualism.
The experiments’ grounding has implications for how experiments are designed and the underlying assumptions about individual and collective behavior that are made. Perhaps the most obvious example of this is that the laureates often argue that specific aspects of poverty can be solved by correcting cognitive biases. Unsurprisingly, there is much overlap between the work of randomistas and the mainstream behavioral economists, including a focus on nudges that may facilitate better choices on the part of people living in poverty.
Another example is Duflo’s analysis of women empowerment. Naila Kabeer argues that it employs an understanding of human behavior “uncritically informed by neoclassical microeconomic theory.” Since all behavior can allegedly be explained as manifestations of individual maximizing behavior, alternative explanations are dispensed with. Because of this, Duflo fails to understand a series of other important factors related to women’s empowerment, such as the role of sustained struggle by women’s organizations for rights or the need to address unfair distribution of unpaid work that limits women’s ability to participate in the community.
Note that there is nothing embedded in RCTs that forces randomistas to assume individuals are rational optimizing agents. These assumptions come from the economics tradition. This is therefore not a critique of RCTs per se, but of the way RCTs are employed in the laureates’ work and in most of mainstream economics.
Method: If You Didn’t Randomize It, Is It Really Knowledge?
While understanding causal processes is important in development economics, as in other social science disciplines, RCTs do so in a very limited way. The causal model underlying RCTs focuses on causal effects rather than causal mechanisms. Not only do RCTs not tell us exactly what mechanisms are involved when something works, they also do not tell us whether the policy in question can be reliably implemented elsewhere. In order to make such a judgement, a broader assessment of economic and social realities is unavoidable.
Assuming that interventions are valid across geographies and scale suggests that micro results are independent of their macroeconomic environment. However, while “effects” on individuals and households are not separate from the societies in which they exist, randomistas give little acknowledgement to other ways of knowing about the world that might help us better understand individual motivations and socio-economic situations. As it is difficult to achieve truly random sampling in human communities, it is perhaps not surprising that when RCTs are replicated, they may come to substantially different results than the original.
Not only do RCTs rarely have external validity, but the specific circumstances needed to understand the extent to which the experiments may have external validity are usually inadequately reported. This has led even critics within the mainstream to argue that there are misunderstandings about what RCTs are capable of accomplishing. A deeper epistemological critique involves the problematic underlying assumption that there is one specific true impact that can be uncovered through experiments.
Recent research has found that alternative attempts to assess the success of programs transferring assets to women in extreme poverty in West Bengal and Sindh have been far superior to RCTs, which provide very limited explanations for the patterns of outcomes observed. The research concludes that it is unlikely that RCTs will be able to acknowledge the central role of human agency in project success if they confine themselves to quantitative methods alone.
There are also serious ethical problems at stake. Among these are issues such as lying, instrumentalizing people, the role of consent, accountability, and foreign intervention, in addition to the choice of who gets treatment. While ethical concerns regarding potential harm to groups is discussed extensively in the medical literature, it receives less attention in economics, despite the many ethically dubious experimental studies (e.g. allowing bribes for people to get their drivers’ licences in India or incentivizing Hong Kong university students into participation in an antiauthoritarian protest). Finally, the colonial dimensions of US-based researchers intervening to estimate what is best for people in the Global South cannot be ignored.
Why It Matters: Limits to Knowledge and Policy-Making
There will always be research that is more or less relevant for development, so why does it matter what the randomistas do? Well, as the Nobel Committee stated, their “experimental research methods now entirely dominate development economics”. A serious epistemological problem arises when the definition of what rigour and evidence means gets narrowed down to one single approach that has so many limitations. This shift has taken place over the past couple of decades in development economics, and is now strengthened by the 2019 Nobel Prize. As both Banerjee and Duflo acknowledged in interviews after the prize was announced, this is not just a prize for them, but a prize for the entire movement.
The discipline has not always been this way. The history of thought on development economics is rich with debates about how capital accumulation differs across space, the role of institutions in shaping behavior and economic development, the legacies of colonialism and imperialism, unequal exchange, the global governance of technology, the role of fiscal policy, and the relationship between agriculture and industry. The larger questions have since been pushed out of the discipline, in favor of debates about smaller interventions.
The rise of the randomistas also matters because the randomistas are committed to provoke results, not just provide an understanding of the situations in which people living in poverty find themselves. In fact, it is one of their stated goals to produce a “better integration between theory and empirical practice”. A key argument by the randomistas is that “all too often development policy is based on fads, and randomised evaluations could allow it to be based on evidence”.
However, the narrowness of the randomized trials is impractical for most forms of policies. While RCTs tend to test at most a couple of variations of a policy, in the real world of development, interventions are overlapping and synergistic. This reality recently led 15 leading economists to call to “evaluate whole public policies” rather than assess “short-term impacts of micro-projects,” given that what is needed is systems-level thinking to tackle the scale of overlapping crises. Furthermore, the value of experimentation in policy-making, rather than promoting pre-prescribed policies, should not be neglected.
The concept of “evidence-based policy” associated with the randomistas needs some unpacking. It is important to note that policies are informed by reflections on values and objectives, which economists are not necessarily well-suited to intervene in. Of course, evidence should be a part of a policy-making process, but the pursuit of ineffective policies is often driven by political priorities rather than lack of evidence.
While randomistas might respond to this by arguing that their trials are precisely meant to de-politicize public policy, this is not necessarily a desirable step. Policy decisions are political in nature, and shielding these value judgements from public scrutiny and debate does little to strengthen democratic decision making. Suggesting that policy-making can be depoliticized is dangerous and it belittles the agency and participation of people in policy-making. After all, why should a policy that has been proven effective through an RCT carry more weight than, for example, policies driven by people’s demands and political and social mobilisation?
While the Nobel Prize does leave those of us concerned with broader political economy challenges in the world anxious, not everything is doom and gloom. Firstly, the Nobel directs attention to the persistence of poverty in the world and the need to do something about it. What we as critical development economists now need to do is to challenge the fact that the Prize also legitimizes a prescriptive view of how to find solutions to global problems.
Secondly, the fact that a woman and a person of color were awarded a prize that is usually reserved for white men is a step forward for a more open and inclusive field. Duflo herself recognizes that the gender imbalance among Nobel Prize winners reflects a “structural” problem in the economics profession and that her profession lacks ethnic diversity.
However, it is obvious that to challenge racism, sexism and Eurocentrism in economics, it is not enough to simply be more inclusive of women and people of color that are firmly placed at the top of the narrow, Eurocentric mainstream. To truly achieve a more open and democratic science it is necessary to push for a field that is welcoming of a plurality of viewpoints, methodologies, theoretical frameworks, forms of knowledge, and perspectives.
This is a massive challenge, but the systemic, global crises we face require broad, interdisciplinary engagement in debates about possible solutions.
Thanks to Carolina Alves, Devika Dutt, Minna Lehtinen and Farwa Sial for helpful discussions on these issues and comments on a previous draft.
21 comments
Comments are closed.
Ms. Kvangraven,
You’ve written a veritable public service here. I’d like to try to reciprocate in some, small way.
You might wish to read Mark Weisbrot’s (of the Center for Economic & Policy Research in Washington, DC) Ph.D. dissertation. (I have.) After explaining in depth why neo-classical economics could not explain what was happening in poorer countries, the field of economic development actually grew out of classical economics (in the 20th Century, no less). From there, other paradigms – Keynsian economics, Institutionalism – weighed in with meaningful contributions. However, the field eventually succumbed to neo-classicism not because of any theoretical superiority. Rather, if I remember correctly (it’s been many years since I read the dissertation), neo-classicism won out for reasons of rank power.
Let me know if you have trouble finding it; reading this dissertation will likely provide you with many more arrows in your academic quiver.
I’m guessing this is it:
Mark Weisbrot – Ideology and method in the history of development economics
https://search.proquest.com/docview/304059655
My contention has always been that the point of neoclassical economics is simply to serve as apologetics for the status quo, and this RCT nonsense is just yet more of that. Don’t question the actual causes of poverty, just fiddle around with some minor interventions that don’t question existing power relations, and whose results aren’t replicable, and collect your accolades for being such a clever economist.
And just one quibble with the author’s attempt to find a silver lining in this absurdity: anyone who needs the awarding of a Nobel to “direct attention to the persistence of poverty in the world and the need to do something about it,” has really, truly, not been paying any attention at all. And I don’t think this prize will lead to anyone actually taking poverty seriously (because if they did they’d be advocating for land reallocation). What it will do is ensure that anyone in the economics profession who wants to work on these issues will be expected to do so using RCT and will be shunned for pursuing other non-award-winning methodologies.
God save us when the elite decide what is best for the poor.
I watched a video on Netflix. “Inside Bill’s Brain “, a while back and it was a good video about Bill Gates trying to solve the problems of dysentery among the world’s poor caused from inadequate sewage treatment. He got the greatest and brightest minds together and developed a whiz bang new fecal waste treatment system and he got it implemented in Dakar. Now this is an excellent system, Bill Gates deserves credit for that, but because he didn’t understand the socio-economic policies in Dakar, he didn’t do a thing to solve the dysentery problems there – a point the video completely ignores – the whiz bang system his team devised was too expensive for the poor to use and according to this study, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3317593/ , 37% of the poor are still throwing their fecal waste into the streets, just as before and dysentery is still a huge problem in Dakar. But Bill Gates got so involved with the technology, I think he forgot his original reasons for doing it – or perhaps he just doesn’t understand that not everybody can pay, even for things they desperately need.
I’ve heard more than one anthropologist who has studied in Africa say that much of what the elite are doing, out of honest and real concern for the poor, backfires, because they just do not understand the needs of the poor, nor the socio-economic reasons that the people are in poverty. Bringing new farming techniques and high technology to some of these areas doesn’t make the impoverisheds’ lives better, but it does make them more dependent. I don’t think the elite understand that. Perhaps they should listen to the anthropologists who try to tell them that they should talk to the people they are trying to help to understand what they need and want, not just decide that because of their own personal socio-economic status, they automatically know what is best for other people.
As an African who lives and works in development in Africa, my most sincere contribution is that these foreign helpers simply have no understanding of our history and culture. They are the most important things to know/understand since they shape our “needs”. I strongly believe the onus is on us (Africans who live and work here) to build (develop) our home because it is only us who can decide the trajectory. But alas, we are waiting for such a time that many Africans will realize and take action; my money is on social media revolutionizing our cultural thinking the way the printing press did in Europe.
A very important macro force here is the distinction between agricultural and resource extraction-based economies. In an agricultural economy, the society requires workers and is required to treat them well. A natural resource-based economy needs few workers, and therefore treats its people like garbage.
To imply that these observations are random or clinical or even strict is a disservice to randomized clinical trials. In real RTCs every possible interference is screened out so that the results are as pure as possible. This kind of result is not possible with randomized economic observations. The idea is good in one way – in that it focuses on a locality, a town, a countryside to diagnose the problem of local poverty. And this can be repeated as a technique anywhere and probably should be. Just seriously looking at it helps immediately to change it. It might make a good ongoing evaluation of the exaggerated claims of economists who never put a foot on the ground. This is kind of groundbreaking even though it is timid. The existing system will think of it as a way to micromanage poverty. That in turn will lead to “prescriptive” action and eventually to economic protocols. An amusingly polite challenge to the “system that produces the problems” in the first place. But, imo, this does introduce a higher state of economic reality.
Unfortunately. RCT is clearly a flawed method because its findings cannot be reproduced. If you are thinking in regression terms, it’s as if you had early run a regression, found the independent variables with really low R-squares, and then did RCT studies using only a couple of them.
I get the same feeling from this “prize” as I did when they gave Obama the peace prize. I hope it marks a turning point, a change of view. Pre-emptive prizing or stg.
There’s a reasonable point lost in all this, but the article is essentially a plea for the primacy of ideology over empirical research, which, ironically is exactly what the author is complaining about. She just wants economics dominated by different ideologies. the last paragraph
“…to challenge racism, sexism and Eurocentrism in economics, it is not enough to simply be more inclusive of women and people of color that are firmly placed at the top of the narrow, Eurocentric mainstream. To truly achieve a more open and democratic science it is necessary to push for a field that is welcoming of a plurality of viewpoints, methodologies, theoretical frameworks, forms of knowledge, and perspectives.”
is essentially meaningless unless you believe that skin colour somehow influences the nature of reality. There’s also a fundamental confusion about a “plurality of forms of knowledge,” since knowledge is one thing, and only one thing, whereas interpretations can be various.
Well honestly your race or ethnicity actually plays a huge role in your reality, did you really say “if you actually believe?” ?
Unless reality is the same for everyone there is no possibility of doing science. If your reality is different from mine then we are condemned to solipsism. Communication itself would be impossible. Indeed language could not exist.
David is referring to objective Reality, which is what these RCTs are supposed to reveal, not subjective “reality” which inevitably scuppers them since social relations (such as an economy) are largely determined by the latter.
This “Nobel” has been given, as always, not for revealing objective Reality but for finding a new way to make obscuring it sound more sciencey.
Well, David did not make clear what he was asserting, just a very clear statement saying “unless you believe” and I think a lot of people out there genuinely believe
I’ve never understood this desire to make economics so “sciency”. It is very possible to study socio-economics, take account of real lived experiences to form new solutions to problems without discounting these “subjective realities”. That’s really the issue with these grand development theories, no one is genuinely asking why Africans are the way they are and an honest way to chart a new trajectory; most seem to just want to make up something that sounds good but is totally impractical. Something can work in one community but be totally useless in another, the culture of a people usually plays a large role in how things turn out but for some reason it’s frowned on to admit that in this field.
Economics strives to ape actual science to further the illusion that the pronouncements from the field are not ideological apologies for wealth, but rather apolitical, objective descriptions of reality… that There Is No Alternative.
That’s why the banks needed to invent a fake Nobel prize in the first place, of course.
The goal, largely successful, is to remove economic power from the sphere of the political, making it immune to democratic forces.
The Bank if Swdeb Prize iam with a few exceptions, mostly a hall of shame, Witness last years awarding to William Nordhaus for “environmental” economics, when his work more than any other economist’s, has obstructed a robust response to the climate crisis. Cf. {Philip Mirowski’s work the the origins of the Faux Nobel.
On Scott Alexander’s Slate Star Codex blog, I recently came across a book review of James Scott’s book, Seeing like a State:
https://slatestarcodex.com/2017/03/16/book-review-seeing-like-a-state/
Essentially, the book is an overview/comparison of the High Modernists ( City Planners, Historian’s example of Bill Gates above, etc.) with the sort of on-the-ground ultra-empirical knowledge or local practical wisdom or “metis”, similar to Historian and Abi’s comments above. Scott seems to favor the more libertarian/anarchism of local knowledge while in his review Alexander sees the benefits of both central authority planning and local practical wisdom.
I tried to read Poor Economics a while ago and could not endure it for the reasons Kvangraven and the NC commentariat mention. I think local practical wisdom is important but as Yves mentions, to think it can be used as some sort of RCT and be reproducible seems foolhardy. One the main things I have gained from reading some of Jerome Kagan’s books such as the Human Spark, is many things depend on context, conditions, culture, time in history, geography, etc. etc.
I think Alexander’s book review is very good and a good supplement to this post. I would like to write more about the connections I see between this post and Alexander’s review but dinner awaits.
Gates example reminds me of the India water well church philanthropy almost 2 decades ago, sadly the geo was not tested and arsenic poisoning resulted. Seems good intentions ran afoul of costing ratios and number of wells provided.
In my case I find the foundations of probability and its extenuation the baseline argument, no amount of maths and physics can substitute for failures in the prior, though some do try and do get accolades for the attempt. Which sadly only reinforces the methodology, reminiscent of the B-17, get it right one day, not that it ever was, but romanticized anyway.
Which brings me back to my first point, funding and expectations supplied through narrow networks and what ever rolls around in those heads.
You may want to read The Nobel Factor by Avner Offer and Gabriel Söderbergh. It supplies a sytematic history of the misnamed “Nobel Prize,” exlains the purpose of its creation, and it provides some ideas of just how unscientific economics really is.
In passing , the Swedish word “vetenskap” which occurs in the proper description of the prize sometimes translates as “scholarship. And note, the certificates and the medals given out do not include the phrase “Nobel Prize.”
It is well writtn, and also provides an interesting history of the development of economics since 1968
Econometrics gives the impression of a very thorough study having gathered together all the available evidence.
A meticulous study of evidence by leading economists, Reinhart and Rogoff in “This time is different” missed the bleedin’ obvious.
When bankers mess up, look at what they have been doing with their debt products.
https://cdn.opendemocracy.net/neweconomics/wp-content/uploads/sites/5/2017/04/Screen-Shot-2017-04-21-at-13.52.41.png
One missing data set and the conclusions are entirely different.
“Recent research has found that alternative attempts to assess the success of programs transferring assets to women in extreme poverty in West Bengal and Sindh have been far superior to RCTs, which provide very limited explanations for the patterns of outcomes observed.”
“Far superior” is surely hyperbole. Yes, ethnographic studies are much better at generating explanations than RCT’s. But ethnographic studies can’t show how important their explanations are. They don’t provide solid tests of whether the generated explanations are correct. Much of their explanations involve making visual correlations like “the successful ones tended to be richer”. They also share the weakness of non-generalizability with RCT’s, if not more so, because they are just as contextual. If you’re not explicitly testing if your explanations work, you are much more likely to fall into the trap of confirmation bias.
All methods have flaws. Let’s not forget that ethnographic work has been going on for a long time without making a dent on poverty, so it is also not the panacea on its own. Development need BOTH kinds of research.