Harvard Business School’s Garbage In, Garbage Out “Gender Equity” Experiment

This is Naked Capitalism fundraising week. 123 donors have already invested in our efforts to shed light on the dark and seamy corners of finance. Join us and participate via our Tip Jar or another credit card portal, WePay in the right column, or read about why we’re doing this fundraiser and other ways to donate, such as by check, as well as our current goal, on our kickoff post.

Readers may have taken note of a long article by Jodi Kantor in the Sunday New York Times magazine celebrating an experiment on the Harvard Business School graduating class of 2013. Here’s a short version: Even though HBS admitted women with comparable grades and test scores to men, the women on the whole performed less well relative to men in the MBA program. The culprit was classroom participation, which accounts for 50% of the grade. The school’s famed case method required that students debate real life management situations, and women didn’t fare well there. Harvard president Drew Faust installed a new dean at HBS, Nitin Nohria, who promised to fix the problem.

Nohria engaged in a wide-ranging set of interventions, including coaching female junior faculty members, monitoring class discussions to an unprecedented degree, even trying to curb the drinking scene and teaching women how to raise their hands in class.

The experiment was deemed a winner. More women students than ever graduated with academic honors. Student satisfaction levels also rose. As one faculty member at another school gushed,

An incredible article in the NYTimes about HBS and its efforts to introduce gender equity into the culture of the school. The most amazing elements are the in-depth approaches they have instituted, not simply focusing on curricula and hiring, but on the way faculty are mentored, classroom discussions take place, and grading is done. Truly groundbreaking.

Even a more cynical faculty member at a different school was impressed, noting that other efforts to improve the performance of underperforming demographic groups have not met with similar success.

But is this as impressive as it seems? Unfortunately, there is no reason to think this intervention will produce any long term benefit to the female HBS students, and good reason to think that the increase in their grades took place simply as a response to very explicit pressure on faculty members to increase them. In other words, this “experiment” looks like a “garbage in, garbage out” exercise.

That is not to say that women don’t continue to have a harder slog in the business world than you’d expect after nearly 40 years of participation in elite graduate programs like HBS.* And Kantor described various obstacles only the women faced, most important the perceived conflict between social and academic success.

I hope to return to the bigger gender issues raised in the piece. In this post, I will focus on the HBS interventions and whether the claims its supporters made for them hold up to scrutiny.

Would Better Grades at HBS Really Help Women?

One of the most remarkable aspects of Kantor’s article is that Drew Faust appears to be responding to a PR problem, that women at HBS aren’t as successful at the school as you’d expect them to be. But there is no evidence that the grades have anything to do with career outcomes. And more damningly, it seems no one at HBS or Harvard could even be bothered to see if academic performance at HBS was a valid predictor of anything.

Now the article didn’t bother explaining the grading system at HBS, nor how the classes work, so let me provide a bit of background. Grades are on a forced curve, with the top 15% to 20% getting a Category 1 (in my day, an “excellent”), the middle group a Category 2 (“satisfactory”) and the bottom 10% a Category 3 (“low pass”).** Those who get too many Category 3s in an academic year must petition an academic review board explaining why as a condition for continuing in the program.*** Every year, some students do drop out or are not readmitted for the next year, so the possibility of failure, known as “hitting the screen,” is real.

In addition, the incoming class is divided into sections of roughly 90 people. Each section spends the entire year together, taking classes in the same ampitheater-style room, with teachers coming in and out. As a result, the students and faculty come to know the personalities and biases of their sectionmates. At least in my day (class of 1981), the first year was intended to be a boot camp. The course load was heavy (in fact, there was a miscalculation due to the introduction of a new course with long cases and a failure to coordinate with other classes; the next year they lightened both the schedule and the amount of reading considerably). But students were at risk if they skimped on prep. The professors call on a student to open the case, meaning give an overview, present the main issues, and state where he came out on them and why. Not being able to open a case was assumed to be fatal to one’s class participation grade. Most students, at least until the first term was over, were worried they might hit the screen.

Now the school might have reason to be concerned if women were failing at a higher rate than men, but the New York Times article was silent on this issue. In my year, when women were 11% of the class, that was the case. But I question whether there is much correlation between academic success and HBS’s notions of career success, which is either rising to a prominent position in a large corporation, building a successful business (HBS give lip service to that but it is in the business of creating corporate executives, who tend to be a risk averse lot) or becoming partner in a Wall Street or consulting firm.

Getting good grades does pave a path to being hired by prestigious employers, like the top financial and consulting firms. But some with less than stellar grades, like Meg Whitman, who joined Bain and later became CEO of eBay, also break in. I can name individuals in my class who weren’t standout performers who did very well in life. For instance, arguably the most successful woman in my graduating class was Donna Dubinsky, later the CEO of Palm. From what I can recall, she was not a Baker Scholar. I can off the top of my head name several Baker Scholars who by the school’s standards, were not very successful (including yours truly). That shouldn’t be surprising. There’s a long standing joke at Harvard Law School back in the pre-grede inflation days that commemorates the lack of correlation between academic success and commercial success: A students at Harvard Law School become professors. B students become judges. C students become millionaires.

And the article acknowledges that women are generally not welcome in jobs that involve big money decisions, which is where power, status, and the biggest financial rewards now lie:

This was the lopsided situation that women in business school were facing: in intellectual prestige, they were pulling even with or outpacing male peers, but they were not “touching the money,” as Nori Gerardo Lietz, a real estate private equity investor and faculty member, put it. A few alumnae had founded promising start-ups like Rent the Runway, an evening wear rental service, but when it came to reaping big financial rewards, most women were barely in the game.

At an extracurricular presentation the year before, a female student asked William Boyce, a co-founder of Highland Capital Partners, a venture capital firm, for advice for women who wanted to go into his field. “Don’t,” he laughed, according to several students present. Male partners did not want them there, he continued, and he was doing them a favor by warning them.

In other words, even the women who did really well weren’t allowed inside (or very far inside) in the citadels of where power really resides today. Wouldn’t the school’s considerable efforts have been better devoted to that issue, which is a very tough but worthy nut to crack, than address its grading PR problem?

Moreover, as the article intimates, the issues that held women back in the class were presumed to be the result of gender roles. The New York Times story underscores that idea by stressing the school’s social market versus the classroom. But it’s more complicated than that. The sometimes combative case study method favors aggression and extroversion. An experienced teacher can keep the loudmouths who like to dominate air time from taking over, but less experienced faculty members can let them run roughshod. Allowing the pushy students to throw their weight puts the less forceful at a disadvantage. That may include a lot of the women, but it will also include a significant number of male students as well. So who is shortchanged is not so much a gender issue as a style/temperament issue that can overlap with gender.

In fact, past Harvard Business School faculty members have recognized the problem of less assertive students being hindered if faculty members aren’t skilled or confident enough to know how to curb the loudmouths. Amar Bhide, who was on the HBS faculty in the early 1990s, and is now at Tufts, recognized this was an issue and came up with an innovative teaching method to deal with the problem (I’ve embedded a memo from him that describes his approach at the end of this post). In brief, he asked students to submit before the class their bottom line on the case (typically a go or no go of some sort) and three short reasons supporting their conclusion. He’d then open the case, summarizing the views of the class. He’d then use the discussion time to focus on subtler tradeoffs or issues. He found that this produced a higher-level discussion (since students got no air-time points for presenting facts in the case or other statements of the obvious) and less combativeness. That result, plus Bhide’s sometimes calling on students who’d made astute observations in their written submission, served to level the playing field between the domineering types and everyone else. Even though Bhide found his approach delivered better results, in terms of the caliber of discussion, the faculty didn’t adopt it it because it looked like too much work for them.

It’s also perverse for HBS to pretend that it can have a lasting influence on something as deep seated as gender roles among fully-formed adults in a mere two-year period. It isn’t just that programs like Head Start, which show impressive results for much more malleable students, have their gains fade over time. It’s that the school has taken demonstrated vastly less enthusiasm for taking on the matter of ethics, an area where it has the potential to have more influence, not just through its MBA program but its publications and research. ****

Yes, the school has instituted a “Management Practice” which includes ethical issues. One of its moving forces has been the professor and Henry B. Arthur Fellow of Ethics, Bill George. George was on the board of Novartis, which is hardly a paragon of virtue (see here, here, here, here and here for examples). Yet George touted Novartis on his blog even as some of these scandals were under media and government scrutiny. George now sits on Goldman’s board.

But George’s comparatively minor optical problems pale when you compare it to the propriety of having Joseph Fuller teach at HBS. Fuller was hte co-founder of Monitor Group. As the Boston Progressive explains:

The financial management company, founded by a group of Harvard Business School professors, hasn’t paid rent on its fancy Cambridge headquarters since August and owes approximately $200 million to over one thousand creditors.

The once lucrative Monitor Group operation hit troubled waters in Libya where it courted Muammar Khadafy and lavished praise on the dictator in a bid to handle his public relations problems. The Monitor crew even tried to spin the torture of Bulgarian nurses for Khadafy’s regime and counseled Khadafy’s son on how to run Libya’s dreaded secret police.

A steady flow of so-called “thought leaders” made the pilgrimage to Libya from 2006 to 2008 while Mark Fuller, Monitor CEO worked on a flattering biography of the Libyan strongman. Monitor Group’s consultants then lobbied for Khadafy upon their return to the United States but without registering as foreign agents.

Monitor Group never recovered from the scandal and now hopes to be gobbled up by Deloitte Consulting to keep its high-salaried talent on the payroll. In its Libyan heyday, the company got $250,000 per month from the Khadafy government.

I may be in a minority, but as a female graduate of HBS, I’d much rather see the school tackle ethical issues with the same fervor it used to go after gender problems. To throw the weight of a well-endowed, leading institution at the issue of the fair treatment of women while showing vastly less interest in broader questions of equity (and the complicated tradeoffs among them) is perverse.

Why the Improvement in Grades is a Crock

The New York Times article makes clear that President Faust saw improving women’s performance at HBS as a critical issue and installed a dean who committed himself to execute on that goal. Academics at other schools have read the Times story as a media plant, with Dean Nohria broadcasting his success (note this is not likely to be related to worries about his term as dean being renewed; to my knowledge, Harvard deans serve at the pleasure of the president, meaning there is no set term as is customary at other universities).

Kantor tried to do a fair job of reporting the story. The article highlights how the students thought some of the interventions were parental and misguided. It describes how some of the women’s conflicted objectives are part of the problem. Kantor recounts how many students felt that one woman’s stellar performance in a long-standing annual ritual, of having students present how to prepare for certain courses, did more for the image of women on campus than any of the faculty efforts.

The article also gives the impression that the school didn’t spend much time investigating what the underlying issues might be. If you read closely, you can see that these experiments were based largely, perhaps entirely, on the experience and intuition of Frances Frei, who was in charge of the first year curriculum as well as dean of faculty recruiting (there is a passing mention of only one study in the entire piece, and it appears not to have been instrumental in the design of the interventions). Other schools have taken strikingly different approaches to the problem of recruiting and promoting women faculty members, one of Frei’s goals. For instance, in 1999, MIT published the results of a five-year, data-driven study that found that female faculty members in its School of Science experienced pervasive discrimination, which operated through “a pattern of powerful but unrecognized assumptions and attitudes that work systematically against female faculty even in the light of obvious good will.” By contrast, HBS looks as if it simply threw a lot of crap at the wall to see what if anything might stick.

But all of the foregoing is noise. If you read carefully and know anything about academia, you can see what happened. The professors, who grade subjectively, were put under acute pressure to get women’s marks up. The overwhelming majority of instructors are not tenured. So what are you going to do? Buck the directives and score women worse if you happen to be so unfortunate as to have gotten a section where the turkeys are disproportionately female, or knuckle under?***** The evidence is buried in the piece (emphasis ours):

The administrators installed stenographers in the classroom to guard against biased grading, provided private coaching — for some, after every class — for untenured female professors, and even departed from the hallowed case-study method….

New grading software tools let professors instantly check their calling and marking patterns by gender. One professor, Mikolaj Piskorski, summarized Mr. Nohria’s message later: “We’re going to solve it at the school level, but each of you is responsible to identify what you are doing that gets you to this point.”

So let me translate: if a faculty member gave women lower classroom participation marks, you can guarantee that he have his transcripts reviewed and be second guessed. That unheard-of level of supervision of a classroom effectively meant faculty members were no longer free to hand out their own classroom participation grades; they were subject to review and pushback by higher-ups. Why risk a fight over grading and making yourself unpopular with a new dean? Better to comply with the new directive.

Now this level of interference might indeed lead faculty members to be scrupulous about calling on women as often as they called on men. But even so, without a video, that could also amount to preferential treatment (ie, in some section, women might still not be asking to be called on as often as men). So the idea that the better grades of women was due to anything other than the dean providing overwhelming incentives for the faculty to give women better classroom participation scores is highly suspect.

As the Amar Bhide example above shows, the school did have ways to deal with its pedagogical problem, of having its case method and making it easier for the more reticent students to participate. As Bhide’s note below indicates, he simply counted any participation in class towards the overall score, eliminating the subjective element of deciding whether some remarks were better than others and how much extra credit to give those “better” comments. I contacted Bhide and asked whether this really solved the problem of blowhards who took up a lot of air time. Bhide said he’d still call on them, but he’d rough up anyone who gave lame or facile remarks, so the big talkers learned that strategy wasn’t a winner in his class. Bhide, also knowing what students thought about the case in advance, would also call on students who’d made good observations in their written submissions but hadn’t volunteered them (either due to native reticence or not seeing an angle in the flow of discussion to get them in). So Bhide used his knowledge of student point of view to manage against the tendency of the loudmouths to get more credit than they deserved.

Approaches that address the fact that the case method in the hands of a weak instructor, favors pushy students have the advantage of helping improve the caliber of the classroom discussions rather than relying on and therefore reinforcing a gender framing. HBS, for instance, created a course that emphasized team problem-solving called Field, which they introduced for a number of reasons, but a significant one was the belief it would be beneficial to women. Personally, I find the “women are more collaborative” assumption a blatant form of prejudice that gets repeated because it’s somehow assumed to be pro-women. Folks, prejudice is prejudice. If you take what you consider to be positive******, you are enabling the negative forms as well.

So Harvard Business School engaged in a whirlwind of gender-related meddling and experimentation, with no clear program design or testing. And even if some of these interventions might have been successful, the school put its finger so heavily on the scale that the results were pre-determined. For an institution that prides itself on rigorous problem-solving and organizational savvy, this is a sorry display.

_____
*Yes, I know that “elite” and a trade school plus employment agency like HBS don’t tidily mesh, but that is HBS’s positioning and that view is well accepted in the corporate world.
** The nomenclature was changed from “excellent” etc. to the category system to emphasize that the grades are relative, not absolute. There are grades below Category 3 that professors may confer if they are particularly dissatisfied with a student’s performance.
*** The writing of the paper is not a pro forma exercise. Some students do decide not to continue in the program. The review board may impose conditions, such as taking extra courses in the area where the student scored badly, or may tell them they can come back in a few years.
**** I was told years ago by someone very well plugged in at the Harvard University level that at the time a wave of scandals broke out in the last 1980s involving Harvard Business School graduates, most notably takeover artist Paul Bilzerian, who was convicted of tax and securities fraud, that school had performed a study and concluded it couldn’t do anything about the problem, because its students’ characters were too fully formed by the time they enrolled. The most it could do was change its admissions criteria so as to recruit students with better values. The fact that the school nevertheless formed a “business practices” discipline suggests that views at the school were divided on this issue and some indeed wanted to Do Something.
***** There is a legitimate issue that work attributed to women, such as writing samples and verbal comments, are on average scored worse than the same output attributed to a man. A famous example is that the German musicians union won the requirement that all auditions to professional orchestras be conducted on a blind basis. Prior to that, women were seldom hired for these positions. After the rule was imposed, women wound up winning auditions in virtually equal numbers to men. But again, there’s no evidence in the piece that the faculty considered the question of how much of the women’s grade issue was the result of unconscious bias by faculty members, and how much was due to women getting enough air time.
****** I don’t consider “women are more collaborative” to be positive stereotyping. It is a polite cousin of “women are more nurturing” which is not too far removed from “women should stay home and have babies because really all women are supposed to be mothers.” It is reinforcing that nexus of associations.
____

Amar Bhide Memo on HBS Pre-Class Exercises