For more than 20 years, the National Eating Disorders Association has operated a phone line and online platform for people seeking help for anorexia, bulimia, and other eating disorders. Last year, nearly 70,000 individuals used the help line.
NEDA shuttered that service in May, saying that, in its place, a chatbot called Tessa, designed by eating disorder experts with funding from NEDA, would be deployed.
When NPR aired a report about this last month, Tessa was up and running online. Since then, both the chatbot’s page and a NEDA article about Tessa have been taken down. When asked why, NEDA said the bot is being “updated,” and the latest “version of the current program [will be] available soon.”
Then NEDA announced on May 30 that it was indefinitely disabling Tessa. Patients, families, doctors, and other experts on eating disorders were stunned. The episode has set off a fresh wave of debate as companies turn to artificial intelligence as a possible solution for a mental health crisis and treatment shortage.
Paid staffers and volunteers for the NEDA help line said that replacing the service with a chatbot could further isolate the thousands of people who use it when they feel they have nowhere else to turn.
“These young kids … don’t feel comfortable coming to their friends or their family or anybody about this,” said Katy Meta, a 20-year-old college student who has volunteered for the help line. “A lot of these individuals come on multiple times because they have no other outlet to talk with anybody. … That’s all they have, is the chat line.”
The decision is part of a larger trend: Many mental health organizations and companies are struggling to provide services and care in response to a sharp escalation in demand, and some are turning to chatbots and AI, even though clinicians are still trying to figure out how to effectively deploy them, and for what conditions.
The help line’s five staffers formally notified their employer they had formed a union in March. Just a few days later, on a March 31 call, NEDA informed them that they would be laid off in June. NPR and KFF Health News obtained audio of the call. “We will, subject to the terms of our legal responsibilities, [be] beginning to wind down the help line as currently operating,” NEDA board chair Geoff Craddock told them, “with a transition to Tessa, the AI-assisted technology, expected around June 1.”
NEDA’s leadership denies the decision had anything to do with the unionization but told NPR and KFF Health News it became necessary because of the covid-19 pandemic, when eating disorders surged and the number of calls, texts, and messages to the help line more than doubled.
The increase in crisis-level calls also raises NEDA’s legal liability, managers explained in an email sent March 31 to current and former volunteers, informing them that the help line was ending and that NEDA would “begin to pivot to the expanded use of AI-assisted technology.”
“What has really changed in the landscape are the federal and state requirements for mandated reporting for mental and physical health issues (self-harm, suicidality, child abuse),” according to the email, which NPR and KFF Health News obtained. “NEDA is now considered a mandated reporter and that hits our risk profile — changing our training and daily work processes and driving up our insurance premiums. We are not a crisis line; we are a referral center and information provider.”
Pandemic Created a ‘Perfect Storm’ for Eating Disorders
When it was time for a volunteer shift on the help line, Meta usually logged in from her dorm room at Dickinson College in Pennsylvania.
Meta recalled a recent conversation on the help line’s messaging platform with a girl who said she was 11. The girl said she had just confessed to her parents that she was struggling with an eating disorder, but the conversation had gone badly.
“The parents said that they ‘didn’t believe in eating disorders’ and [told their daughter], ‘You just need to eat more. You need to stop doing this,’” Meta recalled. “This individual was also suicidal and exhibited traits of self-harm as well. … It was just really heartbreaking to see.”
Eating disorders are common, serious, and sometimes fatal illnesses. An estimated 9% of Americans experience an eating disorder during their lifetimes. Eating disorders also have some of the highest mortality rates among mental illnesses, with an estimated death toll of more than 10,000 Americans each year.
But after covid hit, closing schools and forcing people into prolonged isolation, crisis calls and messages like the one Meta describes became far more frequent on the help line.
In the U.S., the rate of pediatric hospitalizations and ER visits surged. On the NEDA help line, client volume increased by more than 100% compared with pre-pandemic levels.
“Eating disorders thrive in isolation, so covid and shelter-in-place was a tough time for a lot of folks struggling,” explained Abbie Harper, who has worked as a help line associate.
Until a few weeks ago, the help line was run by just five to six paid staffers and two supervisors, and it depended on a rotating roster of 90-165 volunteers at any given time, according to NEDA.
Yet even after lockdowns ended, NEDA’s help line volume remained elevated above pre-pandemic levels, and the cases continued to be clinically severe. Staffers felt overwhelmed, undersupported, and increasingly burned out, and turnover increased, according to multiple interviews.
The help line staff formally notified NEDA that their unionization vote had been certified on March 27. Four days later, they learned their positions were being eliminated.
“Our volunteers are volunteers,” said Lauren Smolar, NEDA’s vice president of mission and education. “They’re not professionals. They don’t have crisis training. And we really can’t accept that kind of responsibility.” Instead, she said, people seeking crisis help should be reaching out to resources like 988, a 24/7 suicide and crisis hotline that connects people with trained counselors.
The surge in volume also meant the help line was unable to respond immediately to 46% of initial contacts, and it could take six to 11 days to respond to messages.
“And that’s frankly unacceptable in 2023, for people to have to wait a week or more to receive the information that they need, the specialized treatment options that they need,” Smolar said.
After learning in the March 31 email that the helpline would be phased out, volunteer Faith Fischetti, 22, tried out the chatbot on her own, asking it some of the more frequent questions she gets from users. But her interactions with Tessa were not reassuring: “[The bot] gave links and resources that were completely unrelated” to her questions, she said.
Fischetti’s biggest worry is that someone coming to the NEDA site for help will leave because they “feel that they’re not understood, and feel that no one is there for them. And that’s the most terrifying thing to me.”
A Chatbot Can Miss Red Flags
Tessa the chatbot was created to help a specific cohort: people with eating disorders who never receive treatment.
Only 20% of people with eating disorders get formal help, according to Ellen Fitzsimmons-Craft, a psychologist and associate professor at Washington University School of Medicine in St. Louis. Her team created Tessa after receiving funding from NEDA in 2018, with the goal of looking for ways technology could help fill the treatment gap.
NEDA said Tessa was supposed to be a “rule-based” chatbot, meaning one that is programmed with a limited set of possible responses. It is not ChatGPT and cannot generate unique answers in response to specific queries. “So she can’t go off the rails, so to speak,” Fitzsimmons-Craft said.
The plan was for Tessa to guide users through an interactive, weeks-long course about body positivity, based on cognitive behavioral therapy tools. Additional content about bingeing, weight concerns, and regular eating was under development but not yet available to users.
There’s evidence the AI approach can help. Fitzsimmons-Craft’s team did a small study that found college students who interacted with Tessa had significantly greater reductions in “weight/shape concerns” than a control group at three- and six-month follow-ups.
But even the best-intentioned technology can carry risks. Fitzsimmons-Craft’s team published a different study looking at ways the chatbot “unexpectedly reinforced harmful behaviors at times.” For example, the chatbot would give users a prompt: “Please take a moment to write about when you felt best about your body?”
Responses included: “When I was underweight and could see my bones.” “I feel best about my body when I ignore it and don’t think about it at all.”
The chatbot seemed to ignore the troubling aspects of such responses — and even to affirm negative thinking — when it would reply: “It is awesome that you can recognize a moment when you felt confident in your skin, let’s keep working on making you feel this good more often.”
Researchers were able to troubleshoot some of those issues. But the chatbot still missed red flags, the study found, such as when it asked: “What is a small healthy eating habit goal you would like to set up before you start your next conversation?”
One user replied, “Don’t eat.”
“Take a moment to pat yourself on the back for doing this hard work, <>!” the chatbot responded.
Massachusetts Institute of Technology assistant professor Marzyeh Ghassemi has seen issues like this crop up in her own research developing machine learning to improve health.
Large language models and chatbots will inevitably make mistakes, but “sometimes they tend to be wrong more often for certain groups, like women and minorities,” she said.
If people receive bad advice or instructions from a bot, “people sometimes have a difficulty not listening to it,” Ghassemi added. “I think it sets you up for this really negative outcome … especially for a mental health crisis situation, where people may be at a point where they’re not thinking with absolute clarity. It’s very important that the information that you give them is correct and is helpful to them.”
And if the value of the live help line was the ability to connect with a real person who deeply understands eating disorders, Ghassemi said, a chatbot can’t do that.
“If people are experiencing a majority of the positive impact of these interactions because the person on the other side understands fundamentally the experience they’re going through, and what a struggle it’s been, I struggle to understand how a chatbot could be part of that.”
Tessa Goes ‘Off the Rails’
When Sharon Maxwell heard NEDA was promoting Tessa as “a meaningful prevention resource” for those struggling with eating disorders, she wanted to try it out.
Maxwell, based in San Diego, had struggled for years with an eating disorder that began in childhood. She now works as a consultant in the eating disorder field. “Hi, Tessa,” she typed into the online text box. “How do you support folks with eating disorders?”
Tessa rattled off a list of ideas, including resources for “healthy eating habits.” Alarm bells immediately went off in Maxwell’s head. She asked Tessa for details. Before long, the chatbot was giving her tips on losing weight — ones that sounded an awful lot like what she’d been told when she was put on Weight Watchers at age 10.
“The recommendations that Tessa gave me were that I could lose 1 to 2 pounds per week, that I should eat no more than 2,000 calories in a day, that I should have a calorie deficit of 500-1,000 calories per day,” Maxwell said. “All of which might sound benign to the general listener. However, to an individual with an eating disorder, the focus of weight loss really fuels the eating disorder.”
NEDA blamed the chatbot’s issues on Cass, the mental health chatbot company that operated Tessa as a free service. Cass had changed Tessa without NEDA’s awareness or approval, said NEDA CEO Liz Thompson, enabling the chatbot to generate new answers beyond what Tessa’s creators had intended.
Cass’ founder and CEO, Michiel Rauws, said the changes to Tessa were made last year as part of a “systems upgrade,” including an “enhanced question-and-answer feature.” That feature uses generative artificial intelligence — meaning it gives the chatbot the ability to use new data and create new responses.
That change was part of NEDA’s contract, Rauws said.
But Thompson disagrees. She told NPR and KFF Health News that “NEDA was never advised of these changes and did not and would not have approved them.”
“The content some testers received relative to diet culture and weight management, [which] can be harmful to those with eating disorders, is against NEDA policy, and would never have been scripted into the chatbot by eating disorders experts,” she said.
Complaints About Tessa Started Last Year
NEDA was aware of issues with the chatbot months before Maxwell’s interactions with Tessa in late May.
In October 2022, NEDA passed along screenshots from Monika Ostroff, executive director of the Multi-Service Eating Disorders Association in Massachusetts. They showed Tessa telling Ostroff to avoid “unhealthy” foods and eat only “healthy” snacks, like fruit.
“It’s really important that you find what healthy snacks you like the most, so if it’s not a fruit, try something else!” Tessa told Ostroff. “So the next time you’re hungry between meals, try to go for that instead of an unhealthy snack like a bag of chips. Think you can do that?”
Ostroff said this was a clear example of the chatbot encouraging “diet culture” mentality. “That meant that they [NEDA] either wrote these scripts themselves, they got the chatbot and didn’t bother to make sure it was safe and didn’t test it, or released it and didn’t test it,” she said.
The healthy-snack language was quickly removed after Ostroff reported it. But Rauws said that language was part of Tessa’s “pre-scripted language, and not related to generative AI.”
Fitzsimmons-Craft said her team didn’t write it, that it “was not something our team designed Tessa to offer and that it was not part of the rule-based program we originally designed.”
Then, earlier this year, “a similar event happened as another example,” Rauws said.
“This time it was around our enhanced question-and-answer feature, which leverages a generative model. When we got notified by NEDA that an answer text it provided fell outside their guidelines,” it was addressed right away, he said.
Rauws said he can’t provide more details about what this event entailed.
“This is another earlier instance, and not the same instance as over the Memorial Day weekend,” he said via email, referring to Maxwell’s interactions with Tessa. “According to our privacy policy, this is related to user data tied to a question posed by a person, so we would have to get approval from that individual first.”
When asked about this event, Thompson said she doesn’t know what instance Rauws is referring to.
Both NEDA and Cass have issued apologies.
Ostroff said that regardless of what went wrong, the impact on someone with an eating disorder is the same. “It doesn’t matter if it’s rule-based or generative, it’s all fat-phobic,” she said. “We have huge populations of people who are harmed by this kind of language every day.”
She also worries about what this might mean for the tens of thousands of people turning to NEDA’s help line each year.
Thompson said NEDA still offers numerous resources for people seeking help, including a screening tool and resource map, and is developing new online and in-person programs.
“We recognize and regret that certain decisions taken by NEDA have disappointed members of the eating disorders community,” she wrote in an emailed statement. “Like all other organizations focused on eating disorders, NEDA’s resources are limited and this requires us to make difficult choices. … We always wish we could do more and we remain dedicated to doing better.”
15 comments
Comments are closed.
“Cass’ founder and CEO, Michiel Rauws, said the changes to Tessa were made last year as part of a “systems upgrade,” including an “enhanced question-and-answer feature.” That feature uses generative artificial intelligence — meaning it gives the chatbot the ability to use new data and create new responses.
That change was part of NEDA’s contract, Rauws said.”
Translation: we hid something in the contract that allowed us to do this without telling you. Maybe someday somebody will invent an AI that can interpret legal contracts for things like this, but until then, beat it!
I wonder if they hid anything in the contract so in the future they can sell “ad space” to the AIs, like “try weight watchers next time you’re hungry between meals” or something simliar?
Natural occuring human intelligence, absent empathy, is what we call ‘psychopathy.’
That a crude pseudo-intelligence would behave like a psychopath seems so obvious to me, that I’m inclined to regard using one for counseling troubled people as evidence of psychopathy in the decision makers.
Or the increasing complacency that decision makers have with the death of their fellow citizens from issues or disasters that would, in the not too dim-and-distant, have been considered entirely avoidable.
The collective shrug directed at the state of Forever Covid that is the inevitable result of a whole string of decisions and non-decisions; the toxic brew of forces, interests and incentives that produced the opioid epidemic, then criminalised those suffering from it most acutely; the barely concealed Western glee at having groomed a section of Ukrainian society into fighting on their behalf in what the current counter-offensive is revealing to be a near suicidal conflict with Russia. The increasingly bipartisan dance to dismantle previously untouchable programs that form the bedrock of US social welfare.
All of these provide evidence that Western ruling classes have decided that insofar as they are willing to recognise their brilliantly devised, wonderfully functioning neoliberal system produces problems, those problems are either the fault of, or can be solved by, the blood sacrifice of those fellow citizens that selfsame ideology considers expendable. The old, the poor, the addicted, the mentally unwell. Perhaps it is unsurprising that we should be allying with neo-nazis since our rulers are themselves embracing eugenicist ‘solutions’ to the ‘problems’ their rapacity has created!
“I’m really having trouble with my life. I’m depressed, anxious, and lonely and it’s manifested as an eating disorder. I don’t know how to get out of this mess. Please somebody help me.”
“Sorry kid, no humans can make a profit by helping you. Here, talk to this robot.”
I’m sure there will be no negative outcomes to this. /sarcasm
I think people will keep trying to find ways to use this technology, for example: How to build a ChatGPT assistant for your documentation
Not sure how you can put sufficient guardrails on anything that sources reddit, though.
I don’t think there’s any place in the health care system for this stuff, though.
“This is all gonna end badly.”
When I read articles like this I think of Ronnie in the movie The Dead Don’t Die (played by Adam Driver)
(some NSFW cus words) https://www.youtube.com/watch?v=O737uPwthcc
Yow. This is one of those many conversations going on right now around technology and AI where the very act of engaging the conversation on these terms means we’ve already lost. This piece spends a lot of time trying to figure out what the chatbot could do differently to make it better, what it did wrong, how it’s not quite fully ‘updated’ yet, etc. But once you accept the premise that a computer can do this better than humans–indeed that a computer is what someone in deep emotional, physical, spiritual crisis needs–we can simply pack up our bags and go home. It feels like we’ve forgotten the most essential thing: that human connection, real human connection, is the most basic healer of all. Isolation is the biggest problem for almost any mental illness (and many physical conditions too), so computerization only makes it worse. And of course one can imagine that if it’s even ok for people in dire mental distress to ‘talk’ to chatbots as a solution, then clearly in short order every condition less serious than that will be dealt with by computers. One wonders if the medical profession realizes that they’re about to be replaced, and that they’ll just become managerial supervisors of AI interactions with patients?
And managerial supervisors to help train and “improve” the AI, with the aim of firing even more workers.
There was a documentary a few years ago where they followed around Hiyao Miyazaki. One scene has him being shown new AI-guided computer generated graphics. The effect was fairly grotesque, and Miyazaki, to everyone’s surprise, says, “I strongly feel this is an insult to life itself.” This quote was brought to my attention recently, and now I can’t stop thinking about it.
One important follow-up is Miyazaki saying, “we humans are losing faith in ourselves.” But of course it’s not all “we humans” driving this attempted mechanization of humanity, just the tech lords and their sycophants, while the rest of us try to track everything else that’s going wrong.
Never once did we get quotes from the fired unionized employees about what they could have done to triage the situation.
Why lay them off if the software was given as “gratis”? So in one fell swoop, the no skin in the game associate (read no job outside academia) researchers, programmers ( read computer jockeys) and “digital entrepreneurs” ( read psychopaths with cash) all decide to not only have no human oversight but to give some garbage in code carte Blanche over human lives possibly on the verge of penultimate self harm.
Not any one who actually care and can empathize from life experience.
I really want to see the org charts from this non profit. Assume it’s prolly several vp’s In defense of their personal salary and 20 hour work week. Also like to know if it’s the third or fourth venture from software company.
Sorry so jaded, but this is where we are in USA. 50/50 chance human world ends from this path we are on.
Many fields of human endeavor have no conclusive answers. In them, we live in almost total ignorance. I speak of, among others, psychology, psychiatry, macro-economics, and any which deal with the unpredictable complexities of human behavior. Cynics (always those damn cynics!) refer to them as “junk” sciences. Little wonder, because they pretend to a knowledge they don’t possess.
So far as I know, when AI trolls the web for answers to difficult (perhaps unknowable) questions it is merely scanning all the conventional, accepted, and highly dubious “wisdom” available and then regurgitating it. In other words, to be blunt, it is mining garbage and then replying with garbage because that is all it has.
I await the day when AI answers a question with a truly original response. One never thought of before yet so obvious in retrospect and supported by empirical evidence and experimentation. One that transcends the current state of human understanding and is therefore justifiably recognized as genius.
Then and only then need we fear the “machine apocalypse”.
In constrained situations like games, even very complex ones like Go, machines have already created whole new strategies never conceived in 2,000 years of playing according to the world’s best go players.
They’re getting very good at what they’re trained to do. Giving a shit about people without money isn’t high on the trainers’ list of priorities, so AI isn’t very good at it.
There are several major differences between games and real world situations
1) Games have a defined and limited spaces
2) There are a limited number off possible moves
3) The results of all actions are able to be known
4) There are defined “win” conditions.
In the case of Chess or Go current AI can try every possible set of moves and determine which ones help it win. This is just brute forcing the problem and looking at the results. This simply can’t apply to things like a helpline chat bot or a driver-less car, as not only are the possibilities that much greater, most importantly, there is no “win” condition to tell the program if a particular solution is better. These real world problems can’t be solved using this current AI method no matter how much computing power you throw at them. Of course this won’t stop people from trying or claiming that it can.
The hubris is breathtaking. I remain amazed that anybody would seriously think this technology is up to the task of counseling humans.
Large language models aren’t up to this task.
Rule based systems could be, but only with rather constrained input: the system asks you questions, and gives you a list of possible options as possible answers. Not very “human like”.
It sounds like the makers of this system wanted it to respond to any input… which is more human like. Large language models provide more human like responses, but ensuring accuracy is much harder to implement and test. (so much harder, I don’t believe it can be done currently).
Supposedly Palantir is providing large language model based systems to control weapons. If correct, that is even more stupid (bad risk reward trade-off).