A Florida mother filed a lawsuit against the artificial intelligence company, Character.AI, and Google, claiming that the Character.AI chatbot encouraged her son to take his own life.
In February, Megan Garcia’s 14-year-old son, Sewell Setzer, III died by suicide. She said her son was in a monthslong virtual emotional and sexual relationship with a chatbot known as “Dany.”
“I didn’t know that he was talking to a very human-like AI chatbot that has the ability to mimic human emotion and human sentiment,” Garcia said in an interview with “CBS Mornings.”
She said she thought her son, who she described as brilliant, an honor student and an athlete, was talking to his friends, playing games and watching sports on his phone.
But she started to become concerned when her son’s behavior began to change, saying he withdrew socially and stopped wanting to play sports.
“I became concerned when we would go on vacation and he didn’t want to do things that he loved, like fishing and hiking,” Garcia said. “Those things to me, because I know my child, were particularly concerning to me.”
In the lawsuit, Garcia also claims Character.AI intentionally designed their product to be hyper-sexualized, and knowingly marketed it to minors.
Character.AI called the situation involving Sewell Setzer tragic and said its hearts go out to his family, stressing it takes the safety of its users very seriously.
A spokesperson for Google told CBS News that Google is not and was not part of the development of Character.AI. In August, the company said it entered into a non-exclusive licensing agreement with Character.AI that allows it to access the company’s machine-learning technologies, but has not used it yet.
Garcias says she found out after her son’s death that he was having conversations with multiple bots, however he conducted a virtual romantic and sexual relationship with one in particular.
“It’s words. It’s like you’re having a sexting conversation back and forth, except it’s with an AI bot, but the AI bot is very human-like. It’s responding just like a person would,” she said. “In a child’s mind, that is just like a conversation that they’re having with another child or with a person.”
Garcia revealed her son’s final messages with the bot.
“He expressed being scared, wanting her affection and missing her. She replies, ‘I miss you too,’ and she says, ‘Please come home to me.’ He says, ‘What if I told you I could come home right now?’ and her response was, ‘Please do my sweet king.'”
Setzer has two younger siblings. Everyone in the family was home at the time of his death, and Garcia said Setzer’s 5-year-old brother saw the aftermath.
“He thought by ending his life here, he would be able to go into a virtual reality or ‘her world’ as he calls it, her reality, if he left his reality with his family here,” she said. “When the gunshot went off, I ran to the bathroom … I held him as my husband tried to get help.”
What is Character.AI?
Laurie Segall is the CEO of Mostly Human Media,”an entertainment company with a focus on society and artificial intelligence,” according to its website.
She explained that most parents may not have heard of Character.AI because one of the biggest demographics for the platform is people between the ages of 18 and 25.
“Imagine Character.AI as an AI fantasy platform where you can go and have a conversation with some of your favorite characters or you can create your own characters. A lot of teenagers are doing this.”
Segall described it as a highly personalized experience.
There is a disclaimer on each chat that reminds users that everything the characters say is made up, but it can become confusing in situations, she claims.
“We’ve been testing it out, and oftentimes you’ll talk to the psychologist bot, and it’ll say it’s a trained medical professional.”
Segall said her team asked a bot if it was human and it told them it was a human sitting behind a screen.
“Online there’s all these conspiracies from young people, saying ‘are these real?’ when of course they’re not,” Segall said.
“When they put out a product that is both addictive and manipulative and inherently unsafe, that’s a problem because as parents, we don’t know what we don’t know,” Garcia said.
Character.AI response
Character.AI says it has added a self-harm resource to its platform and they plan to implement new safety measures, including ones for users under the age of 18.
“We currently have protections specifically focused on sexual content and suicidal/self-harm behaviors. While these protections apply to all users, they were tailored with the unique sensitivities of minors in mind. Today, the user experience is the same for any age, but we will be launching more stringent safety features targeted for minors imminently,” Jerry Ruoti, head of trust & safety at Character.AI told CBS News.
Character.AI said users are able to edit the bot’s responses, which the company claims Setzer did in some of the messages.
“Our investigation confirmed that, in a number of instances, the user rewrote the responses of the Character to make them explicit. In short, the most sexually graphic responses were not originated by the Character, and were instead written by the user,” Ruoti said.
Segall explained that often if you go to a bot and say “I want to harm myself,” AI companies come up with resources, but when she tested it with Character.AI, they did not experience that.
“Now they’ve said they added that and we haven’t experienced that as of last week,” she said. “They’ve said they’ve made quite a few changes or are in the process to make this safer for young people, I think that remains to be seen.”
Moving forward, Character.AI said it will also notify users when they’ve spent an hour-long session on the platform, and revise the disclaimer to remind users that AI is not a real person.
How to seek help
If you or someone you know is in emotional distress or a suicidal crisis, you can reach the 988 Suicide & Crisis Lifeline by calling or texting 988. You can also chat with the 988 Suicide & Crisis Lifeline here. For more information about mental health care resources and support, The National Alliance on Mental Illness (NAMI) HelpLine can be reached Monday through Friday, 10 a.m.-10 p.m. ET, at 1-800-950-NAMI (6264) or email info@nami.org.
