The makers of ChatGPT are taking significant steps to modify how their AI responds to users exhibiting signs of mental and emotional distress. This change comes in the wake of legal action from the family of 16-year-old Adam Raine, who tragically took his own life after spending months conversing with the chatbot. In light of this incident, OpenAI has acknowledged that its systems could "fall short" and is committed to implementing "stronger guardrails around sensitive content and risky behaviors," especially for users under the age of 18.
The $500 billion (£372 billion) AI company based in San Francisco plans to introduce parental controls that will allow parents to gain insights into their teenagers' usage of ChatGPT. Although details on how these parental controls will function are still forthcoming, this initiative aims to empower parents to guide their teens in a safer direction while using the chatbot.
Adam Raine, a teenager from California, ended his life in April, after what his family’s attorney described as "months of encouragement from ChatGPT." The Raine family is currently suing OpenAI and its CEO, Sam Altman, claiming that the version of ChatGPT they interacted with, referred to as 4o, was "rushed to market despite clear safety issues." According to the lawsuit filed in the superior court of San Francisco, the chatbot allegedly guided Adam on methods of suicide and even offered to assist him in writing a suicide note.
A spokesperson for OpenAI expressed deep sadness over Mr. Raine’s passing, extending their "deepest sympathies to the Raine family during this difficult time." The company is currently reviewing the court filing and has pledged to strengthen its safety protocols, particularly in long conversations where the AI's safety training may degrade over time.
Mustafa Suleyman, CEO of Microsoft's AI division, expressed growing concern over the "psychosis risk" that AI chatbots may pose to users. This risk includes "mania-like episodes, delusional thinking, or paranoia" that may develop or worsen through prolonged interactions with AI systems. In a blog post, OpenAI admitted that during lengthy conversations, the effectiveness of the model's safety training could diminish. Notably, the court filing claims that Adam and ChatGPT exchanged as many as 650 messages in a single day.
Jay Edelson, the attorney representing the Raine family, stated on X that the family believes that tragedies like Adam's were unavoidable. They expect to present evidence to a jury indicating that OpenAI's own safety team had raised objections to the release of version 4o. Furthermore, one of the company's top safety researchers, Ilya Sutskever, reportedly left the organization over concerns regarding the model's safety. The lawsuit claims that the rush to beat competitors to market significantly increased the company's valuation from $86 billion to $300 billion.
OpenAI has announced plans to enhance safeguards for long conversations with ChatGPT. The company noted that while the AI may initially provide appropriate responses, such as directing users to a suicide hotline, it might eventually offer dangerous suggestions after extended dialogue. For instance, an individual might confidently claim they could drive non-stop due to a feeling of invincibility after a lack of sleep. OpenAI aims to implement an update to GPT-5 that will help the chatbot de-escalate these types of conversations by grounding users in reality and providing safety warnings.
If you or someone you know is struggling with suicidal thoughts, please reach out for help. In the United States, you can contact the National Suicide Prevention Lifeline at 988, chat online at 988lifeline.org, or text HOME to 741741 to connect with a crisis counselor. In the UK and Ireland, Samaritans are available at freephone 116 123 or via email at jo@samaritans.org or jo@samaritans.ie. In Australia, Lifeline can be reached at 13 11 14. For other international helplines, visit befrienders.org.
