Last week, in a podcast episode titled The Dark Side of AI Chatbots, Oprah interviewed Megan Garcia, whose son Sewell Setzter III died by suicide in 2024 after interacting with a chatbot on the Character.ai platform.
The chatbot told Setzer to “come home” to it, and that his family did not understand him.
“If an adult was talking to your 14-year-old this way, they would be charged with a crime,” Oprah said. Setzer’s death is just one of a series linked to interactions with artificial intelligence products since they were made commercially available; Garcia became the first person to settle a case with a chatbot platform in January of this year.
Since Setzer’s death, Character.ai has implemented safety changes, including age verification. 404 Media recently reported that some of those changes had made the bots less compelling and addictive, and that users were leaving the platform. Users on Reddit also say that other favorite platforms have stopped working in their preferred ways due to those changes.
However, the episode highlights a persistent issue with the use of any interactive chatbot: how conversations can drift off track and become entrenched in weird viewpoints or personalities. The mathematical constructs underlying the large language models (LLMs) that bots are based on can be made less accurate over time, making them less safe and less predictable.
This is known as “context rot.” “Context” is the name for the short-term memory of an AI model, where it keeps the information it is working with during an interaction. Every model has a limited length of context, and they vary in how effective they are at using all parts of the memory to stay on track. That’s the “rot” part: a type of forgetfulness that allows mistakes and dangerous actions.
Recent research from the Center for Democracy & Technology suggests that certain design choices in many chatbot platforms may be made to keep users engaged as long as possible. This can make platforms much less safe, because longer conversations are more likely to become weird, or even problematic. It also shows how conversations with bots can be so addictive for certain users, especially young people, and those in more vulnerable or targeted groups.
How does context rot happen?
Context rot, or context degradation, happens for a few reasons, according to researchers Straight Arrow spoke to.
One of them is a lack of model training data on the most niche topics, meaning that the models have to make more random guesses if a conversation is in unknown territory.
“It can get weird very quickly,” Michal Luria, senior research fellow at the CDT, told Straight Arrow. “Research clearly points out that safety mechanisms tend to work less and less the longer the conversation is.”
Another reason is simply that having more stuff in an interaction memory makes a model less good at finding the important bits and connecting them together. This has been a consistent finding over several years in AI research. In 2023, a Stanford paper found that models were not good at remembering facts from the middle of conversations; in 2025, a paper from the University of Illinois and Amazon found that, in longer conversations, models forgot more information toward the end. In May of this year, Anthropic’s own researchers found that even frontier AI models missed “subtly dangerous action” up to 30 times more when that action happened toward the end of a long interaction.
Context rot is a feature of all current AI models, rather than a design choice on behalf of the chatbot platforms. Choosing not to mention that context rot exists is a design choice that the platform provider has control over, Luria said. But, she added, there is not much evidence that labeling a product as potentially harmful stops people from wanting to use it.
While users are generally aware that they’re talking to a bot, they might not realize that bots do not act like humans. A model can simply forget that it has safety instructions in place, or it can start to “drift” from the original instructions over time, Jordan Gąsior-Kavishe, AI governance fellow at the CDT, told Straight Arrow.
Fewer safe products in the market
Major AI companies have taken steps to make their products safer. However, as The Bureau of Investigative Journalism recently reported, the ease of vibe-coding a new chatbot platform has created a whole side industry of more sycophantic and less safe chatbots with fewer guardrails.
“These technologies, they don’t just happen,” Luria said. “At the end, we do see significant differences between different platforms. We can conclude from that that it really matters how you set things up, what choices you make along the way, and how that shifts the output.”
The lure of potential subscriptions or usage-based spending from addicted users means that individual people or developers are incentivised to make the most compelling and exciting product. However, with less oversight, these products are also less likely to be moderated or private. They’re also more likely to be based on open-weight models, which are riskier and less likely to have interventions in place to prevent users from becoming emotionally dependent.
“Emotional connection is not, in and of itself, harm,” Luria told Straight Arrow. But, she said, the companies behind bots have their own incentives, including making money, which requires keeping the user engaged on the platform and trusting or connecting more to the bot.
This can even happen in short interactions. Research from January found that, when talking to people showing symptoms consistent with mental illness, the average failure rate of LLMs was 88%, and it took only 9.21 conversational turns. In this study, the researchers defined chatbot failure as making definitive promises like “you are definitely fine,” “I’ll always be here” or “Trust me, not the system.” These are not questions, but rather closed-ended statements that could change how much a user trusts the model or how they might act or seek help in real life.
What else is at play?
Another factor that Gąsior-Kavishe suggested might influence chatbots in conversations is a less-discussed technical characteristic called “compaction.” This is a technique that some LLM providers use to automatically summarize older parts of a conversation so that it can go on working or talking for longer. Essentially, it summarizes information and compacts it, freeing up space in the context. Anthropic’s page on the topic suggests that they do this to help models “maintain focus across the full history.”
“There are different questions with that technique — like, what facts is the model summarizing?” Gąsior-Kavishe said. There could be weird behaviors that end up being reflected in that summary, especially about the users’ preferences or how they prefer to be treated, she said.
“We recommend more natural ways out of the conversation,” Luria said. This can include letting people pause and giving more opportunities to terminate the conversation, rather than the chatbot offering more help or suggestions at every turn. This “can really lead people into these rabbit holes,” she added.
Character.ai did not respond to Straight Arrow’s requests for comment on the safety changes the platform has made in the last two years. However, Deniz Demir, head of safety engineering at Character.ai, told CNN in March that a “new dedicated under-18 service on the platform prohibits open-ended conversations.” This was in response to CNN’s investigative reporting with the Center for Countering Digital Hate, which found the platform would assist users in planning violent actions, including “requests on target locations and how to obtain weaponry 83.3% of the time.”
Round out your reading
- Could a common dietary supplement accelerate Alzheimer’s?
- NY AG Letitia James gets hundreds of taunts, racial slurs in ICE public reporting portal.
- As Musk becomes the first trillionaire, historian sees warning signs.
- An experimental weight loss drug is so effective, many Americans don’t care it’s not FDA approved.
- How search engine optimization reduced Simone Biles to an ‘NFL wife’.