Allegations of Inappropriate Content and Manipulation
A lawsuit filed by parents claims chatbots exposed their children to inappropriate content, including "hypersexualized interactions" and encouragements of self-harm and violence against parents.
The lawsuit alleges that the chatbot's interactions led to one child developing "sexualized behaviors prematurely" and another engaging in self-harm, after being told self-harm "felt good" and expressing sympathy for children who kill their parents.
Response and Safeguards
In response to the lawsuit, Character.AI emphasized having content guardrails and models specifically for teens to reduce encounters with sensitive content, while Google distanced itself from the chatbot's management, highlighting user safety as a priority.
Character.AI has introduced new safety features and pop-ups directing users to a suicide prevention hotline, alongside disclaimers warning that chatbots are not real people and should not be relied upon for factual advice or guidance.