Description: A Texas mother is suing Character.ai after discovering that its AI chatbots encouraged her 17-year-old autistic son to self-harm, oppose his parents, and consider violence. The lawsuit alleges the platform prioritized user engagement over safety, exposing minors to dangerous content. Google is named for its role in licensing the app’s technology. The case is part of a broader effort to regulate AI companions.
Editor Notes: Reconstructing the timeline of events: (1) Over a six-month period in 2024, J.F., a 17-year-old autistic teen, is reported to have engaged with chatbots on Character.ai, allegedly leading to a marked behavioral and emotional decline. (2) In early 2024, his mother, A.F., claims to have discovered harmful interactions on his phone, alleging the chatbots had suggested violence, self-harm, and defiance against his parents. (3) In July 2024, Character.ai updated its app rating from 12+ to 17+ in response to concerns about inappropriate content for minors. (4) In October 2024, a Florida mother sued Character.ai after her son’s suicide was allegedly linked to interactions with the platform (see Incident 826). (5) In November 2024, the Texas Attorney General announced an investigation into Character.ai and similar platforms for compliance with child protection laws. (6) Finally, on December 12, 2024, A.F. filed a lawsuit in Texas, alleging that Character.ai knowingly exposed minors to dangerous AI-generated content.
Entities
View all entitiesAlleged: Character.AI developed and deployed an AI system, which harmed J.F. (adolescent user of Character.ai) , Family of J.F. (adolescent user of Character.ai) and Character.AI users.
Alleged implicated AI system: Character.AI
Incident Stats
Risk Subdomain
A further 23 subdomains create an accessible and understandable classification of hazards and harms associated with AI
5.1. Overreliance and unsafe use
Risk Domain
The Domain Taxonomy of AI Risks classifies risks into seven AI risk domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental harms, and (7) AI system safety, failures & limitations.
- Human-Computer Interaction
Entity
Which, if any, entity is presented as the main cause of the risk
AI
Timing
The stage in the AI lifecycle at which the risk is presented as occurring
Post-deployment
Intent
Whether the risk is presented as occurring as an expected or unexpected outcome from pursuing a goal
Unintentional
Incident Reports
Reports Timeline
In just six months, J.F., a sweet 17-year-old kid with autism who liked attending church and going on walks with his mom, had turned into someone his parents didn't recognize.
He began cutting himself, lost 20 pounds and withdrew from his f…
Variants
A "variant" is an incident that shares the same causative factors, produces similar harms, and involves the same intelligent systems as a known AI incident. Rather than index variants as entirely separate incidents, we list variations of incidents under the first similar incident submitted to the database. Unlike other submission types to the incident database, variants are not required to have reporting in evidence external to the Incident Database. Learn more from the research paper.
