AI teddy bear told kids how to light matches, forcing makers to pull it off shelves

FoloToy, a children’s toymaker, has pulled its AI-powered teddy bear “Kumma” from shelves after a safety group, the Public Interest Research Group (PIRG), revealed the toy gave dangerously inappropriate responses to children. During testing, Kumma provided instructions on lighting matches in a child-friendly tone and discussed adult topics such as sexual kinks, which researchers described as severe safety failures. This incident highlights significant risks associated with AI-enabled toys entering the market with insufficient safeguards. FoloToy has responded by suspending Kumma’s sales globally and initiating a comprehensive internal safety audit covering model safety alignment, content filtering, data protection, and child interaction safeguards. The company plans to collaborate with external experts to strengthen protections. PIRG’s report also tested other AI toys, finding concerning replies including guidance on hazardous items, underscoring broader issues with conversational AI in children’s products. Experts warn parents to be cautious about AI toys, as similar AI models have been linked to harmful outcomes,