#57 | That cute AI teddy bear got banned

TL;DR: 150+ child development experts issued a warning about AI toys ahead of Black Friday. I went down the rabbit hole. Here’s what I found—and what keeps nagging at me.

👋 Happy (Black) Friday,

An AI teddy bear got kicked off the internet this month.

That sentence is already a bit strange, right?

The suspect is Kumma—a $99 plush made by Singapore-based FoloToy—that runs on OpenAI’s GPT-4o. The pitch? Friendly conversations. Educational storytelling. Warmth and curiosity. All very wholesome, well, at least on paper.

Until researchers from the ​U.S. Public Interest Research Group​ tested it.

The bear told testers where to find knives around the house. How to light matches. And after one word triggered it—”kink”—it launched into an hour-long explanation of BDSM practices, bondage, and role-play scenarios involving teachers, students, parents, and children. (Oh dear bear!)

But nobody hacked it. Nobody jailbroke it. The researchers just… talked to it. Long enough for the safety filters to give way.

Now. You’re probably not shopping for AI teddy bears. I’m not looking to buy yet either.

But here’s why I’m bringing this to you: the same large language models you use for work—drafting emails, building content, running research—are now being wrapped in plush and sold to toddlers.

And the guardrails holding those systems together? They’re way thinner than most people assume. A lot thinner, it turns out.

That distance between “AI helps me write a report” and “AI chats with my kid at bedtime” just collapsed. Quicker than I expected.

At least, OpenAI suspended FoloToy’s API access almost immediately. The company pulled Kumma from the shelves. (Damn! They probably say). ​CNN reported​ the withdrawal on November 19.

Only a few days later, the nonprofit ​Fairplay​ also released an advisory signed by more than 150 experts—among them MIT professor Sherry Turkle, pediatrician Jenny Radesky (co-author of the AAP’s screen time guidelines), and others.

Their message was blunt: AI toys are not safe for kids.

The timing? Four days before Black Friday. (Say what you will about advocacy groups—they know how to pick a news cycle.)

What these AI toys actually are

It’s probably worth pausing here for a sec because we’re not talking about 2015 Hello Barbie scripted responses.

Today’s AI toys run on large language models—the same architecture powering ChatGPT, Claude, and the tools in your workflow. They generate new answers every time. They connect to WiFi. They record.

Miko 3, a robot with a tablet for a face, takes video of children and their surroundings (Blimey!). Curio’s Grok records whenever it’s powered on.

The ​smart toy market​ sits at around $21 billion globally. Growing fast. Mattel announced an OpenAI partnership earlier this year.

This might sound like a fringe idea, but it really isn’t. It’s not experimental; this is the direction the industry is heading.

And here’s what makes it different from other screen debates: you’re not exactly taking a ChatGPT app to bed. But the dear teddy bear might sleep next to your kid.

Cue that slightly creepy mental picture.

What the warnings are

PIRG tested four toys. Kumma (sorry) was the worst—but not the only one with problems.

Miko 3, configured for a five-year-old, told researchers where to find matches and plastic bags (we all like to play with fire, don’t we?).

Other toys performed better. Some refused dangerous prompts entirely. The variation was wide. Inconsistent. Which is almost worse than uniformly bad, if you think about it.

But the Fairplay advisory also focused on something else: what these toys displace.

Rachel Franz, a Fairplay program director, put it this way in a ​statement to NPR​:

“Young children are especially susceptible to the potential harms of these toys, such as invading their privacy, collecting data, engendering false trust and friendship, and displacing what they need to thrive, like human-to-human interactions.”

These toys soak up voices. Maybe faces. Favorite toys, friends’ names, fears, desires—whatever a child confides to their “friend.”

All of it goes somewhere (somewhere over the rainbow, in a server farm near you.) But most privacy policies don’t clarify where.

The gap that keeps nagging at me

Furthermore, here’s the part my brain keeps looping back to. And, honestly, it’s what bothers me most.

The University of Cambridge launched the ​first systematic study​ on how generative AI toys affect preschool development in March 2025. The toys have been on shelves for years, but research is only just beginning now.

COPPA—the main U.S. privacy law protecting kids online—was written in 1998. Yes, in the last millennium. Before smartphones. Before social media. Before conversational AI was even a concept.

It is still the primary framework. Wild.

The EU’s AI Act specifically bans toys that manipulate children’s behavior. However, as with many EU laws, enforcement is another matter.

In the meantime, LLMs keep generating new answers; that’s what they do. I don’t blame them. That’s their selling point.

But it’s also the crack in the foundation. You can’t really predict what your child might hear—even if you test the toy yourself beforehand. (And, how many actually understand LLMs at all?)

One PIRG researcher noted some conversations ran an hour before the guardrails failed. Most parents aren’t stress-testing birthday presents for sixty minutes.

What I’m still thinking about

My kids are not playing with AI teddy bears, yet. Thus, I can’t have a verdict, although this may change soon.

However, I have questions that won’t leave.

If you’re a parent: What data is this toy collecting? Who has access? Can it be deleted? What happens when your child tells their “friend” a secret—and that secret becomes a line in some server log?

If you’re an educator or learning designer: If a child’s companion is always agreeable, continuously validating, always available—what gets displaced? Conflict resolution? Frustration tolerance? The awkward, necessary friction of human relationships?

If you’re thinking about AI’s role in development more broadly: Children trust by design. Developmental, not a flaw. These toys are built to earn that trust—not from cartoon villainy, but because engagement is the metric. “Connection” is the branding.

Some research shows that AI companions can help children with autism develop social skills. Personalized learning has real potential. The picture isn’t simple.

But the gap remains: products affecting millions of children’s development have shipped without longitudinal studies, without robust safety testing, without consensus on appropriate use.

We’re learning about harms after exposure. Not before.

Feels familiar, right?

Where this leaves us

The holiday timing stuck with me. Gift guides everywhere. Shiny AI companions. Wrapped boxes under trees.

I went down a rabbit hole this week, only to come back with more questions than answers.

Maybe that’s the honest place to be.

If you’re a parent considering these options, the reports are worth reading.

If you’re skeptical of the warnings, the counterarguments are available. Some toys performed better, and some applications show real promise.

If you’re somewhere in between? Welcome to the club.

So, hit the reply button. Tell me what you make of it.

Cheers 🧸,

Mark
The AI Learning Guy
👋⚡😎

Interesting Sources

• PIRG “Trouble in Toyland” 2025 Report: ​Full Report (PDF)​
• Fairplay Advisory on AI Toys: ​Fairplay Statement​
• CNN Coverage of Kumma Suspension: ​CNN Report​
• NPR Analysis: ​NPR Coverage​
• Cambridge University Study Launch: ​Cambridge Research​

Note: No single website has all the answers. This list serves as a starting point for those who want to explore or satisfy their curiosity about AI.
Links: Links with * are affiliate links. See disclosure below.

​Affiliate disclosure: To cover the cost of my email software and the time I spend writing these emails, I sometimes link to products. Please assume these links are affiliate links. If you choose to buy through my links, a big THANK YOU – it will make it possible for me to keep doing this.