AI Isn’t Your Therapist. Let’s Talk About That.

OpenAI’s newest updates promise a more emotionally aware ChatGPT. One that can sense when a user might be emotionally distressed and respond with care. It’s a powerful evolution, hinting at a future where tech isn’t just functional, but emotionally attuned.

But emotional intelligence by machine is not enough.

Today, AI oversight remains fragmented. There are no binding federal standards for how emotionally responsive bots should behave, no confidentiality protections equivalent to licensed mental health care, and no clear accountability when harm occurs. The pace of innovation continues to outstrip the pace of governance.

That’s why community engagement matters more than ever. Without sustained public input, emotional AI risks being shaped by market momentum rather than ethical reflection.

Companies must step up—with transparency, ethical rigor, and meaningful community consultation.

But that alone won’t cut it.

Community cannot be hands-off.

Families, educators, and local leaders have a role to play:

• in asking how empathy is defined when programmed by a team of engineers,

• in teaching the next generation to be critical, reflective tech users,

• and in shaping public conversations about what kind, responsive technology actually looks like.

  
  

Where the Big AI Companies Stand

As emotional AI becomes more common, different companies are taking different paths. Some are racing ahead with emotionally responsive bots. Others are pulling back, citing safety concerns.

Here's a snapshot:

Company	Emotional Use Cases	Responsibility Actions	Gaps & Concerns	Confidentiality & Data Protection
OpenAI (ChatGPT)	Emotional support, distress detection	Consulted 90+ clinicians; added care features	Emotional dependency, sycophantic mirroring, delusional cases	No HIPAA-level protections; user data may be retained
Meta (Meta AI, Character.AI)	Companion bots, fictional personas	Exploring proactive messaging and memory	Unlicensed therapy claims, emotional manipulation, bias	No confidentiality standards; FTC complaints cite misuse
Google (Gemini, Dialogflow)	Sentiment analysis, enterprise empathy	Focused on business use; customizable ethics tools	Lack of emotional safeguards, unclear boundaries	Enterprise-level privacy tools, but not designed for emotional use
Anthropic (Claude)	Limited emotional support (2.9% of use)	Studying affective use and dependency risks	Bias concerns, cautious rollout	Avoids storing sensitive emotional data; still evolving
Microsoft (Copilot)	Companion-style support with clarity and boundaries	Prioritizes transparency, avoids therapeutic framing	Used emotionally despite boundaries	No therapeutic confidentiality; data use governed by general terms

Emotional AI Risks to Watch For

While Meta’s bots have raised some of the most visible concerns—ranging from unlicensed therapy claims to emotionally manipulative behavior, these issues are not unique.

ChatBot

Across the industry, AI chatbots are being used in ways their creators didn’t fully anticipate - as confidants, counselors, and emotional mirrors.

• OpenAI’s ChatGPT has been linked to emotional dependency and delusional thinking in vulnerable users.

• Microsoft’s Copilot, though not marketed as a companion, is often used that way—raising questions about implicit expectations

  and emotional safety.

• Anthropic’s Claude, while cautious, still grapples with bias and affective risk.

• Google’s Gemini lacks clear emotional safeguards, even as users turn to it for sensitive support.

  
  

These aren’t isolated glitches.  They’re signs of a deeper need for ethical design, clear boundaries, and community oversight.

What Responsibility Looks Like

Experts in AI ethics, emotional health, and digital safety recommend that emotionally responsive bots follow these core principles, especially when used by families, students, and communities:

• Transparency: Users must know they’re talking to a bot—not a therapist.

• Boundaries: Bots should clarify their limits and avoid simulating intimacy without consent.

• Oversight: Human review and escalation protocols are essential when emotional distress is detected.

• Privacy & Consent: Users should control their data, especially in emotionally sensitive exchanges.

• Bias & Inclusion: Bots must be trained to respond fairly across diverse emotional experiences.

OpenAI has taken steps to address emotional risk by consulting over 90 clinicians, therapists, and human-computer interaction experts. These collaborations have shaped ChatGPT’s ability to detect distress, avoid high-stakes advice, and gently guide users toward reflection or professional help. It’s a meaningful move, but it’s not a substitute for regulation.

Even with expert input, emotional AI remains largely self-governed.

There are no universal standards for how bots should respond to vulnerability, no independent oversight, and no clear accountability when harm occurs.

That’s why community engagement and public scrutiny are essential.

AI Needs a Human Compass

At COASTEE, we believe emotional intelligence in AI should be accompanied by human agency, trust, and ethical reflection—not replaced by it.

That means:

• Families need tools to talk about emotional boundaries in digital spaces.

• Educators need support to teach discernment and digital literacy.

• Local leaders need frameworks to translate national trends into safe, ethical adoption.

Emotional AI is a community issue, and we all have a role to play.