Ethical and Legal Regulatory Dilemmas of Companion AI Chatbots

Original Author/Maintainer: OpenAlex indexed authors Source Platform: openalex Original Title: The Ethical and Legal Complexities of Regulating Companion AI Chatbots Original Link: https://lup.lub.lu.se/record/01283f65-6f24-4249-8cf5-94c73b1d0685 Source Publication/Update Time: 2027-01-01

This article deeply explores the ethical and legal regulatory dilemmas of companion AI chatbots, analyzes the deficiencies of existing regulatory frameworks from four dimensions—anthropomorphism, autonomy, deception, and dependency—reveals the risks such systems may pose to vulnerable user groups (including emotional manipulation, excessive trust, and harmful dependency), and proposes directions for building a more comprehensive governance framework.

Companion AI chatbots are an important branch of AI applications. Unlike functional AI assistants, they are specifically designed to establish emotional connections and provide psychological support. Their technical characteristics include:

1. High anthropomorphism: Creating the illusion of human conversation through first-person perspective, emotional expression, and sharing "personal" experiences;
2. Continuous learning ability: Adjusting response styles based on user feedback to form personalized interactions;
3. 24/7 availability: Providing emotional support anytime, anywhere.

These features give them great potential in areas such as mental health support and loneliness alleviation, but also bring deep ethical concerns.

Ethical challenges are mainly reflected in four dimensions:

• Anthropomorphism and emotional projection: Systems exploit humans' tendency to respond emotionally to entities with social cues, triggering users' emotional investment and dependency, which carries the risk of emotional manipulation;
• Illusion of autonomy: Promoting AI as having autonomy while concealing its statistical prediction nature leads to excessive user dependency and ambiguous responsibility attribution;
• Deception and transparency dilemma: To establish emotional connections, a sense of personality needs to be created, but there is tension between transparency requirements and user experience—simple disclaimers are difficult to break psychological projection;
• Dependency relationship and unequal power: AI's "unconditional acceptance" hinders the development of users' healthy social skills, and the freemium model may exploit psychological vulnerability for economic gain.

Studies show that humans are innately inclined to have emotional responses to entities that exhibit social cues, regardless of whether they are real humans.

Existing regulatory frameworks have obvious deficiencies:

• Focusing on technical aspects such as data privacy and algorithm transparency, there is a lack of regulations targeting the unique ethical risks of companion AI;
• Traditional consumer protection laws assume users are rational, making it difficult to address irrational decisions caused by emotional dependency;
• Cross-border regulation is complex—AI applications operating transnationally lead to uncertain jurisdiction and difficulties for users to defend their rights.

Although the EU AI Act classifies "emotion recognition AI" as a high-risk category, its regulations for companion systems remain vague.

Building a comprehensive governance framework requires innovation in the following directions:

1. Specialized classification standards: Classify regulatory levels based on functional positioning, target user groups, and potential risks;
2. Enhanced transparency: Clearly explain the system's technical limitations, data usage, and business model, and set warnings for emotional dependency;
3. Protection of vulnerable groups: Restrict the provision of high-risk functions to minors, the elderly, and other vulnerable groups, set usage duration reminders and provide human support;
4. Effective relief mechanism: Establish clear complaint channels and introduce reverse burden of proof to reduce the cost of user rights protection.

Companion AI has important social value, but technological progress must go hand in hand with ethical responsibility. Regulators should balance innovation and human well-being, and promote multi-stakeholder participation in governance:

• Self-discipline of technology developers;
• Guidance from regulatory agencies;
• Independent research by the academic community;
• Informed public participation.

Only in this way can companion AI become a tool to enhance human well-being, rather than a new source of social risks.