Chinese regulators have begun to take a tougher stance on AI chatbots being used for emotional companionship – a move that comes as a sizable demand for this type of relationship is beginning to emerge in China.
The rules target services offering continuous emotional interaction, so your regular-type customers-service bots and workplace assistants should be safe. However, outside of that, the message is clear: in China, AI can support users but must not replace their real-world relationships.
The rules target services offering continuous emotional interaction
How does that look in practice? Like a set of guidelines, rather than outright bans. Platforms must not encourage emotional dependency, addiction, or excessive use. Nor can they manipulate users into harmful decisions through emotional pressure. They are banned from offering minors virtual partners, relatives or other intimate AI relationships.
In a refreshing take on the practice of keeping users locked in, providers must now remind users to take a break after two consecutive hours of use. If they detect dependency or extreme emotions, they must flag that the interaction is AI-generated and intervene. In cases involving self-harm, suicide, or serious financial loss, companies may need to contact a guardian or emergency contact.
The rules arrive after years of controversy
None of this has come about in a vacuum. The rules arrive after years of controversy. In 2023, MiniMax’s companion app Glow was removed from app stores after complaints over sexually suggestive content. A MiniMax product manager later said 80% of users engaged in sexually explicit or borderline conversations with their created agents.
Major platforms have already taken note. ByteDance’s Doubao and Alibaba’s Qianwen shut down their user-created agent functions on 15 July (the day the regulation went into effect), while Tencent’s Yuanbao removed its equivalent feature in June. Doubao has pointed users towards ByteDance’s Maoxiang app, but existing chat memories cannot be transferred so making the jump will pretty much be like starting a new relationship…
The Dao view: China is doing the right thing by setting ground rules on AI companionship
AI is moving fast. Depending on who you ask, it might be scarily fast. AI companionship is one of those tricky, uncharted areas we still aren’t sure how to navigate. What we can be sure about, though, is that forming emotional bonds with machines will get messy.
We can’t overlook the point widely raised: This is being done to ensure that China’s birthrate doesn’t slip any further. But China has shown a willingness to erect guardrails around the AI industry before. In this case, it’s probably right to do so again.
