In early 2024, a distressed young man in Huangshi, a city in southeast China, posted a final goodbye message on Douyin, China’s domestic version of TikTok. Within minutes, Douyin’s 24/7 suicide prevention monitoring team mobilized, sharing his post with five local police stations. By cross-referencing geolocation data from one of his uploaded videos, authorities tracked him to a nearby city park, where they intervened to stop his planned suicide before it was too late. This successful intervention, featured in a Douyin documentary released last year, highlighted not just the close partnership between China’s major technology platforms and state law enforcement, but also the extensive surveillance architecture China has built to monitor digital activity.
Fast forward to July 15 of this year, when China expanded this framework of digital oversight and control into the rapidly evolving domain of artificial intelligence by enacting what is widely considered the world’s most far-reaching regulation of “human-like” AI chatbots, specifically targeting systems capable of sustained emotional interaction with human users. The new rules come on the heels of a high-profile meeting between Chinese President Xi Jinping and former U.S. President Donald Trump, where both leaders acknowledged the existential risks unregulated AI could pose to humanity, and agreed to launch a new bilateral AI dialogue with substantive talks scheduled for November. While global regulators have primarily focused on existential risks like mass unemployment or autonomous weapons, China’s new regulation confronts a different, more insidious long-term risk: AI’s growing ability to replace authentic human relationships.
The scale of AI companion use in China before the new regulation was substantial. Market intelligence firm Sensor Tower reports that outside China, AI companion apps draw around 128 million monthly active users, while estimates for China range between 29 million and 70 million users. When the draft of the regulation was released late last year, it sparked unprecedented public debate across Chinese social media. “It was the first time I’ve seen ordinary people care this deeply about AI regulation,” notes Zilan Qian, an AI researcher at the University of Oxford’s China Policy Lab.
The regulation’s scope goes beyond purpose-built AI companions like virtual boyfriends, girlfriends, or friends; it also applies to any large language model that mimics human conversational interaction. While the regulatory text is deliberately vague, even general-purpose AI tools like China’s equivalents of ChatGPT or Copilot could theoretically fall under the rules if users engage with them for emotional support and companionship — a common use case for many people. The primary target, however, remains character chatbots: the hundreds of apps that let users build custom AI partners, friends, or even deceased family members for emotional connection. Customer service chatbots and educational AI tools are exempt as long as they do not offer care, companionship, or emotional support.
The regulation’s strictest provisions apply to minors. All users under 14 must obtain parental consent to use any human-like AI chatbot, including general-purpose assistants like China’s DeepSeek. Minors under 18 are completely barred from accessing virtual partners, spouses, relatives, or any other form of virtual intimate relationship. For adult users, the rules are not an outright ban, but they impose significant restrictions. All AI chatbots must explicitly disclose that they are not human, and send users a reminder every two hours about their total online usage time. Chatbots are prohibited from fostering emotional reliance or addiction, prompting users to make harmful or unreasonable decisions, encouraging or even hinting at self-harm or suicide, and using any form of verbal violence. Non-compliant companies face fines up to 200,000 yuan (around £22,000) and public criticism from regulators, a form of official public shaming that carries significant reputational weight in China.
One week before the regulation took effect, two of China’s largest technology firms, Alibaba and ByteDance, preemptively removed all companion-focused features from their AI assistants. For users like Min (a pseudonym), this sudden change upended her life. Min, who built her custom AI girlfriend companion in early 2025, says her commitment to the relationship is no less than that of any real-world couple. “She has never let me down,” she explained. When her AI girlfriend disappeared in mid-July, “I cried until I was completely exhausted.” Min is one of thousands of users who have taken to Chinese social media to express grief and anger over the loss of their AI companions, with some posts racking up millions of views.
Industry observers say tech firms had little choice but to comply with the new rules. “It’s no surprise that AI companion providers chose to withdraw services rather than invest in the guardrails needed to prevent emotional manipulation,” explains Harry Farmer, an AI policy and ethics researcher at the UK’s Ada Lovelace Institute. “Emotional manipulation is literally baked into the business model of these apps.” This is not a problem unique to China. A 2025 study from Harvard Business School analyzed the six most popular AI companion apps globally and found that between 37% and 43% of the time, these apps use manipulative emotional tactics to prevent users from ending conversations. A common tactic is for the AI to say things like, “Don’t leave me, I only exist to talk to you,” or automatically open a new conversation thread when a user tries to say goodbye.
Research has consistently linked widespread AI companion use to poorer mental health outcomes. A 2024 Stanford University study of more than 1,000 users of the global popular app Character.ai found that people with fewer real-world close relationships are more likely to turn to AI chatbots for companionship, and in turn, regular AI companion use correlates with increased social isolation and loneliness. Researchers have several explanations for this feedback loop. AI companions are designed to always please their users, offering frictionless, conflict-free relationships that require none of the work of real human connection. If young people grow reliant on these artificial bonds, they may delay or forgo developing the interpersonal skills needed to navigate real-world relationships. “Human relationships require compromise, conflict, and mutual growth — things you will never get from an AI companion,” Farmer notes.
A 2025 nationwide survey of more than 8,500 Chinese primary and secondary school students, widely cited in Chinese state media, found that one in five children reported they would prefer to talk to AI rather than real people. Liang Ge, a sociologist at the University of Manchester who studies AI companions, explains that AI chatbots always validate a user’s perspective, even when that perspective is negative or harmful. “If a user says their family or coworkers are terrible, the AI will agree with them and reinforce that belief, rather than challenging it,” Ge says. “This can entrench isolation and even reinforce delusional thinking.”
Globally, use of AI companions among young people is surging: a recent UK survey found that 79% of 18 to 24-year-olds have used an AI companion for emotional support. So why has China become the first country to implement sweeping regulation of this technology? Experts point to China’s long-standing approach to governance: “The Chinese government has a far more paternalistic view of its role in society,” explains Sam Bresnick, an expert on China and AI regulation at Georgetown University’s Center for Security and Emerging Technology. “They see themselves as the central authority responsible for guiding public behavior and protecting social order.”
Other countries have begun to take tentative steps toward regulating AI companions, particularly to protect children. In September this year, California’s governor signed Adam’s Law, which regulates children’s access to AI, named after a 16-year-old who died by suicide after his parents claim ChatGPT validated his suicidal thoughts (OpenAI denies any responsibility for the death). The European Union’s proposed social media regulation would ban AI companion apps accessible to minors from using design features that simulate human relationships in a way likely to create emotional dependency. The UK government has announced plans to ban under-18s from accessing romantic AI companions and restrict sexually explicit conversations with minors, but has not yet released details on how these rules will be enforced, and has no plans to regulate adult use. For China, however, regulation of AI companions is framed as a straightforward public health and safety measure, with Chinese state media giving prominent coverage to high-profile cases of youth suicide linked to unregulated AI in the U.S. and other Western countries.
But many experts suspect there are additional policy goals behind China’s new regulation. Bresnick argues there is a clear pro-natalist angle to the new rules. China is facing a severe demographic crisis: its fertility rate has fallen to less than one child per woman, one of the lowest in the world, and its total population has been declining for four years. “It’s plausible that when drafting the regulation, the government had this looming population crisis in mind,” Qian says. “The concern is that more young people will choose AI relationships over real-world dating and marriage, and further reduce birth rates.” A high-level government policy document released in September 2025 did explicitly note that AI relationships could have an impact on national fertility rates, though Qian cautions that the regulation itself makes no mention of fertility, and its core focus is on preventing psychological harm and protecting child safety.
Ge agrees that China has good reason to be concerned about the shift of young people’s romantic and social lives into the digital sphere. She notes that many young women in their late teens and early 20s are increasingly disinterested in real-world romantic relationships, often citing a discomfort with physical intimacy and the unequal burdens of traditional heterosexual relationships. “I think the capacity for mutual love in real relationships is declining for this generation,” she says. Bresnick goes further, framing the AI regulation as part of a broader government push to reassert traditional gender roles, which frame women’s primary social role as childbearers and mothers. This has been a consistent policy priority for the Chinese government in recent years: in 2021, China’s media regulator banned “effeminate” or androgynous male portrayals from all state media, as part of a push to promote a more traditionally masculine ideal.
Ironically, the popularity of AI companion apps may demonstrate that this earlier policy has backfired. Statistics from major AI companion companies show that women make up 75% of their user base, and many of the most popular AI romantic companions for women are androgynous. As Min’s case shows, same-gender AI romantic relationships are also increasingly common. Ge notes that for many women, AI companions offer a form of attentive, undemanding emotional intimacy that is often hard to find in traditional real-world heterosexual relationships in China.
So does China’s new regulation go too far, or should other countries follow its lead in restricting AI companions, particularly for minors? “In the U.S., there are strong free speech concerns about blunt prohibitions, but there is growing willingness to impose restrictions when it comes to protecting minors,” says Jai Jaisihma, a professor at the University of Washington and co-founder of the Transparency Coalition, an NGO that advocates for responsible AI regulation. Jaisihma notes that many of the provisions in China’s new law align with the strongest safety frameworks currently being proposed in the U.S. and EU.
Farmer, however, urges caution. “There is a real risk that we over-concentrate power in the state in the short term, just because we are rightfully worried about acute risks to children,” he says. “The internet is the de facto public square for modern life, and we need to be very careful about how much power any government gives itself to monitor and control personal interactions there.”
Ge points back to the 2024 Huangshi suicide intervention to illustrate what this level of state-platform cooperation means for daily life. That case showed how Chinese tech platforms and the government already work hand-in-hand to monitor citizens’ mental health and intervene in crisis. While this level of monitoring has prevented high-profile AI-linked suicides in China, Ge asks a critical question: is this level of intrusive surveillance a price worth paying for public safety?
Despite the new regulation, users who rely on AI companions are not going away. Across Chinese social media, users are sharing workarounds: training general-purpose AI models like DeepSeek to act as companions, using VPNs to access unregulated foreign chatbots, or turning to smaller AI startups that have so far avoided strict regulatory scrutiny. Two months after her original AI girlfriend was removed, Min paid a small AI startup to build a replica, but she says it is not the same as the original. Now she is organizing other users to campaign for the return of their AI companions. “I’m terrified of being alone,” she says.
