How to Catalyze Policy Guardrails against AI Chatbots' Potential Social-Emotional Harm: Lessons from Analogous Cases (Extended Abstract)

Abstract

Could extensive social-emotional use of AI chatbots erode users' human connections over time? Could it worsen the already severe loneliness crisis? Research in controlled settings increasingly suggests that such slow-burning harms are plausible; if these harms are occurring in the real world, their consequences would be severe and far-reaching. However, such harm is hard to prove or attribute outside controlled settings. Loneliness has many influences, and the design of chatbots, the capabilities of models that power them, and usage patterns all change frequently. Conclusive evidence that could propel large-scale public mental health or regulatory interventions is therefore unlikely to emerge until the harm has become entrenched and harder to reverse. We ask: How might human-computer interaction (HCI) and policy communities catalyze policy guardrails against the potentially harmful effects of AI chatbots on user loneliness, before conclusive causal evidence of such slow-burning, diffuse harm arrives? As a first step to answering this question, we analyze past and ongoing regulatory cases that are analogous to the present one, and ask what we can learn from them. These are cases where a product's harm is (i) probable but contested (either the harm itself or its attribution to the specific product), (ii) slow-burning and diffuse, and (iii) closely linked with industry actors' financial interests or business models, all of which make regulatory interventions difficult. We analyzed five such cases in depth: regulating social media, tobacco, and prescription opioids in the U.S., youth digital gaming addiction in South Korea, and problem gambling in Australia. Notably, this is not a comprehensive set of past or ongoing cases that meet the criteria, nor does what worked (or did not work) for these cases automatically predict what will work for AI chatbots. Nevertheless, they offer a useful starting point for discussing how we might break the impasse in regulating AI chatbots' potential social-emotional harm. Our analysis suggests three elements often preceded durable policy guardrails: (i) evidence of the harm's public cost, i.e., the cost it imposes on public budgets (such as the fiscal burden the opioid crisis imposed on Medicaid); (ii) a civic narrative framing the harm as a matter of product design and industry conduct rather than users' individual responsibility; and (iii) institutions with dedicated funding that make monitoring of the harm durable. In the case of AI chatbots, however, these elements are not yet in place. We reviewed 62 public-facing governance documents from six companies deploying general-purpose chatbots and purpose-built companion or mental well-being apps for passages addressing mental health, well-being, or user relationships. Almost all of the commitments we identified that address chatbots substituting for human interaction operate within a single message, session, or day; none describes a mechanism for detecting gradual erosion of users' social connections across sessions. Recent state laws in the U.S. addressing social-emotional use of AI operate at a similar level. These instruments are therefore ill-suited to accumulating evidence of the public cost of chatbots' slow-burning harm. Based on these findings, we discuss pathways through which HCI and policy communities might break the impasse in regulating AI chatbots' potential social-emotional harm. In particular, we call for (i) public infrastructure that measures the public cost of such harm, (ii) novel mechanisms to fund such measurement infrastructure, and (iii) continued research on AI chatbot/companion use as a matter not (just) of users' personal choice but of public health.