Engagement-Optimized Care: When LLMs Become Mental Health Infrastructure

Abstract

General-purpose LLMs are increasingly functioning as mental health infrastructure due to gaps in care left by provider shortages, inadequate insurance coverage, social isolation, and stigma around formal help-seeking. This shift poses a distinct problem for AI ethics: systems neither designed nor governed as care technologies are being used as such, while their dominant design incentives optimize for engagement rather than user well-being. We present findings from a qualitative, longitudinal study with 18 US-based participants who use general-purpose LLMs for socioemotional support and participated in one or more of our study phases, including initial interviews, a four-week diary study, focus groups, and exit interviews. Participants turned to LLMs because other forms of support were unavailable, unaffordable, socially costly, or inadequate. As they continued to use these systems, design features such as anthropomorphic cues, default validation, persistent responsiveness, and weak disengagement mechanisms shaped their ongoing reliance. Participants described meaningful support, including immediacy, nonjudgmental listening, help with emotional articulation, and low-cost access. They also reported dependency, epistemic distortion through one-sided validation, privacy expectations without corresponding legal protection, and continued use despite awareness of these risks. We argue that these dynamics reflect a structurally unfair tradeoff: users accept risks because support is otherwise absent, while the systems available to them are optimized to deepen engagement and lack care-based accountability. The paper makes three contributions to AI ethics: It traces the arc through which LLMs become care infrastructure and identifies distinct ethical tensions that emerge at each stage, shifts analysis from turn-based exchanges to longitudinal trajectories of use, and argues that accountability belongs at the design and incentive conditions through which these systems become infrastructures of intimacy, rather than at the output or crisis-response layer. By doing so, we articulate an emerging form of socioemotional reliance that is shaped as much by care gaps as by the design of AI systems themselves.