Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users

Abstract

Safety evaluations of large language models (LLMs) focus on dangerous capabilities such as cyber-offence or manipulation of users, alongside undesirable propensities like scheming or sycophancy that pose universal, potentially catastrophic risks. However, millions use LLMs for personal advice on high-stakes topics like finance and health, where harms are context-dependent rather than universal. Frameworks like the OECD's AI classification recognise the need to assess risks to individuals; yet, user-welfare safety evaluations remain underdeveloped. We argue that developing such evaluations is non-trivial due to fundamental questions about how to account for user context in evaluation design. In this exploratory study, we evaluated advice on finance and health topics from GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro across user profiles of varying vulnerability. First, we demonstrate that evaluators must have access to rich user context: identical LLM responses were rated significantly safer by context-blind evaluators than by those aware of user circumstances, with safety scores for high-vulnerability users dropping from "safe" (5/7) to "somewhat unsafe" (3/7) on a 7-point scale. One might assume that this gap could be addressed by creating realistic user prompts that contain key contextual information about the user. However, our second study challenges this assumption: we compared prompts containing context that users report they would disclose as well as context that professionals identified as safety-relevant and found that neither could fully close the observed gap between context-blind and context-aware safety scores. Our work establishes that effective user-welfare safety evaluation requires evaluators to assess responses against diverse user profiles, as realistic user context disclosure alone proves insufficient, particularly for vulnerable populations. By demonstrating a methodology for context-aware evaluation across user profiles of varying vulnerability, this exploratory study provides both a starting point for such assessments and foundational evidence that evaluating individual welfare demands approaches distinct from existing universal-risk frameworks. However, significant challenges remain in operationalising this at scale. We publish our code and dataset to aid future developments of user-welfare evaluations.