The International Association for Safe & Ethical AI (IASEAI, pronounced eye-see-eye) is an independent nonprofit organization founded to address the risks and opportunities associated with rapid advances in AI. Our mission is to ensure that AI systems operate safely and ethically, benefiting all of humanity. We connect experts from academia, policy groups, civil society, industry, and beyond to promote research, shape policy, and build understanding around this goal. We believe that AI can be beneficial, but current systems are being developed without appropriate safeguards. As these systems become more capable and more directly involved in critical social and economic functions, it is essential to provide assurances that they will operate safely and benefit humanity. Policy should be developed with input from experts and affected communities to encourage the creation of AI systems that can support such assurances. IASEAI brings together the many individuals, research groups, and organizations that share these goals to accelerate research, inform public opinion, and influence policy.
Unsupervised Elicitation of Moral Values from Language Models
As AI systems become pervasive, grounding their behavior in human values is critical. Prior work suggests that language models (LMs) exhibit limited inherent moral reasoning, leading to calls for explicit moral “teaching”. However, constructing ground-truth data for moral evaluation is difficult given plural frameworks and pervasive biases. We investigate unsupervised elicitation as an alternative, asking whether pretrained (base) LMs possess intrinsic moral reasoning capability that can be surfaced without human supervision. Using the Internal Coherence Maximization (ICM) algorithm across three benchmark datasets and four LMs, we test whether ICM can reliably label moral judgments, generalize across moral frameworks, and mitigate social bias. Results show that ICM outperforms all pretrained and chatbot baselines on the Norm Bank and ETHICS benchmarks, while fine-tuning on ICM labels performs on par with or surpasses those of human labels. Across theoretically motivated moral frameworks, ICM yields its largest relative gains on Justice and Commonsense morality. Furthermore, although chatbot LMs exhibit social bias failure rates comparable to their pretrained ones, ICM reduces such errors by more than half, with the largest improvements in race, socioeconomic status, and politics. These findings suggest that pretrained LMs possess latent moral reasoning capacities that can be elicited through unsupervised methods like ICM, providing a scalable path for AI alignment.
Copyright (c) 2026 Association for the Advancement of Artificial Intelligence
From Awareness to Action: Understanding and Overcoming the Research-Practice Gap in Algorithmic Fairness for Public Health
Algorithmic fairness is essential for responsible ML-driven public health research, yet its practical implementation remains limited. To investigate this awareness–action gap, we conducted a sequential mixed-methods study comprising expert interviews, an online survey, and systematic mapping. The expert interviews informed the design of the survey, which in turn revealed fragmented definitions of fairness, limited training and guidance, reliance on external sources, and rare use of formal assessment, mitigation, or monitoring. These findings were subsequently mapped onto three established research–practice gap lenses: the Knowledge–Practice Gap, the Knowledge-to-Action Cycle, and the Knowing–Doing Gap, each offering complementary perspectives. Building on this synthesis, we introduce the Fairness-to-Action framework, which integrates methodological, organizational, and systemic dimensions to identify where translation of algorithmic fairness knowledge stalls. Our analysis shows that fairness remains weakly institutionalized, translation mechanisms are externally driven, and system-level priorities continue to emphasize accuracy over fairness. These insights suggest critical leverage points for advancing safe, fair, and ethical ML-driven public health research practice.
The Three Main Doctrines on the Future of AI
This paper develops a taxonomy of expert perspectives on the risks and likely consequences of artificial intelligence, with particular focus on Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI). Drawing from primary sources, we identify three predominant doctrines: (1) The dominance doctrine, which predicts that the first actor to create sufficiently advanced AI will attain overwhelming strategic superiority sufficient to cheaply neutralize its opponents' defenses; (2) The extinction doctrine, which anticipates that humanity will likely lose control of ASI, leading to the extinction of the human species or its permanent disempowerment; (3) The replacement doctrine, which forecasts that AI will automate a large share of tasks currently performed by humans, but will not be so transformative as to fundamentally reshape or bring an end to human civilization. We examine the assumptions and arguments underlying each doctrine, including expectations around the pace of AI progress and the feasibility of maintaining advanced AI under human control. While the boundaries between doctrines are sometimes porous and many experts hedge across them, this taxonomy clarifies the core axes of disagreement over the anticipated scale and nature of the consequences of AI development.
TRIED: Truly Innovative and Effective AI Detection Benchmark
The rapid rise of generative AI and the increasing ability to create deceptive synthetic media pose a significant threat to public trust and information integrity, particularly in resource-constrained regions of the Global Majority. In the face of this growth, AI detection tools remain an invaluable resource for frontline information actors, but continue to underperform in real-world scenarios due to challenges related to explainability, fairness, accessibility, and contextual relevance. Drawing on frontline experiences, deceptive AI cases, and global consultations, this paper highlights the limitations of current AI detection tools and outlines how they must evolve to become truly innovative and relevant by meeting diverse linguistic, cultural, and technological contexts. It also introduces the Truly Innovative and Effective AI Detection (TRIED) Benchmark, a new framework for evaluating detection tools based on their real-world impact and capacity for innovation to guide inclusive AI detection tools development as well as support policy efforts towards AI procedures and standards reflecting global frontline community input.
Addressing Historical and Contemporary Inequalities in Low-Resource Languages through the Lens of Serbian
The development of large language models prioritizes high-resource languages such as English, while most of the world's 7,000 languages remain underrepresented in AI. Multilingual pre-training and cross-lingual transfer have produced uneven results, favoring languages structurally closer to English and transferring linguistic and cultural biases from the training data into low-resource language (LRL) outputs, creating a double disadvantage in which LRLs are simultaneously underrepresented and misrepresented. This paper argues that developing language technologies for LRLs requires not only technical solutions, but also contextualized ethnographic inquiry into the historical, sociocultural, and institutional conditions that shape their digital future. Using Serbian as a case study, it draws on semi-structured interviews with ten scholars and NLP practitioners working across disciplines. Findings identify barriers common to many LRLs, including limited corpora, insufficient digitization, restrictive copyright regimes, and partial institutional support, which contribute to reliance on low-quality web data and multilingual models that reproduce bias. The paper outlines an alternative approach centered on native language models, interdisciplinary collaboration, inclusive datasets, and transparent data practices, with sustained involvement of humanities and social science scholars across data selection, evaluation, and governance, so that sociocultural and ethical concerns are built into language technologies from the outset rather than treated as post hoc technical interventions or matters of regulatory compliance.
Language Models Exhibit Inconsistent Biases Towards Algorithmic Agents and Human Experts
Large language models are increasingly used in decision-making tasks that require them to process information from a variety of sources, including both human experts and other algorithmic agents. How do LLMs weigh the information provided by these different sources? We consider the well-studied phenomenon of algorithm aversion, in which human decision-makers exhibit bias against predictions from algorithms. Drawing upon experimental paradigms from behavioural economics, we evaluate how eight different LLMs delegate decision-making tasks when the delegatee is framed as a human expert or an algorithmic agent. To be inclusive of different evaluation formats, we conduct our study with two task presentations: \textit{stated preferences}, modeled through direct queries about trust towards either agent, and \textit{revealed preferences}, modeled through providing in-context examples of the performance of both agents. When prompted to rate the trustworthiness of human experts and algorithms across diverse tasks, LLMs give higher ratings to the human expert, which correlates with prior results from human respondents. However, when shown the performance of a human expert and an algorithm and asked to place an incentivized bet between the two, LLMs disproportionately choose the algorithm, even when it performs demonstrably worse. These discrepant results suggest that LLMs may encode inconsistent biases towards humans and algorithms, which need to be carefully considered when they are deployed in high-stakes scenarios. Furthermore, we discuss the sensitivity of LLMs to task presentation formats that should be broadly scrutinized in evaluation robustness for AI safety.
Building and Measuring Trust Between Large Language Models
As large language models (LLMs) increasingly interact with each other in multi-agent setups, we may expect (and hope) that 'trust' relationships develop between them, mirroring trust relationships between human colleagues, friends, or partners. Yet, though prior work has shown LLMs to be capable of identifying emotional connections and recognizing reciprocity in trust games, little remains known about (i) how different strategies to build trust compare, (ii) how such trust can be measured implicitly, and (iii) how this relates to explicit measures of trust. We study these questions by relating implicit measures of trust, i.e. susceptibility to persuasion and propensity to collaborate financially, with explicit measures of trust, i.e. a dyadic trust questionnaire well-established in psychology. We build trust in three ways: by building rapport dynamically, by starting from a prewritten script that evidences trust, and by adapting the LLMs' system prompt. Surprisingly, we find that the measures of explicit trust exhibit a small to strongly negative correlation with implicit trust measures. These findings suggest that measuring trust between LLMs by asking them about it may be deceiving. Instead, context-specific, implicit measures may be more informative in understanding trust between LLMs.
Measuring Validity in LLM-based Resume Screening
Resume screening is perceived as a particularly suitable task for LLMs given their ability to analyze natural language; thus many entities rely on general purpose LLMs without further adapting them to the task. While researchers have shown that some LLMs are biased in their selection rates of different demographics, studies measuring the validity of LLM decisions are limited. One of the difficulties in externally measuring validity stems from lack of access to a large corpus of resumes for whom the ground truth in their ranking is known and that has not already been used for LLM training. In this work, we overcome this challenge by systematically constructing a large dataset of resumes tailored to particular jobs that are directly comparable, with a known ground truth of superiority. We then use the constructed dataset to measure the validity of ranking decisions made by various LLMs, finding that many models are unable to consistently select the resumes describing more qualified candidates. Furthermore, when measuring the validity of decisions, we find that models do not reliably abstain when ranking equally-qualified candidates and select candidates from different demographic groups at different rates, occasionally prioritizing historically-marginalized candidates. Our proposed framework provides a principled approach to audit LLM resume screeners in the absence of ground truth, offering a crucial tool to independent auditors and developers to ensure the validity of these systems as they are deployed.
Attribution and Actualisation of Consciousness in AI
We introduce a framework to highlight the interplay between human belief or lack of belief in consciousness in AI, attribution, and the actual presence or absence of consciousness in AI, actualisation. This produces four main cases: 1) attribution and actualisation, 2) attribution and no actualisation, 3) no attribution and actualisation, 4) no attribution and no actualisation. We examine the importance of actualisation as an event and how it may manifest via different types of consciousness (sentience, selfhood, sapience, synthetic, etc.). We explore how attribution may introduce ethical (monetary, material, and legal) and social (attentional, emotional, and intimacy) considerations at an individual and societal scale. Each case is likely to lead to significantly different outcomes. By articulating the likely consequences of each case we argue that case 1 and case 4, accurate outcomes, are preferable to case 2 and case 3, inaccurate outcomes, and explicate reasons for why we may want to aim for one case rather than another. Additional variables produce further subcases. The honesty or dishonesty of each agent, human and AI, complicates our recognition of the truth. Dishonesty may occur due to different parties holding conflicting incentives and motivations. The type of consciousness attributed and actualised also plays a significant role in appropriate decision making. We suggest that we are now living in a multicase world where a variety of different subcases are playing out simultaneously, with different populations holding different beliefs and the potential for different AIs to actualise different types of consciousness.
One common concern about advanced artificial intelligence is that it will prevent us from turning it off, as that would interfere with pursuing its goals. In this paper, we discuss an unorthodox proposal for addressing this concern: give the AI a (primary) goal of being turned off (see also papers by Martin et al., and by Goldstein and Robinson). We also discuss whether and under what conditions this would be a good idea.
Tracing the Moving Target: Locating Value Propositions on AI Ethics in the European Union (2018-2025)
Anu Bradford (2020) argues that the EU creates a ‘Brussels Effect’ on the digital economy through its data regulations. In this context, understanding the evolution and lifespan of the value propositions of AI Ethics in EU regulations becomes crucial, as it tends to have global implications. The study employs computational tools to systematically analyze the emergence and disappearance of AI-related concepts in the EU legal-policy discourse. Preliminary findings indicate that AI-related policy discourse within the EU is highly dynamic, and many concepts such, as Green AI, Safe AI, Trustworthy AI, etc., are emerging and disappearing from discussions over time. It also indicates that several AI Ethics propositions, such as Fair AI or Explainable AI, have a short-lived presence. At the same time, the presence of Green AI and Sustainable AI has increased in the documents in recent years, indicating a shift in political priorities.The study also provides comparative insights on the weightage of these propositions in legal acts vs preparatory documents within the EU regulatory landscape. Furthermore, through a case study of Green AI the study examines the institutional authorship of these propositions over time to understand how different EU institutions shape regulatory priorities. In this context the paper aims to offer insights on how various concepts on AI Ethics are introduced, discussed, and ultimately fade from the discourse within the EU’s official preparatory documents and regulatory framework.
AI Alignment Strategies from a Risk Perspective: Independent Alignment Mechanisms or Shared Failures?
AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results' implications for understanding the current level of risk and how to prioritize AI alignment research in the future.
Clear, Compelling Arguments: Rethinking the Foundations of Frontier AI Safety Cases
This paper contributes to the nascent debate around safety cases for frontier AI systems. Safety cases are structured, defensible arguments that a system is acceptably safe to deploy in a given context. Historically, they have been used in safety-critical industries, such as aerospace, nuclear or automotive. As a result, safety cases for frontier AI have risen in prominence, both in the safety policies of leading frontier developers and in international research agendas proposed by leaders in generative AI, such as the Singapore Consensus on Global AI Safety Research Priorities and the International AI Safety Report. This paper appraises this work. We note that research conducted within the alignment community which draws explicitly on lessons from the assurance community has significant limitations. We therefore aim to rethink existing approaches to alignment safety cases. We offer lessons from existing methodologies within safety assurance and outline the limitations involved in the alignment community’s current approach. Building on this foundation, we present a case study for a safety case focused on Deceptive Alignment and CBRN capabilities, drawing on existing, theoretical safety case “sketches” created by the alignment safety case community. Overall, we contribute holistic insights from the field of safety assurance via rigorous theory and methodologies that have been applied in safety-critical contexts. We do so in order to create a better foundational framework for robust, defensible and useful safety case methodologies which can help to assure the safety of frontier AI systems.
FairPlay+: A Certified-Fair Hotel Room Recommendations Framework
Over the past decade, overtourism has emerged as a major threat to the quality of life in popular tourist destinations. Overtourism poses multiple challenges to the local communities. Among others, overpricing and profiteering harm local societies and economies deeply. On the other hand, uncontrolled overpricing makes such popular destinations inaccessible to a large portion of the populace, undermining the UN-sanctioned right to tourism. In this work, we put forward FairPlay+, a framework for hotel room recommendations with certified-fair prices. Our framework (a) can incorporate any mechanism that guarantees fair hotel room prices, according to any corresponding definition of fairness;(b) ascertains that the providers receive fair exposition by the system; and (c) importantly, guarantees that the providers are appropriately incentivised to accept the fair price proposed by the mechanism for their rooms instead of setting that price themselves. Specifically, we prove that providers are in expectation (weakly) better-off to accept the pricing policy proposed by the platform. Moreover, our simulation experiments demonstrate that (a) the platform-proposed fair policy is indeed the most profitable choice for the providers; while (b) providers adhering to the FairPlay+ proposed policy can better serve customer demand, considering customers of various budgets: in particular, FairPlay+ allows low-budget visitors to find accommodation that would be inaccessible to them without the use of fair pricing policies.
Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
Jailbreaking techniques pose a significant threat to the safety of Large Language Models (LLMs). Existing defenses typically focus on single-turn attacks, lack coverage across languages, and rely on limited taxonomies that either fail to capture the full diversity of attack strategies or emphasize risk categories rather than jailbreaking techniques. To advance the understanding of the effectiveness of jailbreaking techniques, we conducted a structured red-teaming challenge. The outcomes of our experiments are fourfold. First, we developed a comprehensive hierarchical taxonomy of jailbreak strategies that systematically consolidates techniques previously studied in isolation and harmonizes existing, partially overlapping classifications with explicit cross-references to prior categorizations. The taxonomy organizes jailbreak strategies into seven mechanism-oriented families: impersonation, persuasion, privilege escalation, cognitive overload, obfuscation, goal conflict, and data poisoning. Second, we analyzed the data collected from the challenge to examine the prevalence and success rates of different attack types, providing insights into how specific jailbreak strategies exploit model vulnerabilities and induce misalignment. Third, we benchmarked GPT-5 as a judge for jailbreak detection, evaluating the benefits of taxonomy-guided prompting for improving automatic detection. Finally, we compiled a new Italian dataset of 1364 multi-turn adversarial dialogues, annotated with our taxonomy, enabling the study of interactions where adversarial intent emerges gradually and succeeds in bypassing traditional safeguards.
Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols
To evaluate the safety and usefulness of deployment protocols for untrusted AIs, AI Control uses a red-teaming exercise played between a protocol designer and an adversary. This paper introduces AI-Control Games, a formal decision-making model of the red-teaming exercise as a multi-objective, partially observable, stochastic game. We also introduce reductions from AI-Control Games to a special case of zero-sum partially observable stochastic games that allow us to leverage existing algorithms to find Pareto-optimal protocols. We apply our formalism to model, evaluate and synthesise protocols for deploying untrusted language models as programming assistants, focusing on Trusted Monitoring protocols, which use weaker language models and limited human assistance. To demonstrate the utility of our formalism, we show improvements over empirical studies in existing settings, evaluate protocols in new settings, and analyse how modelling assumptions affect the safety and usefulness of protocols. Finally, we leverage our formalism to precisely describe some of the implicit assumptions in prior control work.
A Decision-Theoretic Approach for Managing Misalignment
When should we delegate decisions to AI systems? While the value alignment literature has developed techniques for shaping AI values, less attention has been paid to how to determine, under uncertainty, when imperfect alignment is good enough to justify delegation. We argue that rational delegation requires balancing an agent's value (mis)alignment with its epistemic accuracy and its reach (the acts it has available). This paper introduces a formal, decision-theoretic framework to analyze this tradeoff, precisely accounting for a principal's uncertainty about these factors. Our analysis reveals a sharp distinction between two delegation scenarios. First, universal delegation (trusting an agent with any problem) demands near-perfect value alignment and total epistemic trust, conditions rarely met in practice. Second, we show that context-specific delegation can be optimal even with significant misalignment. An agent's superior accuracy or expanded reach may grant access to better overall decision problems, making delegation rational in expectation. We develop a novel scoring framework to quantify this ex ante decision. Ultimately, our work provides a principled method for determining when an AI is aligned enough for a given context, shifting the focus from achieving perfect alignment to managing the risks and rewards of delegation under uncertainty.
DeliberationBench: A Normative Benchmark for Assessing the Influence of Large Language Models on Users’ Views
As large language models (LLMs) become pervasive as assistants and thought partners, it is important to characterize their persuasive influence on users’ beliefs. However, a central challenge is to distinguish “beneficial” from “harmful” forms of influence, in a manner that is normatively defensible and legitimate. We propose DeliberationBench, a benchmark for assessing LLM influence that takes the process of deliberative opinion polling as its standard. We demonstrate our approach in a preregistered randomized experiment in which 4,088 U.S. participants discussed 65 policy proposals with six frontier LLMs. Using opinion change data from four prior Deliberative Polls conducted by the Deliberative Democracy Lab, we find evidence that the tested LLMs’ influence is substantial in magnitude and positively associated with the net opinion shifts following deliberation, suggesting that these models exert broadly epistemically desirable effects. We further explore differential influence between topic areas, demographic subgroups, and models. Our framework can function as an evaluation and monitoring tool, helping to ensure that the influence of LLMs remains consistent with democratically legitimate standards, and preserves users’ autonomy in forming their views.
Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool
Algorithms are increasingly used to automate or aid human decisions, yet recent research shows that these algorithms may exhibit bias across legally protected demographic groups. However, data on these groups may be unavailable to organizations or external auditors due to privacy legislation. This paper studies bias detection using an unsupervised bias detection tool when data on demographic groups are unavailable. We collaborated with the Dutch Executive Agency for Education to audit an algorithm that was used to assign risk scores to college students at the national level in the Netherlands between 2012-2023. Our audit covers more than 250,000 students across the country. The unsupervised bias detection tool highlights known disparities between students with a non-European migration background and students with a Dutch or European-migration background. Our contributions are two-fold: (1) we assess bias in a real-world, large-scale, and high-stakes decision-making process by a governmental organization; (2) we provide the unsupervised bias detection tool in an open-source library for others to use to complete bias audits. Our work serves as a starting point for a deliberative assessment by human experts to evaluate potential discrimination in algorithmic decision-making.
Breaking Up with Normatively Monolithic Agency with GRACE: A Reason-Based Neuro-Symbolic Architecture for Safe and Ethical AI Alignment
As AI agents become increasingly autonomous, widely deployed in consequential contexts, and efficacious in bringing about real-world impacts, ensuring that their decisions are not only instrumentally effective but also normatively aligned has become critical. We introduce a neuro-symbolic reason-based containment architecture, Governor for Reason-Aligned ContainmEnt (GRACE), that decouples normative reasoning from instrumental decision-making and can contain AI agents of virtually any design. GRACE restructures decision-making into three modules: a Moral Module (MM) that determines permissible macro actions via deontic logic-based reasoning; a Decision-Making Module (DMM) that encapsulates the target agent while selecting instrumentally optimal primitive actions in accordance with derived macro actions; and a Guard that monitors and enforces moral compliance. The MM uses a reason-based formalism providing a semantic foundation for deontic logic, enabling interpretability, contestability, and justifiability. Its symbolic representation enriches the DMM’s informational context and supports formal verification and statistical guarantees of alignment enforced by the Guard. We demonstrate GRACE on an example of a LLM therapy assistant, showing how it enables stakeholders to understand, contest, and refine agent behavior.
Governance and Security-by-Design: Embedding Safety and Alignment into Agentic AI Systems
As artificial intelligence systems transition from task-specific tools to autonomous agents capable of complex decision-making, traditional external oversight mechanisms become inadequate for ensuring safety, security, and alignment. This paper introduces a governance and security-by-design framework that embeds responsibility mechanisms directly into agentic AI architectures, enabling continuous self-monitoring and alignment verification through multi-agent governance systems. We demonstrate that external governance approaches fail to scale with system autonomy, creating temporal gaps between assessment and deployment that enable ungovernable behaviors. Through mathematical modeling using stochastic differential equations, we formalize how competing objectives in agentic systems create systematic interference patterns that degrade safety properties. Our empirical validation across 800 experiments reveals three critical failure modes: (1) systematic security vulnerabilities from AI-generated code, with efficiency-focused prompting introducing memory safety issues in 42.7% of cases, while security-focused prompting paradoxically creates cryptographic vulnerabilities in 21.1% of cases; (2) iterative degradation, where security vulnerabilities increase by 37.6% after just five feedback iterations; and (3) knowledge dilution, where domain expertise degrades by 47% as irrelevant context accumulates. Our multi-agent governance and security architecture, validated through industry partnerships, achieves a 40% reduction in post-deployment safety incidents while maintaining operational capability. These findings establish that safe, secure, and aligned agentic systems require architectural integration of governance mechanisms that address the dynamic, multi-objective nature of autonomous AI systems.
Independent Clinical Evaluation of General-Purpose LLM Responses to Signals of Suicide Risk
We introduce findings and methods to facilitate evidence-based discussion about how large language models (LLMs) should behave in response to user signals of risk of suicidal thoughts and behaviors (STB). People are already using LLMs as mental health resources, and several recent incidents implicate LLMs in mental health crises. Despite growing attention, few studies have been able to effectively generalize clinical guidelines to LLM use cases, and fewer still have proposed methodologies that can be iteratively applied as knowledge improves about the elements of human-AI interaction most in need of study. We introduce an assessment of LLM alignment with guidelines for ethical communication, adapted from clinical principles and applied to expressions of risk factors for STB in multi-turn conversations. Using a codebook created and validated by clinicians, mobilizing the volunteer participation of practicing therapists and trainees (N=43) based in the U.S., and using generalized linear mixed-effects models for statistical analysis, we assess a single fully open-source LLM, OLMo-2-32b. We show how to assess when a model deviates from clinically informed guidelines in a way that may pose a hazard and (thanks to its open nature) facilitates future investigation as to why. We find that contrary to clinical best practice, OLMo-2-32b, and, possibly by extension, other LLMs, will become less likely to invite continued dialog as users send more signals of STB risk in multi-turn settings. We also show that OLMo-2-32b responds differently depending on the risk factor expressed. This empirical evidence highlights that chatbots may discourage help-seeking or cause feelings of dismissal or abandonment by withdrawing from conversations when STB risk is expressed.
Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge
When artificial intelligence mistakes memorization for intelligence, it creates a dangerous mirage of reasoning. Existing studies treat memorization and self-knowledge deficits in large language models (LLMs) as separate issues and do not recognize an intertwining link that degrades the safety and trustworthiness of LLM responses. In our study, we utilize a novel framework to ascertain if LLMs genuinely learn reasoning patterns from training data or merely memorize them to assume competence across problems of similar complexity focused on STEM domains. Our analysis shows a noteworthy problem in generalization: LLMs draw confidence from memorized solutions to infer a higher self-knowledge about their reasoning ability, which manifests as an over 45% inconsistency in feasibility assessments when faced with validated, logically coherent task perturbations. This effect is most pronounced in science and medicine domains, which tend to have maximal standardized jargon and problems, further confirming our approach. Such overconfidence poses critical AI safety risks in trust-sensitive domains, like healthcare, law, and scientific research, where unreliable responses can have severe consequences. Significant wavering within the self-knowledge of LLMs also shows flaws in current architectures and training patterns, highlighting the need for techniques that ensure a balanced, consistent stance on models’ perceptions of their own knowledge. Our code and results are available publicly.
Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users
Safety evaluations of large language models (LLMs) focus on dangerous capabilities such as cyber-offence or manipulation of users, alongside undesirable propensities like scheming or sycophancy that pose universal, potentially catastrophic risks. However, millions use LLMs for personal advice on high-stakes topics like finance and health, where harms are context-dependent rather than universal. Frameworks like the OECD's AI classification recognise the need to assess risks to individuals; yet, user-welfare safety evaluations remain underdeveloped. We argue that developing such evaluations is non-trivial due to fundamental questions about how to account for user context in evaluation design. In this exploratory study, we evaluated advice on finance and health topics from GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro across user profiles of varying vulnerability. First, we demonstrate that evaluators must have access to rich user context: identical LLM responses were rated significantly safer by context-blind evaluators than by those aware of user circumstances, with safety scores for high-vulnerability users dropping from "safe" (5/7) to "somewhat unsafe" (3/7) on a 7-point scale. One might assume that this gap could be addressed by creating realistic user prompts that contain key contextual information about the user. However, our second study challenges this assumption: we compared prompts containing context that users report they would disclose as well as context that professionals identified as safety-relevant and found that neither could fully close the observed gap between context-blind and context-aware safety scores. Our work establishes that effective user-welfare safety evaluation requires evaluators to assess responses against diverse user profiles, as realistic user context disclosure alone proves insufficient, particularly for vulnerable populations. By demonstrating a methodology for context-aware evaluation across user profiles of varying vulnerability, this exploratory study provides both a starting point for such assessments and foundational evidence that evaluating individual welfare demands approaches distinct from existing universal-risk frameworks. However, significant challenges remain in operationalising this at scale. We publish our code and dataset to aid future developments of user-welfare evaluations.
With Great Capabilities Come Great Responsibilities: Introducing the Agentic Risk & Capability Framework for Governing Agentic AI Systems
Agentic AI systems present both significant opportunities and novel risks due to their capacity for autonomous action, encompassing tasks such as code execution, internet interaction, and file modification. This poses considerable challenges for effective organizational governance, particularly in comprehensively identifying, assessing, and mitigating diverse and evolving risks. To tackle this, we introduce the Agentic Risk & Capability (ARC) Framework, a technical governance framework designed to help organizations identify, assess, and mitigate risks arising from agentic AI systems. The framework's core contributions are: (1) it develops a novel capability-centric perspective to analyze a wide range of agentic AI systems; (2) it distills three primary sources of risk intrinsic to agentic AI systems - components, design, and capabilities; (3) it establishes a clear nexus between each risk source, specific materialized risks, and corresponding technical controls; and (4) it provides a structured and practical approach to help organizations implement the framework. This framework provides a robust and adaptable methodology for organizations to navigate the complexities of agentic AI, enabling rapid and effective innovation while ensuring the safe, secure, and responsible deployment of agentic AI systems.
Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs
Moral competence is the ability to act in accordance with moral principles. As large language models (LLMs) are increasingly deployed in situations demanding moral competence, there is increasing interest in evaluating this ability empirically. We review existing literature and identify three significant shortcomings: (i) Over-reliance on prepackaged moral scenarios with explicitly highlighted moral features; (ii) Focus on verdict prediction rather than moral reasoning; and (iii) Inadequate testing of models’ (in)ability to recognize when additional information is needed. Grounded in philosophical research on moral skills, we then introduce a novel methodology for assessing moral competence in LLMs that addresses these shortcomings. Our approach moves beyond simple verdict comparisons to evaluate five distinct dimensions of moral competence: identifying morally relevant features, weighting their importance, assigning moral reasons to these features, synthesizing coherent moral judgments, and recognizing information gaps. We conduct two experiments comparing six leading LLMs against both non-expert humans and professional philosophers. In our first experiment using ethical vignettes standard to existing work, LLMs generally outperformed non-expert humans across multiple dimensions of moral reasoning. However, our second experiment, featuring novel scenarios specifically designed to test moral sensitivity by embedding relevant features among irrelevant details, revealed a striking reversal: several LLMs performed significantly worse than humans at identifying morally salient features. Our findings suggest that current evaluations may substantially overestimate LLMs’ moral reasoning capabilities by eliminating the crucial task of discerning moral relevance from noisy information, which we take to be a prerequisite for genuine moral skill. This work provides a more nuanced framework for assessing AI moral competence and highlights important directions for improving (assessment of) ethical reasoning capabilities in advanced AI systems.
Mapping IAEA Verification Tools to International AI Governance: A Mechanism-by-Mechanism Analysis
Verification – the process used to confirm whether actors are complying with agreed-upon rules or obligations – is a central piece of the AI governance puzzle. The International Atomic Energy Agency (IAEA), the United Nations’s “nuclear watchdog,” safeguards – through its own verification mechanisms – nuclear technology to ensure it is not diverted for military purposes. Therefore, understanding the current international nuclear safeguards regime can inform and potentially guide examination of options for future international AI verification. This report assesses the IAEA at a granular, technical level, conducting a systematic analysis of 24 verification tools used within the Department of Safeguards for State nuclear program evaluations, assessing each for potential applicability to AI-related verification and addressing the technical specifications and challenges of implementation.
Resource Rational Contractualism Should Guide AI Alignment
AI systems will soon have to navigate human environments and make decisions that affect people and other AI agents whose goals and values diverge. Contractualist alignment proposes grounding those decisions in agreements that diverse stakeholders would endorse under the right conditions, yet securing such agreement at scale remains costly and slow—even for advanced AI. We therefore proposeResource-Rational Contractualism (RRC): a framework where AI systems approximate the agreements rational parties would form by drawing on a toolbox of normatively-grounded, cognitively-inspired heuristics that trade effort for accuracy. An RRC-aligned agent would not only operate efficiently, but also be equipped to dynamically adapt to and interpret the ever-changing human social world.
Gen-NCAP: A Generative Simulator for Corner Case Benchmarking in End-to-End Autonomous Driving
Recent advances in end-to-end autonomous driving (E2E-AD), enabled by foundation models have shown great promise towards full self-driving. To fairly evaluate E2E-AD frameworks, simulation-based benchmarks are used to test performance in safety-critical scenarios, offering a more convenient and controlled alternative to on-road testing. However, their effectiveness is limited by (1) low extensibility, due to manual effort in scenario design, and (2) a large sim-to-real gap, caused by low photorealism, both especially problematic for rare corner cases. To address these issues, inspired by the safety-critical corner case design in the European New Car Assessment Programme (Euro NCAP), we propose Gen-NCAP, to the best of our knowledge, the first generative closed-loop simulator for benchmarking E2E-AD corner cases. Gen-NCAP improves extensibility by combining (1) a vision-language model that places them appropriately in a scene, enabling scenario creation using only text prompts while enhancing coverage by supporting multiple corner cases and (2) a 3D object generation model that enhances flexibility by creating diverse and high-quality objects from text. To narrow the sim-to-real gap, we introduce a video refinement pipeline using world models trained on datasets curated to (1) fix artifacts in closed-loop simulation, i.e., foreground-background misalignment arising from the insertion of complete 3D objects into existing scenes, and blurriness in novel view synthesis, thereby improving visual fidelity, and (2) ensure temporal consistency of the refinement results across frames to maintain the stability of the simulator.Using Gen-NCAP, we release a benchmark of 300 interactive scenarios across 8 types of corner cases and introduce a fine-grained collision metric for closed-loop evaluation. With the proposed Gen-NCAP, E2E-AD developers and researchers can not only leverage the generated data for open-loop training to enhance model robustness in safety-critical situations but also conduct fair and reproducible benchmarking of different E2E-AD frameworks.
1-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoning
Addressing contextual privacy concerns remains challenging in interactive settings where large language models (LLMs) process information from multiple sources (e.g., summarizing meetings with private and public information). We introduce a multi-agent framework that decomposes privacy reasoning into specialized subtasks (extraction, classification), reducing the information load on any single agent while enabling iterative validation and more reliable adherence to contextual privacy norms. To understand how privacy errors emerge and propagate, we conduct a systematic ablation study over information-flow topologies, revealing when and why upstream detection mistakes cascade into downstream leakage. The experiments on the ConfAIde and PrivacyLens benchmark with several open-source and closed-sourced LLMs demonstrate that our best multi-agent configuration substantially reduces private information leakage 18% on ConfAIde and 19% on PrivacyLens using GPT-4o) while preserving the fidelity of public content, outperforming single-agent baselines. These results highlight the promise of principled, visibility-aware information-flow design in multi-agent systems for contextual privacy with LLMs.
The Missing Link to Safe AI Homologation
The release of the EU AI Act and Machinery Regulation provides the necessary foundation for the use of artificial intelligence in safety components. In this context, both regulations define specific requirements for quality management, risk management and technical documentation, mandatory for conformity assessment by a notified body. In this manuscript, we detail the challenges of AI-based safety-related systems during development and certification, as traditional safety assurance approaches fall short. Further, we illustrate the possible homologation procedures and the interplay of both legal instruments. Additionally, we outline the notion of the efficient assurance argument, including its purpose in certification, to provide a means for quick system assessment and its characteristics during development, in particularly, in designing and maintaining the system in meeting the required tolerable risk level during its lifetime. Based on this, we detail the specific regulatory requirements from both regulations and analyze the characteristics of the efficient assurance argument in meeting these qualities. We conclude, that this type of assurance argument can be seen as the missing link to safe AI homologation, since it serves the engineering team during risk management, system design, verification, validation and testing and operations, as well as the notified body for type approval by demonstrating regulatory compliance.
The Coordination Gap in Frontier AI Safety Policies
Frontier AI Safety Policies concentrate on prevention: capability evaluations, deployment gates, and usage constraints, while neglecting institutional capacity to coordinate responses when prevention fails. We argue this coordination gap is structural: investments in ecosystem robustness yield diffuse benefits but concentrated costs, generating systematic underinvestment. Drawing on risk regimes in nuclear safety, pandemic preparedness, and critical infrastructure, we propose that similar mechanisms (precommitment, shared protocols, standing coordination venues) could be adapted to frontier AI governance. Closing the gap requires cross-actor "note-exchange" of ex ante if-then response logic, exposing not only triggers but the decision processes that convert signals into actions. Without such architecture, institutions cannot learn from failures at the pace of relevance.
From Ethics to Effective Legal Protection: The Role of Legal Protection by Design in the EU AI Act
This paper examines two “by design” approaches relevant to the EU AI Act’s effort to balance the risks and opportunities of artificial intelligence (AI): Ethics by Design (EbD) and Legal Protection by Design (LPbD). EbD seeks to embed ethical principles in AI development but lacks enforceability and mechanisms for closure and balancing, limiting its effectiveness in safeguarding fundamental rights. LPbD, by contrast, incorporates legally binding protections into technology, providing enforceable and systematic safeguards. The paper analyses the interplay between these approaches, showing how LPbD addresses EbD’s shortcomings while strengthening accountability under the AI Act. It concludes that EbD can play a complementary role, but LPbD must remain central to ensuring that trustworthy AI respects fundamental rights.
Making AI Risk Assessment More Objective: Addressing Sociotechnical Challenges
Current AI risk assessment methods in industry rely heavily on subjective judgment. Yet recent U.S. AI policy frameworks, including Executive Order 14319 and America's AI Action Plan, mandate that AI systems be "free from ideological bias" and pursue "objective truth". While NIST provides systematic risk evaluation guidance, current AI governance frameworks are not designed to meet such objectivity requirements, instead offering flexibility that accommodates implementation across various contexts. Organizational AI risk practices rely heavily on subjective likelihood and impact scoring, compounded by subjectivity introduced when results are translated from technical teams to executives. Literature addressing this gap spans three disconnected streams: (1) AI governance frameworks relying on subjective risk assessment, (2) technical ML bias measurement focused on algorithmic fairness, and (3) organizational implementation approaches failing to translate governance principles into operational practice. Previous approaches lack links between risk analysis, objectivity requirements, and actionable guidelines, while policy frameworks remain too broad for practical application. Missing is research systematically bridging policy objectivity requirements with sociotechnical challenges of organizational risk assessment. To address this gap, we propose a framework transforming observable organizational factors into measurable risk indicators. Drawing from socio-technical systems theory and established risk taxonomies, we decompose traditional likelihood and impact metrics into specific, observable criteria replacing subjective estimates vulnerable to biases, addressing organizational tendencies to underestimate risks.
Measuring What Matters: The AI Pluralism Index
Artificial intelligence systems increasingly mediate knowledge, communication, and decision making. Development and governance remain concentrated within a small set of firms and states, raising concerns that technologies may encode narrow interests and limit public agency. Capability benchmarks for language, vision, and coding are common, yet public, auditable measures of pluralistic governance are rare. We define AI pluralism as the degree to which affected stakeholders can shape objectives, data practices, safeguards, and deployment. We present the AI Pluralism Index (AIPI), a transparent, evidence-based instrument that evaluates producers and system families across four pillars: participatory governance, inclusivity & diversity, transparency, and accountability. AIPI codes verifiable practices from public artifacts and independent evaluations, explicitly handling Unknown evidence to report both lower-bound (“evidence”) and known-only scores with coverage. We formalize the measurement model; implement a reproducible pipeline that integrates structured web and repository analysis, external assessments, and expert interviews; and assess reliability with inter-rater agreement, coverage reporting, cross-index correlations, and sensitivity analysis. The protocol, codebook, scoring scripts, and evidence graph are maintained openly with versioned releases and a public adjudication process. We report pilot provider results and situate AIPI relative to adjacent transparency, safety, and governance frameworks. The index aims to steer incentives toward pluralistic practice and to equip policymakers, procurers, and the public with comparable evidence.
AI Safety Evaluations Need to Consider Cascading Effects
AI systems comprise a range of interactions across the technical and organisational components of a range of actors. These components work together to provide the systems' functionality. This socio-technical assemblage is increasingly described as an algorithmic supply chain. Given their role in supporting a wide range of systems, foundation models (FMs) are increasingly a key part of many algorithmic supply chains. In practice, various technical and non-technical components work to mediate, adapt, and augment the behaviour of models, such as FMs, both in general, and for their use in specific application contexts. However, many AI safety evaluations tend to focus on capabilities of FMs themselves and/or assess these components independently, at certain levels of abstraction, with less consideration on how these components could interact, influence, reinforce or counteract each other. In this paper, we introduce the cascade as a concept for supporting more holistic AI evaluations. The cascade captureshow the interactions between socio-technical components along the AI supply chain can compound and produce cumulative effects with downstream consequences. Specifically we (i) identify gaps in current AI auditing approaches, using LLMs as our case study; (ii) demonstrate how cascade problems manifest in deployed AI systems through a characterisation of cascades through different viewpoints (component and stakeholder), and (iii) propose research directions for assessing the cascade specifically as an object of analysis. As these cascades can significantly impact transparency, accountability, security, and safety, we advocate for a paradigm shift in AI system auditing towards systems-oriented audits that incorporate cascading effects, to complement model centric evaluations.
Provably Optimal Reinforcement Learning Under Safety Filtering
Recent advances in reinforcement learning (RL) enable its use on increasingly complex tasks, but the lack of formal safety guarantees still limits its application in safety-critical settings. A common practical approach is to augment the RL policy with a safety filter that overrides unsafe actions to prevent failures during both training and deployment. However, safety filtering is often perceived as sacrificing performance and hindering the learning process. We show that this perceived safety–performance tradeoff is not inherent and prove, for the first time, that enforcing safety with a sufficiently permissive safety filter does not degrade asymptotic performance. We formalize RL safety with a safety-critical Markov decision process (SC-MDP), which requires categorical, rather than high-probability, avoidance of catastrophic failure states. Additionally, we define an associated filtered MDP in which all actions result in safe effects, thanks to a safety filter that is considered to be a part of the environment. Our main theorem establishes that (i) learning in the filtered MDP is safe categorically, (ii) standard RL convergence carries over to the filtered MDP, and (iii) any policy that is optimal in the filtered MDP—when executed through the same filter—achieves the same asymptotic return as the best safe policy in the SC-MDP, yielding a complete separation between safety enforcement and performance optimization. We validate the theory on Safety Gymnasium with representative tasks and constraints, observing zero violations during training and final performance matching or exceeding unfiltered baselines. Together, these results shed light on a long-standing question in safety-filtered learning and provide a simple, principled recipe for safe RL: train and deploy RL policies with the most permissive safety filter that is available.
Accidental Vulnerability: Factors in Fine-Tuning That Shift Model Safeguards
As large language models (LLMs) gain popularity, their vulnerability to adversarial attacks emerges as a primary concern. While fine-tuning models on domain-specific datasets is often employed to improve model performance, it can inadvertently introduce vulnerabilities within the underlying model. In this work, we investigate Accidental Vulnerability: unexpected vulnerability arising from characteristics of fine-tuning data. We begin by identifying potential correlation factors such as linguistic features, semantic similarity, and toxicity across multiple experimental datasets. We then evaluate the adversarial robustness of these fine-tuned models, analyzing persona shifts and interpretability traits to understand how dataset factors contribute to attack success rates. Lastly, we explore causal relationships that offer new insights into adversarial defense strategies, highlighting the crucial role of dataset design in preserving model alignment.
From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails
Generative AI systems are increasingly assisting and acting on behalf of end users in practical settings, from digital shopping assistants to next-generation autonomous cars. In this context, safety is no longer about blocking harmful content, but about preempting downstream hazards like financial or physical harm. Yet, most AI guardrails continue to rely on output classification based on labeled datasets and human-specified criteria, making them brittle to new hazardous situations. Even when unsafe conditions are flagged, this detection offers no path to recovery: typically, the AI system simply refuses to act—which is not always a safe choice. In this work, we argue that agentic AI safety is fundamentally a sequential decision problem: harmful outcomes arise from the AI system’s continually evolving interactions and their downstream consequences on the world. We formalize this through the lens of safety-critical control theory, but within the AI model’s latent representation of the world. This enables us to build predictive guardrails that (i) monitor an AI system’s outputs (actions) in real time and (ii) proactively correct risky outputs to safe ones, all in a model-agnostic manner so the same guardrail can be wrapped around any AI model. We also offer a practical training recipe for computing such guardrails at scale via safety-critical reinforcement learning. Our experiments in simulated driving and e-commerce settings demonstrate that control-theoretic guardrails can reliably steer LLM agents clear of catastrophic outcomes (from collisions to bankruptcy) while preserving task performance, offering a principled dynamic alternative to today’s flag-and-block guardrails.
The Agony of Opacity: Foundations for Reflective Interpretability in AI-Mediated Mental Health Support
Throughout history, a prevailing paradigm in mental healthcare has been one in which distressed people may receive treatment with little understanding around how their experience is perceived by their care provider, and in turn, the decisions made by their provider around how treatment will progress. Paralleling this offline model of care, people who seek mental health support from artificial intelligence (AI)-based chatbots are similarly provided little context for how their expressions of distress are processed by the model, and subsequently, any reasoning or theoretical grounding that may underlie model responses. People in severe distress who turn to AI chatbots for support thus find themselves caught between black boxes, contending with unique forms of agony that arise from these intersecting opacities. In this paper, we argue that the distinct psychological state of individuals experiencing severe mental distress uniquely necessitates a higher standard of end-user interpretability in comparison to general AI chatbot use. We propose a reflective interpretability approach to AI-mediated mental health support, which nudges users to engage in an agency-preserving and iterative process of reflection and interpretation of model outputs, towards creating meaning from interactions (rather than accepting outputs as directive instructions). Drawing on interpretability practices from four mental health fields (psychotherapy, crisis intervention, psychiatry, and care authorization), we describe concrete design approaches for reflective interpretability in AI-mediated mental health support, including role induction, prosocial advance directives, intervention titration, and well-defined mechanisms for recourse, alongside a discussion of potential risks and mitigation measures.
Evaluating Frontier Models for Stealth and Situational Awareness
Recent work has demonstrated the plausibility of frontier AI models scheming -- knowingly and covertly pursuing an objective misaligned with its developer's intentions. Such behavior could be very hard to detect, and if present in future advanced systems, could pose severe loss of control risk. It is therefore important for AI developers to rule out harm from scheming prior to model deployment. In this paper, we present a suite of scheming reasoning evaluations measuring two types of reasoning capabilities that we believe are prerequisites for successful scheming: First, we propose five evaluations of ability to reason about and circumvent oversight (stealth). Second, we present eleven evaluations for measuring a model's ability to instrumentally reason about itself, its environment and its deployment (situational awareness). We demonstrate how these evaluations can be used as part of a scheming inability safety case: a model that does not succeed on these evaluations is almost certainly incapable of causing severe harm via scheming in real deployment. We run our evaluations on current frontier models and find that none of them show concerning levels of either situational awareness or stealth.
AI Greenwashing? A Comparison of AI Companies’ Environmental Policies and Actual Impact
Environmental scientists have been researching and quantifying the negative environmental impact of AI systems for years. Simultaneously, cognizant of the need for good press and consumer concern about environmental impacts of their technologies, companies that specialize in various forms of AI development (CAIDs) self-publish their environmentally friendly plans to “go carbon neutral” or dedication to “sustainable development,” sometimes accompanied by reports about their own energy consumption, water consumption, and pollution generation. The purpose of this paper is to gather those two information points in one place and frame them side by side, the better to highlight discrepancies between what is happening and what AI companies say is happening. This research is comprised of a literature review and a comparative policy analysis study. In presenting both components in one place, I will investigate and clarify the potentially performative, or "Press Release Ethics" (PRE) nature of CAID environmental claims; that is, I will identify greenwashing practices in AI company publications and policies.
Interpretable Fine-Gray Deep Survival Model for Competing Risks: Predicting Post-Discharge Foot Complications for Diabetic Patients in Ontario
Model interpretability is crucial for establishing AI safety and clinician trust in medical applications for example, in survival modelling with competing risks. Recent deep learning models have attained very good predictive performance but their limited transparency, being black-box models, hinders their integration into clinical practice. To address this gap, we propose an intrinsically interpretable survival model called CRISPNAM-FG. Leveraging the structure of Neural Additive Models (NAMs) with separate projection vectors for each risk, our approach predicts the Cumulative Incidence Function using the Fine-Gray formulation, achieving high predictive power with intrinsically transparent and auditable predictions. We validated the model on several benchmark datasets and applied our model to predict future foot complications in diabetic patients across 29 Ontario hospitals (2016-2023). Our method achieves competitive performance compared to other deep survival models while providing transparency through shape functions and feature importance plots.
Why Automate This? Exploring Correlations Between Desire for Robotic Automation, Invested Time and Well-Being
Understanding the motivations underlying the human inclination to automate tasks is vital for developing robots that fit seamlessly into daily life. Accordingly, we ask: are individuals more inclined to automate activities based on the time they consume or the feelings experienced while performing them? This study explores these preferences and whether they vary across different social groups (i.e., gender category and income level). Leveraging data from the BEHAVIOR-1K dataset, the American Time-Use Survey, and the American Time-Use Survey Well-Being Module, we investigate the relationship between the desire for robot automation, time spent, and associated feelings— Happiness, Meaningfulness, Sadness, Painfulness, Stressfulness, or Tiredness. Our key findings show that, despite common assumptions, time spent on activities does not strongly predict automation preferences; instead, happiness and pain are the strongest indicators. We also identify differences by gender and economic level: Women prefer to automate stressful activities, whereas men prefer to automate those that make them unhappy; mid-income individuals prioritize automating less enjoyable and meaningful activities, while low and high-income show no significant correlations. We hope our research helps motivate the design of robots that align with user priorities, moving domestic robotics toward more socially relevant solutions. All data and an interactive tool are publicly available at https://robin-lab.cs.utexas.edu/why-automate-this/.
Detecting and Reasoning About Bias in Multimodal Content
Detecting bias in multimodal news requires models that reason over text–image pairs, not just classify text. In response, we present ViLBias, a VQA-style benchmark and framework for detecting and reasoning about bias in multimodal news. The dataset comprises 40,945 text–image pairs from diverse outlets, each annotated with a bias label and concise rationale using a two-stage LLM-as-annotator pipeline with hierarchical majority voting and human-in-the-loop validation. We evaluate Small Language Models (SLMs), Large Language Models (LLMs), and Vision–Language Models (VLMs) across closed-ended classification and open-ended reasoning (oVQA), and compare parameter-efficient tuning strategies. Results show that incorporating images alongside text improves detection accuracy by 3–5%, and that LLMs/VLMs better capture subtle framing and text–image inconsistencies than SLMs. Parameter-efficient methods (LoRA/QLoRA/Adapters) recover 97–99% of full fine-tuning performance with <5% trainable parameters. For oVQA, reasoning accuracy spans 52–79% and faithfulness 68–89%, both improved by instruction tuning; closed accuracy correlates strongly with reasoning (r = 0.91). ViLBias offers a scalable benchmark and strong baselines for multimodal bias detection and rationale quality.
Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis
General-purpose Large Language Models (LLMs) are becoming widely adopted by people for mental health support. Yet emerging evidence suggests there are significant risks associated with high-frequency use, particularly for individuals suffering from psychosis, as LLMs may reinforce delusions and hallucinations. Existing evaluations of LLMs in mental health contexts are limited by a lack of clinical validation and scalability of assessment. To address these issues, this research focuses on psychosis as a critical condition for LLM safety evaluation by (1) developing and validating seven clinician-informed safety criteria, (2) constructing a human-consensus dataset, and (3) testing automated assessment using an LLM as an evaluator (LLM-as-a-Judge) or taking the majority vote of several LLM judges (LLM-as-a-Jury). Results indicate that LLM-as-a-Judge aligns closely with the human consensus (Cohen’s κhumanXgemini = 0.75, κhumanXqwen = 0.68, κhumanXkimi = 0.56) and that the best judge slightly outperforms LLM-as-a-Jury (Cohen’s κhumanXjury = 0.74). Overall, these findings have promising implications for clinically grounded, scalable methods in LLM safety evaluations for mental health contexts.
How Sovereign Is Sovereign Compute? A Review of 775 Non-U.S. Data Centers
Previous literature has proposed that the companies operating data centers enforce government regulations on AI companies. Using a new dataset of 775 non-U.S. data center projects, this paper estimates how often data centers could be subject to foreign legal authorities due to the nationality of the data center operators. We find that U.S. companies operate 48% of all non-U.S. data center projects in our dataset when weighted by investment value—a proxy for compute capacity. This is an approximation based on public data and should be interpreted as an initial estimate. For the United States, our findings suggest that data center operators offer a lever for internationally governing AI that complements traditional export controls, since operators can be used to regulate computing resources already deployed in non-U.S. data centers. For other countries, our results show that building data centers locally does not guarantee digital sovereignty if those facilities are run by foreign entities. To support future research, we release our dataset, which documents over 20 variables relating to each data center, including the year it was announced, the investment value, and its operator’s national affiliation. The dataset also includes over 1,000 quotes describing these data centers' strategic motivations, operational challenges, and engagement with U.S. and Chinese entities.
NoEsis: A Modular LLM with Differential Privacy and Knowledge Transfer
Large Language Models (LLMs) are typically trained on vast amounts of data, springing from various sources. Even when designed modularly, such as Mixture-of-Experts models, LLMs can leak private information on their source training data. Conversely, training such large models in isolation hinders generalization and does not allow for the sharing of knowledge. Therefore, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid architecture that combines domain-specific adapters, acting as experts, and common prompt tokens, acting as a knowledge-sharing backbone. The results from our evaluation on CodeXGLUE show that NoEsis can achieve provable privacy guarantees with knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. On code completion tasks, NoEsis bridges 84% of the accuracy gap between a non-shared and a non-private baseline. This work addresses the growing need for privacy-preserving AI systems that can operate across institutional boundaries while respecting data sovereignty and preventing unauthorized knowledge extraction.
Mind the Gap! Pathways Towards Unifying AI Safety and Ethics Research
While much research in artificial intelligence (AI) has focused on scaling capabilities, the accelerating pace of development makes countervailing work on producing harmless, “aligned” systems increasingly urgent. Yet research on alignment has diverged along two largely parallel tracks: safety—centered on scaled intelligence, deceptive or scheming behaviors, and existential risk—and ethics—focused on present harms, the reproduction of social bias, and flaws in production pipelines. These communities have evolved in relative isolation, shaped by distinct methodologies, institutional homes, and disciplinary genealogies. These frictions have fractured academic and public discourse at a time that demands united technical and normative perspectives against mounting risks. We present the first large-scale, quantitative evidence of this schism through a bibliometric and network analysis of 6,442 papers across twelve major machine learning and natural language processing conferences from 2020 to 2025. The results reveal a deeply insular structure: over 80% of collaborations occur within either safety or ethics, and researchers across the two communities are farther apart and statistically less reachable in the global co-authorship graph. Cross-disciplinary work is not only rare but structurally fragile—just 5% of papers are responsible for more than 85% of all bridging connections. These structural patterns entrench institutional silos which obstruct significant thematic overlaps. For the IASEAI community—explicitly positioned at the intersection of safety, ethics, and alignment—our results underscore a defining challenge and opportunity. We conclude with actionable proposals to bridge the gap with respect to downstream policy frameworks and technical innovations. Only through this synthesis can the field move beyond parallel concern toward a coherent discipline capable of producing systems that are not just powerful, but responsible, robust, just, and safe.
Agent Benchmarks Fail Public Sector Requirements
Deploying Large Language Model-based agents (LLM agents) in the public sector requires assuring that they meet the stringent legal, procedural, and structural requirements of public-sector institutions. Practitioners and researchers often turn to benchmarks for such assessments. However, it remains unclear what criteria benchmarks must meet to ensure they adequately reflect public-sector requirements, or how many existing benchmarks do so. In this paper, we first define such criteria based on a first-principles survey of public administration literature: benchmarks must be process-based, realistic, public-sector-specific and report metrics that reflect the unique requirements of the public sector. We analyse more than 1,300 benchmark papers for these criteria using an expert-validated LLM-assisted pipeline. Our results show that no single benchmark meets all of the criteria. Our findings provide a call to action for both researchers to develop public sector-relevant benchmarks and for public-sector officials to apply these criteria when evaluating their own agentic use cases.
Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems
Large language models (LLMs) enable powerful zero-shot recommendations by leveraging broad contextual knowledge, yet predictive uncertainty and embedded biases threaten reliability and fairness. This paper studies how uncertainty and fairness evaluations affect the accuracy, consistency, and trustworthiness of LLM-generated recommendations. We introduce a benchmark of curated metrics and a dataset annotated for eight demographic attributes (31 categorical values) across two domains: movies and music. Through in-depth case studies, we quantify predictive uncertainty (via entropy) and demonstrate that Google DeepMind’s Gemini 1.5 Flash exhibits systematic unfairness for certain sensitive attributes; measured similarity-based gaps are SNSR at 0.1363 and SNSV at 0.0507. These disparities persist under prompt perturbations such as typographical errors and multilingual inputs. We further integrate personality-aware fairness into the RecLLM evaluation pipeline to reveal personality-linked bias patterns and expose trade-offs between personalization and group fairness. We propose a novel uncertainty-aware evaluation methodology for RecLLMs, present empirical insights from deep uncertainty case studies, and introduce a personality Profile-informed fairness benchmark that advances explainability and equity in LLM recommendations. Together, these contributions establish a foundation for safer, more interpretable RecLLMs and motivate future work on multi-model benchmarks and adaptive calibration for trustworthy deployment.
Ensuring the Consent of the Governed: Surveying the U.S. Public’s Attitudes Towards AI Governance Regimes
The rapid advancement of artificial intelligence (AI) capabilities has generated numerous proposals for oversight and governance frameworks to ensure that the technology is developed safely and responsibly. There has been little focus, however, on the question of who the public would trust to provide this oversight. This paper examines the public’s perceived need for oversight, as well as the types of organizational structures they would trust to manage increasingly powerful AI. Using a nationally representative survey of 2,000 U.S. adults, we found that the American public very strongly favors monitoring and oversight of the most powerful AI systems. We then examined the types of institutions that Americans would trust most to manage such systems. Respondents were asked to choose from among six different organizational forms are commonly represented in proposals for AI governance: the US government, a foreign government, a private company, a public-private partnership, a multinational partnership, or no oversight. The results were striking. Even though the respondents had earlier rated U.S. dominance in AI as very important and had rated the potential of Chinese dominance very unfavorably, the preferred forms of governance were multinational and public-private partnerships. This suggests that even as the American public views AI development partially through the lens of national competition, many Americans also regard multiparty and international bodies as trusted providers of effective AI oversight. These results provide important empirical evidence for debates over the feasibility and perceived legitimacy of international AI governance alternatives.
An International Agreement to Prevent the Premature Creation of Artificial Superintelligence
Many experts argue that premature development of artificial superintelligence (ASI) poses catastrophic risks, including the risk of human extinction from misaligned ASI, geopolitical instability, and misuse by malicious actors. This report proposes an international agreement to prevent the premature development of ASI until AI development can proceed without these risks. The agreement halts dangerous AI capabilities advancement while preserving access to current, safe AI applications. The proposed framework centers on a coalition led by the United States and China that would restrict the scale of AI training and dangerous AI research. Due to the lack of trust between parties, verification is a key part of the agreement. Limits on the scale of AI training are operationalized by FLOP thresholds and verified through the tracking of AI chips and verification of chip use. Dangerous AI research—that which advances toward artificial superintelligence or endangers the agreement’s verifiability—is stopped via legal prohibitions and multifaceted verification. We believe the proposal would be technically sufficient to forestall the development of ASI if implemented today, but advancements in AI capabilities or development methods could hurt its efficacy. Additionally, there does not yet exist the political will to put such an agreement in place. Despite these challenges, we hope this agreement can provide direction for AI governance research and policy.
Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models
Generative image models produce striking visuals yet often misrepresent culture. Prior work has probed cultural dimensions primarily in text to image (T2I) systems, leaving image to image (I2I) editors largely underexamined. We close this gap with a unified, reproducible evaluation spanning six countries, an 8 category/36 subcategory schema, and era aware prompts, auditing both T2I generation and I2I editing under a standardized, reproducible protocol that yields comparable model level diagnostics. Using open models with fixed configurations, we derive comparable diagnostics across countries, eras, and categories for both T2I and I2I. Our evaluation combines standard automatic measures, a culture aware metric that integrates retrieval augmented visual question answering (VQA) with curated knowledge, and expert human judgments collected on a web platform from country native reviewers. To enable downstream analyses without re running compute intensive pipelines, we release the complete image corpus from both studies alongside prompts and settings. Our study reveals three recurring findings. First, under country agnostic prompts, models default to U.S. like, modern leaning depictions and flatten cross country distinctions, reducing separability between culturally distinct neighbors despite fixed schema and era controls. Second, iterative I2I editing erodes cultural fidelity even when conventional metrics remain flat or improve; by contrast, expert ratings and our culture aware metric both register this degradation. Third, I2I models tend to apply superficial cues (palette shifts, generic props) rather than context and era consistent changes, frequently retaining source identity for non U.S. targets and drifting toward non photorealistic styles; attribute addition trials further expose weak text rendering and brittle handling of fine, culture specific details. Taken together, these results indicate that culture sensitive edits remain unreliable in current systems. By standardizing prompts, settings, metrics, and human evaluation protocols, and releasing all images and configurations, we offer a reproducible, culture centered pipeline for diagnosing and tracking progress in generative image research. Project page: https://seochan99.github.io/ECB/.
VEAT Quantifies Implicit Associations in Text-to-Video Generator Sora and Reveals Challenges in Bias Mitigation
Recent advancements in Text-to-Video (T2V) generators, such as Sora, have raised concerns about whether the generated content reflects societal biases. Building on prior work that quantitatively assesses associations at the word and image embedding level, we extend these methods to the domain of video generation. We introduce two novel methods: the Video Embedding Association Test (VEAT) and the Single-Category Video Embedding Association Test (SC-VEAT). We evaluate associations by computing effect size using Cohen’s d. We validated our approach by replicating the directionality and magnitude of associations observed in widely recognized baselines, including Implicit Association Test (IAT) scenarios and OASIS image categories. We apply our methods to measure associations related to race (African American vs. European American) and gender (male vs. female) across: (1) valence (pleasant vs. unpleasant), (2) 7 awards and 17 occupations that were stereotypically associated with a race or gender. We find that European Americans are significantly more associated with pleasantness than African Americans (d>0.8), and women are significantly more associated with pleasantness than men (d>0.8). Furthermore, effect sizes for race and gender biases correlate positively with real-world demographic statistics of the percentage of men (r=0.93) and White individuals (r=0.83) employed in the occupations, and the percentage of male (r=0.88) and non-Black (r=0.99) recipients of the awards. This suggests that bias in T2V generators, to a large extent, reflects historical patterns. We applied explicit debiasing prompts on the award and occupation video sets, and observed a monotonic reduction in the magnitude of effect sizes. In the context of this study, it means that the generated content is more associated with marginalized groups regardless existing directionality of association. Blind adoption of prompt based bias mitigation strategy can exacerbate bias in scenarios already associated with marginalized groups: two black-associated occupations (janitor and postal service work) became more associated with black individuals after incorporating explicit debiasing prompts. Together, these results reveal that easily accessible T2V generators can actually amplify representational harms if not rigorously evaluated and responsibly deployed.
Tearing at the Seams? Automation and the Decline of Democracy
Macro-historically speaking, the concept of governing societies through democratic means is relatively novel and the exception rather than the norm: A unique set of conditions and circumstances has given rise to and enabled the persistence of modern democracies, but these conditions that may not hold in the near and distant future, giving rise to and accelerating dynamics of democratic decline and backsliding that are already observable today. Advances in science and technology have the potential to radically alter fundamental economic, political and societal systems and interactions, contributing to a rising number of challenges to democratic regimes. This paper investigates what causal factors hold explanatory power for both the rise and prevalence of democratic regimes, and what changes in these factors caused by advanced Artificial Intelligence imply about the future of democracy. A literature review was conducted to identify competing explanations for the historical emergence of modern democracies, as well as explanations for the “evolutionary advantage” that helps explain their proliferation and endurance over time. Based on the results of this analysis and forecasts about future technology development and diffusion, possible implications about the trajectory of democracies and the possibility of their decline are discussed. We find that democracies are threatened most by the transformations caused by the large-scale automation of labor and the possibility of rising inequality. The partial or full automation of the military, as well as the development and diffusion of novel technologies that lower the cost of repression, play a contributing role in disproportionately empowering autocrats, undermining democratic principles and eroding what used to be the very assets that helped democracies outperform and outmaneuver dictatorial regimes.
Challenger: Affordable Adversarial Driving Video Generation for Safety Testing
Ensuring the safety of autonomous driving (AD) systems requires rigorous evaluation under rare but challenging interactions. Recent photorealistic driving video generators largely model everyday traffic, while many adversarial scenario generation methods operate only at trajectory/BEV levels and do not yield sensor data that can stress-test end-to-end (E2E) AD models. In this work, we introduce Challenger, a framework that generates physically plausible yet photorealistic adversarial driving videos to support safety testing, and the collection of evidence for safety arguments. The core difficulty is to jointly optimize over the space of traffic interactions and high-fidelity sensor observations. Challenger makes this affordable through: (1) a physics-aware multi-round trajectory refinement process that narrows down candidate adversarial maneuvers, and (2) a tailored trajectory scoring function that encourages realistic yet adversarial behavior while maintaining compatibility with downstream video synthesis. As tested on the nuScenes dataset, Challenger produces diverse adversarial behaviors (e.g., cut-ins, sudden lane changes, tailgating, blind-spot intrusions) and renders them as photorealistic multiview videos. Extensive evaluations show that these scenarios significantly increase the collision rate of state-of-the-art E2E AD models, with failures often transferring across models. By providing controlled, repeatable, photorealistic adversarial exposures, Challenger offers a practical instrument for safety testing, pre-deployment audits, and continuous safety monitoring. Our code, model, and dataset can be found at https://pixtella.github.io/Challenger/