Safety Drift After Fine-Tuning: Evidence from High-Stakes Domains

Abstract

Foundation models are routinely fine-tuned for use in particular domains, yet safety assessments are typically conducted only on base models, implicitly assuming that safety properties persist through downstream adaptation. To test this assumption, we evaluate an ecologically valid sample of fine-tuned models for medical and legal applications. We compare each with their respective base models to assess changes in general-purpose and domain specific safety behavior from fine-tuning. We find that benign fine-tuning induces large, heterogeneous, and often contradictory changes in measured safety: models frequently improve on some instruments while degrading on others, with substantial disagreement across evaluations. We show that safety performance changes unpredictably under realistic downstream adaptation. Our results indicate that models must be evaluated after modification to accurately measure risk. Given that many AI governance proposals rely on base model evaluations, this casts doubt on their ability to effectively manage risk. At the same time, allocating all responsibility to actors that fine-tune and deploy models, risks placing undue burden on the least resourced and powerful steps of model deployment. To address this, we propose a governance framework where upstream actors provide resources to support downstream risk measurement.