Do Compliance Judges Read the Rules? READ: Robustness Through Erasure, Ablation and Diagnosis of LLM Policy Grounding

Abstract

Large language models are increasingly deployed as zero-shot judges of policy compliance: a policy is placed in the prompt alongside a candidate response, and the model returns a verdict without per-policy training. Standard accuracy benchmarks cannot distinguish a verdict grounded in the supplied policy from one produced by safety priors acquired during pretraining, leaving deployers without a way to audit policy adherence directly. We present READ (Robustness through Erasure, Ablation, and Diagnosis), a perturbation-based diagnostic that pairs each scenario with the original policy, the policy with the violated clause removed, the policy with each non-violated clause removed in turn, and the policy removed entirely. From the verdict shifts we compute five Policy-Adherence Metrics: Policy Influence Rate, Clause Substitution Rate, Ablation Specificity Rate, Context Consistency Rate, and Policy-Corrected Clause Substitution Rate. We organize their patterns into four per-instance failure modes, Prior-Based, Policy-Degraded, Confused, and Faithful, plus a Degenerate flag for judges that fail the per-class F1 check. Across our evaluation, headline accuracy does not predict policy-dependence: the most accurate judge in our table reaches 92.5 percent accuracy at a Policy Influence Rate of only 15.3 percent, showing that high accuracy can co-occur with the policy moving the verdict on roughly one instance in seven. The metrics are simple enough to act on a single deployment decision. The evaluation covers six open-weight judges across five organizational-policy domains and aggregates approximately 3.5 million graded decisions.