Challenger: Affordable Adversarial Driving Video Generation for Safety Testing

Abstract

Ensuring the safety of autonomous driving (AD) systems requires rigorous evaluation under rare but challenging interactions. Recent photorealistic driving video generators largely model everyday traffic, while many adversarial scenario generation methods operate only at trajectory/BEV levels and do not yield sensor data that can stress-test end-to-end (E2E) AD models. In this work, we introduce Challenger, a framework that generates physically plausible yet photorealistic adversarial driving videos to support safety testing, and the collection of evidence for safety arguments. The core difficulty is to jointly optimize over the space of traffic interactions and high-fidelity sensor observations. Challenger makes this affordable through: (1) a physics-aware multi-round trajectory refinement process that narrows down candidate adversarial maneuvers, and (2) a tailored trajectory scoring function that encourages realistic yet adversarial behavior while maintaining compatibility with downstream video synthesis. As tested on the nuScenes dataset, Challenger produces diverse adversarial behaviors (e.g., cut-ins, sudden lane changes, tailgating, blind-spot intrusions) and renders them as photorealistic multiview videos. Extensive evaluations show that these scenarios significantly increase the collision rate of state-of-the-art E2E AD models, with failures often transferring across models. By providing controlled, repeatable, photorealistic adversarial exposures, Challenger offers a practical instrument for safety testing, pre-deployment audits, and continuous safety monitoring. Our code, model, and dataset can be found at https://pixtella.github.io/Challenger/