Gen-NCAP: A Generative Simulator for Corner Case Benchmarking in End-to-End Autonomous Driving

Abstract

Recent advances in end-to-end autonomous driving (E2E-AD), enabled by foundation models have shown great promise towards full self-driving. To fairly evaluate E2E-AD frameworks, simulation-based benchmarks are used to test performance in safety-critical scenarios, offering a more convenient and controlled alternative to on-road testing. However, their effectiveness is limited by (1) low extensibility, due to manual effort in scenario design, and (2) a large sim-to-real gap, caused by low photorealism, both especially problematic for rare corner cases. To address these issues, inspired by the safety-critical corner case design in the European New Car Assessment Programme (Euro NCAP), we propose Gen-NCAP, to the best of our knowledge, the first generative closed-loop simulator for benchmarking E2E-AD corner cases. Gen-NCAP improves extensibility by combining (1) a vision-language model that places them appropriately in a scene, enabling scenario creation using only text prompts while enhancing coverage by supporting multiple corner cases and (2) a 3D object generation model that enhances flexibility by creating diverse and high-quality objects from text. To narrow the sim-to-real gap, we introduce a video refinement pipeline using world models trained on datasets curated to (1) fix artifacts in closed-loop simulation, i.e., foreground-background misalignment arising from the insertion of complete 3D objects into existing scenes, and blurriness in novel view synthesis, thereby improving visual fidelity, and (2) ensure temporal consistency of the refinement results across frames to maintain the stability of the simulator.Using Gen-NCAP, we release a benchmark of 300 interactive scenarios across 8 types of corner cases and introduce a fine-grained collision metric for closed-loop evaluation. With the proposed Gen-NCAP, E2E-AD developers and researchers can not only leverage the generated data for open-loop training to enhance model robustness in safety-critical situations but also conduct fair and reproducible benchmarking of different E2E-AD frameworks.