On the Utility and Limitations of the MSTAR Dataset for Deep Learning-Based SAR Target Recognition - Milligan, Charles (Proceedings of the AAAI Symposium Series, 2026)
The DARPA Moving and Stationary Target Acquisition andRecognition (MSTAR) dataset has served as a foundationalbenchmark for syntheticaperture radar (SAR)–based automatic target recognition(ATR) research for more than two decades. In recent years,it has been widely adopted for training and evaluating deeplearning models, with reported classification accuraciesoften exceeding 95%. While these results demonstrate theeffectiveness of modern neural architectures, they alsoraise concerns regarding the representativeness andcontinued utility of MSTAR as a proxy for operational SARATR tasks. In this paper, we examine the strengths andlimitations of the MSTAR dataset in the context of deeplearning–based model development. We present representativeperformance results from a convolutional neural networktrained on MSTAR imagery, illustrating how datasetcharacteristics—such as centered targets, fixed chipdimensions, limited clutter, and single-object scenes—leadto artificially easy classification problems. We furthercontrast these results with preliminary experiments on theATRNet-STAR dataset, where modestperturbations including target offset and rotation producesubstantial performance degradation. Collectively, thesefindings highlight the need for more realistic SAR datasetsand evaluation protocols. This paper is intended as a workin progress, with ongoing analysis of embedding-spacestructure and robustness metrics to be reported in futurerevisions.