On the Utility and Limitations of the MSTAR Dataset for Deep Learning-Based SAR Target Recognition

Abstract

The DARPA Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset has served as a foundational benchmark for synthetic aperture radar (SAR)–based automatic target recognition (ATR) research for more than two decades. In recent years, it has been widely adopted for training and evaluating deep learning models, with reported classification accuracies often exceeding 95%. While these results demonstrate the effectiveness of modern neural architectures, they also raise concerns regarding the representativeness and continued utility of MSTAR as a proxy for operational SAR ATR tasks. In this paper, we examine the strengths and limitations of the MSTAR dataset in the context of deep learning–based model development. We present representative performance results from a convolutional neural network trained on MSTAR imagery, illustrating how dataset characteristics—such as centered targets, fixed chip dimensions, limited clutter, and single-object scenes—lead to artificially easy classification problems. We further contrast these results with preliminary experiments on the ATRNet-STAR dataset, where modest perturbations including target offset and rotation produce substantial performance degradation. Collectively, these findings highlight the need for more realistic SAR datasets and evaluation protocols. This paper is intended as a work in progress, with ongoing analysis of embedding-space structure and robustness metrics to be reported in future revisions.