Position: The Iceberg of Pitfalls in LLM-Based Secure Code Generation

Abstract

Secure code generation techniques have significantly improved the security of code produced by large language models (LLMs), achieving notable vulnerability reductions on standard benchmarks. We critically review this trajectory. We primarily question whether these gains reflect genuine security reasoning or merely the masking of vulnerabilities through syntactic pattern matching. We further scrutinize the existing decoupling of security and functionality, contending that current evaluation protocols often overlook cases where security constraints result in non-functional code. We additionally highlight that existing methods employ fundamentally incompatible evaluation protocols including distinct datasets and analyzers that preclude fair comparison and render reported gains incomparable. Finally, we identify how secure code generation risks magnifying systemic research failures, specifically latent data contamination and synthetic-data-induced drift, which threaten the long-term robustness of these systems. By formalizing these concerns, we propose a research agenda that prioritizes joint security-functionality validation and adversarial stress testing to move the field toward more robust and practically deployable secure code generation.