Towards Reliable Lung Cancer Prediction: A Hybrid Framework for Noise Reduction and Uncertainty Control

Abstract

Uncertainty remains a critical challenge in healthcare AI, since predictive errors can directly compromise patient safety and undermine trust. Structured clinical datasets in healthcare are frequently characterized by heterogeneous acquisition protocols, incomplete records, and inconsistent or noisy encodings. This inflates aleatoric uncertainty and weakens calibration. These challenges are exemplified in lung cancer risk modeling, where small cohorts, variable collection practices, and limited feature quality make the problem especially acute. Significant advances in uncertainty quantification (UQ) have been achieved in imaging and signal processing through Bayesian inference, evidential learning, and robust architectural designs. In contrast, tabular clinical datasets remain a critical yet underexplored domain. Addressing this gap requires methods that are lightweight, certifiable, and effective on noisy datasets without relying on large models or data. Considering this challenges, we propose a frequency-aware hybrid representation that combines Principal Component Analysis (PCA) with the Discrete Cosine Transform (DCT). Using mutual information (MI)–based feature ordering, the framework suppresses high-frequency artifacts while preserving discriminative structure. As the framework was applied to a publicly available lung cancer dataset, it demonstrated an accuracy improvement from 98.1% to 99.7%, reduced Negative Log-Likelihood (NLL) by 82% from 5.25% to 0.94%, lowered aleatoric uncertainty from 10.50% to 3.35% (68% reduction), and preserved AUROC at 99%. We evaluated the framework across three publicly available lung cancer datasets where it demonstrated a reduction in aleatoric uncertainty by 7% on an average, confirming generalizability. The Wilcoxon signed-rank test confirms that the results are statistically significant. This work shows that part of the ‘irreducible’ variability is actually compressible noise, thereby facilitating more reliable and uncertainty-aware AI for healthcare.