Safer On-Premise Medical AI: A Hybrid Framework for Hallucination Detection in Local LLMs

Abstract

LLMs (Large Language Models) are increasingly integrated into medical workflows to provide decision-making support, for example as Question Answering (QA) systems. However, in medical context, privacy requirements and resource constraints require the use of local and small (1B-parameter) LLMs. 1B-parameter LLMs are highly susceptible to hallucinations, and existing hallucination detection frameworks often suffer from poor calibration in these low-resource settings. To address this problem, we introduce a lightweight explainable hybrid detection method that integrates semantic entropy to quantify the uncertainty at the meaning-level in diverse response samples with an internal signal probe trained on the hidden states of the final layer of the model. The framework operates without external retrieval and consistently improves the detection of factual errors across biomedical benchmarks, outperforming single-signal baselines most clearly in calibration stability, the property that enables safe abstention in clinical workflows. These findings contribute to efforts dedicated to building transparent, safe, and privacy-preserving AI systems in medical decision-making support.