When Reasoning Hurts Confidence: Demographic Calibration of Vision-Language Models in Medicine (Extended Abstract)

Abstract

Vision-language models (VLMs) are increasingly evaluated as diagnostic aids in medical imaging, yet clinical deployment requires confidence estimates that are reliable and equitable across patient demographics. When a model reports 80% confidence, it should be correct about 80% of the time, for every patient group and not just on average. We evaluate the demographic calibration of three architecturally distinct VLMs (GPT-4o, R1-Onevision-7B, and Qwen2.5-VL-7B) across skin lesion and chest radiograph classification tasks, stratified by Fitzpatrick skin type and patient sex, using three confidence signals, namely token-level probability (TLP), verbalized confidence (CE), and self-consistency (SC). In six of nine model-task combinations (all involving GPT-4o and R1), the best-calibrated signal is not the most equitable, and in the most extreme case it carries a 2.6× ECE ratio across skin tones, while post-hoc temperature scaling reduces ECE by up to 96% but leaves equity gaps unchanged. The R1/Qwen same-backbone ablation reveals a sharp architecture dependence. R1's TLP becomes anti-discriminative on CheXpert while SC at high temperature remains the best-performing signal, whereas for Qwen, SC collapses uniformly and TLP is the only discriminative signal. In our study, no single confidence method transfers reliably across model architectures and clinical tasks, and signal selection has direct implications for equitable deployment.