How Can Large Language Models Be More Reliable?

Abstract

Large Language Models (LLMs) are increasingly integrated into everyday applications, yet their reliability remains inconsistent, even for seemingly simple queries. By “scaling up” and “shaping up”, these models have improved average accuracy and robustness to prompt variations, but they continue to display “difficulty discordance”: they solve harder tasks while making errors on easier ones. Moreover, they show a marked reluctance to refuse answers even when uncertain. Such behaviour deprives users of clear cues about when outputs can be trusted. This work explores strategies to enhance LLM reliability through confidence-based abstention, combining uncertainty estimation techniques with measures of question difficulty to define a model’s “safe operating area”. By ensuring that queries are either answered correctly or explicitly declined, the approach aims to enhance predictability, transparency, and user trust, while providing a framework for managing model limitations.