Evaluating and Improving Value Judgments in AI: A Scenario-Based Study on Large Language Models’ Depiction of Social Conventions

Abstract

The adoption of generative AI technologies is swiftly expanding. Services employing both linguistic and multimodal models are evolving, offering users increasingly precise responses. Consequently, human reliance on these technologies is expected to grow rapidly. With the premise that people will be impacted by the output of AI, we explored approaches to help AI output produce better results. Initially, we evaluated how contemporary AI services competitively meet user needs, then examined society's depiction as mirrored by Large Language Models (LLMs). We did a query experiment, querying about social conventions in various countries and eliciting a one-word response. We compared the LLMs' value judgments with public data and suggested a model of decision-making in value-conflicting scenarios which could be adopted for future machine value judgments. This paper advocates for a practical approach to using AI as a tool for investigating other remote worlds. This research has significance in implicitly rejecting the notion of AI making value judgments and instead arguing a more critical perspective on the environment that defers judgemental capabilities to individuals. We anticipate this study will empower anyone, regardless of their capacity, to receive safe and accurate value judgment-based outputs effectively.