Comparing Human and LLM Ethical Analyses: A Case Study in Computational Social Science Research

Abstract

As researchers increasingly engage with ethically complex digital phenomena, timely and accessible support for ethical reflection is essential---yet often unavailable beyond formal institutional review processes, which are more focused on regulatory compliance than ethics. This paper investigates the potential of large language models (LLMs) to serve as research ethics support tools by providing immediate, context-sensitive feedback on draft research protocols. We analyze a draft research proposing to scrape digital platforms for data on ``Sephora Kids''---a trend in which minors promote beauty products on platforms like YouTube and TikTok---as a case study to explore this possibility. Two human ethicists and two LLMs (GPT-4o and Claude 3.7 Sonnet) independently reviewed the proposal and produced ethical evaluations. We then compared the outputs to assess whether LLMs could meaningfully assist researchers in identifying and engaging with ethical issues. Our findings suggest that LLMs can already offer valuable support.