SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs

Abstract

Existing LLM safety benchmarks focus primarily on English and Western contexts, often overlooking the linguistic diversity and culturally grounded safety risks present in other languages. We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios, explicitly designed for ten major Indian languages - Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu, along with English. We benchmark state-of-the-art LLMs on SurakshaEval, establishing safety baselines and identifying recurring failure modes, including over-refusal and insufficient contextual awareness in regionally sensitive settings, particularly in native scripts. These findings underscore the need for region-specific data and structured evaluations to build safe, ethical, and culturally aligned AI systems. Warning: This paper contains potentially offensive or unsafe content.