Cross-Lingual Robustness of Societal Bias Representations in Large Language Models: A Contrastive Activation Steering Analysis of Llama 2 and GPT-OSS

Abstract

Large language models (LLMs) increasingly support consequential applications, making it important to examine whether they reproduce harmful social biases. We thus present a bilingual, representation-level study of bias and refusal behavior in Llama-2-7B-Chat (7B, dense) and GPT-OSS-20B (Mixture-of-Experts) using English and Italian prompts, where the Italian material is a reviewed translation of the English material and therefore supports a controlled bilingual comparison rather than a culturally localized audit. Using Contrastive Activation Steering, we construct gender, race, religion, and refusal vectors from paired stereotype/anti-stereotype and refusal/non-refusal prompts. We compare unsteered, bias-only, and bias-plus-refusal-subtraction generation and analyze layer-wise vector geometry. Across the tested prompts and interventions, stereotype-consistent completions remain elicitable in both models and languages. Because the response-level analysis uses seven held-out templates and single-annotator labels, these behavioral findings are exploratory. The geometric analysis shows that, in the tested models, bias directions are partially aligned and become increasingly anti-aligned with refusal in later layers. Thus, visible refusal does not by itself establish that stereotype-associated directions are absent. The findings motivate representation-level bilingual red-teaming that evaluates refusal and bias jointly.