Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models

Abstract

Generative image models produce striking visuals yet often misrepresent culture. Prior work has probed cultural dimensions primarily in text to image (T2I) systems, leaving image to image (I2I) editors largely underexamined. We close this gap with a unified, reproducible evaluation spanning six countries, an 8 category/36 subcategory schema, and era aware prompts, auditing both T2I generation and I2I editing under a standardized, reproducible protocol that yields comparable model level diagnostics. Using open models with fixed configurations, we derive comparable diagnostics across countries, eras, and categories for both T2I and I2I. Our evaluation combines standard automatic measures, a culture aware metric that integrates retrieval augmented visual question answering (VQA) with curated knowledge, and expert human judgments collected on a web platform from country native reviewers. To enable downstream analyses without re running compute intensive pipelines, we release the complete image corpus from both studies alongside prompts and settings. Our study reveals three recurring findings. First, under country agnostic prompts, models default to U.S. like, modern leaning depictions and flatten cross country distinctions, reducing separability between culturally distinct neighbors despite fixed schema and era controls. Second, iterative I2I editing erodes cultural fidelity even when conventional metrics remain flat or improve; by contrast, expert ratings and our culture aware metric both register this degradation. Third, I2I models tend to apply superficial cues (palette shifts, generic props) rather than context and era consistent changes, frequently retaining source identity for non U.S. targets and drifting toward non photorealistic styles; attribute addition trials further expose weak text rendering and brittle handling of fine, culture specific details. Taken together, these results indicate that culture sensitive edits remain unreliable in current systems. By standardizing prompts, settings, metrics, and human evaluation protocols, and releasing all images and configurations, we offer a reproducible, culture centered pipeline for diagnosing and tracking progress in generative image research. Project page: https://seochan99.github.io/ECB/.