What Should AI Generate? Moving Beyond Bias in Generative Systems

Abstract

Generative AI systems seamlessly transform underspecified prompts into embodied outputs, masking the numerous decisions required to instantiate a representation. These decisions not only shape what is depicted, but render implicit statements about what is possible, likely, and necessary. Existing work has largely evaluated such patterns through notions of representational bias. However, conflicting critiques of generated representations suggest a broader normative ambiguity: what should generative representations embody? In this work, we provide a conceptual framework that separates the descriptive aspects (what is instantiated by model outputs) from the normative (what we aspire for representations to depict and make possible). Drawing on longstanding work on modality in philosophy and linguistics, we characterize patterns of instantiation as implicit possibility assertions and use varieties of modality to structure distinct normative foundations for what representations should do. This framing reveals how the same representation may be evaluated differently depending on whether one prioritizes historical fidelity, conceptual possibility, aspirational aims, or other representational purposes. Rather than seeking a single ``unbiased'' representation, we argue for plurality in the normative expectations governing generative representations. We conclude by considering affordances for rendering these commitments explicit, critically engaging with them, deliberating over contested defaults, and providing users greater maneuverability over what is rendered possible.