Explainable CLIP-Guided 3D-Scene Generation in an AI Holodeck

Abstract

This paper describes the AI Holodeck, a co-creative software prototype that creates virtual scenes from the user input text, inspired by the fictional Holodeck virtual reality device from the science fiction series Star Trek. This application collects common-sense knowledge from annotated datasets and reference images. It uses this knowledge to populate scenes with objects found in selected environments alongside those explicitly mentioned by the user. We present the system design of the AI Holodeck, and a proposed study to measure the effects of its visualizations on user perceptions of the system's creativity.