A Multi-User Virtual World with Music Recommendations and Mood-Based Virtual Effects


  • Charats Burch Wentworth Institute of Technology
  • Robert Sprowl Wentworth Institute of Technology
  • Mehmet Ergezer Wentworth Institute of Technology




Music Mood Recognition, Music Mood Classification, Music, Music Recommender Systems, Music Recommendation Algorithm, Mel-spectrograms, Virtual Environment, Content-based Music Recommendation


The SEND/RETURN (S/R) project is created to explore the efficacy of content-based music recommendations alongside a uniquely generated Unreal Engine 5 (UE5) virtual environment based on audio features. S/R employs both a k-means clustering algorithm using audio features and a fast pattern matching (FPM) algorithm using 30-second audio signals to find similar-sounding songs to recommend to users. The feature values of the recommended song are then communicated via HTTP to the UE5 virtual environment, which changes a number of effects in real-time. All of this is being replicated from a listen-server to other clients to create a multiplayer audio session. S/R successfully creates a lightweight online environment that replicates song information to all clients and suggests new songs that alter the world around you. In this work, we extend S/R by training a convolutional neural network using Mel-spectrograms of 30-second audio samples to predict the mood of a song. This model can then orchestrate the post-processing effect in the UE5 virtual environment. The developed convolutional model had a validation accuracy of 67.5% in predicting 4 moods ('calm', 'energetic', 'happy', 'sad').




How to Cite

Burch, C., Sprowl, R., & Ergezer, M. (2023). A Multi-User Virtual World with Music Recommendations and Mood-Based Virtual Effects. Proceedings of the AAAI Conference on Artificial Intelligence, 37(13), 16063-16069. https://doi.org/10.1609/aaai.v37i13.26908



EAAI Symposium: Human-Aware AI in Sound and Music