During my Ph.D., I explored the usage of projected Virtual Reality (VR) technologies for immersive live musical performances.
Differently from previous VR music research, my work aimed at maximizing spectators‘ feeling of presence. Indeed, since the experimentations in the early 90’s] artists have recognized the potential VR has as a means to explore unprobed musical territories and innovative audio/visual experiences. However, I can say that, until the time I started my Ph.D., all the research in the field had always primarily targeted the musicians’ experience, by building virtual instruments and virtual world exclusively designed for the performers on stage. This was due to technological limitations, both in terms of rendering and tracking capabilities; in other words, there is not much to do to invite the audience in the virtual world when your workstation supports only one low res screen [most likely an HMD] and your tracking system is a cyber-glove. But you can still make very good music.
When in the late 2000’s I started to work with projected VR, I was immediately stricken by the incredibly strong perceptual effects this technology has on our brain. The simple possibility to see my hand overimposed on the virtual environment and directly interacting with virtual objects made every experience in the ADVR VR room so real, no matter how simple it was. However, in the beginning, I started to focus as everyone else on the classic user-centered paradigm, designing virtual instruments and installations. it took me a whole year of my Ph.D and several studies and experiments to gradually realize the unexplored musical potential of projected VR. In particular, it was a dance+projections performance I randomly stumbled upon on Saturday, February 20th 2010 that made me think that VR could be used in a different way from what done so far.
From that point on, the leit motiv of my Ph.D. research became the search for novel visualization and interaction paradigms for VR musical performances. Throughout the following years, this plan translated into the design of setups to display interactive virtual environments that surround the audience, the performer, as well as the on stage physical instrumentation. This paradigm is defined in literature as Hybrid Reality, as proposed by Milgram and Kishino in the paper “A Taxonomy of Mixed Reality Visual Displays”. To use the authors’ words, in Hybrid Reality setups “real physical objects in the user’s environment play a role in (or interfere with) the computer generated scene”. However [and not surprisingly], Milgram and Kishino described only user-centered displays, while my research focused on exocentric setups. In the scenario, the primary visual feedback is moved from the user [the performer] to an external observer [the audience], while interaction is still mainly carried out by the user. This feedback decoupling leads to a number of issues, that I tried to address through a series of user studies on the role of multimodality in interaction. The outcome was a set of metaphors for Hybrid Reality Performances, that allow both the performer and the audience to engage in musical interaction with virtual objects, while immersed in an audio/visual scenography.
This long journey is documented in my dissertation, entitled “Multimodal VR Interface for Music and Visuals Creation”.
It consists of 3 intense years of study of the human body, software and hardware experimentation, user studies on perception and musical interaction, installations and performances; all of which would have not been possible without the team of engineers and psychologists I was part of and, above all, without the collaboration with the talented musicians from the underground music scene in Genova. Other than consisting of my first step into the world of academia and research, this experience also deeply influenced my vision as avant-garde musician and new media artist.
[For sake of curiosity, the dance performance that triggered my search for novel paradigms was by a famous Italian dancer/TV host, which also inspired Beyoncé’s choreography for the 2011 Billboard Music Awards. You can watch excerpts of both here]