Here we go, the second episode of my tutorial on how to implement real-time physical modelling synthesis on the GPU, using C++ and OpenGL shaders. The first episode can be
ContinueDIGITAL_DUETS is a multidisciplinary project I am currently leading, as part of a Marie Curie Fellowship I was awarded in 2014. As suggested by the fancy acronym, the project deals with realistic vocal synthesis, Digital Musical Instrument (DMI) design and multimodal control.
Human voice is an extremely fascinating instrument for expression. Our brain is tuned up to perceive and enjoy the tiniest nuances in voice production and, as hinted by the complexity of the vocal apparatus, humans are capable of an outstanding level of control over the involved organs. In a world where HCI is focusing more and more on communication between humans and computers and where people are striving for the exploration of new forms of digital expression, the possibility to synthesize and extend natural sounding voice is an exciting challenge.
DIGITAL_DUETS aims at pushing the boundaries of vocal synthesis and control, leveraging on the investigation of the physiological and acoustic mechanisms behind speech and singing voice. The project runs in synergy with 3 other ongoing HCT projects, ArtiSynth, OPAL and DIVAs, and its agenda is composed of 3 main phases.
The first part of the project consists of designing and evaluating a novel real-time acoustic model, to be coupled with the advanced bio-mechanical models of the human vocal tract developed in ArtiSynth by the OPAL group. The core of the acoustic model is an innovative 2D airwave propagation system, running on the GPU. The aim is to obtain unprecedented quality in real-time vocal synthesis, trying to simulate the physiological processes underlying voice production in the most realistic and accurate way.
During the second part of the project, the resulting articulatory vocal synthesizer will be turned into a DMI, replacing the bio-mechanical control with the expressive gestural real-time control developed for DIVAs and focusing on singing voice simulation. This entails the remapping of vocal articulators onto other parts of the human body, a challenging goal that involves many aspects of HCI.
Finally, the last part of the project aims at making use of novel Virtual and Mixed Reality technologies to extend the instrument with multimodal control, specifically designed for audio/visual performances where voice is synthesized both sonically and visually.
This research was supported by a Marie Curie International Outgoing Fellowship within the 7th European Community Framework Programme.