DIGITAL_DUETS – DIGITAL Devices for mUltimodal Expression Through vocal Synthesis

2015-ongoing _ HCT lab |UBC|, department of Advanced Robotics |IIT|

Image module

DIGITAL_DUETS is a multidisciplinary project I am currently leading, as part of a Marie Curie Fellowship I was awarded in 2014. As suggested by the fancy acronym, the project deals with realistic vocal synthesis, Digital Musical Instrument (DMI) design and multimodal control.

Human voice is an extremely fascinating instrument for expression. Our brain is tuned up to perceive and enjoy the tiniest nuances in voice production and, as hinted by the complexity of the vocal apparatus, humans are capable of an outstanding level of control over the involved organs. In a world where HCI is focusing more and more on communication between humans and computers and where people are striving for the exploration of new forms of digital expression, the possibility to synthesize and extend natural sounding voice is an exciting challenge.

DIGITAL_DUETS aims at pushing the boundaries of vocal synthesis and control, leveraging on the investigation of the physiological and acoustic mechanisms behind speech and singing voice. The project runs in synergy with 3 other ongoing HCT projects, ArtiSynth, OPAL and DIVAs, and its agenda is composed of 3 main phases.

The first part of the project consists of designing and evaluating a novel real-time acoustic model, to be coupled with the advanced bio-mechanical models of the human vocal tract developed in ArtiSynth by the OPAL group. The core of the acoustic model is an innovative 2D airwave propagation system, running on the GPU. The aim is to obtain unprecedented quality in real-time vocal synthesis, trying to simulate the physiological processes underlying voice production in the most realistic and accurate way.

During the second part of the project, the resulting articulatory vocal synthesizer will be turned into a DMI, replacing the bio-mechanical control with the expressive gestural real-time control developed for DIVAs and focusing on singing voice simulation. This entails the remapping of vocal articulators onto other parts of the human body, a challenging goal that involves many aspects of HCI.

Finally, the last part of the project aims at making use of novel Virtual and Mixed Reality technologies to extend the instrument with multimodal control, specifically designed for audio/visual performances where voice is synthesized both sonically and visually.



This research was supported by a Marie Curie International Outgoing Fellowship within the 7th European Community Framework Programme.


marie_curie_logo   eu_logo

_media and links
_related artworks
_related publications
  • Zappi, Victor, Arvind Vasuvedan, Andrew Allen, Nikunj Raghuvanshi, and Sidney Fels, “Towards real-time two-dimensional wave propagation for articulatory speech synthesis”, Proceedings of Meetings on Acoustics, 171st edition, Vol. 26. [in press], ASA, 2016.
_development and updates
OpenGL FDTD Tutorial: Episode 2 – Simulation Cycle and Shader Programs

  Here we go, the second episode of my tutorial on how to implement real-time physical modelling synthesis on the GPU, using C++ and OpenGL shaders. The first episode can be

OpenGL FDTD Tutorial: Episode 1 – Simulation Initialization

Welcome to the first episode of my tutorial to implement real-time physical modelling synthesis on the GPU, using C++ and OpenGL shaders. More context can be found in this introduction. We’re

OpenGL FDTD Tutorial Introduction

This post is an introduction to a 3-episode tutorial that explains how to implement real-time physical modelling synthesis on the GPU, using OpenGL shaders. It is part of a full paper titled

SFU Colloquium and Vancouver Pro Musica – Talks, Panels, Demos…

Fall 2016 has been an important period for DIGITAL_DUETS, for both development and dissemination. While I’ll save updates about technical stuff for another, later, beefier post, here I’d like to

Performing in Flatland @ CAA meeting 2016

Finally some music! Last week I played a show with Arvind Vasudevan [one of the students I work with/co-supervise] and Sidney Fels [the main professor I work with], showcasing a

2D Vowels Followup – Curvature and Asymmetries

As introduced in this post, I am currently exploring novel ways of representing 3D Vocal Tract [VT] geometries in 2D, by means of extending the base concept of area function. This is

2D Vowels Presentation at ASA Meeting – Who needs Area Functions?

This is a long post. I don’t like long posts, but it’s research, so it is motivated.   In 2 days I’ll fly to Salt Lake City [Ut] to join the

Invited Talk at Moving Stories May Residency

I have been invited to give a talk at the Moving Story May Residency that will take place May 24th-28th at Emily Carr University, here in Vancouver. I am scheduled for Thursday

2D Propagation on GPU – It all started like this…

  It all started like this, with a puff of air.     This little gif is the result of my first real-time simulation of pressure propagation in open space.