In this website, we are demonstrating an in depth investigation for the realization of a system, able to combine gesture and vocal recognition for interactive art, live events and speech based expressive applications. Practically, our goal is to create a flexible platform for collaborative and improvisatory storytelling combining voice and movement. Our work advances both conceptual and technical research relating speech, body, and performance using digital technologies and interactive media.

We create an immersive audio-visual Story Telling Room that responds to voice and sound inputs. The challenge is to set up an efficient speech feature extraction mechanism using as good microphone conditions as possible. Leveraging the TML’s realtime media choreography framework, we map speech to a wide variety of media such as animated glyphs, visual graphics, light fields and soundscapes. Our desiderata for mapping speech prosody information centre on reproducibility, maximum sensitivity, and nil latency. Our purpose is not to duplicate but to supplement and augment the experience of the story that is being unfolded with a performer and audience in ad hoc, improvised situations using speech and voice for expressive purposes. 

Please note that the work presented here is a work in progress and the experimentation are not to be considered as final performance pieces.