VVV: Volumetric Video in Videogames is a practice based research inquiry that uses full motion video (FMV) videogame design patterns to scaffold the design of new games using volumetric (spatial 3D) video. It aims to advance critical discourse and design knowledge surrounding volumetric video and other emerging forms of hybrid captured media, within videogames and related immersive experiences. VVV is a multi-institution collaboration between experimental game designer Dr. Cindy Poremba (OCAD University), game historian Dr. Carl Therrien (Université de Montréal), and Prof. Nicolas Hesler (Sheridan College); in partnership with Simile, the engineers of the volumetric video platform DepthKit. VVV leverages an extensive historical archive of cinematic videogames housed at Université de Montréal, the advanced game development skills of Sheridan's students and faculty, and Dr. Poremba's research-creation work exploring captured media in game worlds (Poremba 2011), as well as her prior exploration into volumetric capture (Eyebeam 2013, Anderson Ranch 2016). It will result in opportunities for student training at both the undergraduate and graduate level, presentations to both academic and public audiences, the publication of two journal papers, the delivery of an advanced workshop, contributions towards a scholarly manuscript, and a creative work that will be showcased at high profile events locally and internationally.
Volumetric video is a computational fusion of digital video recording and depth sensor data, resulting in a spatialized, and potentially navigable, 3D captured moving image. Volumetric images are becoming increasingly prevalent and sophisticated, driven by interest in augmented reality (AR) and virtual reality (VR) production. Although there is an emerging maker community surrounding this technology, creators have yet to explore its full expressive potential, particularly in procedurally intense, interactive forms such as videogames. Canada's videogame industry contributes $3 billion annually to our GDP, and 54% of Canadians identify as videogame players (ESA 2015). But games struggle to diversify their content, creator, and player base. Volumetric video content, appearing snatched from the world and evoking feelings of presence and actuality, could be a powerful addition to a game development toolkit, but it has properties that differ from what audiences and creators expect to encounter in games. A poor understanding of the use of this material, in both design and critical contexts, can damage our perception of the value and potential of emerging works before they have the opportunity to develop. How can we take an informed, critical approach to the integration of volumetric video in games? What theoretical and design resources can we draw upon to scaffold experimentation, critical understanding, and ethical critique in this area?
VVV: Volumetric Video in Videogames will investigate how design patterns (formal solutions and responses to common design challenges) from underexamined game genres such as fullmotionvideo (FMV) videogames can suggest future paths for design using captured content such as volumetric video. Taking a new materialist approach to hybrid capture technologies, this research aims to make visible the increasingly complex entanglements that make up digital media forms we casually refer to as "real." In doing so, it will support critical dialogue around hybrid captured media, particularly involving real people and places, and further reinforce Canada’s creative leadership in this emerging area. VVV will also contribute to a growing body of game design research on the application of design patterns to new work, and mobilize design knowledge from an underutilized body of research into early videogame design. It has the potential to expand and enrich artistic, documentary and commercial game development and audiences, and to broaden the expressive palette for other forms of media using volumetric content, including nongame AR and VR applications.
This research was supported by the Social Sciences and Humanities Research Council of Canada.