VVV: Volumetric Video in Videogames

VVV: Volumetric Video in Videogames is a practice based research inquiry that uses full motion video (FMV) videogame design patterns to scaffold the design of new games using volumetric (spatial 3D) video. It aims to advance critical discourse and design knowledge surrounding volumetric video and other emerging forms of hybrid captured media, within videogames and related immersive experiences. VVV is a multi-institution collaboration between experimental game designer Dr. Cindy Poremba (OCAD University), game historian Dr. Carl Therrien (Université de Montréal), and Prof. Nicolas Hesler (Sheridan College); in partnership with Simile, the engineers of the volumetric video platform DepthKit. VVV leverages an extensive historical archive of cinematic videogames housed at Université de Montréal, the advanced game development skills of Sheridan's students and faculty, and Dr. Poremba's research-creation work exploring captured media in game worlds (Poremba 2011), as well as her prior exploration into volumetric capture (Eyebeam 2013, Anderson Ranch 2016). It will result in opportunities for student training at both the undergraduate and graduate level, presentations to both academic and public audiences, the publication of two journal papers, the delivery of an advanced workshop, contributions towards a scholarly manuscript, and a creative work that will be showcased at high profile events locally and internationally.

Volumetric video is a computational fusion of digital video recording and depth sensor data, resulting in a spatialized, and potentially navigable, 3D captured moving image. Volumetric images are becoming increasingly prevalent and sophisticated, driven by interest in augmented reality (AR) and virtual reality (VR) production. Although there is an emerging maker community surrounding this technology, creators have yet to explore its full expressive potential, particularly in procedurally intense, interactive forms such as videogames. Canada's videogame industry contributes $3 billion annually to our GDP, and 54% of Canadians identify as videogame players (ESA 2015). But games struggle to diversify their content, creator, and player base. Volumetric video content, appearing snatched from the world and evoking feelings of presence and actuality, could be a powerful addition to a game development toolkit, but it has properties that differ from what audiences and creators expect to encounter in games. A poor understanding of the use of this material, in both design and critical contexts, can damage our perception of the value and potential of emerging works before they have the opportunity to develop. How can we take an informed, critical approach to the integration of volumetric video in games? What theoretical and design resources can we draw upon to scaffold experimentation, critical understanding, and ethical critique in this area?

VVV: Volumetric Video in Videogames will investigate how design patterns (formal solutions and responses to common design challenges) from underexamined game genres such as fullmotionvideo (FMV) videogames can suggest future paths for design using captured content such as volumetric video. Taking a new materialist approach to hybrid capture technologies, this research aims to make visible the increasingly complex entanglements that make up digital media forms we casually refer to as "real." In doing so, it will support critical dialogue around hybrid captured media, particularly involving real people and places, and further reinforce Canada’s creative leadership in this emerging area. VVV will also contribute to a growing body of game design research on the application of design patterns to new work, and mobilize design knowledge from an underutilized body of research into early videogame design. It has the potential to expand and enrich artistic, documentary and commercial game development and audiences, and to broaden the expressive palette for other forms of media using volumetric content, including nongame AR and VR applications.

 

This research was supported by the Social Sciences and Humanities Research Council of Canada.

SSHRC Logo

Creator: 
A screenshot of volumetric capture of a figure
Sunday, July 1, 2018 - 4:00am
Lab Member: 
Cindy Poremba

Vernacular Volumetric

Vernacular Volumetric is a research-creation project that uses speculative design to explore how volumetric video recording might look as a mundane future technology. It is part of a larger research investigation into the techno-social entanglements surrounding hybrid captured images.

The project seeks to envision and investigate the future of digital capture technologies in the creation, understanding, and use of everyday images. The project considers:

  1. How social and vernacular practices may impact volumetric capture
  2. The key affordances of a volumetric image
  3. The opportunities and challenges for this technology in relation to broader adoption and use
  4. How this technology may be appropriated and subverted by amateur practitioners

Vernacular Volumetric is a pilot project intended to develop into a broader scale research agenda exploring digital capture.

Hybrid image capture practices, including volumetric capture, are becoming increasingly common. Such projects showcase the incredible potential for this mode of computationally-enhanced capture, but also present a number of emerging issues. These technologies are, for the most part, restricted to technologically-savvy and well-funded art technologists, with the most common path of exploration being high-end production. Additionally, the rapid adoption and development of this technology has not been accompanied by a substantive reflective practice or analysis. Important questions remain with regard to our understanding and relationships to these images, and what this technology might look like upon wider consumer adoption.

Vernacular Volumetric is an initial exploration into hybrid image capture technology, specifically volumetric video recording, as it might look as a mundane future technology. The project examines the future of everyday hybrid image capture in the shift from professional to consumer adoption, with a specific focus on future vernacular (everyday) practices in the context of post-photographic discourse. This work project is also contextualized within discourse surrounding image creation, reception, and everyday use.

Using speculative scenarios, Vernacular Volumetric creates a series of small-scale image prototypes using volumetric capture. This series serves as a prompt for further discussion and research development in hybrid image capture technologies and practices. The project utilizes speculative design, a discursive approach which focuses on using the design process as a tool for envisioning and interrogating potential design futures. 

This research caters to a growing interest in the impact of emerging technology on approaches to non-fiction media production. It forms the basis for a broader examination of how digital technologies impact capture practices, and helps ensure that the development and conceptualization of hybrid image technology is not limited to high-end approaches and can instead be engaged by diverse socio-cultural practices.

Creator: 
Image prototype using volumetric capture: two views from different angles of a figure standing in a rectangular prism
Photograph of the team working in the lab, two people discussing content of a large computer screen amidst notes
Friday, November 24, 2017 - 12:00pm
Lab Member: 
Cindy Poremba