Mixed Reality Studios: A Primer for Producers

The following is an excerpt from an article by Nils Porrmann posted to the dandelion + burdock website

Situation

In 2020 Mixed Reality has become a much discussed topic in media production. Its increased application has undoubtedly been catalysed by the Corona pandemic. Uncertainties in travel, restricted venue capacities, social distancing, and quarantine rules have globally halted live events.

Nonetheless brands, artists, and live shows want to retain a market presence, and therefore solutions for compelling visual effects and narratives in controlled environments are desirable. Mixed Reality endeavours to deliver this presence with immediacy and emotional value, while distributing the composited product via media streaming or broadcasting.

In our opinion some important factors have aligned in 2020:

1. Realistic real time rendering for small to medium size canvases (roughly up to UHD level) are maturing to be considered technically and artistically viable. This is due to hardware and software improvements
2. Media servers are focusing to provide unified platforms to manage rendering, tracking, spacial-, temporal- and colour- calibrations
3. Producers and engineers are adapting workflows and retraining in response to new remote working conditions
4. Clients are willing to consider new risks and investments into cutting edge production technologies, when the alternative is a diminished media presence
5. Remote work is increasingly accepted
6. Due to lockdowns and social distancing, audiences welcome streaming formats
7. Media on demand and home offices require updates to data infrastructure globally (fibre and 5G)

The XR Set

In Mixed Reality live performers and props are filmed inside an illuminated setup of LED walls, which are mapped with a corresponding virtual environment. This environment is rendered in real time to match the perspective of a tracked camera. The actual stage and performers are lit corresponding to the virtual scenery. In this way real and virtual elements blend into a single comprehensive mise-en-scene.

Different levels of immersion of virtual productions have different names. An XR Space is a somewhat encompassing wall, while XR Stages have an added walk-on floor, and XR volumes almost entirely cocoon the production.

In XR spaces and stages, props and performers need to be conventionally lit to achieve immersion. A volume, on the other hand, can be considered to create its own light space, a condition in which a lighting rig is more auxiliary.

Typically XR stages are laid out as three sided cubes (diamond configuration) or curved screens atop a walk-on LED floor. There are variations and hybrids of these concave spaces. All of them typically open to align the floor’s diagonal as the main camera – or viewing axis (see diamond illustration above) analogue to the upstage-downstage direction. The arrangement encompasses screens, lights, cameras, performers and props.

In contrast to green screening and post production, XR workflows can achieve a true interaction of analogue and virtual elements in real time. Instead of compositing shots after the fact, all effects are presented to one camera and video stream.

Whatever type of XR set, it is a part of the real world production space. The virtual reality is a conceptual overlay to the real space. Their amalgamation is then made visible through the camera on set.

To plan the XR design, lighting and in camera result, it is useful to simulate the LED set, real props and performers in 3D. The real and virtual components of the set and scenery are becoming one creative domain.

Please follow this link to read the full article or visit us at Medium