Barriers to XR Adoption in Virtual Production

And how to get past them – A guide for users

This article is an extension of an article by Nils Porrmann, Thomas Kother and myself at dandelion + burdock

While live event production has largely been shuttered due to the coronavirus pandemic, one opportunity to produce performance content has seen a rapid growth in development over the last 8 months, and that is XR. The term ‘XR,’ short for miXed Reality, is used to describe a host of technologies that allow for the deployment of real-time generated images that are manipulated based on the position of the viewer, which is the camera.  Camera POV makes these images seem to coexist naturally with the performers, allowing for sets, lights and even other performers to be placed virtually in the scene.  

While these tools were in development prior to the pandemic for their ability to create natural lighting and reflections, they are seeing a rush to use as they solve many Covid compliance issues that make current filmmaking and television production exceedingly complex. However, while XR has been seen as having the potential to save live events, tv and film production, the rate of adoption has been a challenge. 

Of the factors that impact the ability to use XR successfully, budget and time are primary limitations. But there are other factors that include skills acquisition, team structure, production planning and communication that add further complexity.  Some of these issues existed prior to XR Production and have only been exacerbated by the rush to adapt to this very different working style. In this article we will break down issues and strategies to plan for future discussion with clients about successful use of XR.

Where we’ve come from

A bit of history for perspective

Terms like XR & Virtual Production have become industry buzzwords which unfortunately means they are easily interchanged and misapplied. XR is a subset of Virtual Production technologies and applies specifically to instances of combining AR or Scenic Extension with background replacement tools.  Background replacement can be the result of live green screen compositing or real-time video output to LED walls and is the backbone of Virtual Production.  When the background is rendered and delivered as the scene is shot and overlaid with foreground elements, and all imagery driven by camera POV, you have XR.

This animatic from Nils’ article is a great visual explainer:

XR is the result of decades worth of technology coming together into a new production paradigm. AR (Augmented Reality) has been used in sports for over twenty years going back to the introduction of the 10 yard line overlay.  3D tools have been combined into video playback and media servers to support the popularity of projection mapping. Tracking tools used to place an object, person or camera in 3D space have been implemented and developed for the last twenty years.  Who expected sports graphics, video mapped cathedrals, and moving lights following a stage performer to all meet as one production toolset? 

It took the language and practice of real-time content generation, as well as the use of previsualization tools in film production, to bring all these pieces together.  

In the live events community, most every media server employed some kind of effects tools to live augment video content. Colorizing a video file or treating it with some distortion is a form of real-time content manipulation. Several media servers took the next step, generating content from a bit of code and user parameter definitions rather than starting with a video file.  But it took the software, Notch, to rapidly advance the use of code generated imagery for use in media playback for live events.

If you are familiar with Notch, you will quickly understand the value of a real-time visual system when it comes to responding to external information and controls. Sound, light levels, physical forms, mocap data can all be employed to impact the way an image was generated. It is a natural evolution of this product for users to experiment with camera position data within the media servers to impact the real-time production of content. 

The disguise platform has led the charge of a very short list of XR solutions, providing users an ecosystem that blends spatially driven content mapping tools with camera tracking and real-time content generation. While there are a few virtual studio solutions for AR & Green Screen, the disguise xR solution provides an LED screen + AR + scenic extension toolset.  This combines the best of the benefits of shooting in an LED volume, while having the immersive feel of a live green screen replacement workflow. There is an evolving list of Virtual Production solutions and their capabilities at the end of this article.

Over in the film industry, VFX is a history book unto itself.  I will call out the advancement of previsualization, motion capture and robotic camera controls was essential to creating more realistic visual effects.  What is critical is the collision of these previz tools with gaming engines like Unity and Unreal, along with the improvement of projection and LED Screen quality. Armed with all these technological advancements, I assume someone in a dark office surrounded by computer monitors thought, “wouldn’t this shot be easier to capture in camera than replace in post?” Ah, I wish it was as easy as saying that, but we are getting there.

What we have now is the beginning of a spatially sensitive production workflow with high quality background replacement and real-time content generation technology. My hope is the pre-production shot planning and previz of film production finds its way into the multi-camera live event workflow. This hybrid process is the full realization of XR Production.

We need time and understanding to build out this new approach. When a client calls asking for XR, it is possible they mean any one or all of these tools. Don’t assume they know exactly what they want or why they want it. You have to engage your client to speak to the production goals to fully understand what they actually want and how to deliver that, no matter what your client asked for. So, let’s talk about how to deliver what your client asked for when the best solution is XR.

Skills Acquisition

How to build your skills and prepare to use XR

Much like projection mapping inspired learning 3D software, XR has a host of knowledge challenges that face the community.  One does not need to be an expert in all aspects of XR Production, but it is beneficial to be able to speak to the various technologies and issues that come with each one.  XR requires a number of unique skills and at first it will seem overwhelming.  There are new tools to learn on the servers that drive XR as well as the complimentary real-time software applications we’ve discussed.  There are third party systems for camera tracking that require specialized skills.  And then there are the detailed intricacies of understanding the cameras themselves.  

Skills acquisition in any one of these areas can be managed with self study and time commitment. The trick is building expertise when you are learning about a process that requires access to specialized gear. This gear is expensive to set up and requires its own expertise to operate.  While you might have access to the media server or software of your choice, you may not have access to the screen, studio or camera equipment necessary to see all these elements work together.  And if you can get access to a full testing environment, you may not be able to simulate or anticipate full show conditions to see the stress points to the XR workflow and prepare for them.

You are in good company. There are only a handful of teams working that are able to fund a fully operational studio for testing and workflow development.  Don’t let that intimidate you.  You have to start somewhere if you intend to start at all.  Yes, there is more to learn than when you first started out with your current expertise. Yes, there is risk involved. It’s not the cutting edge because it’s easy, or even consistently functional. But the path forward means you have to start somewhere.  

Focus on a natural progression of your current expertise.  If you are a media server programmer, start learning the new features that drive XR for that system. Build test projects. Get in touch with a rental house and see if they have built a studio that you can work in when it’s idle.  If you are 3D software savvy, but don’t build content, learn enough Unity/Notch/Unreal to have intelligent conversations with content creators. You will need to advise them on how the visuals will be used in a production environment and therefore, how to optimize the content team’s process. You will also need to understand what an experienced content team is telling you about how to use their content file as designed.  This is a two-way conversation.

If you are a system engineer, you will want to learn about the different camera tracking protocols.  There are a variety of systems in use, each with different approaches to tracking. Depending on the market you are working in, one might be favored over another.  Also, the gear you currently use to build a media server system, some switchers and networking gear have already been tested by other teams.  Talk to people and share your approach when building a system. Others may catch red flags that you don’t expect.

Another important engineering expertise in XR is video over IP protocols.  If you are confused about how NDI, Unreal’s nDisplay and disguise’s RenderStream are ‘same, same but different,’ start reading up on the protocols and the hardware that supports them. Knowing the application, advantages and disadvantages of each are valuable skills. 

Maybe you are reading all this and thinking, “I didn’t exactly learn any 3D software yet.”  Here is the link to download Blender. It’s free and has a huge user community.  Both sites have tutorials for beginners. The future of our production work is dependent on 3D skills.  More and more of the tools used to build a live event are spatially aware. It is never too late to start learning; you can only put a moment on the calendar from where you built your experience.  Mine just happens to be June 2007 when I first learned Cinema 4D.  In ten years, you will be advising someone the same.

For those of you who have started down this path and feel stuck, don’t let this stop you.  If you have been asking for help and insight from others and feel left behind, it’s a strange time for everyone out there. We are typically a very open and sharing community.  We love to discuss how we solved some impossible problem or found a clever workaround.  Because work is so scarce these days, the unfortunate side effect is less openness. It’s not personal; it’s business. If you let it stop you from exploring this technology and getting involved, that’s on you. In time, that openness will return.

As you dive into learning, you will quickly find this is a team effort. Sure there are some rocket scientists out there who can build all the parts and get pretty pictures, but this in practice is a bad idea. This isn’t said to denigrate any one person’s amazing skills; it is said as a call to build teams to protect those geniuses to do excellent work as our roles have become far more demanding than work prior to XR. Let’s take a look at an ideal XR Production team.

Team Structure

What a good XR team should look like

Prior to XR, media servers were programmed by representative of one of three groups: a creative team as an extension of content production and display, an engineering team as an extension of screen rental and/or playback control engineering, or as an operations team, hired specifically for a programmers ability to manage and run a media server based playback environment.  This latter approach is where I started in the field and my perspective on Screens Producing and Media Operations.  This is my soapbox, my textbook and my perspective on XR team structure.

I start with the assumption that most teams currently working in XR are too small to manage the responsibility now expected of this department. These teams were too small before XR and have an order of magnitude more work and responsibility with XR Production. The XR team must manage not only the complex infrastructure of a good system, but educate the full production team on the disruption to their existing approach to work. It is not enough to only manage and play back content as the XR team is now responsible for any visibility of the shooting environment. The XR set exists mostly if not entirely in virtual space. If the team is bogged down with any one of a number of engineering issues, who is left to talk to the director or the content team about issues related to their departments? That is the role of the XR Producer.

For example, let’s say the director is experimenting with a shot and goes outside the allowable zoom range or pans the camera exposing part of the virtual set where there is no content. Typically in a normal production, the set designer or art director would be called over to see how that space could be incorporated into the shot. With XR, there are now technical limitations to camera zoom that will continue to limit this director’s choices. In the case of a pan, maybe it’s a budget decision that limited how much of the virtual world was built. In either case, the XR Producer needs to be available to represent the team in these discussions while other work continues, and guide directors and producers to choices that result in workable solutions.

An XR Producer or a Screens Producer, sometimes a production will need both, these roles are analogous to the role of a Set or Lighting Designer in terms of seniority.  What we don’t have in our community is a creative led position to drive discussion about how this technology integrates into production. Most live event work outside theater exists without a video designer role. Instead, Creative Producers or Directors might demand XR for a project, and leave the creative development to the domain of multiple content teams and their individual leaders. Someone needs to collect all these choices and drive the conditions under which they can succeed. It takes a strong team leader in the XR department to drive discussions on budget and schedule as well as creative implications to using these tools

As an XR Producer, any production meeting that would typically involve a Set or Lighting designer should now include you as well.  I believe that to be true of a Screens Producer in a non-XR environment, but this is an evolving discipline and the role of Screens Producer is still unknown to many production teams. As a community, we can’t even quite agree what this leadership role should be called (I’m talking to you JT). Here is an organizational chart that Nils Porrmann and I collaborated on to summarize a production team, and your production team on XR.  Please note the cross disciplinary impacts of introducing XR to your project.

Aside from the XR Producer, other roles on the XR Production team may include or have multiple positions for: Programming, Content Workflow Developer, Content Coordination, Media Server Engineering, System Integration Engineering and Broadcast Integration Engineering. There are no small XR shows right now so I caution against allowing one person to wear too many hats.  

Other examples to consider: if a content team needs attention because a file is not delivered properly while the programmer is unable to access existing files, your two content positions are busy.  If camera tracking data seems off and you want to troubleshoot sync to make sure the symptom is pointing you in the right direction, suddenly most of your engineers are busy.  If the director wants to change the camera frame rate, better the XR Producer handle that conversation so that everyone else can continue working.

I know the first concern will be the budget to cover this large team. Remember, you are highly skilled professionals. You have been overworked with too many responsibilities for most of your career. As a comparison, look at a well staffed broadcast truck with its clearly defined roles and large support team of utilities. I don’t know how long it took to evolve a broadcast truck to its current staff size, but that’s what it takes to mix playback, camera and graphic overlays down one broadcast feed, maybe two if they are switching IMAG separately from Program.  Now look at our team sizes and the amount of signals we manage. We need larger teams.

Build out your team size. Partner with other people and teams to fill out your expertise if you need to. This is what it takes to do this job well in late 2020.  In time things will get faster/cheaper/leaner, but for now, be clear with your clients about the full team needed to make XR run well. A poorly staffed XR team might mean you personally keep working after a show that doesn’t go as planned, but it sets XR adoption back 6 – 12 months.  

Another XR challenge is thinking XR is only one incremental step in production technology. Nothing could be farther from the truth. Every department will need to be prepared to rethink how they do their jobs.  Before you assume I’m exaggerating, please read on.

XR Production

It may look like a duck and sound like a duck; it’s not a duck

If you take away one thought from this article, let it be this: do not underestimate how little you know about XR Production.  It cannot be understated how much of the production playbook must be completely torn out by the roots and re-written to use these tools effectively.  Existing budgets and schedules no longer apply. Team hierarchy no longer applies. That engineer in the back the Executive Producer barely knew existed is now telling the Director what shots will and won’t work. 

This will be hard for your client to understand. It’s the same team with the same gear running the same screens, right? All you did was plug in some camera data and what’s so hard about that? Also, the last time you told me you needed more time and money, you were fine, the show was fine so this is clearly just more of the same. 

Have you bid an XR project? Does this sound familiar yet?

That’s just one side of this problem. If the client cannot see the implications of how hard this is, please assume you might be making similarly poor assumptions about the implications to the rest of the production. We are often too willing to find a way to deliver to our clients without clearly stating the risks we are taking on to do so. Sometimes we overlook the risks entirely. 

As video screens have dominated event production over the years, every risk our team’s take has larger and larger impact.  Faults are harder to hide and failures can be showstoppers when video screens tip over 70% of the visual real estate.  In the past, a major issue might mean the lighting department steps up their look or you shoot around the problem. At the very least, rehearsal can continue.

When there is an issue with XR, the entire production stops.  Everything.  No more camera rehearsal and no more set until the issue is resolved. We’re used to being in the hot seat, some might even question the sanity of working under the stress of our normal production conditions. XR is another beast entirely. We cannot be cavelier about the implications of using XR without a backup plan.

I know we are fighting for budget, but it is worth considering having a parallel system to run traditional content while an XR issue is sorted out.  This is especially true of shows that use a single volume for both XR and 2D content (2D content in this case refers to spatially static content). It will seem like an impossible ask, but running a parallel system is cheaper than overtime for the entire production team.  At least discuss this failsafe and its implications when bidding a show.

As for explaining to your client what they don’t know about the job they have done for decades, that’s a far more delicate discussion.  Having a clear and thoughtful conversation reviewing the reasons behind the large increase in your team size, the budget and schedule implications will help, but you need a willing listener.  We need simple tools to explain the significance of choosing XR on a production.

Many XR production teams are producing Behind The Scenes clips to sound this message.  In other cases, manufacturers are providing documentation to help outline the changes to the production workflow when working in XR.  You can also forward this or Nils’ article. 

Sometimes the only path to fully realizing how different XR Production is, is to go through it. This is why I encourage you to stick to your guns about building up your team, budget and time requests to ride out the challenges of taking a client, or your team, through these first XR Production experiences.  You will need to be a clear communicator and thoughtful production partner to get through this XR Production development phase.

Inter-Department Communication

Preparing for your seat at the table

For an XR production, XR teams should prepare the client for a far expanded presence in production meetings and creative discussions as early as possible. Many choices and discussions that impact the XR process will occur without the production team understanding those impacts unless a central XR team member, like the XR Producer is there to capture those discussions and raise issues as they come up.  The trick of course is establishing this new position and dynamic within an existing group of professionals.  

I have long advocated for a greater presence of the Screens Producer role in production planning with mixed results.  I have experienced great collaborations and other times, I have felt like a diner waitress of video content. What matters is I kept working to evolve the role. Every producer or director will eventually have that light bulb moment that shows the value of collaborating with someone in the Screens Producer role.  Are you ready for that moment for XR? Is someone on the team ready?

Leadership is not a function of expertise, although that clearly helps with XR. Good leadership is being excellent at communicating, and good communication is a function of listening more than it is speaking.  You listen for the problems, you listen for the goals, you speak solutions and improvements. If you can capture the issues facing your clients early, you can solve potential issues before they become major problems by shaping your advice to known working solutions. Solving problems and making yourself a valuable partner will encourage production leadership to listen to you.

Good leadership is also building relationships with your partner video departments. The camera department is not thrilled with their new virtual overlords mucking about with their lenses. We have to show them we are there to support their work, not take it over.  We have to show the broadcast engineers and screen engineers we care about their success too.  We have to be respectful of how we integrate XR into these existing roles by educating ourselves about those roles and become valued partners.

Finally, good leadership is being thoughtful about the disruption of XR Production to the rest of the production team. They don’t know what they don’t know about Virtual Production and XR Production is the most technologically complex approach to unfamiliar territory. Production teams instead focus on what appears to be familiar while adapting to the difficulties of getting back to work at all in the pandemic.  We have to lead with a firm voice, clear information and some awareness not every production will adapt to XR easily.  Change is hard.  And budgets, they have a lot of changing to do.

Budget

Yes, this is (*&$ing expensive, don’t shy away from that fact

I expect the client may have laughed at your first bid.  Or possibly you were told your proposal was more than the entire shoot budget. Perhaps a forward thinking line producer doubled the budget you used on the last project together. It’s still not enough.  Worse, you haven’t been working much lately and really can’t afford to not do this gig. So what now?

Don’t hide the real costs.  You never want to be in a position that you overrepresented your capabilities on an underfunded budget. 

Start with the most reliable, well backed up system you can design staffed by enough team members to keep a project in motion even when some of you get bogged down solving technical issues.  If you are new to XR, add 20% and another engineer.  You’ll need it. Itemize everything clearly but as concepts, not individual pieces of gear, and be prepared to explain the impacts of not having each line item.  Explain where compromises can be made the result of those choices.  Also, prepare a budget for a 2D version of the project, one without tracked cameras and spatially responsive content. Use this to represent the cost difference between XR and a traditional screen control team and system.

This only covers engineering and operations. Content budgeting for XR has its own escalation of costs and the content studios with the expertise to build worlds for XR, like the teams that build and support XR infrastructure, are in short supply. Advise your producer handling budget to expect these costs if content production comes from another team outside your budget scope.

You will also need to clearly outline the time factor of XR, even when you can’t represent the full costs in your own budget.  Those camera calibration days? Clarify the other production team members not represented in your budget that you will need to participate on those days so the line producer can fully understand the cost implications. One hour ESU? Maybe you need two hours to start the day correctly or an extra morning set aside every few days to keep your system running smoothly. 

One of the complexities of building a budget is clear understanding of the project requirements.  A challenge of XR budgeting is the layers of assumptions made by both client and provider to the needs of a production. This is often complicated by the interconnectedness of the camera(s) to the XR system. In the past, I would be concerned I didn’t have adequate coverage in the signal flow to allow any truck signal to hit any screen. Now I’m concerned the director doesn’t know the camera restrictions. One of those conversations is a much easier fix.

My point is shot planning, while familiar territory for the film world, is not a thing for live events. Typically the live event world feels more like a documentary of a staged performance. Sure there are planned, even choreographed shots and I’ve worked with excellent directors. But much of the finesse of those shots evolves in the room, during rehearsal, where the camera is a creative extension of the director’s process.  The XR team is now stomping all over the director’s process, so it’s going to go about as well as you think it will when you say, “yeah, you can’t pan that far, we ran out of content budget.”

I believe the solution to the budget problem is spending more money on previsualization. Don’t look at previz as an added layer of complexity; it is a natural extension of content creation for XR. Previz will encourage critical decision making that will improve the XR Production process on site.

To bring production costs down, we need to solve problems before we are on site together. That way we can focus on only the problems that come up from building a defined plan, rather than discovering the plan while we are concurrently solving problems.  We’ll get back to the live-ness of live, but it’s an evolution.  We have incredible tools to engage directors, show them the world they’ll be shooting in, both on a monitor and via VR, and let the big creative choices happen before the real expense of production starts. 

I don’t have a bullet point list to give you on how to sell your budget. You will unfortunately have to make compromises and you will get frustrated. Your clients are frustrated too. Their bosses think Virtual Production is a suped-up Zoom call and have slashed budgets 40%. But start with clarity, ask your client to understand XR Production is new, expensive and as exciting as they think it is, but you need resources to do it well, unlike what you’ve needed in the past.  And then you can tackle the schedule discussion.

Schedule

It’s also ($&@ing time consuming, what about it

Clearing stating schedule needs will be new territory for most of us working in XR.  Most of us are given a schedule and told to stick to it.  Occasionally I might control the start date for my team and be asked to advise on how much time we need to build our media server system and prep content for rehearsal.  XR Production requires that we advise a production on our critical time requirements to be successful.

Each system and approach will determine how much time you need, but I often hear two days per tracked camera for calibration.  That’s after you have installed your system and have it fully functional, another two to three days at least. This describes building a volume from scratch, as opposed to renting an XR Production Studio where these tools have already been installed.

Once you have an operational system, maintenance time should be scheduled into the production calendar to offset unplanned recalibration needs or technical issues.  For the time being, we need to assume XR Production is slower and plan that way. What you are providing is the tools of post production into the shooting process, so in truth it’s not as slow as it feels for what is being achieved.

Of course, these time requirements are expensive.  It’s one thing to have a projection team align projectors overnight on a skeleton crew. It’s quite another endeavor to require the full XR, camera and engineering departments to participate in days of calibration and tests.

We expect these time demands will become greatly reduced in the future.  What takes days to calibrate will become a 15 minute warm up cycle at the start of the day.  Currently, we have to understand the implications of doing this work now, in its early development.  Hiding or avoiding these time considerations will further impact XR adoption by having productions fail or go way over budget during the shoot.

Time is the safety net to XR Production success.

Skills Acquisition Part II

Final thoughts on XR evolution

Those of you have now skimmed to the end to look for a simple ‘how to’ summary, I will leave you sadly unsatisfied.  What I will review is the outline of learning ahead of you:

New Software
New Hardware
Expanded features of software/hardware you already work with
Budgeting strategy
Schedule strategy
Team Building 
Leadership Training

You are not going to learn all these things at once nor should you expect to be an expert in all of these factors. I only want to reinforce that there is a significant amount of learning for the community to adapt to these tools and change effectively for XR Production to be successfully adopted.

Instead of a ‘how-to’ let’s look at why we should go through this learning process.

Your old job isn’t coming back anytime soon. Broadway is slowly going to start reestablishing shows for reduced audience sizes in June 2021. Live concert touring will start to recover with small shows in the same timeframe.  TV & Film production has some life, but nothing like it was 10 months ago. Many talented production professionals will not be able to wait. Many will find other jobs, and won’t come back to their prior fields. This will mean a years long recovery process to the quality of live entertainment that was produced in early 2020.

When shows do come back, audiences will lag. We have to rebuild trust about safe social activities, large crowds and travel. This means the focus on entertainment production will be on delivering engaging experiences people can enjoy and interact with at home.

We still have yet to solve that audience interaction piece and Zoom is wearing thin.  I think great discoveries are still ahead of us as we build virtual productions for truly live events. 

I see a glimmer of hope that production companies are willing to invest to make Virtual Production and XR Production possible.  Every successful event means more producers want these tools. These first ones are hard. I see my peers going through unimaginable levels of exhaustion blazing this trail. But momentum is building. XR will get easier thanks to their hard work, and the hard work of the developers creating the systems that make XR possible.

When the entertainment world can get back to normal, these tools won’t go away.  They will keep evolving into hybrid forms for live experiences.  Virtual Production will easily keep growing in the film world. Learning these tools can only expand your opportunities.

There is no short term lifespan for this learning investment. This is a revolution. Join us.

Laura Frank
Founder
frame:work

frame:work is a community platform for video professionals working in live and virtual production

—————————————————————–

My thanks to Nils Porrmann of dandelion + burdock and Matthew Ward of Superlumenary for helping review this article

—————————————————————–

Further Reading

Virtual Production: Exactly Where to Start by Drew Viehmann
Mixed Reality Studios: a primer for producers by Dandelion & Burdock

GLOSSARY

Virtual Production
The use of one or any combination of real-time content manipulation technologies that make background and foreground content responsive to camera point of view

Background Replacement
Real-time content generation of the background of a scene using camera position data. The resulting content can either be fed to an LED Wall or used to live composite a scene using a green screen.

AR
Real-time generation of content in the foreground of a scene, overlaid onto a shot using camera position data.

XR (miXed Reality)
A subset of Virtual Production that mixes background replacement with AR. Sometimes referred to as MR.

Scenic Extension
Expansion of a LED Background Replacement workflow that digitally continues the background image that would exist beyond the LED Wall

Virtual Production Solutions:

The following matrix is an outline of current virtual production solutions and their capabilities.  This list is based on the company websites of the products listed below.  I expect there are errors and welcome your feedback to make this list an exact outline of the current state of Virtual Production technologies.