When Is Production in Mixed Reality Sets Worth It?

Virtual (VR) or Mixed Reality (MR) can do more than replace physical sets. What interests us most is the creative freedom. Because this is where the way we think about and experience events could truly be revolutionized.

We Need to Rethink Events

Videos and events produced in front of a greenscreen or an LED wall and then combined with 3D worlds from Unreal Engine look really impressive by now. The way we produce visual content - not just in the high-level segment, but across the board - is currently at a turning point with enormous opportunities for all of us.

And this doesn’t merely mean that physical sets can be replaced by virtual ones to save costs and simplify production conditions. What’s truly interesting are the creative freedoms that emerge for live or partially pre-produced events.

Case Study: Virtual Panel and Keynote

For the 10th anniversary of Jakobs Medien, we offered our guests a moderated live talk show production on the topic of “Next-Level Storytelling Through Virtual Production” in our greenscreen studio - including a company keynote and a stand-up comedy performance.

WYSIWYG: Production on our greenscreen stage with preview monitor showing the live-keyed virtual sets

The setup: Greenscreen stage with lighting, audio, and remote-controlled PTZ cameras open toward the office area, seating for 30-40 guests facing the stage, mixing in the control studio beneath the stage. Via preview monitors, our guests in the audience could watch live as the panelists merged with the virtual environment.

The idea: We provide the experts with a stage and use the case for a product presentation. The goal was to show what’s already possible with us, what’s already working elsewhere, and to provide inspiration for how events can be built in virtual or mixed reality sets.

Storytelling: Planning Keynotes as Emotional Journeys

Of course, a presentation can simply be delivered in a mixed reality set - with a visualized PowerPoint on a virtual screen in the background and a person moving on a virtual stage, delivering the keynote.

But the real opportunity that virtual sets offer is the content and visual fusion of backgrounds, animations (graphics, logos, diagrams) and the presentation. These components are like protagonists in a stage play, telling a story together and taking the audience on a journey.

Presentations Staged Like Theater Pieces

Seamless transitions between pre-produced and live-generated content. Video scene from the Elbe Sandstone Mountains…

Through the ability to swap keynote sets in quasi-real time, fluid transitions from video inserts to the next point of the presentation can be staged. For example, in our case, when the image film ends with a team event mountain climb in the Elbe Sandstone Mountains and the presentation then picks up at that very location in the virtual set, because that’s where the idea for a new product was born, which is then presented.

… initiates the setting change. The keynote continues in the virtual Saxon Switzerland.

Likewise, changing backgrounds can elevate the protagonist’s presentation to a new emotional narrative level. In the JM keynote, there was a segment where light sources gradually went dark until complete darkness, symbolically representing the cancelled contracts during the lockdown. This was followed by a dialogue with displayed messenger messages that led to a new wave of commissions, crowned by the liberating moment when the showreel video started and celebrated the project highlights of the years that followed.

Background, motion graphics, and protagonists create the staging together

Different Workflows Needed in Event Planning

All of this naturally requires a shift in planning. Keynotes are then no longer just presentations - they are planned like shows or theater pieces.

This changes the traditional planning steps, requires more effort at the conceptual level, needs a detailed storyboard, and a more intensive rehearsal phase. But the result is an emotional journey, content, and a presentation that stay in the audience’s memory. And that should be the goal of any event - especially given the prevailing content overload of information.

Why Live and Not Pre-Produced?

Producing live means determining the moment when the message lands, when the discussion in the desired community is kicked off. We all compete for the attention of our target audiences. Without the live moment, there’s no necessity to watch right now. And the longer people wait, the faster the relevance fades.

Being live also means being there in the moment when something special happens. It gives you the feeling of being part of the event. And when audience interaction is added, an event becomes our shared experience.

This works with a clapping and stomping audience rocking Wembley Stadium together with Queen, just as it does when theater actors break the fourth wall with questions to the audience or pull someone onto the stage.

It’s about building a relationship with the audience to have their full attention for our message - for the story we want to tell. That’s why interaction is the key to more attention… and to contemporary events.

Interaction Is the Key to Contemporary Events

For corporate events or events in a more conservative context (political talks, delegation meetings, publication launches), remote speaker connections or claim-the-stage situations from the audience (via avatar or live cam) offer an interesting possibility for active participation.

Give feedback via conference tool or become part of the panel

Live polls or sentiment readings could in turn be visualized through 3D animations in real time, while haptic feedback like applause can be expressed via keyboard or through graphical components (applause-o-meter) displayed on screen.

The Gamification of Event Formats

The playful co-design and control of a virtual set is also possible. Starting with the collective selection of virtual backgrounds, the time of day, the weather, or the lighting mood, but also triggering animated set elements (a commuter train, a labeled hot air balloon, or a flock of birds flying through the background). The keyword here is gamification.

And where the journey will ultimately lead is shown by recent events in the metaverses of gaming communities, where for example virtual avatars of DJs perform in Fortnite levels, uniting up to a billion gamers in front of their screens who co-create the moment with their own creative in-game actions. A generation growing up with such gamification features is already making - and will increasingly make - exactly these demands of special live events.

Which Formats Make an MR Production Worthwhile?

In principle, VR and MR are suitable for all event formats to tell a far more emotional story that should leave a lasting impression on the audience. But also for events that, through the interplay of changing locations or settings that partly don’t exist in reality (journey into the self, Martian landscape, complex moderation studios, underwater show, etc.), want to create a connection between the visual language and what is being presented.

However, one must not forget that such event productions in virtual or mixed reality sets involve significantly more effort in terms of planning and preparation. If this doesn’t develop into a recurring format that becomes more cost-effective with each subsequent edition, then such events are particularly worthwhile for exclusive occasions. For example, corporate events intended to inspire a company’s workforce or express a certain appreciation toward the audience. Or product presentations/launches that want to raise the visual bar because they want to determine the starting point of public discussion (creating buzz) through a remarkable appearance.