Building an alternate reality in augmented reality: UCLA redefines theater with Unreal Engine

 

Setting the scene

“We first became interested in AR because it aligns with our values,” begins co-director of the production, Mira Winick. “REMAP was built to encourage collaborative innovation and creativity amongst both students and faculty. With AR, the audience could get involved too. Students, faculty, and audience members could have a unique shared experience in a non-physical space. We knew this would be a great medium for art with a message, and instantly decided to build our next project around it.”

With its narrative involving overlapping timelines and prescient socio-political themes, an original story set in the world of The Man in the High Castle seemed like a perfect direction for the AR project. Having dreamt up a unique reimagining of Philip K. Dick’s 1962 novel, REMAP set about acquiring sponsors and applied for an Epic MegaGrant.'

The pitch was for an AR immersive theater performance set in the world of The Man in the High Castle, in which the audience would carry an AR application called The Grasshopper Lies Heavy, allowing them to view a parallel story unfolding in another timeline. For example, the audience can see actors depicting the oppressive reality in which the Axis powers won World War II right in front of their eyes and use their AR devices to glimpse a world more like our own, in which protests against facism begin to inspire hope for a better day.

While REMAP had previously used Unreal Engine 2.0 for a live theater production in 2006, the team found Unreal Engine 4.24 to be far more accessible than earlier versions, meaning students with little to no Unreal Engine experience could get to know its capabilities. “When A Most Favored Nation was first conceived of it was decided that Unreal Engine would provide the necessary performance, features, and stability to incorporate AR elements,” says Burke, explaining the production was part of a cluster of activities around Unreal Engine at UCLA, which included film previsualization courses, remote virtual performances during the pandemic, and immersive learning environments.

In the midst of the pandemic, the extensive previs process required to produce great AR also meant that the team could understand how 3D and real-life elements would interact without being able to physically access the set. As soon as the set became available, the team had only to confirm that things worked as expected rather than design from scratch.

“Previs became essential both from a social distancing point of view and an accessibility point of view,” says lighting designer, Ben Conran, who was also an Unreal Engineer on the project. “We became very reliant on showing AR through previs for directors and cast members who couldn’t be in the space or as a way of circumventing a build procedure when the time was short. When shooting our green screen footage of what would be the actor’s AR performances, the real-time previs allowed us to match and adjust the lighting to fit the tone of the AR spaces and gave the director a chance to edit and refine the performances live during the shoot.”

Next, all the action was performed across a large theatrical stage that had been configured as a sound stage. Each storyline had a separate physical set, in some cases two, with a corresponding AR set to allow different timelines to be shown to audiences in the same space without people seeing and hearing the same thing.

The live-to-tape approach meant that scenes didn’t run them concurrently, but the intent was to use standard installation and theatrical techniques to provide sound isolation, moving the audience through interlude scenes in which they could make choices about where to re-enter the story.
 

Showtime

In the end, the show’s audience consisted of people who knew the show by virtue of being in its COVID bubble and guests who had no knowledge of the piece but came to participate in the live-to-tape process. “One consistent takeaway was that the live experience was really appealing and very difficult to convey in recording, even as much as we tried,” adds Burke.

Despite this admission, Burke also acknowledges that the mandatory distance between the real-life and AR performances added a compelling emotional layer to the show, with the feeling of two separate worlds proving to be a nice thematic fit. Of course, delivering a great show wasn’t the team’s only goal. They also wanted to find out how integrating AR into a story would impact the creative process and audience experience. They emerged with three key findings.

First, it’s clear that AR can be effectively integrated into immersive performance and provide a new and interesting approach for engaging audiences. The technology seems within reach to offer commercial experiences within a few years. Secondly, REMAP found that the results of the work and many conversations with those involved in the project validated the benefit of having the technology exist in some form within the fictional world as well as within the real world. This made it central to the drama, relevant to the characters, and more than just a gimmick.