How to teach virtual production

How to Get Started 

Know the options

There are many roads to successfully teaching virtual production. A great place to start, however, is by looking at what you know. In most cases, a VP program can be slotted into what the school already teaches. 

“If you have been teaching games, then take a game level, look at how it differs from what is needed for VP, and work from there,” explains Mark Flanagan, Education Partner Manager at Epic Games. “If you’ve been teaching on-set lighting, start by exploring how to apply those same theories to lighting in Unreal Engine.” 

The type of VP you choose can also make a huge difference to your curriculum. There are various different types of VP, including: 

• Visualization: using imagery to convey the creative intent of a sequence. Visualization can take the form of previs, virtual scouting, stuntvis, and more.
• Motion capture: recording the movements of objects or actors, then using that data to animate digital models (think Gollum in Lord of the Rings).
• Hybrid virtual production: using camera tracking to composite green screen cinematography with CG elements.
• LED virtual production: replacing green screens with LED walls that can display CG elements and images from real-time engines. This means everyone on set can see and capture final-pixel imagery, completely in camera. Popularized by The Mandalorian, the use of in-camera VFX (ICVFX) will only continue to grow. 

The right VP for you can depend on many factors, from your budget to the physical space at your school. According to Manon Hartzuiker, Virtual Production Producer at Framestore, visualization is a good place to start. “It sits at the top of the production process and leads into other areas of the workflow,” she explains.

Finding and training educators

The last piece of the puzzle is the most important to get right. Whatever type of VP you choose and tech you buy, it’s crucial to hire new staff with VP experience from the industry, or train your current team to a high level.

For Spencer Idenouye, Virtual Production Lead at SIRT, the Unreal Fellowship program was the perfect way to train. “It was a fantastic opportunity,” he remembers. “In five weeks, I was able to construct a short film almost entirely using Unreal Engine and Unreal Marketplace assets. Participating in this sort of exercise, while documenting the process along the way, helped me to gain an in-depth understanding of the skills needed for virtual production.” 

Planning Your Curriculum

The top seven topics to teach VP students

With a technology that spans across as many applications as virtual production, it can be challenging to decide what to teach. Here are the most important topics to cover:

1. The fundamentals: It’s critical for anyone working with VP to understand the correlation between film production concepts and their game engine equivalents. That means learning about editing, directing, acting, and composition. “Learning VP does not mean you don’t have to know how to make movies,” adds Flanagan.

2. On-set lighting: Learning about lighting is also essential for VP students to create realistic backgrounds. “Matching the lighting of the physical foreground to the virtual background really brings everything together,” says Joerg Schodl, Cinematographer at SCAD.

3. Scheduling and production: Virtual production is making the industry re-evaluate pipelines that have been in place for decades. When VFX no longer happens in post, planning needs to be airtight. That’s why VP students need to learn everything they can about scheduling and production.

4. Virtual design: It’s not enough to teach students about onset filmmaking. They need to learn about CG too in order to create and prep content for real-time environments. “It’s important to teach students how 3D assets are built for both static meshes and skeletal-based characters through geometry, materials, rigging, blendshapes, and more,” explains Spencer Idenouye, Virtual Production Lead at SIRT.

5. Stage operations: In LED VP, the setup and engineering of the LED wall including color calibration, nDisplay setup, and more is critical. For students, learning the basics of how these work will give them a deeper understanding of what goes on behind the scenes.

6. Performance capture and camera tracking: If you want to animate 3D characters in real-time, or film actors on the moon, you need to learn all things mocap. Students should know how to ensure cameras are properly calibrated, which solutions they can use for streaming mocap data into Unreal Engine, and understand facial capture work.

7. Visual scripting: Familiarity with game logic and mechanics will help students develop or customize tools that will support their virtual production projects, and help them solve complex problems once they move into the industry.

Educators need to consistently maintain close relationships with technology companies in order to keep up with hardware and software updates. “As new hardware is released, schools could even request early access and give feedback to developers as a way to stay on track,” Meadows explains. “Creating and participating in user groups and forums is also a great way to stay current.” 

According to Framestore’s Hartzuiker, it’s also important to keep in touch with VFX production teams. “Every production is different, and we're always finding new ways of making our workflows more efficient,” she says. “When schools reach out, we’re happy to share our findings.”

Setting Job Expectations

Following virtual production training, students will likely take on a variety of roles, such as:

• Virtual Production Supervisor: Manages the on-stage component of virtual production technology to achieve fully integrated live action and CG hybrid shots. 
• Technical Artists: Uses Unreal Engine to create virtual set extensions, optimize data for real-time playback, animate with Blueprints, and more.
• Production Designer or VAD Artist: Takes advantage of virtual set scouting to finalize environments via VR & AR technologies. 
• Previsualization Supervisor or Artist: Conceptualizes film projects, including establishing the creative direction of key sequences.
• Mocap Animator: Captures, refines, and delivers realistic animation based on motion-captured data.

As well as the jobs above, a virtual production generalist will also have access to many other traditional filmmaking roles. “Everything in virtual production stems from a solid understanding of filmmaking, and conversely, any new knowledge or techniques obtained from learning virtual production can inform new processes for traditional filmmaking,” explains Brian Pohl, M&E Technical Program Manager at Epic Games. 

After VP training, for example, a student could decide to be a Director of Photography or gaffer after doing things virtually, before ever even touching a real light. Graduates could also expand into the role of the Digital Imaging Technician, or cinematography for the same reasons.