Building Environments in UE5 for LED Walls

With the rise in popularity of Virtual Production, it is more important than ever to optimize your UE5 environments for use on an XR virtual production stage.

As the team here at CoPilot has worked to develop environments, implement workflows, and refine a pipeline for XR Virtual production, we've picked up a few tips and tricks to optimize a virtual scene for placement on an LED Volume.

Let's get into it.

If you've done all of your pre-production and effectively mapped out your shots, you're in a great place for one of the biggest performance savers, and that is only building out your scene around your framing.

Don’t overpopulate the environment if you aren’t going to be seeing it.  Building everything is unnecessary! To make your project shine you’ll need to optimize as you work.Imagine the pain you’d feel if you built a massive cityscape, only to find that just a single corner of it was being used— be efficient, find your shots, and only build what you see.

So what’s the best way to achieve this? Just like in traditional filmmaking, you storyboard or plan your shots. Place virtual cameras in the environment to what the camera will see - and make sure to account for any camera movements you might be making that will reveal more of the environment. 

If you know your LED Volume layout, your camera movements, and the lens data you can frame your shot in-engine and only build out the scene based on what is being captured and rendered.

Subtlety in world building reveals certain things about your world that you want the audience to know— in this method, less is often more!

Just because you've mapped out a massive landscape doesn’t mean every single shot has to reveal all the work you’ve done. Doing a lot of work in a small area of your world can reveal more about your scene than hours put into details that are lost in a wide shot. 

Small details give your world character, and that character can work in tandem with the theme of your shot, sequence, or even the whole movie.

Knowing your story and shots can help direct where you need to put your energy and time when developing your environment.

There’s a paradox to this one— yes, you CAN build whatever you want, explore your creativity and make an awesome space that's never been seen or thought up before, but even in elements of fantasy, there has to be grounding elements that connect the audience to the visual narrative and a degree of environmental storytelling that authenticates the world and the experiences within.

Let’s say you’re building your own house— you’ll know every nook, scratch in the floors, and chips in the ceiling that bring realism to your world. Going with what you know reveals to the audience your mind, your perspective, and what matters to you.

This level of environmental storytelling puts a piece of yourself into the story adding a layer of perceived emotional complexity to your scene.

When you look at a virtual person— your mind goes two ways— it’s either a depiction of what a person should look like, or its a real person. But there is a weird space where your depiction looks about 90% real, and that’s where we get into uncanny valley. It makes people feel uncomfortable because things aren’t quite what a human should look like to our human brains… the same can happen with environments. 

When making an environment in UE5, yes things look hyper realistic, but not true to life. I know people are going to disagree with me on that one, but I have yet to see a TRUE UE5 render that looks absolutely real to me— and its because our brains are really good at depicting what’s real or unreal. SO, when making your environment, keep this paradox in mind. Trees in unreal look really good for a video game, but if you just saw one in real life you’d notice the textures and lights reflecting off of it look different. 

Sometimes the depiction of a tree, something far away that looks like and acts like a tree, is better than a hyper realistic tree. 

So keep this tip in mind — When building out your environment it’s not so much about making sure the environment looks real to you, but how the different elements are picked up/understood by the camera later.

A major difference when building an environment for an LED Volume instead of say a stylized animated short or a video game is that you want the scene to emulate photorealism, while at the same time acknowledging that the environment is being used to extend the background and won't necessarily be an object of focus in the final shot. This means that it'll be largely impacted by depth of field falloff from the practical camera.

To merge the virtual and the real world together, one of the biggest things we can do is to bridge the divide between physical objects and textures and the ones used in our environment. 

Know what environment props realistically work in the scope of your project, and build your environment accordingly

Similarly, Don’t make lights in unreal that aren’t replicable in real life, consider the source of the lights illuminating your environment, how many lights need to be matched on-set

The virtual environment needs have matchable colour to your physical environment. Find the balance between limitless and realistic. You can even use photogrammetry to make that happen.

Once you have done all this, here’s the final touches needed to prepare it for a volume: 

1. Make sure the computer that’s outputting the project to an LED volume is suitable. Check out our other PC build video
2. As an environment artist, aim for a target frame rate of 2-3 times your camera’s frame rate. If 24fps, aim for a 50-60fps project
3. Ray tracing is fun, but if you can avoid it…. BAKE. YOUR. LIGHT. 
4. Set any rendering/dynamic/high-quality features early as any adjustments could impact other settings along the way
5. If you need dynamic lighting, ask yourself if you want dynamic shadows. Because the dynamic shadows are truly what sucks the performance out of your project.
6. Remember, a static set extension is less expensive than a dynamic background

Getting a project prepared for a volume is all about performance. You want to do anything to reduce the power being sucked from the PC’s processors and graphics cards to allow for the LEAST latency possible. Latency is key.