Problem:
It’s hard to reproduce natural, cinematic camera movements both quickly and at a high level of quality in a game engine, especially while in VR.
Goals:
Build a simple virtual camera system that allows us to pilot a cinematic film-style camera using tracked VR controllers in order to capture in-game video footage.
Be able to change camera settings in real time while filming.
Allow a user to see what they’re filming while in VR, while still being able to progress through the game.
Also allow external monitors to display the cinematic camera feed for external viewers.
Overview:
We approached this as a variation of a VR spectator camera. We needed to have VR enabled since we were using the tracked controllers, but the screen output needed to be a camera separate from the player view. We developed this as a modular blueprint that could be dropped into any similar VR compatible project and used as-is. This tutorial from Epic is a great jumping off point.
In Unreal 4, this meant using a SceneCaptureComponent2D render target system. We also used many of the available post process settings so that we could closely match the look of our regular in-game cameras. Since this would result in a 2D video, we could also leverage fancy effects like depth of field and lens flares that we normally can’t use in VR.
In order to simulate cinematic camera operation, we added controls to the Xbox controller including field of view angle, depth of field focal distance, aperture and a global exposure setting. We also had the ability to control the camera movement smoothing speed. Since humans are inherently jerky and imprecise in their movements, this was a way to simulate a “Steady Cam” type of system. This was useful depending on whether we were shooting close or far. The closer the shot, the more responsive we needed the camera to be.
We had access to some great hardware which is not necessary but certainly helped. We mounted an Odyssey 7Q+ wireless display capture device and a Teradek Bolt 1000 wireless display receiver onto a director’s mount along with a Vive controller and an Xbox controller. The weight actually helped to make the system move like a real camera and feel tactile.
For our next iteration of this system, we plan to use a shoulder mount system instead of the director’s mount. Or we’ll at least have something similar to the director’s mount attached to the shoulder mount. The Xbox controller will also be directly attached. This will allow one person to have full control while operating the virtual camera.
Our capture computer was running a GTX 1080ti with 64GB of RAM and an i7 5960X. We were almost always able to maintain a consistent 60fps capture rate.
Our captured footage was 1920x1080 at 60fps.
Lessons We Learned:
We originally intended for this to be operated by one individual while in VR. This did technically work, however, even with the high-res displays of the Rift S and Vive Pro, it was hard to tell when things were properly in focus while in VR. There would have to be more time spent increasing the fidelity of the in-game camera monitor. But it was still very cool to be able shoot like a real cinematographer and see what was being filmed while in VR.
The lack of a shoulder mount in this first iteration was also frustrating. The system was ~10 pounds and it made certain shots tough due to shoulder strain. A shoulder mount is essential in our opinion, both for comfort and shot quality.
For certain shots, since we didn’t have the Xbox controller mounted to the camera mount, a second person was required to pull focus and control the camera settings. This actually worked really well but having the controller mounted could allow everything to be managed by one camera operator.
Beware the output color setting of the SceneCaptureComponent2D in UE4. We ran into this after our first shoot and realized that the setting we chose resulted in footage that was too dark. It was then difficult to brighten without adding noisy artifacts. The setting that worked best for us was “Final Color (LDR) in RGB”. It’s probably best to shoot a test scene and then run it through your video post production process to see the results. UE4’s SceneCapture output color might be slightly different from your expected video output. Depending on the setting chosen, this will also determine whether your SceneCapture actor captures your post process effects.
Conclusion:
We’re excited to build upon the first iteration of our DIY steady-cam for VR. There are plenty of improvements to be made in both the physical hardware that we use, and our virtual cam support in-engine. Quality capture of VR content is a difficult dragon to slay, but we think the footage we got for our Holodexxx Home teaser shows the promise of this technique. We’re looking forward to experimenting with more technically complicated shots, and thinking bigger for our next trailer.
Hardware Buyer’s Guide:
If you are interested in recreating this virtual steadycam, we’ve included a part’s list for our wildly extravagant rig, and a much cheaper alternative that could get the job done just as well for a fraction of the cost.
Best Option, but $$$:
HTC Vive (Oculus Rift S can also work but controller tracking is worse)
Camera shoulder mount system (https://www.bhphotovideo.com/c/product/1531534-REG/camvate_c2247_handheld_shoulder_mount_rig.html)
Choose one that can mount both the wireless display and VR controller at the front, rather than the back
This could be a different type of mounting system to hold the components together but we feel a modest shoulder mount would produce the best results
Odyssey 7Q+ wireless display and recorder
Monitor or TV (1920x1080)
Cheaper Option:
HTC Vive (Oculus Rift and also work but controller tracking is worse)
Camera shoulder mount system (https://www.bhphotovideo.com/c/product/1531534-REG/camvate_c2247_handheld_shoulder_mount_rig.html)
Technically, you just need a mounting system to hold the components together but a shoulder mount would produce the best results
You could probably DIY this with common items and save the cost of purchase
Video capture card (AVer Media Live Gamer Portable 2 Plus or similar card)
This could be used in place of the expensive wireless display+capture device but it would require someone to manually start/stop the recording. With some dev effort you could probably also develop a command system so that the Xbox controller could trigger start/stop.
Screen capture software like OBS could also be used but will place more strain on your PC, potentially causing a choppier recording since it’s running alongside the game engine.
A mobile app like RemoteMouse (~ $4) could also be used instead. It uses WiFi to connect to a PC in order send mouse and keyboard commands
This works fine but is more cumbersome to use since it can’t be finely tuned like the thumbsticks of the Xbox controller
Monitor or TV (1920x1080)
