Lytro’s Immerge VR demo shows the promise of lightfield video

An astronaut steps out from the lunar lander in front of me, beginning the familiar “one small step for man” line before a voice shouts “cut!” from behind me. The lights cut on and the scene opens up to a full studio — I’m on the set of the famed urban legend of Kubrick faking the lunar landing. At first glance it appears to be fairly similar to any 360 video — something that some would argue may not even really be “VR.”

But this video has one key difference: I can lean in around the scene. I’m witnessing one of the first ever videos to come out of Lytro’s upcoming Immerge lightfield camera.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":2046002,"post_type":"feature","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"arvr,games,media,pc-gaming,","session":"A"}']

Ever since announcing the Immerge camera in November of last year, the company has remained fairly quiet outside of teasing the Lytro Cinema camera at the NAB conference this April. But after months of silence, Lytro has finally given us a glimpse in what it means to experience a true “lightfield” video in virtual reality, and while it is not quite ready yet, it offers a promising look at what is to come.

What’s a lightfield?

A quick primer on lightfields, which can be a bit of a tough topic. When you look at an object, you aren’t really seeing that thing, but rather each ray of light as it bounces off of it and into your corneas. As you shift position, the angles at which that light hits your eyes changes, resulting in the subtle changes in reflections and shadows that give objects depth and realism. A lightfield camera captures every ray of light as it passes through an area or volume, like an array of image sensors. When rendered properly, this produces an effect that allows one to view a live action scene from multiple perspectives with true depth.

Lightfield is quite a buzzword, consisting of cameras and displays being touted as “lightfield” devices. What Lytro means when it says it’s producing a 360-degree lightfield camera is that the video that you’ll watch is effectively a capture of thousands of different perspectives within a given radius. What that means effectively is that you’ll be able to move your head and change your perspective within the 360-degree video.

The effect feels quite natural, displaying an impressive amount of depth and parallax to the left and the right. Even the audio takes space into account of the experience, and I could locate sounds quite well within the scene without having to look at them. However, while there were a lot of positives to the progress of the tech, it wasn’t perfect. I noticed visible artifacts around the live-action cast, resulting in less than crisp outlines of the figures in the scene. That said, it wasn’t enough to be tremendously distracting, and a more recent demo (the Moon scene was six months old, according to Lytro) showed significant improvement in that regard.

According to Tim Milliron, Lytro’s VP of engineering, the scene blended rendered lightfields with the live action lightfield captured by the camera. This means the CG objects reflect light differently from every perspective you watch from. Having the light flickering off of the astronaut’s helmet match the movements of my head throughout the scene was a nice touch of detail that adds such inherent value to the realism.

The tech

 

The real meat and bones of Lytro is way beyond the camera hardware. In addition to its camera hardware technology, the team is also building the lightfield processing and the software tools for creating true immersive 3D videos. The development plugin works with Nuke, which is a standard tool in the film industry.

The rig that created this video was Lytro’s first iteration of their Immerge camera. For that reason, this setup was built on a plane of 60 cameras, positioned in a staggered hexagonal pattern creating a flat plane of cameras, called a “wedge.” The entire array is .75 meters high and .75 meters wide, connected to a big server rack processing all video feeds. By rotating the rig and shooting the scene five times from five different rotational positions of the rig, it created a 360-degree video scene from essentially 300 camera perspectives. It’s common practice to shoot only portions of 360-degree videos in multiple takes, in order for the production crew not to have to hide every time the director yells “Action!,” so this fits into the current pipeline.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":2046002,"post_type":"feature","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"arvr,games,media,pc-gaming,","session":"A"}']

Due to the nature of lightfield video captured in this way, the moveable volume from the viewer’s perspective is restricted to the physical volume of the camera. In the case of this prototype, the camera offers a sphere with about a half meter diameter, though Milliron tells us that they are expanding that for the next iteration. Hypothetically, one could continue adding more cameras to the rig, increasing its volume potentially limitlessly — something that Milliron says speaks to the rig’s configurability.

Lightfield video is still in its earliest stages, despite the promise this demo showcased. The file sizes for these videos are still astronomical — requiring an entire server cabinet to record. The 30-second clip I was shown was 50GB alone, and required an extremely powerful computer to run at 90 frames per second. Milliron says they plan to optimize the compression for these videos quite a bit (lightfield video is actually highly compressible) and are targeting a 5GB file size for the same video by the end of the year. The size alone will restrict its ability to scale in the short term, but it is a solvable problem, Lytro says.

The company is in talks with a number of “major content creators” about applying its technology to their content, with plans to start shooting more within the year. For now, however, this tech demo will remain just that — a tech demo. There is no word yet on when Lytro plans to release the demo to the public; for now the plan is to showcase it only at events that the company will announce at a later date.

Main image is a rendering of Lytro’s Immerge camera, not the final product — Ed. Additional reporting by Will Mason.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":2046002,"post_type":"feature","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"arvr,games,media,pc-gaming,","session":"A"}']

This post first appeared on UploadVR.