How filmmakers are inventing the language of VR

Imagine you’re a filmmaker. Cinema is your craft, an art honed by more than a hundred years of brilliant ideas and happy mistakes. With it, you can do something that probably once felt like magic: capture any moment, idea, or feeling so anyone can relive it.

Now picture something new. It’s a lot like film, only it puts the audience inside your story — and it feels like magic, when it works. With it, you can create entire worlds for your audience, but none of the original rules of cinema apply. How do you create your art when all of your tools have changed?

This is how some filmmakers see virtual reality. And at least one director, Penrose Studios‘ Eugene Chung, likens virtual reality to the invention of the moving picture — and how it usurped the stage.

“I don’t think anyone alive really knows much about the language of VR,” Chung said in an interview before his company debuted its second VR animated feature, Allumette, at the 2016 Tribeca Film Festival. The film explores the sacrifices a mother makes for her daughter.

“In the last great art transition, the major stage play directors tried to become film directors, and a lot of them didn’t succeed because they did things like put the camera in front of the stage,” said Chung.

Today’s recipe for great film doesn’t work on an Oculus or HTC headset. The medium, in its infancy, may look like film (or like someone crammed a theatrical release inside a clunky headset), but really it’s something entirely different. In VR, the audience controls the camera. There is no language — no rulebook or established tools — for telling stories, so directors and technologists are inventing it as they go.

“Like the stage and theater, there’s a unity of time and space in VR in a way that there isn’t in cinema,” said Chung, “so things just tend to take longer to do. You have to actually wait for that person to walk across the room. You can’t go: ‘OK, she’s here” and then you cut to another [scene] where she’s across the room.

“We have experimented with lots of things — including scale. … For us, we want to create these big open worlds. We actually think about ourselves as creating worlds first, and then the stories and characters as secondary,” said Chung.

In Allumette, the viewer sees each scene as if they were perched in the clouds, a god-like view. You move the camera by looking around, like any other VR experience out today, but you can also change the scale of the set and characters simply by walking toward them. You watch it standing up.

“You can’t try to fit the square hole of cinema into the spherical hole of VR,” said director Rose Troche, whose second VR piece, Perspective Ch. 2: The Misdemeanor, also debuted at the 2016 Tribeca Film Festival. “It’s not like … ‘Oh my god, I have a script laying around, I’m going to do this in VR.’ If you’re going to do it in VR, then you have to rewrite it for VR.”

In stark contrast to Allumette, Troche’s film explores bias by capturing the perspective of police officers, suspects, and bystanders as an altercation spirals out of control. The viewer sees the film from the perspective of each character — you’re the cop pulling the trigger of a gun, or a 15-year-old collapsing onto the ground.

You really have to watch it sitting down.

In each film there’s this sensation of being there — and yet not. “There’s a really interesting question of identity in VR,” said Chung. “Have you ever seen Patrick Swayze in the movie Ghost? In VR, you kind of feel that way. Let’s say there’s a dog here, and I’m like, ‘hey dog,’ and the dog doesn’t respond to me the way I think it would, you feel this weird effect — you feel like Patrick Swayze in Ghost. It’s ‘the Swayze Effect.’

“That’s the weird thing about working in these new mediums,” said Chung. “You work in a brand-new medium, you don’t have a name for things, right? When you don’t have a name for them you just have to make them up.”

The ghostly sensation Chung described persists in The Misdemeanor, and both cases point to a defining characteristic of virtual reality mostly absent in VR film today: interactivity.

Neither The Misdemeanor nor Allumette are significantly interactive, but each work successfully transports the viewer somewhere else. In both films, you eventually leave the room you’re in — you’re no longer worrying about sitting there awkwardly with a goofy headset on while onlookers take Snaps.

Pearl, Google Spotlight Stories’ latest animated VR short, broadly lacks interactivity, too, but Google and director Patrick Osborne did manage to insert subtle doses of viewer control into the piece. It debuted alongside Allumette and The Misdemeanor at the festival.

“With Pearl, it’s a story of a dad and his daughter growing up, and it happens in a car, throughout their lifetime in a sense — at least, the growth of his daughter. … There are moments where, based on how you look or who you’re looking at, we move the story … or not, or we take you somewhere else, so you see a surprise — an Easter egg in a way — but it makes sense in the moment,” said Rachid El Guerrab, a project lead working on Google Spotlight Stories.

You can bemoan the lack of complex interactivity in VR films today, but simply shooting 360-degree video presents major challenges for filmmakers. While recording continuous 360-degree scenes for The Misdemeanor, Troche and collaborator Morris May, founder of VR studio Specular Theory, had to blend in as extras in the film.

“We’re hiding [in the scene]. I’m in the crowd of people … the sound guy, he’s in a cyclist uniform. You just make your crew extras — whoever’s willing to come on board,” Troche said. “Try directing that way — it’s really crazy. You’re like, ‘Did we get it?’ It reminds me of when I shot in Super 8, and I’m like, ‘Well, I guess I’ll see it when it comes back from the lab,’ you know?”

Thinking about the possibility of not just watching a movie, but somehow existing inside it, is the easiest way to realize that VR filmmakers have yet to truly realize the full potential of the medium. Yet that’s what makes VR’s evolution so mesmerizing. Measured in terms of polish, VR cannot yet compete with cinema. Measured by impact, it absolutely can.

As Hayao Miyazaki, a master of traditional animated film, transports the audience inside his stories (see: Spirited Away, a story about crossing from childhood into adulthood), so can Chung’s team pull the viewer into a story about coming to terms with loss — as can Troche and Google Spotlight Stories, in diametric ways.

The medium is a work in progress — and maybe it always will be — but it’s also already proven itself worthy of attention, regardless of its longevity or potential to replace cinema, like the moving picture eclipsed the stage a century ago.

The ideas and idea makers are there — directors and technologists hard at work, crafting the medium before our eyes.