When we get our first box of crayons in preschool, the primary rule that the teachers drill into us is to draw within the lines of a flat piece of paper. Our imaginations may be infinite, but our two-dimensional representation of it has a limit that ends at the four borders of a rectangle.
As the Oculus Rift and virtual-reality technology matures, artists whose traditional craft requires respect for the frame are going to find themselves in a new world that expands far beyond the edge of the 2D canvas.
It’s alluring to see this 360-degree medium as limitless, but this borderless world is not as free of structure as it first appears. Artists face a new set of rules to play by if they intend on creating an intentional, visual, virtual-reality composition. The people working with the virtual-reality crayon box are scribbling away, using techniques from film and gaming to try to unlock what practices are considered pleasant or harsh when directing an audience that’s wearing a set of goggles or a helmet on their heads.
The new visual language
One of the creative linguists attempting to translate virtual reality’s new visual language is Dennis Danfung. Traditionally a cinematographer, Danfung is currently working on Zero Point, a documentary film about virtual reality, shot for virtual reality. After giving me a demonstration of the film using Oculus Rift hardware (my first experience with the headset), I was struck by the new challenge artists like Danfung faced. Externally, I kept my cool, but I desperately wanted to discuss all of the details of what he has discovered so far about the new challenges of composition and pacing.
“All we know is that traditional rules of filmmaking no longer apply,” Danfung said. “We’re focused on the technical aspect to begin with as there are still big challenges in the capture, workflow, and playback … but I think the even bigger challenge is the new visual language that is going to emerge.”
Danfung explains that shooting a scene for Zero Point requires several cameras attached almost spherically to a special rig, pointing outward. These cameras capture the scene from several different angles, encompassing what would be the viewer’s left, right, front, back, up, and down vantage points. This footage is then stitched together and projected in a device like the Oculus Rift, off of the inside of a polygonal sphere, with the viewer’s controllable viewpoint being in the center.
Danfung adds, “We’re actively learning, researching, and experimenting. As a cinematographer, this is completely new ground – the essence of the frame is gone. Now instead, we’re capturing and delivering a massive amount of raw information to allow the viewer to decide what the frame is.”
My mind boggles at how this process completely obliterates the concept of the 2D “frame.”
The boundary of the image is not the only concept being completely reinvented by this 360-degree screen. Film, like video games, is an artistic collaboration of several different disciplines (theatrical arts, cinematography, graphic design, special effects, etc.). One of the few unique things film does that all of these other mediums do not is the instant manipulation of time and space through editing. Visual pacing within the animated two-dimensional frame, using one shot followed by an abrupt change to another shot, has had a century to develop into its own language. Artists have spent that time crafting the edit, learning what is considered pleasing or disturbing for audiences, building a foundation that film makers after them utilize to perfect their cinematic expressions.
Within a virtual-reality setting, however, Danfung is finding that the laws of visual pacing are completely different in the audience-encompassing canvas, “The cut, the most fundamental, basic technique of editing … it’s gone. It’s too abrupt [for virtual reality]. We’re finding that if you cut from one scene to another, people will all of a sudden become disoriented and ask, ‘Where am I now?’”
An interactive medium may hold some clues
There is another young medium, however, that has had a few decades head start in tackling the problems of pacing and audience manipulation in a 3D space: video games. Like film, the final visual product of a video game has been traditionally confined to a rectangular border, but 3D video games do not have the luxury (or confines) of a static frame. Similar to the basic challenges in virtual reality, there are games where a player can manipulate their camera’s viewpoint and focus on whatever they want within their space. These are also occasions where you can’t apply the traditional film edit. The player needs to be visually coerced and guided to move the camera where the designer wants them to go. Surely, 3D game designers have discovered rules of visual-audience manipulation that they can apply to virtual-reality narratives?
“We actually think we’re going to have to learn a lot from gaming, how narrative is conveyed in games, and possibly merge the two mediums [games and film],” Danfung said. “These new, immersive experiences may be inspired from both films and games, but fundamentally we’re building something that’s entirely new.”
When it comes to specific games that have inspired Danfung and may hold clues to transitioning into this new visual language, he gives a unique example, “I’ve been looking at Gone Home, as it’s an excellent example of exploring and discovering the story as you move through the spaces.”
Gone Home is a 2013 independent title created by The Fullbright Company, an independent game developer started by three former 2K Games developers. Players take the role of Kaitlin, a young woman who returns to an abandoned family mansion in Arbor Hill, Ore., in the summer of 1995. Through a first-person view, players explore the mansion and uncover small personal items that help unfold the narrative. Although the concept on paper doesn’t sound ground-breaking, the execution of drawing the player’s attention throughout the environment and enabling the story to blossom at an independent pace has been well received in many corners of the game industry.
“I think there’s a lot to learn from that game. If that was an [Oculus] Rift experience it would be highly compelling, as you’re deciding what spaces to go into and explore and the story slowly revealed,” Danfung said. “It would work really well in VR. It’s a great example of a nonlinear narrative, but one in which you’re piecing together the narrative as you explore.”
The magic story goggles of the future
Like film in the late 1800s, it feels as if we are staring at devices like the Oculus Rift from the viewpoint of a traveling side-show wonder. The virtual-reality goggles are being showcased as a curious piece of future technological magic, making the rounds of convention floors and back room press gatherings, showing off the 2014 equivalents of Arrival of a Train at La Ciotat. We think we can imagine what this new world of sensory input can provide, but it’s going to take a group of creative artists and thinkers experimenting the hard way, willing to take their crayons and draw outside the traditional lines, to find out.
“It’s going to take a new generation of storytellers.” Danfung said, “[creative individuals] who have backgrounds in film, in gaming, who will be able to bring all these traditional notions of story, of experience, and invent an entirely new form of communication.
“This is just the beginning. Zero Point is just the beta of our technology and a signpost of what is to come.”
What is on the horizon, according to Danfung, is not just a change in how we consume entertainment but perhaps how we will interact with the technological conveniences we use in our modern lives.
“I think there’s the potential for a fundamental shift in how we interact with our computer.” Danfung ponders, “Soon we may no longer be tied to a keyboard and mouse. We’re going to be interfacing directly with the content as an environment, in the same way we naturally interface with the world around us. The implications go far beyond gaming, entertainment or even communication, but may help us even understand the human experience.”
How close are we to this vision? “It will take many years before the technology is perfected, but when it happens, virtual reality will be indistinguishable from reality.” Danfung said. “I have no idea what that means for society, but the hope is that we will be able to share our personal consciousness and experienced to create a better understanding and empathy with one another.”