VR’s inevitable move to eye control: Why the industry will be taking a lesson from Iron Man

Imagine a world where you can control objects with your eyes instead of your hands. With virtual reality moving into the mainstream, this could become an everyday activity. Marvel has already shown us how it could work in its Iron Man movies: Tony Stark frequently interacts with his armor through eye-tracking and voice control, freeing him up to save the world.

So how soon until we get this kind of capability?

There are currently no “control scheme” standards in the VR world. In other words, every company is creating its own method for users to navigate and interact with virtual spaces. Handheld controls are currently the most prevalent option, although eye-tracking has emerged as an alternative more recently. At the moment, all three major high-end virtual reality headsets — Rift, Vive, and Project Morpheus — rely on traditional game controllers, wands, or a combination of the two. On the opposite end of the spectrum, VR startup Fove is working on eye-tracking.

None of the control tools currently on the market has emerged as the all-around “best.” Each has its own pros and cons.

The least immersive way to interact with a virtual reality experience is through a traditional game controller. Companies are using these types of controllers because consumers are already familiar with them, but also because one of the biggest applications for VR to date has been video games. But a traditional game controller seriously limits immersion by locking a user’s hands to a device.

Additionally, anyone who has used a traditional handheld controller in VR knows that it can create a disconnect between what you see and what you are doing with your hands. This pulls you out of the experience.

Wands are significantly more immersive and better suited to VR than traditional gaming controllers, as they allow users to move their arms and hands to interact with the virtual environment. It appears that the VR industry is moving in this direction. Still, wands can be clunky and unnatural.

For the time being, these types of controllers are the best we have, at least until technology reaches the point where controllers morph into usable gloves. But where the industry really needs to be headed in terms of the most lifelike immersive experience is some form of eye tracking. After all, isn’t this how we take in everything around us in real life? We identify what’s interesting through our eyes and then decide if it’s something we want to interact with.

Fove raised about $490,000 this summer to create a head-mounted display with eye-tracking capabilities. The two biggest advantages to eye-tracking as a navigation and interaction tool is that it removes unnatural head movement and frees up viewers’ hands.

Next-generation headsets will likely combine multiple navigation and interaction schemes that include a mix of traditional controllers, wands, eye-tracking, and voice control. But what might this look like?

Returning to the Iron Man metaphor, users could combine voice control and eye-tracking to remain in the virtual experience while interacting with menus. For example, users could use voice control to bring up a settings menu and then rely on eye-tracking to adjust volume and clarity.

This type of interaction and navigation scheme could also work in reverse. A user could focus on an area inside the virtual experience and then use voice control to interact with various in-experience elements. This would eliminate the need to navigate with your eyes and let you fully take in your surroundings.

In general, voice controls would greatly increase the immersiveness of a virtual experience. If users were able to toggle voice control on or off, they could use that option to adjust the virtual experience without pausing or stopping.

More likely than voice control, though, headset manufacturers will probably first combine eye-tracking with handheld controllers. This combination would work in much the same way that voice control and eye-tracking might function together: Eye-tracking would act as a cursor while the controller would function as a selection tool.

It’s far from certain which potential navigation and interaction schemes will dominate. However, as virtual reality becomes more widely adopted, there is a growing need to standardize navigation and interaction, as well as to find ways to increase immersion. While some manufacturers will continue to cling to traditional game controllers, the very best will include eye-tracking or voice-control or some combination of both.

Endri Tolka is cofounder and COO of YouVisit.