Tobii eye-tracking adds a new layer to game control (impressions)

By watching you, Tobii can improve your gaming experience.

The eye-tracking technology has been honed over the past decade to a point where it is now a viable method for controlling games. Building from experience in the marketing and user-experience fields and later in assistive technology, the Stockholm-based company has created a focused platform that will let it move into the game space, giving the company an avenue into PC gaming — a market that research firm Newzoo expects to hit $30.7 billion by 2017.

The company is off to a great start. The 2015 International CES marked its debut in gaming, kicking off with a partnership with peripheral maker SteelSeries for the release of the first consumer eye tracker, the $200 SteelSeries Sentry. And while the device has some level of support for most PC games, Tobii recently announced official support for Assassin’s Creed: Rogue in cooperation with Ubisoft Kiev, making it the first to embrace the technology.

Tobii showcased its technology at the 2015 Game Developers Conference in San Francisco last week, showing both the Sentry and its dev-focused cousin, the Tobii EyeX. GamesBeat met with products and marketing vice president Henrik Johansson there to see first-hand how eye-tracking can improve gaming control.

The technology

The eye-tracking device is a small, thin bar that’s similar to Sony’s PS4 PlayStation Camera in shape, though it’s strictly an infrared affair, connected to a computer via a USB3 cable. It’s light enough that a thin peel-and-stick magnet placed on the bottom ridge of a notebook screen or monitor can hold it up. Aside from a one-time calibration that has the user looking at a grid of dots, no other setup is required.

The eye-tracker’s three illuminators beam out IR light to the user’s face while the bar’s camera takes pictures of the lit face at a rate of 60 hertz. Johansson explains that the camera finds your face and eyes, and from there it can lock into where the pupils are. It tracks the pupils specifically — what they’re looking at becomes the target.

Above: Assassin’s Creed: Rogue supports eye tracking for camera control and other functions.

Image Credit: Ubisoft

How it works

Ubisoft put the tracking tech to a different use in Assassin’s Creed: Rogue in what they call an “infinite screen” capability. While playing, it only took me a few seconds to get used to moving Rogue’s camera with my eyes, not the right thumb stick or mouse. Scanning the horizon felt just as natural as looking, though the game camera moved slower than my eyes did. Looking toward the edge of the screen had it panning that direction, allowing the field of vision to move anywhere — no mouse required.

This capability to guide the camera with my eyes tied nicely to character movement. By holding forward and simply looking toward where I wanted the character to move, it followed. If my eyes caught on a point of interest, the character would naturally move toward it. And if at any time finer control was needed, either the mouse or controller stick was always available.

Johansson explained why three controllers running in parallel worked so well in the demo.

“When you move, instead of listening to the mouse, [the game] listens to the tracker instead, he explained. “This is nicely integrated in Assassin’s Creed because it has a very intelligent engine and it actually prioritizes the different controls. In this case, the key map is first, controller is second, and eye-tracking third. Using eye tracking doesn’t take anything away.”

Substituting eye-tracking for mouse movement means that the device is already compatible with just about any PC game. Johansson says that there’s not too much work involved in deeper implementation.

“For Assassin’s Creed, it took them two to three days to implement it in. And then it took about two weeks of tweaking and fine tuning,” he said. “There are some considerations, but nothing too heavy.”

Another game, PC adventure Son of Nor, uses eye-tracking in a way that made me feel like I have a superpower. Normally, in third-person games, players would need to move a character to an object, face it, and then press a button to pick it up. Son of Nor separates character movement and object interaction, with the latter tied to eye tracking.

This let me look at large boulders and click a mouse button to pick it up, and then look at a target and click again to throw it. Doing so while moving (with keyboard control) was no problem. I found that the tracking was so quick and accurate that I could throw a boulder and actually target it mid throw to grab it again. While it wasn’t the point of Son of Nor, this mechanic was so fun and worked so well that I could see it being used as the core of another title.

Eye-tracking: a boon for e-sports and streamers?

The infinite screen and eye-targeting are just the beginning for eye-tracking technology. Johansson says that it can be used in storytelling. For example, a nonplayer character could know when you were looking at it, changing behaviors as the system recognizes eye contact.

The Sentry shipped with two applications that can be of great use to professional gamers and streamers. The first is a training tool that uses eye-tracking overlays for recalled sessions.

“There are a lot of tools to recall your game,” says Johansson.” But what we recall as well is where you’re looking. You can go back and track, say, how many times you looked at the mini map, or how many times you looked at your item count. So you can go back and look at why you actually lost.”

For streamers, eye-tracking adds a third helpful layer that they can convey to an audience.

“What we can do with eye-tracking is, when you stream your game, we will plot the gaze point, so it work as a pointer for where you’re looking,” he said. “Now the audience for streamers can not only see and hear what they’re doing, but they can also actually see where they are looking.”

Above: An example of streaming overlay.

It works with virtual reality, too

The technology that Tobii has developed fits perfectly in the virtual reality space. Given that its cameras can track infrared light in darkness, the system could work within a set of goggles. The only challenge is fitting the IR emitters and cameras in such a small space. But Tobii is already at work on this, and it’s using second-generation technology that it has previously built for its research business.

Johansson showed GamesBeat prototype glasses that featured a small frame around each eye that houses IR emitters and tiny cameras. He says that this frame could easily be placed in existing VR headsets.

Speaking of the prototype’s miniature design, he said, “We did this all deliberately, keeping in mind that it would also go to the consumer market. You can take this and it can be implemented correctly in a VR headset. … We are working with VR partners for integration and we will release an SDK by the end of Q2”

Moving forward

My experience with Tobii’s tracking tech went off well. I came away impressed with how accurate and natural this additional layer of control felt between the two game experiences. It was nice to know that, while both hands and most fingers are were busy, I could still impact the game’s world through another means. It’s a powerful feeling, and that alone shows that device has potential.

Johansson says that it works that well for everyone. He noted that among the roughly 5,000 it has tracked between CES and GDC, only one person with exceptionally thick eye glasses had an issue, as reflections can create a challenge.

While it’s still too early to know how the SteelSeries Sentry is faring, Johansson is confident in the technology. He says that this platform has been in development for roughly two years, and that the company’s history since 2001, though the marketing and medical fields, has also been implemented.

While just about all games can use eye tracking in some fashion, Tobii would greatly benefit from more showpieces like Ubisoft’s Assassin’s Creed: Rogue. Johansson says that the company began distributing dev kits to developers around May of last year, and that he hopes more will implement features soon. Other games are on the way, he promises.