The MindLeap virtual reality game system runs on your brainwaves — no controllers needed

Click here for all of GamesBeat’s 2015 Game Developers Conference coverage.

Virtual reality gaming goes hands-free.

Neurotechnology company MindMaze announces a thought-powered virtual reality game system called MindLeap today. Based on its proprietary system originally developed for use in the medical field, MindLeap detects brain and muscle activity, using mind power along with motion-capture cameras for gameplay in both virtual and augmented reality.

The Switzerland-based company rides the towering VR wave led by Facebook’s Oculus with today’s accompanying announcement that it closed an $8.5 million angel-funding round, which it will use to build on its efforts in the medical field, where its technology accelerates recovery in patients with neurological deficits. It plans to bring this brain-powered virtual reality control to gamers for the first time with MindLeap.

MindMaze says that this system introduces a new way for players to experience the games they already know. The head-mounted display that contains the “NeuroGoggles” and 3D motion cameras should be available in the coming months. This controller-free, brain-powered, latency-free VR will be available on multiple game platforms, including the Xbox and PlayStation (though MindLeap wouldn’t say if it meant the PlayStation 4 and Xbox One or older systems), iOS, and Android.

The engine behind MindLeap will be available to game developers via an SDK in the coming months, says MindMaze, with the system itself arriving late this year.

GamesBeat met with MindMaze during the 2015 Game Developers Conference to see how neurology fits into virtual reality.

From medicine to games

MindMaze founder and CEO Dr. Tej Tadi says that the human brain has the jump on any given movement. The decision to, say, move an arm comes milliseconds before the action takes place. Being able to track brain and muscle activity lets the company tune into the signals that would go out to an arm to make it move. The neurobiologist explained that motions, movement, everything you see and touch, and more are all integrated in one place in the brain, and MindMaze tracks signals originating from there.

It seems that the timing of the signals from this place in the brain is a key part of MindMaze’s technology. Tadi says that we know that sense of touch takes 20 milliseconds to relay to the brain, or that vision comes in at 70 milliseconds. Knowing these set timings “enables this experience perfectly,” he says.

Initially, MindMaze developed the technology for the rehabilitation purposes. Tadi shared a demonstration of a patient that had just lost his left arm and suffered from phantom pain. By tracking the brain signals that would have that missing limb move, and by displaying a corresponding visual representation of the moving missing limb, patients would be relieved of their phantom pains. But this only works in a latency-free experience — one that comes in at 20 milliseconds or less. MindMaze’s work on this front eventually led it to gaming.

The idea behind MindLeap is that it would combine this brain wave-tracking element with existing virtual reality and augmented reality technology to create a gaming platform with a new dimension of control.

“Now we can get a mental health factor into it,” Tadi told GamesBeat. “You’re training the brain, you’re monitoring the brain at the same time. And that can add a whole new element to games. Content creators now have a whole plethora of things to do — it’s incredible.”

Above: MindLeap’s motion sensing camera.

Image Credit: MindMaze

The MindLeap platform

MindMaze built the gaming platform from scratch — the motion capture systems, the brain-reading EG systems, the integration platforms, and everything else. Just as with any VR platform, a headset serves as the centerpiece. Its face features motion capture cameras and depth sensors that provide for a mix of virtual reality and augmented reality. The prototype shown uses a skullcap laced with sensors, but Tadi says that the final version will be closer to a visor headband.

The platform also features a wireless platform that sends back 3D motion capture data to complement the headset’s tracking functions. The unit features head tracking, a full 3D point cloud for body tracking, and skeleton extraction to track body movements . Tadi explains that it’s somewhat like Microsoft’s Kinect, though they had to make some adjustments for the original medical use.

“We had to build it from scratch. You’ve seen the Kinect, right? If you put it in front of a patient, it’s not going to work from 2 meters away, or not track if you’re sitting in a bed,” he said. “So we had to do this ourselves.”

The eventual goal is to work the prototype into a smaller format that will work with mobile devices.

“This is going to be a wireless camera sitting on a table streaming 3D motion capture data to your mobile phone. So you could actually get motion capture on your tablet device,” Tadi said.  “We made this so we might fit this on the back of a phone. That’s where we’re going — that’s what’s coming next.”

Within the next six months, the SDK for MindLeap launches. It includes software, the goggles, and the camera unit. The company hasn’t disclosed the price yet.

Above: MindLeap’s brain tracking technology used in a multiplayer demo.

Image Credit: MindMaze

Brain power in use

A brain-controlled demonstration had me playing against an opponent, both of us using alternating states of calm and tension to either defend an orb base or attack the opponent’s. We both wore headbands equipped with forehead sensors that let the software read our mental state. Relaxing the mind increased a force field around the orb to protect it. Stress had a reverse tug-of-war style attack pushing closer toward the opponent’s orb. This demonstration did not feature virtual reality elements — the demo isn’t ready for prime time yet —  but it did serve as a good example of how mind states could be made into game elements.

It was interesting to be fully engaged in a gaming experience without the need for controllers, let alone movement. Aside from the squinted eyes and forced frowns I ended up doing to transition from relaxed to tense states, I sat perfectly still while playing — a first for me. This sort of sending an action to a game came so easy and worked so well that this simple experience was considerably more entertaining than it would have been as a controller-based game.

A hands-off demonstration of a prototype of the VR rig had one of MindMaze’s staff members holding out his hand in front of his face. The headset’s front cameras sent an HD image to a screen showing his hand and the room around it, with augmented reality fire coming from his fingertips. Brain sensors on the prototype’s headband let the user change his mental state to change the color of the fire from red to blue. Turning around in place, the environment on the screen transitioned from augmented reality to virtual reality. The user’s hand was still there, but it was represented graphically in a futuristic space. Changing orientation let the user change between the AR and VR environments freely.

The future of this futuristic tech

MindMaze wants to get MindLeap out and into the open so that developers can take over with the hopes that t the addition of mind control to virtual reality will open up a new spectrum. Tadi feels that this was the eventual next step.

“If you forecast the way the goggle manufacturers are going, you want finger-tracking, you want gesture recognition, and you want to see your body,” he said “And now there’s the whole neural layer where people who are interested in creating new experiences will say, ah, there’s something cool to do there.

“We had to make the jump to make it more game friendly. I think we’ve enabled that now. It’s more usuable and more fun, too.”

Tadi says that MindLeap would be the first time that virtual reality, augmented reality, 3D motion capture, and neuroscience will come together for gaming purposes.

“You see the capability right now. The next thing we want to do is partner with content creators and game devs. We’ll offer them a kit so they can start building stuff,” he explained. “What we want to do is make it open source. What we want to do is make this available cross-platform and wireless. So people can do a new way of gaming that’s more accurate.”