Click here for all of GamesBeat’s 2015 Game Developers Conference coverage.
Virtual reality gaming goes hands-free.
Neurotechnology company MindMaze announces a thought-powered virtual reality game system called MindLeap today. Based on its proprietary system originally developed for use in the medical field, MindLeap detects brain and muscle activity, using mind power along with motion-capture cameras for gameplay in both virtual and augmented reality.
The Switzerland-based company rides the towering VR wave led by Facebook’s Oculus with today’s accompanying announcement that it closed an $8.5 million angel-funding round, which it will use to build on its efforts in the medical field, where its technology accelerates recovery in patients with neurological deficits. It plans to bring this brain-powered virtual reality control to gamers for the first time with MindLeap.
AI Weekly
The must-read newsletter for AI and Big Data industry written by Khari Johnson, Kyle Wiggers, and Seth Colaner.
Included with VentureBeat Insider and VentureBeat VIP memberships.
MindMaze says that this system introduces a new way for players to experience the games they already know. The head-mounted display that contains the “NeuroGoggles” and 3D motion cameras should be available in the coming months. This controller-free, brain-powered, latency-free VR will be available on multiple game platforms, including the Xbox and PlayStation (though MindLeap wouldn’t say if it meant the PlayStation 4 and Xbox One or older systems), iOS, and Android.
The engine behind MindLeap will be available to game developers via an SDK in the coming months, says MindMaze, with the system itself arriving late this year.
GamesBeat met with MindMaze during the 2015 Game Developers Conference to see how neurology fits into virtual reality.
From medicine to games
MindMaze founder and CEO Dr. Tej Tadi says that the human brain has the jump on any given movement. The decision to, say, move an arm comes milliseconds before the action takes place. Being able to track brain and muscle activity lets the company tune into the signals that would go out to an arm to make it move. The neurobiologist explained that motions, movement, everything you see and touch, and more are all integrated in one place in the brain, and MindMaze tracks signals originating from there.
It seems that the timing of the signals from this place in the brain is a key part of MindMaze’s technology. Tadi says that we know that sense of touch takes 20 milliseconds to relay to the brain, or that vision comes in at 70 milliseconds. Knowing these set timings “enables this experience perfectly,” he says.
Initially, MindMaze developed the technology for the rehabilitation purposes. Tadi shared a demonstration of a patient that had just lost his left arm and suffered from phantom pain. By tracking the brain signals that would have that missing limb move, and by displaying a corresponding visual representation of the moving missing limb, patients would be relieved of their phantom pains. But this only works in a latency-free experience — one that comes in at 20 milliseconds or less. MindMaze’s work on this front eventually led it to gaming.
The idea behind MindLeap is that it would combine this brain wave-tracking element with existing virtual reality and augmented reality technology to create a gaming platform with a new dimension of control.
“Now we can get a mental health factor into it,” Tadi told GamesBeat. “You’re training the brain, you’re monitoring the brain at the same time. And that can add a whole new element to games. Content creators now have a whole plethora of things to do — it’s incredible.”
Above: MindLeap’s motion sensing camera.
The MindLeap platform
MindMaze built the gaming platform from scratch — the motion capture systems, the brain-reading EG systems, the integration platforms, and everything else. Just as with any VR platform, a headset serves as the centerpiece. Its face features motion capture cameras and depth sensors that provide for a mix of virtual reality and augmented reality. The prototype shown uses a skullcap laced with sensors, but Tadi says that the final version will be closer to a visor headband.
The platform also features a wireless platform that sends back 3D motion capture data to complement the headset’s tracking functions. The unit features head tracking, a full 3D point cloud for body tracking, and skeleton extraction to track body movements . Tadi explains that it’s somewhat like Microsoft’s Kinect, though they had to make some adjustments for the original medical use.
“We had to build it from scratch. You’ve seen the Kinect, right? If you put it in front of a patient, it’s not going to work from 2 meters away, or not track if you’re sitting in a bed,” he said. “So we had to do this ourselves.”
The eventual goal is to work the prototype into a smaller format that will work with mobile devices.
“This is going to be a wireless camera sitting on a table streaming 3D motion capture data to your mobile phone. So you could actually get motion capture on your tablet device,” Tadi said. “We made this so we might fit this on the back of a phone. That’s where we’re going — that’s what’s coming next.”
Within the next six months, the SDK for MindLeap launches. It includes software, the goggles, and the camera unit. The company hasn’t disclosed the price yet.