Watch Grateful Dead’s Mickey Hart play drums with Oculus Rift — and see how his brain reacts

SAN JOSE, Calif. — Grateful Dead drummer Mickey Hart gave an incredible performance of the upcoming brain-training rhythm game Neuro Drummer while wearing an Oculus Rift headset — and showed how virtual reality could be a part of future treatments for degenerative conditions such as Alzheimer’s disease.

The performance onstage at the Nvidia GPU Technology conference showed that Hart is a virtuoso percussionist. (Duh) But it also showed what was happening inside his brain in real time during the performance, and it conveyed exactly what he was seeing inside the virtual-reality headset. It is the kind of experience that researchers expect will help restore cognitive health to aging patients and that game companies hope will entice players to lose themselves in virtual worlds.

If every experience or game based on these technologies works as well as this demo, the gaming, entertainment, virtual reality, health, and music rhythm industries could very well draw in many more people into immersive virtual experiences.

The demo of was part of the keynote speech from Adam Gazzaley, the director of the Neuroscience Imaging Center at the University of California at San Francisco. He showed off how the Neuro Drummer game and real-time monitoring of Hart’s brainwaves could exploit the processing power of graphics processing units (GPUs) to create “neurogaming” experiences that haven’t been seen before.

Above: Adam Gazzaley of the University of California at San Francisco.

Image Credit: Dean Takahashi

Hart played the virtual drum set (see the videos for the full performance below) based on what he was seeing. His actions could make things happen in the virtual world of Neuro Drummer, which is like a 360-degree version of the once-popular music game Guitar Hero. He could wave his hands to make objects explode and hit the virtual drum to match the beat of the music. He used a Roland HandSonic HPD-20 Digital Hand Percussion Controller.

He played the game in “Master Level,” but the cognitive health-treatment version would be more basic. It riffs off the theory that music rhythms can help restore connective pathways in the brain that have degenerated in older people through aging or Alzheimer’s disease.

Gazzaley said that Neuro Drummer was built with the help of Mike Gonzales and Matt Whitman of the Studio Bee digital animation house in San Francisco, Hart’s team, and Oculus VR’s Laird Malamed, whose company is now under Facebook’s umbrella after a $2 billion deal earlier this week. The Glass Brain visualization tool, which showed what was happening inside Hart’s brain as he was playing, was built by a team of people from UC San Diego, Gazzlab & Friends at UCSF, Cognionics, BrainVision, and Nvidia.

In his keynote speech, Gazzaley said that a company he cofounded and is advising Akili Interactive Labs, which wants to build “the first therapeutic games for mobile” — new kinds of games with cognitive benefits. Gazzaley has been working on the problem of applying gaming to brain research for the last five years.

“We need to develop an effective approach to enhance cognition in both healthy and impaired people,” he said. “The goal is to improve quality of life. Many in medicine are working on extending our lifspan. We pay a price for that with neurodegenerative diseases. Our focus should be on living a better life.”

The Glass Brain part of the demo was important to Gazzaley because it creates a “closed loop.” The game produces a reaction, or change in behavior, in the brain. The Glass Brain software then records that in real time and isolates which parts of the brain are active. The game can then use that information and adapt so that it can be more effective. That can result in targeted, personalized, multimodal, and closed-loop treatment for brain patients.

As for games, he said, “There are a global phenomenon. They are more gender equal than they have ever been. Interactive. Fun. They are a powerful form of media, and we know they have a strong impact on behavior. They can have a positive impact.”

So far, the results of one game, NeuroRacer, a racer that challenges you to do multiple things at once, have been positive. After one month of training, patients who are in their 70s can show multitasking performance that rivals those of 23 year olds, who are at their peak in brain abilities.

Over the next decade, Gazzaley hopes to create games that can actually can receive federal approval to be prescribed as medical solutions for people. The company is already testing how some of its projects can help with depression, ADHD, dementia, and autism. Pfizer and Shire Pharmaceuticals have invested in Akili.

The Oculus Rift came into the picture because it can provide deeper engagement. Akili is also looking at using Microsoft’s Kinect motion-tracking camera as an input system for a brain game. He said he now needs to take the game and combine it with a feedback system, like Glass Brain.

“By 2019, we hope to use a game to affect parts of the brain to work on specific neural processes, using sensors,” he said. “It could help with traumatic brain injury, and would be much better than a blunt instrument such as a pill, which treats the whole brain.”

At UCSF, researchers are building a new Neuroscape Lab devoted to this research.

Hart said after his performance that his grandmother had Alzheimer’s. When he played the drums for her, she was able to recognize him.

“What we have on top of four bodies is a super-organism,” Hart said. “When I played the drums for my grandmother, she spoke my name. That’s medicine. I knew that music reconnected those broken pathways and brought her back, at least for the time I was drumming.”

Above: Neuro Drummer Mickey Hart

Image Credit: Dean Takahashi