The DeanBeat: How Oculus’ Mike Abrash took my mind off Palmer Luckey

If anyone could take my mind off the missing-in-action Palmer Luckey at the Oculus Connect conference, it was Michael Abrash, chief scientist of Oculus, the maker of the virtual reality headset Oculus Rift. Even though he was as sick as a dog on stage, Abrash gave a brilliant talk that laid the future path for virtual reality and inspired developers to join in the VR crusade. That is exactly what Oculus needed to do after Luckey angered many developers.

A lot of people were upset with Luckey after a report from news site The Daily Beast that revealed he spent $10,000 to fund a pro-Trump group. Nimble America wants to influence the election by running anti-Hillary Clinton internet memes and images on billboards to reach people in swing states like Pennsylvania. With the election coming up, Luckey wisely chose to stay off stage. Max Cohen, vice president of mobile at Oculus, told me, “He didn’t want to be a distraction.”

Abrash did his best to return the focus to the wonky subject of the technologies that will move VR forward. Much of what Abrash had to say went far over my head, and I really had to go to the bathroom after a keynote session that had dragged on for more than two hours. I really didn’t want to listen anymore. But he captured my imagination when he told a story about a conversation he had five years ago with engineer Atman Binstock, over why to get involved in remaking reality, which would happen anyway if he worked on it or not.

He told Binstock there was a “myth of technological inevitability, this idea that because technologies are possible, they will just happen naturally.” Rather, Abrash said, as recounted by Binstock, “Instead, the way technological revolutions actually happen involves smart people working hard on the right problems at the right time. And if I wanted a revolution, and I thought I was capable of contributing, I should be actively pushing it forward.”

Above: Mike Abrash, chief scientist at Oculus, showing us social VR.

Image Credit: Dean Takahashi

Binstock joined Oculus five years ago because of that conversation.

“Everyone in this room has jumped in to make VR happen, and our reward is we are on the leading edge of one of the most important technological revolutions of our lifetime,” Abrash said. “Thanks to all of our efforts, VR is going to leap ahead in the next five years.”

It was a fine ending to a remarkable session where Oculus really made a big impression. Mark Zuckerberg, CEO of Facebook, which bought Oculus for $2 billion in 2014, was on stage earlier to say that VR would become the next big computing platform. He said that Facebook had invested $250 million in VR content, and it would invest another $250 million.

On top of that, he showed off what social virtual reality would be like, with lip-syncing avatars, parties, and virtual rooms. He shared a vision that one day we would wear common eyeglasses with augmented reality and virtual reality built into them. And one of the next big products would be Santa Cruz, the code name for a tetherless, standalone virtual reality headset.

This standalone device would have its own sensors and computing power and display. And it would fit in between mobile VR and PC-based VR, as something we could use while on the run. All of this, Zuckerberg, Brendan Iribe, and Abrash said, was just the beginning of a revolution that the audience could make a reality. I wore this headset, and it actually worked, as there were no wires for me to trip over.

“The reason we are all working on VR now is because of our vision of what VR will become,” Abrash said.

Above: The future technologies that will influence virtual reality.

Image Credit: Dean Takahashi

Abrash also drew me in with his bold predictions about what VR would be like in five years. Rather than be vague, he was mathematically precise about what could be accomplished over the next five years in seven key underlying platform technologies: optics, graphics, eye tracking, audio, interaction, ergonomics, and computer vision. Oculus Research is working on challenges in these areas, and there’s no guarantee they will be solved, Abrash said.

“Frankly, talking about this in public, wasn’t an easy decision to make,” he said. “However, all of you are working on VR right now, at the very beginning, out of faith that it will be incredibly cool and important. I think you deserve a glimpse of just how great the future you are working toward is. VR five years from now will make today’s VR look like something out of prehistory.”

Abrash said we need display with eight times the pixel density of today’s display, three times the field of view, and variable focus. He thinks resolution will be at 4K by 4K per eye in five years. The wider field of view of about 140 degrees will make VR much more immersive and realistic. That requires a breakthrough in optics as current lenses can’t do this without distortion, he said. You need to be able to focus on near and far objects to be able to stay in VR for longer periods.

Graphics is another category that needs to advance. Abrash looks for some relief in costs and rendering requirements in foveated rendering, something that Nvidia recently talked about. This technique focuses on rendering the things that your eye sees at a given moment, and it leaves blurry the things that are on the periphery of your view. It can reduce the amount of processing multiple times, potentially lowering performance and cost requirements. That will help VR reach lower price points and more mainstream audiences. Abrash said that foveated rendering could reduce the number of pixels rendered by an order of magnitude. It has a lot of challenges that have to be overcome, possibly leading to a redesign of graphics pipelines.

Above: Virtual avatars in VR.

Image Credit: Dean Takahashi

But eye tracking has to work perfectly for foveated rendering to work. This challenge is the greatest risk factor, as the pupils of our eyes vary wildly in shape and size. That makes them very hard to track. Abrash thinks it will get solved.

Better audio is another challenge, but Abrash thinks we’ll be able to do surround sound in our headsets that reflects how sound bounces around us before we hear it. If you can hear someone ‘s footsteps coming up behind you, it’s a much more immersive experience.

Interacting with the virtual world is also important. Oculus Touch controllers could be the “mouse of VR,” Abrash said. The replacement for them could be using your hands as direct physical manipulators, as you do in the real world, and tracking that movement precisely. It involves haptics and kinematic technology that isn’t on the horizon yet, but Abrash said avatars will eventually reflect your exact hand movements.

“My prediction is in five years we’ll see good avatar hand tracking and gesture-based simple interface control, but touch-like controllers will still” rule, he said.

Ergonomically, we won’t be able to walk into a place like the Star Trek Holodeck anytime soon, but there are ways to make VR more comfortable so people stay in it longer. Eliminating the tether will certainly enable you to move about freely in VR, he said. But there’s no real existing consumer electronics link that is up to the task of delivering enough wireless bandwidth to a VR headset as will be needed in the future, especially not at 4K by 4K per eye resolutions.

The last major area of invention needed is computer vision, as we’ll be playing in mixed reality environments where recognizing objects should be easy and quick. Abrash calls that mixed reality “augmented VR.” To do this mixed reality, we’ll need reconstructing the real world in a virtual way, and creating virtual humans. Neither is easy. Scanning a scene in real-time with a consumer device is hard, and so is making a truly lifelike human.

“Augmented VR is so important that I’m confident the barriers will be overcome,” Abrash said.

Above: Mike Abrash, chief scientist at Oculus.

Image Credit: Dean Takahashi

As for virtual humans, Abrash said the development of virtual humans will be the single most important and single most challenging task of the future. Capturing the subtle movements of fingers, hands, facial expressions, and even eye movements is really difficult.

In other words, solving the problems of VR is hard. It will attract the best and the brightest, and that’s why it will get done. When those problems are solved, we’ll enjoy things like virtual work spaces, holograms, and social VR. We won’t need monitors anymore. And we’ll see huge gains in gaming, education, entertainment, medicine, and more.

In the end, Abrash expressed a technological optimism that the tough problems would get solved because the audience would take up the call and help solve them all. What Abrash laid out for the audience was a modern-day equivalent of the Manhattan Project, the effort the build the atomic bomb. There are fundamental breakthroughs required to make VR into the next big computing platform. And I’m impressed at how much one company is doing to get there, and I have to remind myself that Oculus is just one of many companies working on these problems.

But, as I take Abrash’s own words to heart, it isn’t inevitable that this vision of VR in five years will happen. It depends on that one smart person out there who makes a breakthrough in one or more of these fields, just as Luckey was that one person who a few years ago believed enough in VR to give the dead 30-year-old technology a second life.