A look inside Microsoft’s HoloLens holographic headset

Nick Baker doesn’t get out much. The distinguished engineer at Microsoft helped create the hardware and silicon brains behind Web TV, Ultimate TV, the Xbox 360, and the Xbox One. But he got out Monday to describe the inner hardware of Microsoft’s newest toy, the HoloLens holographic computer headset — one of the most sophisticated new tech products.

Speaking at the Hot Chips semiconductor conference in Cupertino, California, Baker said that HoloLens is the spearhead for Microsoft’s vision for mixed reality, which combines virtual and physical worlds via a rich set of sensors into a digital experience. Such augmented reality experiences could generate a $120 billion business by 2020, according to tech adviser Digi-Capital.

“We don’t have Minecraft out yet, but we’ve got a game called Fragments,” where animated characters appear in your home, Baker said. “This tells you what we mean by mixed reality.”

Above: Fragments gives you a good idea of what mixed reality is about.

Image Credit: Microsoft

HoloLens has been available since March as a developer kit for $3,000, and Microsoft continues to work on bring the costs down for the headsets, which enable you to see 3D holograms on top of what’s in the real world. So you can see an animated version of a skeleton in the middle of a laboratory as part of an anatomy lesson.

Based on what Baker showed, HoloLens is a technological beast, filled with sensors, on-board processing, communications, custom nano-optics, spatial sound, and a high-definition microdisplay. It is an untethered experience, with no wires leading to a PC, and it has see-through lenses. The processing takes place in the headset, and a chunky battery wraps around your head as a counterbalance to the headset display. It handles gaze, gestures, and voice as input systems. The device has dozens of parts.

“With HoloLens, we knew had to innovate in all three areas at once,” with input, output, and form factor, Baker said. “HoloLens enables holographic computing natively. You don’t have to place any markers in the environment. No reliance on external cameras. Or other devices. No tethers. Don’t need a cell phone. Don’t need a laptop. All you need to interact with the full Windows 10 built-in computer is gaze, gestures and voice” inputs.

Above: Nick Baker, distinguished engineer at Microsoft, spoke about making HoloLens at Hot Chips 2016.

Image Credit: Dean Takahashi

“Because we have the sensing technology built in, you are free to move around your environment,” Baker said. “I can walk around my home, from one room to the next. Holograms come to life and integrate seamlessly with the objects around you. It’s mixed reality, a more natural way of interacting.”

Baker said that the holographic dog that he left there the other day is still on top of a piano, even when he comes back the next day.

“Mixed reality and augmented reality have come to mean so many different experiences,” Baker said.

A variety of Microsoft divisions had to come together as one company in order to make HoloLens happen, Baker said. The chip design team in Mountain View originally came from the acquisition of Web TV. It has been one of the company’s engineering jewels, because it enabled the software giant to move into hardware in the Xbox era. But in contrast to past projects, where they designed the Xbox 360 silicon, this time the Mountain View engineers were also responsible for the system design for HoloLens.

Above: An exploded view of Microsoft’s HoloLens holographic computer headset.

Image Credit: Dean Takahashi

Microsoft’s team had to work on its own custom optics, custom silicon, and custom hardware. To make it work, Microsoft had to go beyond a typical graphics processing unit (GPU) paired with a central processing unit (CPU). The device has a 14-nanometer Intel Cherry Trail system-on-a-chip CPU. The main logic board has 64GB of flash memory and 2GB RAM. There are stereo speakers, as well as WiFi and Bluetooth connectivity.

But it also has what Microsoft calls a custom Holographic Processing Unit (HPU). Microsoft’s own hardware engineering team in Mountain View had to design the HPU internally. The HPU chip is very sophisticated, with 65 million logic gates and 8MB SRAM, and it is built by the Taiwan Semiconductor Manufacturing Co. in a 28-nanometer manufacturing process.

Above: The visor subsystem for the Microsoft HoloLens holographic computer.

Image Credit: Dean Takahashi

Microsoft created various subsystems, including imaging optics and combiner optics. The latter combines the world with a virtual image. The company had to create a display, head-tracking, and gesture sensors. The display consists of two liquid crystal on silicon (LCoS), like what’s in projectors. If any one of these systems is too slow and the images don’t move as fast as your head does, then you’ll get sick. The front visor is there for protection.

The HoloLens sensor bar has a 3D time-of-flight custom depth camera (for scanning your environment), four environment-sensing cameras, an ambient light sensor, and a 2-megapixel high-definition camera. There’s also an inertial measurement unit, which is there to provide fast updates to the head-tracking information. If you move around your head quickly, it feeds information in single-digit milliseconds (a split second) to the display.

Above: The main logic board of Microsoft HoloLens.

Image Credit: Dean Takahashi

Microsoft worked on the technology for over five years, and Baker said there were around five major prototypes.

“It took us a few years to get to an industrial design that we thought we could ship,” he said. “It’s very important to keeping the world from swimming.”

The engineers had to take care that you would really be able to see a detailed hologram, so they made sure the imagery had as much resolution as possible. The whole assembly slides backward and forward to accommodate different head sizes and preferences. The design also needs to fit for those wearing glasses. There are no fans or heat sinks, so everything has to be energy-efficient.

Back in February, a HoloLens unit was sent to the International Space Station, where astronauts used them to make the first Skype call from space.

“As you can tell, work can be a blast sometimes,” Baker said.

Over time, Microsoft will take Windows Holographic and make it available as a platform for others to use. So other makers of VR headsets can make use of it.

As an industry, Baker said technologists from all companies can work cooperatively on improving the field-of-view of head-mounted displays. Most of the headsets can barely do 100 degrees, compared to the wide 180-degree field of view that humans have. Foveated rendering, where you only render sharply the images that your eye can see, can help. And there are other problems to be solved, like haptics.

“You can’t touch or feel holograms,” Baker said. “Those are a few thoughts on what we can all be working on together for the future.”

Above: Microsoft HoloLens lets you see virtual images in the real world.

Image Credit: Dean Takahashi