The guided missile technology that makes HoloLens work

Microsoft’s augmented reality development kit costs $3,000, and that’s because it shares some components that you’d normally find in spacecraft, guided missiles, and drones.

HoloLens, the head-mounted display that mixes digital imagery with the real world, starts shipping to developers March 30, and preorders are open right now. Of course, you’ll need to drain a few grand from your bank account to get one, and now specs reveal why that’s the case. HoloLens features a custom-built Microsoft “holographic processing unit” (HPU), four environment-understanding cameras, and an inertial measurement unit (IMU). This is some serious hardware that enables the wearable computer to track your location in 3D space without needing external sensors or a wired connection to a computer. That last bit, the IMU, is also one of the key pieces that helps the device keep track of where you are in the world. It’s a technology that humans developed to emulate a type of navigational intuition that you find in animals like geese and ants, and now it’s going to ensure that your virtual windows stay exactly where you put them in your office.

Microsoft is solving the problem of positional tracking with hardware and software. Right now, that means the HoloLens is one expensive beast at $3,000. But the company has said that its AR device’s path to consumers is going to take several years. With investment banks like Goldman Sachs predicting that the combined market for virtual and augmented reality will reach $110 billion by 2020, that means Microsoft has both time to figure this out and the motive to get the price down for everything that enables HoloLens to keep Conker, the star of a series of games on Xbox consoles, running around on this table:

Positional tracking is one of augmented reality’s and virtual reality’s biggest hurdles. GamesBeat recently spoke with developer relations boss Nizar Romdan at chip and sensor designer ARM to get an idea of what makes this so difficult. He explained that smartphones, like the Galaxy S7, don’t include big, expensive sensors that take up space and only perform functions for VR or AR. And even when they do, that extra data will put a huge strain on a battery that is only meant to hand around 3 watts to 10 watts.

But Microsoft is building HoloLens specifically for AR, and it is only selling the device to developers at this point. That means it can throw extra hardware at this problem. In this case, that’s an IMU that uses a form of navigation tracking called “dead reckoning” or “path integration.” This is a type of tracking that can deduce your current position based on your inertia and acceleration compared to your last known position.

With an IMU, you can turn on a device or a craft, and it will be able to generally tell where you move at any point after that. In this way, it works like an animal who knows where its home is and is building a path back to that base every time it moves.

Humans originally developed this technology for seagoing vessels, and it often works in coordination with global positioning systems. Today, the IMU is a central technology in all aircraft and spacecraft. They are the devices that enable the artificial intelligence of an unmanned, automated drone to always know where it is on the Earth. It is one of the chips that enables a missile to fly from one continent to another without drifting off course.

For HoloLens, the IMU will most likely not work in concert with a GPS — although that’s still possible. Instead, Microsoft has included “environmental-understanding cameras.” These will likely build a quick understanding of the environment like a next-generation version of the Kinect camera for Xbox One. HoloLens’s software can then take that data and combine it with the IMU to keep your position.

The best thing about this is that Microsoft’s demand for these IMU devices could drive down their cost and size, and we probably aren’t far away from seeing them in a Galaxy S8 or S9 that could work with the much less expensive Gear VR head-mounted display for mobile virtual reality.