An interview with a Microsoft HoloLens true believer

Michael Hoffman is a true believer when it comes to the potential of Microsoft’s HoloLens holographic glasses technology. So much so that he quit his job on the HoloLens team.

Hoffman was part of the Microsoft Research that created HoloLens, which just started shipping in developer kit form last week for $3,000. But he decided to leave the team last year to create Object Theory, a startup devoted to making applications based on HoloLens. Those apps are focused on enterprise applications, such as visualizing what a building would look like so that architects, engineers, and construction workers can communicate precisely on a project.

Hoffman’s startup is just getting off the ground, but he thinks that mixed reality, augmented reality, and HoloLens are going to change the world, and not just for games.

I sat down with Hoffman to talk about his company and the future of HoloLens. Here’s an edited transcript of our interview.

Above: Microsoft’s HoloLens Development Edition.

Image Credit: Microsoft

VentureBeat: What are you guys doing? You’ve become HoloLens experts?

Michael Hoffman: I was a principal lead on the actual HoloLens studio team at Microsoft. I was heavily involved with Trimble, Autodesk, the Fusion 360 proof of concept, the Autodesk Maya motorcycle concept. I don’t know if you’ve seen any of the videos, these use case videos. Then I was also involved in the NASA collaboration, the Curiosity mission planning.

I absolutely believe that HoloLens is the future. I absolutely believe that Microsoft has a great road map and is going to make a big difference in revolutionizing AR. I ended up leaving Microsoft to start Object Theory with Raven Zachary, in part because I lived in Oregon and was commuting regularly up to Microsoft, and also because I’m a startup guy. I’m comfortable starting companies and this felt like a big opportunity. It solved two different things — taking advantage of a great opportunity and being able to operate out of Oregon.

I left Microsoft end of last June. We’ve been building the company ever since. We hired Caleb Cannon, another engineer, and a creative director just a few months after that, the October time frame. Right around that time we were already starting to engage with our first client, CDM Smith. We announced today that we have this partnership developing HoloLens solutions for them.

VB: Who are they?

Hoffman: CDM Smith is one of leading consulting firms in architecture, engineering, and construction. A lot of people in the industry know it as AEC. It’s the epicenter of where 3D matters, because it’s all about 3D design. It’s designing big infrastructure projects. CDM Smith designs wastewater treatment plants, dams, traffic interchanges. They do a lot of oil and gas, other energy sector consulting.

We’re focused on helping them solve some of their collaboration challenges. They have an engineering design team that designs these complex models for building something as massive as a water treatment plant, which, you can imagine, has many millions of moving parts. Then the construction crew starts building it.

Above: Object Theory’s work for CDM Smith shows collaboration with HoloLens.

Image Credit: Object Theory

The collaboration between the engineers and the construction crew can often be a challenge. It’s very error-prone. You do things by phone, whatever it takes to solve the problem. The engineers are rarely on site. They’re almost always at one of CDM Smith’s offices, which are geographically dispersed. You have a dispersed team all trying to solve problems. Their main mission is reducing costs, reducing errors, making better decisions.

We’re at the epicenter of trying to help them figure that out. We’re focused on what it means to take design models and bring them into HoloLens and have multiple people all across the globe engage in a conversation around that model to solve a real problem. Recently they’d ordered a piece of equipment, and what was delivered was bigger than they expected. It didn’t fit in the allotted space.

They had to make a decision. Do you reroute pipes? Do you rip out the old platform that houses it? Do you move a wall? They make those kinds of decisions all the time — last-minute accommodations. It requires those engineers to be part of the conversation. One big thing we’re showcasing here is the ability to use remote collaboration in HoloLens with avatars that let you have shared conversations around a single point of interest.

We’ve had a lot of interesting realizations. With HoloLens you know where someone is looking. Not necessarily where the eyes are looking, but where the head’s pointing. You have this precise hands-free way of pointing at things. It’s great for remote because we may not be able to communicate where your eyes are looking across a phone call, but we can pass across where in a three-dimensional model, where exactly you’re looking at. We take advantage of that.

We started loading these complex models into HoloLens. One of them was a 60-acre construction site. It started off as billions of data points from a laser scan. Those billions of points were reduced to a mesh of that 60 acres and we would just show the 60-acre mesh with fully detailed texture of what it looks like. One of the guys on that site at the time to take the scan, he saw his own safety vest in the back of his pickup in the model. That gives you an idea of the detail.

We’d have conversations where he’d look at that thing in the pickup and I’d know he was looking at the pickup. He could show me the safety vest. Within a second we’re both sharing a conversation with high accuracy and fidelity. It was an eye-opener. HoloLens is amazing for collaborating around anything that’s fundamentally a three-dimensional problem. That industry has lots of three-dimensional problems.

Above: Object Theory shows how you can collaborate on a building design in HoloLens.

Image Credit: Object Theory

VB: This could go to different levels. There are more things you could do, it seems like. If the construction engineer on site had one of these on his head, he could be looking in a certain direction and you could overlay what’s supposed to be there. It gives you the right position, where it’s supposed to fit.

Hoffman: Absolutely. There’s always a challenge between “as designed” and “as constructed.” It’s a dance between the original design intent and the actual construction. To be able to put the original design models on top of the construction and see the deltas, the things that are right or wrong — that’s very powerful. You can immediately say, “Wait a minute, that pipe will be in the way of this other pipe.” It’s a powerful way to make quick assessments that are much more efficient and accurate than has ever been possible before.

VB: Is there also a difference between doing this in AR versus doing it in VR?

Hoffman: VR is good for some phases of this, but not many. One thing about VR, and I’m sure you already know this, is that with VR you’re not able to do anything without worrying about falling or running into something. You’re stuck with a completely immersive experience. Very few of those are good at helping you navigate your environment in a meaningful way.

For the initial understanding of a space in a model where you don’t necessarily need to navigate it effectively — The difference with navigation in VR is that you don’t get a sense of space quite as intuitively as if you’re literally in the space walking around. That has its limits, because the space you can walk around has to be big enough that you can get around your model, which is a challenge if you’re building something as big as a water treatment plant. We would end up having some of the same challenges.

But in AR, there are two big differences. One, we can overlay in the real world. The other is that you can have a conversation with a person. I can look at a model in a fishbowl, so to speak, the war room scenario planning type of thing. I can look at you and have a conversation. We can both point at a nonexistent object in front of it. Everything is aligned accurately enough that we’ll be seeing the same thing to such fidelity that we’ll point at the same thing. That lets you have these conversations around something that doesn’t physically exist.

You can’t do that in VR. You and I could not both have Oculus headsets on and have a conversation about a shared model. It’ll get there. To be honest, eventually all of this will converge. But right now AR, or what Microsoft more often refers to as mixed reality, to differentiate from all these other versions of the concept — with mixed reality, collaboration is one of the biggest opportunity spaces for HoloLens.

VR is a subset of AR. Anything you can do in VR, you can do with an AR headset. But you can’t do everything an AR headset can do with a VR headset. HoloLens is actually a completely applicable VR device. We use VR all the time in our scenarios for walking through an environment.

Above: Microsoft HoloLens

Image Credit: Dean Takahashi

VB: Where does this go from here? Does your client productize it for architects and construction people?

Hoffman: CDM Smith does everything from original conception of a project all the way through construction. They’re starting to get into operating the facilities they build as well. They see opportunities. We’ve helped them identify opportunities across the gamut of what they do. But we’ve decided with them that the one area of focus is this emerging field called virtual design construction.

VDC is an industry term that’s coming to mean, “Don’t have such a big handoff between design and construction. Have it be more seamless and collaborative than it’s been before.” But there are missing pieces for making it seamless and collaborative. Together we’re focused on how to solve some of the challenges in making it seamless, that handoff between design and construction.

That’s the initial focus, but we’ve identified other areas that are also valuable. In the future, there’s going to be a lot of IoT (Internet of Things) in these facilities. There’s already some, but IoT is going to completely take over things like water treatment plants, power plants, you name it. Even highways will eventually have IoT components embedded along the way for things like detecting damage and planning maintenance.

All that telemetry feeds heuristics that can then feed back into the experience of identifying problems and training people. If you need to fix a particular valve, you can lead that person to the valve, give them the complete history of the valve, and then provide a three-dimensional walk-through of how to fix it. The HoloLens could literally say, “Turn this screw. Dial this value.”

VB: What form does your software have now? Is it available for dev kit users to start looking at?

Hoffman: We’re focused almost entirely on custom engagements. We’re building a huge arsenal of foundational components — for avatars, for annotating things in 3D space with voice and photos and text. That tool chest we’re only making available right now to our clients.

During the initial phases of engaging with us they don’t need dev devices, because we have devices. That’s not a requirement. They’ll obviously want to acquire their own eventually, but during the phase where we’re designing the software solution for them, they don’t need to have engineers or devices necessarily. We’re more than happy to augment a customer’s team if they want to build expertise internally, but our primary focus is to be their HoloLens solution architect.

We’re in a position to take on clients. We’re building some real solutions. That said, we want to be good community citizens. We plan on open-sourcing some parts of what we have so everyone can benefit. We believe the pie will be big enough that we can afford to contribute to the larger community and still be a viable company.

VB: What are some other scenarios you’re designing for across the board?

Hoffman: Some of the categories we see repeating patterns in — in medicine there are two very clear areas that keep coming up. One is in the operating theater. A surgeon wants to be able to see all the scans and relevant data on the patient without having to constantly look up at monitors. Instead, while you’re operating, around you all the information is readily accessible and it’s in 3D. The human body is a 3D challenge. You can see the scans in 3D, rotate the scans in 3D, and have a lot more information. The amount of additional intuiting you can do is ridiculous. The other medical scenario is training. It’s very much like the Case Western scenario that was showcased at the keynote.

I can’t wait until everybody experiences a three-dimensional problem in 3D, so they can see how much easier it is to understand. It has applications for everyone from surgeons to baristas to car mechanics. Anybody who has work to do, who has information they have to look at.

Above: Avatars for Microsoft HoloLens project.

Image Credit: Object Theory

Another scenario is retail space planning. Retail spaces change all the time and someone has to decide what works. You can use it for user research. You can use it for just the initial design. It’s hard, in a physical space, to know if a layout is too constraining to walk through. In retail spaces you think about things like that. What first grabs your attention walking into a space? They can use HoloLens to experience all of that before they ever build a mockup or an actual store. They sometimes do life-size mockups before they ever build a store. We allow them to do that experimentation much more efficiently.

There’s premium assisted selling, like the Volvo scenario. You want to design a custom Nike shoe at a flagship Nike store. You can just design it in 3D. Again, it’s fundamentally a 3D problem. You can imagine, in the car scenario, seeing different treatments of all the choices you’ve made. How does the interior of the car look with wood paneling versus carbon fiber versus leather? You can try all those options without having every model of the car available in the showroom.

Related to that is assisted manufacturing. One of the biggest movements in industry is “everything is different.” Every car is different. Every plane is different. We’re moving to a world where you make one of each thing. You don’t make 100 or 1,000 or 1,000,000. Anywhere you make very few of the exact same thing, it’s hard to keep your manufacturing personnel trained.

One of the big opportunity spaces is using HoloLens to walk through all the right steps in the manufacturing process so you don’t make mistakes. That way you can use less skilled labor to accomplish the same outcome. There’s an efficiency play. Boeing did a substantial research project to prove that augmented reality increases efficiency and reduces errors in manufacturing. I’ve been trying to post everything relevant that I can find on my LinkedIn, and that’s one I uncovered recently.

VB: Do you do any work in games as well, or are you leaving that to others?

Hoffman: As Object Theory, we’re staying away from games. The closest we’ll come to games is the larger entertainment industry. A couple of sweet spots we’re seeing — I don’t know if you’ve heard of The Void, but it’s this concept of merging virtual reality with the physical world and have a sort of hybrid experience. The theme parks are thinking about this. How can a theme park create an interesting AR experience? Maybe it’s some combination of animatronics in the physical world, plus virtual reality.

We’re starting to see some interest in those entertainment scenarios. The other possibility is premium VIP experiences for, say, a showcase movie. You’d have the premium version of a movie that HoloLens augments in some way. Or a concert’s playing, but you can experience that concert in 3D as if you’re there, even if you can’t be there.

We’re seeing lots of those kinds of things. They’re not as interesting to us as the founders of this company. We’re more interested in solving the hardcore industrial challenges. But there are some interesting projects that we would probably do if they ended up in our lap. We’d be excited about that.

Above: HoloLens demo from Case Western University.

Image Credit: Dean Takahashi/VentureBeat

VB: Is what you’re doing easier as a startup than it would be inside Microsoft?

Hoffman: Not necessarily. I love startups. I’m a startup guy. But the HoloLens team at Microsoft rocks. It’s the most amazing place I’ve ever worked, and I’ve been a lot of places. I’ve been an early player in a lot of companies. It’s the most high-functioning team I’ve worked on. It feels like an extended family. Everywhere I go I’m hugging all these people I worked with at Microsoft because they’re all awesome. They’re all human. They’re all team players. We all constantly looked out for each other and helped each other. It was magical.

It was really hard to leave that job. I was so happy. But the commute was killing me. It was unfortunate that it didn’t work out. That said, I’m a startup guy. I think this is the future. I’m the kind of guy that’s willing to take a big bet. I absolutely believe that as a startup we’ll be helping invent the future.

VB: How many people do you have?

Hoffman: We’re only five full-time permanent right now, but we use a lot of outside contractors for art and sound and video and storyboarding. We do a lot of outsourcing. But we’ve been interviewing like maniacs. We’re ready to grow. We’re pretty sure that after this Build, now that we’re out in the open about having devices and being a Microsoft partner, the floodgates are going to open. We want to be ready for that.

We want to have at least two to three studios that can handle two to three simultaneous projects. There’s some efficiency in scale because you can have specialists. One of the hardest hires is a technical artist. They’re hard to find and they’re relatively expensive. The expense isn’t the issue as much as they’re just hard to find. You don’t need a technical artist per studio, though. You need one for two or three studios. To have one on staff you actually need more than one studio.

We’re looking at ramping up to two to three studios, maybe three to four, that can each take on projects and work relatively autonomously, but share specialty capabilities. A sound engineer is the other big one. You can’t afford a sound engineer if you just have a single studio.

VB: Have you already raised money, or do you need to raise money?

Hoffman: Right now Raven and I are funding the company through putting money in and working for free. I’m a platform guy. My entire career I’ve built platform solutions for business. We’re both always looking for that platform play. If we think there’s a certain opportunity to bring the world’s best collaboration solution for HoloLens to market as a product, we would do that. That’s when we’d be looking for venture capital.

Right now we’re not that attractive because we’re mainly a cost plus service business. We’re like a digital agency in how we operate. We don’t actually need venture capital. But that could change very fast. Within a year we might be saying, “Okay, let’s run with this opportunity as fast as we can and we need money to do that.”