Magic Leap’s Graeme Devine on how to design for mixed reality and the impossible

Graeme Devine has had a storied career in games, working on projects such as The 7th Guest at Trilobyte, Age of Empires 3 at Microsoft, and cool game technology Apple. He is now the chief creative officer at Magic Leap, which is creating “mixed reality” glasses that overlay animations onto the real world.

Magic Leap, which raised more than $542 million from investors led by Google, hasn’t fully described how it will create “cinematic reality” and other experiences with its glasses or when it will ship them. But it is one of the startups that has ignited a lot of interest from gamers and game developers about the frontiers of gaming. Magic Leap’s goal is to create animations that are indistinguishable from real life and to insert game characters and gameplay into our everyday lives.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

Devine didn’t think the tech was possible. But when he saw a demo of it in Florida at Magic Leap’s headquarters, he was hooked. Other leading thinkers like sci-fi novelist Neal Stephenson have also joined the company.

I sat down with him at GamesBeat 2015 for a fireside chat. Here’s an edited transcript of our talk.

GamesBeat: For people who may not know, tell us a bit about your career in games.

Graeme Devine: Oh, golly. You all know me, right? I’ve been making games since the 1970s. My first game was on the TRS-80. If you remember the resolution of the TRS-80, you remember it was black and white and didn’t have a lot of pixels.

I very quickly moved on to work at Atari for a little while. Worked for Lucasfilm in the ‘80s. I moved to the states and started a company called . Made a game called The 7th Guest, 11th Hour. Helped kickstart the whole CD-ROM industry. You can debate whether it’s Myst or 7th Guest that was the best. It’s obvious, but I’m just saying. Moved on to id Software as the lead designer on Quake III, which kicked off internet gameplay a little bit. Did well there. Went from there on to Microsoft, where I was the lead designer and lead writer on Halo Wars, the best of the Halo series. I’m gonna get killed for that.

From there I moved on to Apple and helped games at Apple. I was the guy that was like, “If you’re putting a virtual D-pad in your game, you’re saying your game is better on the Nintendo DS. Do you want to say your game’s better on a Nintendo DS?” Encouraging people to use this marvelous touch device that had this incredible touch screen and make games for that. Helped launch the iPad with some very cool games and get people to think about touch as an interface.

After that I kind of retired, walking my dogs on the beach in the morning, having my own little company, doing the occasional talk and so forth. Then along came Magic Leap.

GamesBeat: You seem to like to be on the cutting edge. What was attractive about Magic Leap?

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

Devine: The fact that I thought it was impossible. Rony Abovitz, the CEO, called up and said, “I think I need a games guy.” “Okay? Where are you?” “Florida!” “Not interested.” Then he calls up again and has me sign the NDA. He sends me a video of the technology. I said, “That is impossible! You can’t do that! You can’t insert characters into environments with occlusion and things. I’ll come to Florida.”

GamesBeat: Can you describe this a bit for our audience? You aren’t using the term “augmented reality”?

Devine: Mixed reality. We think of mixed reality as the placement of objects in the real world as you interact with objects in the real world. We think of augmented reality as things that are just on top of the world. There was this mixed reality demo of robots actually running around the whole scene.

I went down to Florida and took a look. I stuck my head into this machine that was about the size of a refrigerator. Right away I’m thinking of that movie Brainstorm. Once I joined I had to buy everyone in the company that blu-ray, because apparently nobody’d seen it. There was a monster running around the desk in front of me. It was this blue monster, and I could control it with an Xbox controller. It was looking just like it was real, on the desk. I’m thinking, “That is weird!”

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

Then, in the back of the room, a much larger monster stood up and waved at me. I hadn’t realized it was there to begin with. I didn’t notice that monster. It was so naturally fit into the environment. When I went and focused on it, the monster on the table changed and went fuzzy. “Golly.” Well, I thought more than “Golly.” Mostly “Golly.” But this was insane. I’d never seen anything like it. I went from not moving to Florida to being in Florida and learning about making content for this, what it meant to make content in mixed reality.

GamesBeat: Magic Leap went on to raise more than half a billion dollars from Google and others. It seems like that completely changed the level of attention on you.

Devine: It’s still a startup. When I look at my window, the window in my office, I see rocket scientists. There’s the guy who wrote the Guardians of the Galaxy comic book. There’s optical people doing all of our optics. There’s Neal Stephenson. He’s hanging out. He’s a futurist. He’s talking to the optical people who do plasma weapons at something or other. I think, “I’m just a guy making games for this!” These people are buzzing. There is a visible buzz coming off of them. I just want to bask in that. That startup still exists. That’s the exciting thing about Magic Leap. It’s still that startup.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

GamesBeat: You were in a similar situation at Apple, where everyone wanted to know about what you were doing. You had to be fairly secretive. What’s it been like to have to communicate out of that bubble?

Devine: At first it’s very hard to communicate out of that bubble. But at Apple, you learn how to work around it. And at Magic Leap, I can’t understand a lot of it, so I can’t tell you about it. But I can tell you about what it’s like to make games in which reality—I can tell you about the language we use. I can tell you about what it is to make something exist in the world. I can tell you about the innovation that our thinking’s had because, golly, two years ago we thought it was the bee’s knees putting something on a table. And then two years later, “Oh, that’s easy. Let me tell you about the evolution now.” That’s been interesting.

GamesBeat: What are some pros and cons of mixed reality when it comes to games?

Devine: Mixed reality and game design is really hard. One of the biggest challenges is making a mixed reality game design, because you can’t just be a better console game. You can’t just be a virtual reality game.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

A true mixed reality game is something that takes the reality in front of you, takes this water bottle. It adds virtual content to it so the water bottle is involved in the game and there are characters doing something to that, so the tools can interact with each other. Or it can be playing cards or wooden blocks. It could be objects around your house. Then we can have something interesting that can’t exist anywhere else. You can’t have that game in VR. You can’t have that game on console. Only in mixed reality can that game exist. Then you’re on to something interesting.

GamesBeat: Does the technology have to be seamlessly perfect for this to work for the consumer?

Devine: The experience, for me, when I put it on, is so incredible. My memory is still, “Oh my God, this monster’s in the room with me.” And yet you know in your head that there’s no such thing as monsters. It is very compelling. It seems as if it is real.

GamesBeat: We have a war going on outside of Magic Leap now. Nintendo showed this video of Pokémon out in the real world, sitting on top of Google’s Ingress location-based gaming platform. It seemed very inspirational, to be able to join with a bunch of other people hunting down Pokémon in real places.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1822174,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"C"}']

Devine: If anyone can get me on the beta for that, I’d love to. That seems to be what everyone wants and expects, right? I want to see X-Wing fighters and Marvel superheroes and Superman and have them interact with the world around me. I think that demonstrated very well the fantastic dream of mixed reality.

GamesBeat: How would you see the difference between AR and VR?

Devine: VR takes you somewhere else. It transports me and puts me in some other place. AR presents information on top of the world. Mixed reality adds to the world, adds photons to the world.

Why would I want photons added to the world? What are the applications that I actually want to add on to the world? I think of a five-mile application. A five-mile application is, I’m five miles away from my house. I’ve forgotten my smartphone. Do I go back to my house and get my smartphone? Yeah, I’ll do that, because my smartphone makes me better at what I do. There’s games on it.

What if I’m five miles away from my house and I’ve forgotten my smartwatch? Do I go back and get it? Well, not right now. There’s no game I want to play. There’s no compelling application I need. I’m 50 feet away from my house, will I go back get it? Nope. So what is our game, our application that we have in mixed reality that makes you turn around after five miles? That is the key application we need to be thinking about. That’s why you wind up at the store for four days in the rain. We’re thinking a lot about that application, that experience is so compelling that you’ll turn around for it.

GamesBeat: Tim Sweeney had a very interesting comment about augmented reality. He said that when it’s perfected in the next 10 years, we won’t need displays anymore. No smartphones, no TVs. You’ll have your 40-foot screen right in front of your eyes.

Devine: We talk a lot about that, about a world without atoms. Unless you actually need those atoms, we can make photons work. That sets off a weird round of, well, what needs to be real in my world? My coffee, my coffee cup. Other things – displays, my work, the people in my environment, the experiences I have in the evening, the entertainment I watch, the sports I engage with, the content I go through – I don’t generally interact with the atoms of them. We’ll be changing how you perceive reality.

GamesBeat: Mixed reality versus VR — it almost seems like it’s easier to make games in VR. Maybe you’re more used to or familiar with it. Augmented reality seems like it’d be useful for a lot of non-game applications. Do you foresee that kind of world coming, where maybe AR is more useful for non-game purposes?

Devine: For a while it was very hard to think about what a game could actually be in augmented reality. We started with doing games on tabletops, placing things on tabletops and making the game there. That’s awesome, because now there are games I really like on tabletops. Dungeons & Dragons is cool and fantastic, all those things.

Then we moved on to putting games in your house. You start playing a game using wooden blocks. You spell things with letters on them and look at those words and learn how to actually interact with the world. Then things move in your house. You start to see lights and hear sounds going off in a room. Eventually you go and look in that room and see what’s there. In front of you is a ghost, a ghost standing in your house, in the middle of your room. It points directly at you and directly beyond you. You look behind yourself and there’s a body on the floor. Then you look back and the ghost is gone, but in your ear you hear, “Please help me.”

What you realize is, “I’m having a story in my house. What I’m going to be talking about at work tomorrow is all the haunting going on in my own environment. My gosh, I’m not going to be able to sleep at night. This thing might still be there.” That kind of experience is something that can’t be had in VR, because VR takes you somewhere else. We’re giving you experiences that happen inside environments you’re already very familiar with. That’s incredible. It’s not been done before. That kind of design and narrative is incredibly difficult. It’s the hardest thing I’ve ever worked on, but it is so awesome to work on that problem.

GamesBeat: Who has the hard job when it comes to making something convincing – the mixed-reality game designer or the virtual-reality game designer?

Devine: We both have very hard jobs. I’m already focused on my space. I’m focused on making things appear in rooms and making that compelling. I can’t wait to drive spaceships around in VR, but it’s not something I’m thinking about. I don’t think about that problem space.

GamesBeat: Have you figured out the part about how everyone has a different house?

Devine: Learning to see what is in your house, getting the idea of where spaces are at on the floor, where the walls are at, where objects are—The idea of the ghost is to help you with that. You’ll talk to artificial intelligence people and they’ll say, “We can work it all out. We can work out where everything is at in the house.” But I can also just have a game character ask, “Hey, where is a good place to put this game? Is that a floor? Is that the best wall to put the screen on?”

People love to interact with characters. They love to interact. We don’t need to know that much about the house because we can have someone virtually asking you there, “Hey, is this the right thing to do?”

GamesBeat: How much time for experimentation do we have here? There are different lines. CastAR is a different version of mixed reality. People are going to be experimenting with different platforms. How much time do we have for experimentation before companies start sending out commercial products?

Devine: We think a lot about three things that we do at Magic Leap. We have a team that just looks at Lego bricks, looks at small experiences. What’s music like? What’s it like to touch the world? What’s it like to go and do these small things in Lego bricks, so that we can learn what games are like in mixed reality? We think about that a lot. Those Lego bricks will be helpful to everyone.

Then there’s something we call the incubator lab. It’s in the final week of our internal process of asking everyone across the company, “What is the most awesome thing you could do in mixed reality? Tell us what it is and you’ll have a chance to go prototype it.” We’ve gathered 250 ideas from across the company. 22 of them we asked to go and make better presentations. Eight of them were put together with a team to go and prototype them. The incubator lab is something we work with across the whole company, because we have rocket scientists and Neal Stephensons.

Then we look at what is a large triple-A game in what we’re doing. We take a chance. We try to make something. It might be wrong. It might be bad. But we have a pipeline, moving from the art department to the animation department to programming and so on. What is it like to run on a device that moves around? What’s it like to place something on the table? We think about experimenting in that group quite a bit.

GamesBeat: I’m sure everyone in the audience is thinking this, so I’ll ask: What are you doing up until February, and when is your product coming?

Devine: We’re working day and night, I’ll tell you that. We work around the clock. I’ve never seen a harder-working set of people on the planet. All I can tell you is, it’s very exciting.