The tantalizing promise of augmented reality games

Above: Microsoft’s HoloLens in action.

Image Credit: Microsoft

Question: I work for Ashley Furniture. We’re working on an augmented reality where you can scan a room and take products from our catalog and put it in that scene. What we’re struggling with is getting the CEO of the company to understand that I can’t have that perfect fidelity, because it has to happen in real time. Have you seen any constraints with regard to level of visual quality in certain projects? How do you deal with other stakeholders, getting them to understand?

Ellsworth: It’s interesting that we’re at CES, where everything is about specs and the tangible aspects of everything. One of the lessons I learned working in the toy industry — this was actually beat into me by executives — is that when it’s fun, it’s fun. I’d come in with this super complicated piece of electronics — hey, I just made this better! — but it was fun before when it was half the price and easier to manufacture.

It’s going to be an evolution, I think, for AR gaming and AR applications. It has to function seamlessly enough that the end user will accept it. Some of the struggles with AR now on phones and tablets — you have to download an app and go through all this friction to make it happen. Once some of that gets smoothed out and it’s just persistent all the time in your OS or your glasses, then we’ll start to transition into a phase where it’s a specs war. Glasses manufacturers will push the boundaries of optics. Tracking developers will push the boundaries of machine learning.

Beliaeff: AR right now, because it’s not popularized out there yet — it’s so experiential. A lot of the communication issues you have revolve around how you quantify something that’s experiential. Taking a screenshot of AR doesn’t do you a lot of good. An example of something that’s working out right now in VR is the Samsung commercials, the Samsung VR that was running every second over Christmas. You get the reactions, the oohs and aahs. It does a good job of communicating that you have to try it. For your CEO, it’s experiential. You have to get them to try it. On the graphics standpoint, you have to start thinking about how you preload and cache stuff. Getting graphics up on a webpage quick is a challenge.

GB: Is it a tradeoff against what you’d call latency, maybe? I already hate how long it takes for things to happen in Pokemon Go sometimes. I’m still playing every day, but I feel like I’m being awfully patient, waiting for the loading times.

Question: In retail, you have to be very quick, or customers are just gone.

Beliaeff: If you’re doing stuff in the store, buy some really kickass hardware to render the crap out of it. If you’re doing stuff in the cloud, send an email later after you’ve rendered it. But that’s not really an AR question anymore. It’s a graphics fidelity challenge.

Question: With this overlay that’s being put on the real world, you’re creating a sort of digital landscape. Is there something in the pipe legally, any white papers in the works, as far as creating digital real estate, so to speak? If I own a store, I don’t want a competitor being able to put up his augmented-reality sign in my space.

Beliaeff: Just as a general statement, nothing to do with AR specifically, technology always moves faster than law. The law reacts when it has a need to react. Until court cases start going — that’s what’s going to do it. For us as content creators, we’re trying to make fun things. It’s a legislative issue. Or it’s a Google problem. [laughs]

Ellsworth: It brings up an interesting point. You mentioned science fiction. What does that teach us about AR? For the sake of coming up with really awesome movie plots, they always make everything go horribly wrong, and these writers are actually predicting some of the stuff we have to think about when we’re designing systems.

Beliaeff: There might be a decent example in Pokemon Go. In the beginning, there were certain places — you’d hear the stories about how there was an awesome Pokestop in a graveyard or someplace like that. People would be wandering around this place at night in droves. That forced them to add the feature where people can lodge complaints and go from there. There are forcing functions that may happen on a per-app basis.

Question: As VR and AR become more accessible and available, as much so as any other video game platforms, do you think we’ll hit a point where people may want to just go back to the simplicity of sitting on the couch and watching a screen? Or are these platforms going to entirely replace that old paradigm?

Beliaeff: I’m never a believer in a binary outcome. You had early PC games. Consoles came along. Mobile and tablet came along. There’s been space for each of them throughout. Games in general went from being a boutique hobby to the highest-grossing form of entertainment there is. There’ll be a certain level. The percentage that a particular facet has, console or PC, might go up or down. But in general, whenever we add a new method for people to be entertained, it ends up broadening the entire market. Not necessarily immediately. But it’s another way for games and entertainment to expand.

Ellsworth: I totally agree. It’s going to broaden the experience. I often think about how my father’s going to interact with AR. He’s not a very technical guy. But some of the experiences we’re working on are so direct and so intimate — holographic content you can reach out and touch with our wand — these are all things that my father understands how to do. Launching applications from our system is as simple as a tablet. If I gave him an Xbox controller he would never feel comfortable with it. He’d probably hold it upside down. It’s going to pull some of these people who don’t play games into playing games, because it’s more direct and more intimate. You can do it with your family and friends.

Above: You play CastAR with a wand-like controller.

Image Credit: CastAR

GB: I’m a gamer. I don’t really have a favorite platform. I care about my favorite games. Maybe eight out of my top 10 favorite games of the year were console games, but the others were mobile games. Each year that makeup changes. Platforms come and go, but I still enjoy something from all of them.

Beliaeff: In 2016 I probably didn’t play a lot of PC, but then Civilization VI came out. Then my PC hours went through the roof. When it comes to phone and tablet, I play a lot because I travel a lot.

Question: Are any of you thinking about more invasive technologies, things that might relate to other senses? Like a horror game that would surprise you with a shock or a gust of wind. Or even something very simple, like a paintball game.

Ellsworth: When we were doing R&D at Valve we came up with some pretty crazy stuff. We were given free rein to explore all of these different ways to do inputs and outputs. One of my favorites was a remote-control human, where we ran a bit of electric current behind her ears. There are research papers on this. You can make it feel like people are tipping left and right, so you can steer them down the hall that way.

There’s a lot of interesting techniques out there. Working with Playground Global, there’s a company that has a backpack called SubPac, which is just amazing. It has all these transducers in it that take audio and impart it onto your body. They’re out on the show floor. There are some pretty cool headphones that are 3D, moving sound around. As you said, some of these things will come and go, while some of them will consolidate down into smaller packages and become what we use to play games or collaborate over distances in AR.

Beliaeff: It’s really a matter of what can become integrated. So long as everything is a separate peripheral — the more peripherals you have to buy a have an experience, the fewer people are going to try it. You get a subset of a subset of a subset. You look at the evolution of PC games. At first you had monochrome graphics. Then you had color. Then sound cards and graphics cards and physics cards. All that came together over time, but the process took 20 years. Building in all that sensory stuff, you’re going to look at a similar arc, assuming there’s a killer app that drives it.

I’ve seen VR apps at GDC where the company’s gone all out. There’s one where you’re in Norse times, driving this wagon along, and they did a whole thing where you’re actually sitting in a wagon and holding a set of reins to control it. They have fans set up for the feeling of wind in your face, water and steam, the whole thing. It’s really cool, but it’s a hell of a lot of work for one game. You have to get to a point where there’s engineering efficiency. It can’t just be a labor of love.

Ellsworth: There are going to be location-based experiences just like that. VR lends itself quite well to that. But in the home, you have to think about what people are going to bring into their homes. Most people aren’t going to move their furniture around to have an experience. That’s why we’re focusing on such a compact experience — flip it over and go. As the technologies evolve, maybe holodeck rooms will be what everyone stalls in their house 20 years from now. But it’s hard to imagine that happening anytime soon.

GB: What do you think is a very fun game that already exists, but would be more fun in AR? Or what’s the AR experience that you’d want to ultimately have?

Ellsworth: I love board games, but board games are frustrating for me, especially when it’s a new game where I have to learn all the roles and set up all the pieces. For our platform, I’m excited about making some of those games more digital, allowing them to be stable. If I’m playing with my friends, we can stop for the night and it remembers the game state. Then we come back and resume with no problem.

I’m also looking forward to playing over large distances. With hand tracking and eye tracking built into the headset, I can be playing with my friends across the world. If I need to move a piece I can point and move it. They’ll see a holographic version of my hand. It’ll be more intimate, more like we’re in the same room together. In the short term, board games are going to be pretty amazing.

I also want to see an RTS game, a Command & Conquer type of game, where I can direct my troops. That’ll be amazing. I can have all my friends around the table and we’re playing together with the same big map. When we all meet up in the middle to fight, I can look across the table at my friends and see them react when I get a big kill. Those are the types of experiences that are really magical with our system. When we bring people in to playtest, sometimes it’s less about the game and more about the social aspect. Because you’re sitting over there, I know you can’t see parts of my virtual world. I can run and hide my character, see you approach, and jump out to snipe you in a way I never could have done on a flat screen. A lot of interesting twists on existing games become possible with AR.

Beliaeff: In our drone project, we create these 3D landscapes. Part of it is these alien hives, with things hidden behind buildings. People have to walk around to see behind the buildings and pilot over there. That’s part of the fun of mixed reality.

A lot of things happening right now, things that are very experiential, will lend themselves well to augmented reality. Escape rooms, for example. Maybe you’ve done an escape room experience where you have to figure out what’s going on and how you get out. That would be great for AR. You can turn your apartment or your house into the escape room, making it meaningful and real for you. Laser tag, any sort of game like that where you can visualize the gameplay. Racing, where you can spawn AR goals to race around. There are so many logical extensions, where people already know how to play a game, and now it’s that much cooler.

Ellsworth: You can take a physical, tangible object, like a figurine, and put it down on the table. Then the essence of the character jumps out to battle and level up. What’s interesting is that when you start having those types of experiences, this little plastic toy suddenly becomes mine. It has all these markers of the experiences I’ve had. I’m more connected to this physical thing than if it were just a digital object on a screen. The whole toys-to-life phenomenon is going to be taken to another level. Each of your little gadgets or toys will be unique to you, because it’ll be persistent.