How CastAR’s Jeri Ellsworth will use augmented reality for fun tabletop gaming

Jeri Ellsworth and Rick Johnson have been toiling away at augmented reality for years at their startup, CastAR. Their ambition to make a fun AR product has been cooking for a while, and they’re taking a very different approach to the next-generation gaming platform.

And given Ellsworth’s background as an underdog, it’s no surprise she’s trying to think different. Ellsworth taught herself how to design chips and became known in 2004 for creating a Commodore 64 system on a chip with a joystick. She went on to become a hardware hacker and was part of a team of researchers at Valve, the maker of the Half-Life games and the new SteamVR virtual reality technology.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

Ellsworth left Valve in 2013 to cofound Technical Illusions with fellow Valve technologist Rick Johnson to create the CastAR system. They raised money on Kickstarter and eventually moved from Seattle to Mountain View, Calif., in 2014 with their team. CastAR raised $15 million last August, and it is preparing to ship its first generation product.

Ellsworth wants to create a new kind of gaming experience with augmented reality glasses. The CastAR glasses can overlay animations and other imagery on top of the real world. And for Ellsworth, the focus is on fun for the mass market, not making a really expensive toy that you never play with. She spoke at our GamesBeat Summit 2016 event with Will Mason, cofounder and editor-in-chief of UploadVR.

Here’s an edited transcript of their conversation.

GamesBeat: Today, I want to dive in a bit on mixed reality’s role in the big future we keep hearing about. It seems today you can’t throw a stone in Silicon Valley without hitting someone who wants to talk about AR and VR. Jeri, before we get into the story of CastAR, can you talk about how you got into mixed reality in general? You were working at Valve on VR in 2012, right?

Jeri Ellsworth: I have an interesting background. I’ve been fearless as far as getting into areas I don’t know about. I dropped out of high school and started building and racing quarter-mile dirt track cars with everyone telling me I couldn’t do it.

In the ‘90s, the computer store business was booming. [In 1995] I opened a computer store. I was so poor at the time that I lived out of the back of the store, but I grew it into more and more locations. Eventually, I had five. Then, that market fell apart. In 2000, I thought I’d always enjoyed electronics, so I taught myself chip design. I bought a bunch of books, found some mentors, and started learning about [very-large-scale integration] design.

I came down to Silicon Valley. That was my first introduction to the culture here, which was eye-opening coming from rural Oregon. It was important to get to events like this and start networking if I was going to be discovered as a self-taught chip designer, and so I’d go to events and shake everyone’s hand and show them some circuit boards with [field-programmable gate arrays (FPGAs)] on them. “Look at this video controller I built. Here’s a sound controller. This one takes joystick input.” From there, I slowly launched my career in the early 2000s.

What got me into games and got me closer to working at Valve was, I had done some blog posts about reverse-engineering old 8-bit computers and putting them in these FPGAs. I’d reverse-engineered the old Commodore 64 and stuck it into a chip and blogged about it. A toy company contacted me and said, “We want to make a toy where we have 30 retro video games built into it.” At this point, I’d only done very small designs on bigger teams. They asked me if I could do the whole thing. I took a deep breath and said I could. I’d never manufactured anything before, never done a full chip before. But it turned out well. It was a huge viral hit, and it put me on the map.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

From there, I did a bunch of other toys and got really immersed in low-cost product design. My career afterward centered on customer-facing entertainment devices, hardware that kids could afford and adults could easily purchase. Some years went by until Valve was interested in expanding beyond the PC platform. They were concerned about the Windows platform remaining viable. They hired me as their first hardware person to put together a team and research other ways to get them off the PC platform and into the living room.

This was a dream job. It was magical. They gave me an unlimited budget. I could hire anyone I wanted. I split the research into three pieces. First, I’d get pure researchers. We’d get some maker types, people who are into building things. And then, we got some product people, which I fell into. We were given free rein to research whatever we wanted.

We went down some crazy paths. We had electrodes hooked to people’s heads trying to read their brain waves. We had galvanic skin response to read people’s emotions while they played games, and we fed that back into the games. We even ran electricity through people’s inner ears. We did this thing we called “the remote-controlled human,” where we’d push buttons and make people steer. But our main objective was, “How can we bring new gamers into the living room and attach them to the Steam platform?”

We splintered into some different areas of research. Some folks worked on the Steam Box. Some folks worked on the game controller. And then some of us worked on AR and VR. That’s where I got interested in mixed reality. We were doing a lot of AR experiments, and they were difficult. It was difficult at the time to put photons in the world and wrap them around plastic toys. But we were getting some of the early prototypes going.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

It was quite different from the VR effort because this actually ties back to what the executives at the toy companies were always beating into my head. When it’s fun, it’s fun. Stop when it’s fun, and make sure it’s affordable. Looking at what we could do with AR, it hit a broad demographic. It was inclusive. Someone like my father could play a mixed-reality game with his grandkids. Whereas when I was looking at some of the VR stuff and hardcore gaming like the Steam Box, I could never see new game players coming in to that platform unless it’s something more approachable, more direct.

That was a long story, but that’s how I got to Valve. Valve didn’t want to proceed with their AR efforts, with the CastAR prototype, so Rick Johnson and I spun the company out of Valve, and here we are today.

GamesBeat: You guys emerged out of a Kickstarter, similar to how Oculus came to market. Oculus now has the resources of Facebook. HTC and Valve are established and so is Sony. As you’re trying to bring CastAR and mixed reality to the market, what are some of the strategies that you’re going to use as an underdog in this market?

Ellsworth: We’re trying to keep things simple. It goes back to the toy guys. Put the bare minimum in to make a fun experience. We see a lot of the other guys trying to go huge and grand, creating these huge walk-around, room-scale experiences. That’s not going to be very tractable in the living room with families. That’s one of our strategies. The core of our system is this game board, which is about three quarters of a meter on a side. We’re trying to pack as much experience as we can into this space and make it social and fun.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

Always fun, always cost-effective, zero friction. Friction is another thing we’re focused on. Christmas day, when people get this for the holidays, we want them to unfold their game board, hit the power button, and be playing within minutes. We surveyed Xbox, Nintendo, all these game consoles. How long does it take to get to your first game? Some of them are terrible. It’s almost an hour for some of the game consoles. Even the handhelds, it’s as much as 15 minutes. We want to get a lot of traction in the marketplace by having a zero-friction experience. Hit the power button, slip the glasses on, and start playing.

Keep it social, too. Your friends come over, and you’re looking into the same space. You’re looking at each other. You can see each other. You don’t have to clear out your furniture and make a dedicated VR room. It’s right there.

GamesBeat: There’s been a lot of talk about the gap of disappointment that John Riccitiello mentioned. Things might not be adopted as quickly as some people might think. What are some of the ways you guys are looking to sustain success through that period?

Ellsworth: We’re constantly thinking about what other people have done, other companies, in the past. We look back at Nintendo a lot. They often don’t have the most cutting-edge technology, but they keep it fun and keep it simple. They focus on what counts. The black and white Game Boy went up against the color Atari Lynx. For a tech nerd like me, the Lynx was amazing. But the Game Boy was so much fun. It had Tetris. It had Mario. It ran for a really long time, even with its limitations.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

We’re not trying to make a new tech gadget. We’re trying to make something that my dad can use, that a kid can use. That’s a natural type of experience. I’m a bit concerned that there’s this arms race in VR right now. Who has the most pixels? Who has the most expensive tower sitting next to their VR chair? You’re not going to reach hundreds of millions of users when you have to make that kind of investment. That’s why our price point is so critical and narrowing down to the game experience.

GamesBeat: From a macro perspective, there’s a bit of an issue of rhetoric within the industry. I’d love to get your opinion. How do you define the differences between AR, VR, and mixed reality? Do you think that, in the next five years or so, we’ll still be using those different terms?

Ellsworth: In the next three or four years, we’ll start to see a lot of merging of technologies. VR is the tip of the iceberg. There will be new display technology that will do AR and VR seamlessly, switching between them. Looking out a bit further, there are holographic displays in the prototype stage that don’t require eyewear. If you start projecting out five years, six years down the road, all the lines will be blurred. People will interact in these natural holographic experiences.

Whether it’s some kind of display that’s on a table and projecting holographic images or if it’s eyewear that I put on to add annotations to the world, it’s all going to blend together. This will come in time. Who knows what we’ll call it eventually?

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1944934,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

GamesBeat: The word I keep hearing you say is “fun.” A lot of people talk about VR and AR in these platform-shift terms — “this is going to change everything.” Why is your focus on fun rather this whole world-changing dynamic everyone else talks about?

Ellsworth: When I take a look at VR experiences — what we were researching at Valve, what we see out there now — the first thing you see is, “Oh my God, I had this amazing experience riding a roller coaster; it’s great!” That’s what I look for. I’m a thrill seeker. I have that experience, and it’s fun in that instant. But then I put my VR headset away for months until something new comes along.

What we’re focusing on is how do we find sustained fun? How do we have an experience that you want to revisit every single day? You want to grab your CastAR system and check in on what your friends are doing or invite people over and have this prolonged game experience. That’s a more subtle experience. It’s not as intense.

That’s one of the things we discovered at Valve. When we were doing these experiments with gameplay, we were feeding people’s emotional states back into the game. If you give them super high intensity for prolonged period of times, it’s not as fun as something more mild. But if you survey folks as they leave the different game experiences, if you throw 5000 zombies at them and they die instantly, they have fun in that instant. If you give them a more subtle experience that goes through peaks and valleys, they engage longer and come back to it. That’s what we’re looking for. What are experiences that are natural and get people to come back?

Some of our core values — we look at everything. Do we want to have this type of tracking system in here? Is it going to work 100 percent of the time? Can my father pick up our wand controller and know how to poke at the game character and move him around? Could a five-year-old do that? Could anyone else do that? We want to make sure everything we put into it works seamlessly and isn’t confusing to the end user, especially when it comes to input.

GamesBeat: When you’re building a system like that — obviously, it’s a brand new platform. When you have a new platform, you have to have content to keep people coming back to it. Whose responsibility is it to help in the creation of that content?

Ellsworth: Some of that, going forward, is going to have to be stimulated by us. We’re going to have to go out and commission some content. We have some internal research going on, which is going to feed into our developer network. We’re already working with developers bringing their content over.

We had a game team with a little green army men game that they wanted to port to our platform. It took them a day or so to port it over, build some UI around it, and get it working on the table. It was just amazing to watch this small game development team port their game, an Xbox or Steam type of game, to the platform. New game mechanics that they didn’t expect just emerged out of the blue.

When they first ported it over and you could play solo, it was about the same as playing it on an Xbox or whatever. You could look around these little buildings as your character ran around. But adding more players — two, four, five — the gameplay totally changed. It was hard to pry the headset off of folks. If you’re sitting over there and I know my character is short enough that you can’t see him, but I can peer over that building and see your character, I can run around and snipe him. The alliances that form around the table are social and fun. A lot of content may be familiar but unique.

GamesBeat: Do you think the advantage of mixed reality, especially in terms of games, is the ability to have that kind of transformative social experience in a way that you might not necessarily have with a VR headset?

Ellsworth: Right. In the first generations, we want to touch hundreds of millions of users. We want to be that approachable. I don’t think it’s practical for most people to clear out their furniture and set up a VR room. Although personally, I would.