Epic’s graphics guru Tim Sweeney shares his vision for perfect augmented reality

Disclosure: The organizers of ChinaJoy paid my way to Shanghai. Our coverage remains objective.

SHANGHAI — Tim Sweeney, chief executive of Epic Games, is one of the rare seers who understands both technology and games. He’s been pioneering graphics for 25 years, and his company of game designers has brought us titles such as Unreal Tournament and Gears of War.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

Epic also creates the Unreal Engine 4, a tool that makes it easier for other game developers to create beautiful games with amazing worlds and realistic human characters. Epic’s Unreal is the engine of the game industry, and Sweeney is the one of the brains that makes it happen.

That’s why he’s one of the most interesting seers in games. He gave a talk at this year’s ChinaJoy 2015 game trade show, an event attended by an estimated 250,000 people in Shanghai. At first, as I sat in the room in the Kerry Hotel, there were just nine people in the audience. Sweeney, of all people, had technical difficulties. And by the time he was ready to talk, the room had magically become full of listeners.

Tuning in via live translation, they were a lucky bunch. Sweeney talked about his predictions for mobile game graphics, better web games, virtual reality, the game engine wars, and augmented reality. By his calculation, computers have gotten 100,000 times faster while he’s been making games. But augmented reality is going to be the best revolution yet. And he figures that if augmented reality is perfected, we’ll have the equivalent of 40-feet displays on our eyeballs. That will make TVs, tablet displays, and smartphone displays of all kinds obsolete.

Afterward, in Epic’s booth in one of ten giant halls, I interviewed Sweeney about his predictions. Here’s an edited transcript of our talk.

GamesBeat: It was an interesting talk. That was a big thought again, about AR possibly eliminating all kinds of screens. It’s not good for the glassmakers of the world.

Sweeney: If you think about it, it’s going to be a real boon. Everyone will have an awesome television that’s 40 feet wide and you can display it anywhere in your house. The amount of material needed to manufacture AR will be much smaller than TVs or computers or any of these other devices. It’ll be very economical.

GamesBeat: You suggested that the quality is getting to a point where screens are less and less something we need to look at. The images on the glasses can be as good as real life.

Sweeney: We’re definitely on track to that. There have been a lot of studies on exactly what is the resolution of the human visual system. It’s fairly clear that it’s less than 8000 pixels by 8000 pixels. That’s the maximum quality of graphics we can appreciate. You could go to a display like that right now if you were sufficiently motivated.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

Chipmaking has been driven by Moore’s Law. They’re building billion-transistor chips. But there hasn’t been as much drive behind displays until recent times, until the Retina display revolution. As that technology is pushed harder, we’re just a couple of generations away from displays that are 8K by 8K per eye times two eyes. There you have a display that’s indistinguishable from reality.

The real interesting things with VR and AR start to happen when you have custom-manufactured display devices aimed just at those applications. Right now everybody’s devices are reusing mobile phone displays, augmenting them with lenses and other systems to purpose them for VR. Once you have custom display systems, the level of realism is going to go up very quickly.

GamesBeat: After 10 years, say we don’t need screens anymore. What are some of the steps along the path to making that happen?

Sweeney: The immediate step for content creators is VR. VR is manufacturable. We can build devices with high quality and a very large field of view. The augmented reality content revolution starts with VR, and that’s probably the home of it for the next three or four years. Then we can expect to have very high quality AR devices coming online.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

The industry is going to have to go through a period of reinvention. Every application designed for 2D interfaces will be obsoleted by AR. You’ll be able to achieve far better usability if you design everything for the AR experience using AR input — motion and gestures and other very fine input sources. Think of every type of application you use in daily life and the types of applications used by professionals. You can start to imagine how they might work in the future.

Think of modeling, for example, which is done in 2D with programs like 3D Studio Max and Maya, using very complicated user interfaces with very complicated mouse commands. Instead, you’ll be able to reach out and sculpt your object directly, paint on it with your paintbrush, and zoom in and out in a very natural way.

GamesBeat: I’ve heard people talk about trying to do glasses with VR on one side and AR on the other. They become see-through at some point. Is that another step in the middle that somebody could achieve?

Sweeney: Perfection is a long way off, just because of the difficulty of building these things. We’re going to see a lot of progress and a lot of interesting intermediate steps. Right now with VR we have HTC, Oculus, Sony’s Morpheus, and Samsung with Gear VR, which is probably the most interesting idea, reusing an existing cell phone with some lens hardware to give a VR experience where you don’t have to buy a computing device. That’ll combine with the research into HoloLens and MagicLeap and a lot of other creative ideas.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

It’s going to be like any other period of revolution. We’ll see a lot of experimentation. A lot of people will try ideas. Some of them will work and some won’t, but everyone will learn rapidly. In 10 years, you’re at the destination.

GamesBeat: It sounds like there’s a lot of breakthroughs you could plan for or expect. It’s not just a linear path that you can see from Moore’s Law. There’s a lot of inventing along the way that will have to make this happen.

Sweeney: Mark Rein and I visited MagicLeap a few months ago. I had no idea that some of the things they were doing were even possible. I can’t talk about some things because of the NDA, but it was like stepping into a Harry Potter movie, seeing all these magical inventions that we hadn’t seen before. There will be a lot of very rapid progress there in all areas. Their light field cameras and light field displays, some of the long term solutions for displaying a 3D image with proper depth of field, new display devices, new input devices, cameras that can capture scenes in full 3D –They’ve put together all the different components you need.

All of our game engines and all our technology is output-oriented. We’re good at rendering beautiful scenes, but we have no support for input in 3D. If you want to walk around and digitally scan your house as you go so you can represent that in the game engine and modify it, place paintings on your wall — there’s going to be a whole revolution on the software side.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

GamesBeat: The tool-making that will make it easier to create the content and allow more people to become creators, is that another parallel path that needs to happen?

Sweeney: Absolutely. Empowering everyone to create content is a big step here. Eliminating the technical complexity of content creation — if you can paint a painting or build a sculpture, you should be able to do the same thing in the virtual world without having to spend months learning complicated software. That’s the ultimate destination for all of this.

GamesBeat: I have a kid who’s a very good artist, but she doesn’t want to learn computer animation, computer graphics, complex math. That’s the kind of future she needs.

Sweeney: It’s all about reducing the barriers to entry. The YouTube revolution isn’t a revolution in content consumption, although there’s a huge number of content consumers. It’s about how anybody with a camera or a smartphone can create a video and share it with the whole world. The revolution around alternate reality will be around enabling anyone to become a content creator too.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1777577,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,mobile,","session":"C"}']

GamesBeat: In 2015, I’m pretty impressed with the graphics of human faces I see in games like Batman and Call of Duty. It seems to stem with the face capture technology that you talked about a bit. Are you happy with where that is right now, the ability to motion-capture faces and turn them into something animated with less effort?

Sweeney: The state of the art now, from the game industry to Hollywood, has reached an astonishing level. The challenge right now is that the processes used to create very high quality facial animation in games and movies are very expensive. The next step is to use digital scanning techniques in mainstream cameras instead of specialized motion capture to put that within everyone’s reach.

When you’re in augmented reality having a chat with someone, you want to see their face moving as they talk. You want them represented in 3D so you can walk around them. That means it has to happen at the consumer level, not just the professional level. A lot of the effort we’re putting into developing new Unreal Engine 4 systems is in enabling this world class quality of content with greatly reduced cost and complexity.

GamesBeat: I wondered if the face capture becomes so good that some of the artistry involved isn’t necessary anymore. You do want to stylize a lot of these faces — create an orc out of someone’s scanned face, maybe. It almost seems like the face capture could supersede some of the human contribution.

Sweeney: That’s the interesting thing. Right now, to get the best quality of facial animation in a game, you bring on actors. You make sure the actors’ faces map naturally to the characters in the game. Then you capture their performance — the lines, the face, the emotions — and that goes into the game. It’s not a human animation problem so much.

Every Pixar movie, though, or something like Epic’s A Boy and His Kite cinematic — those are not photorealistic characters. You really want the artistry that only traditional animators can bring. I think there’s going to be a wide range of possibilities. In all cases we want to reduce the cost and the barriers to entry.

GamesBeat: This competition with Unity — you had an interesting explanation there. You consider the convergence on C++ to be one of the core advantages Epic has in the game engine war.

Sweeney: As games get larger, the performance of all the code in the engine becomes much more important. Also, as games get more visually realistic and complex, the ability to access all of the engine from the gameplay code becomes more important as well.

Epic started out with scripting languages in the first generation of the Unreal engine, in 1998. I wrote that. There’s a place in my heart that comes along with the simplicity of programming in a scripting language. But once you go beyond relatively small-scale games to larger experiences with more realism and more characters involved, there’s a strong benefit to C++ — its performance, its complete access to the engine and the operating system.

GamesBeat: That low-level API technology that you were talking about fits in with this strategy. That combination enables mobile devices to exploit graphics to the fullest?

Sweeney: Apple’s Metal already reduces the CPU overhead of graphics by a factor of 10. You can generally render 10 times more objects in a game in real time than you could with previous OpenGL calls, which had much higher overhead. That will all come to other smartphones in the form of the Vulcan API. It enables a revolution in game complexity, especially on mobile, when it comes to object counts and realism. To access that power, you have to have a high-performance programming language and that low-level access to the whole engine.

GamesBeat: So we have to shut down the LCD manufacturers. What else are you predicting?

Sweeney: Augmented reality will change the world more than a lot of other technologies. Traveling around to meet people will be much less important when you can stand in a room and chat with a virtual representation of a person that’s so close to reality — it’ll be a whole new level.

1950s movies always talked about video phones. Skype and other services have shown that it really isn’t all that exciting. You’re looking at a little postage-stamp view of a person animating. It’s not the same as being with them. AR will give you genuine presence, the sensation that you’re really there.

The ability to create shared experiences—It’s not just about single-player games and single-player experiences. It’s going to be social as well. Oculus created the Toy Box demo using Unreal Engine 4, which puts you in a virtual environment with another player. You can see their head and their hands. You can play tetherball with them using the virtual rackets. You can throw objects to them and play catch and do a lot of other things. It creates an awesome social experience that hasn’t been captured in traditional multiplayer games.

GamesBeat: Will we be talking about Unreal Engine 20 at some point?

Sweeney: We’ll be evolving this engine forever. It’s gone through four major generations so far. Computers have gotten approximately 100,000 times faster in that time period, going from software rendering on the CPU to multi-teraflop GPUs nowadays. The rate of progress in the industry is astonishing. We’ve been there at every step.