Epic’s graphics guru Tim Sweeney on AR/VR, game engine wars, and simulating humans

Tim Sweeney, the chief executive of Epic Games, has seen computers become 100,000 times faster during the 25 years he has been making games. And he’s one of the few people in the world who knows what to do with all of that computing power and can predict where we’re heading.

His company of game designers has created titles such as Unreal Tournament and Gears of War, and it’s working on the upcoming Fortnite title. Epic also creates the Unreal Engine 4, a tool that makes it easier for other developers to create beautiful games with amazing worlds and realistic human characters. Epic’s Unreal is the engine of the game industry, and Sweeney is the one of the brains that makes it happen.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

Yet Sweeney is concerned that all of the indie game developers that the Unreal Engine is enabling could face hard times if the industry doesn’t change. Getting games discovered when there are a million competitors is a huge problem, particularly as gamer attention is focused on just the top hit titles. Sweeney has given thought to this potential “indiepocalypse” and how to make life easier for developers in the game engine wars.

I caught up with him at our GamesBeat 2015 event, where we also talked about augmented reality, virtual reality, and the potential to create believable simulated humans. Here’s an edited transcript of our talk.

GamesBeat: You mentioned in one of your talks that computers have become 100,000 times more powerful over the time you’ve been making games.

Tim Sweeney: That’s right. It’s an amazing transition, to go from an Apple II as my first computer to what we have now. In my pocket here I have a device that’s 100,000 times faster than my first PC. But we’re still largely on the same paradigm as we were 25 years ago. We still have a screen. It’s gotten smaller. We still have a keyboard. It’s gotten crappier. We’re still in this old-fashioned computing paradigm.

Over the next decade I can see more progress in the human use of computers than we’ve seen so far in my lifetime prior.

GamesBeat: It would be interesting to see the console gamemakers continue to drive down on realism, making existing console games much more realistic — human faces and such. But you’ve also been very excited about VR and AR. It seems like a divergence from the world you’ve been driving. Why are you so excited?

Sweeney: Unlike previous advances, where Moore’s Law has driven incremental improvements, we’re seeing a complete revolution in the human-computer interface. To go from an experience where you’re look at a little monitor to one where you’re completely immersed in a realistic experience that combines objects in the real world and synthesized images—It’s going to enable entirely new kinds of applications.

It’s not just a revolution for gaming, but for all forms of human-computer interaction. Architects, industrial designers, social network users, we’re all going to be immersed in this in our daily lives in the future.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

GamesBeat: In Shanghai, you said that about 10 years from now, perfect AR lenses are going to replace tablets, televisions, screens of all kinds. Do you really believe that?

Sweeney: It can be hard to extrapolate through that sort of technological change. This is not Moore’s Law. This is not incremental improvement. Let’s imagine an 8K display. They’re being manufactured now in small quantities. Imagine an 8K display per eye that follows your motion as you look around. Imagine a 120-degree field of view, filling your entire field of view, and imagine this miniaturized to the form factor of sunglasses.

We’re not talking about a big helmet that fits over your head. We’re talking about a device that could easily reach a billion consumers, or five billion, over the course of the next decade. That’s going to be a freeing experience. It has applications for all forms of software. Any sort of 2D desktop application you’re using now will be reinvented around the capabilities of AR.

Games are the easy problem right now, because we already know how we’re going to take a 3D game and translate it into AR. All the other aspects of it, though—Imagine social networks and chat, talking person to person over large distances when it looks like you’re really there. It’s going to change the world in many different ways.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

GamesBeat: What about traditional game development? You brought up your own feelings about the potential for the “indiepocalypse.” Others have written about this, saying there are too many indie game developers out there. We have a few million of them. Are those folks going to be able to thrive in this world that’s coming?

Sweeney: We’ve gone through several revolutions in the game industry. The most recent has been the indie revolution, where it’s become possible for a couple of people in a couple of months to create a stand-alone game. With the move to VR and AR, it’s going to be a much harder environment. Everyone’s going to need tools to be able to build photorealistic experiences, to scale them up to the interactions you’ll expect in a physically realistic simulation. It’s going to be a competitive market. It’s going to lead to very high-quality game experiences surfacing.

This doesn’t necessarily mean huge experiences. We have this bullet train demo that we showed at Oculus Connect. With 12 developers working for 10 weeks, we made an awesome small experience. The initial VR market, since there are only going to be a few million units in the market by the end of 2016, will have mostly small experiences. But they’ll be awesome. We’ll see a move away from the simple 2D games that have dominated mobile development for a long time, toward very rich experiences with fairly stringent technological requirements.

GamesBeat: The app stores, over the last few years, have been dominated by just a few games. That doesn’t bode well for all those indies out there. What would you like to see this market evolve into?

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

Sweeney: The market needs more structure. It’s insane that there are more than a quarter million games being released in the app stores every year. They’re each bought by two or three people. The average project is very small and has essentially a one percent chance of providing a living wage to its developers. That’s an unfortunate situation.

What we need is a much more stratified ecosystem. We need more large teams producing large, high quality experiences, and we need indie developers to be able to build on top of that. Ark: Survival Evolved, a recent Unreal Engine 4 game that was released, sold more than two million copies on Steam in the first few months. It has a thriving mod community around it with mod developers building things on top of that.

The advantage in that is that you’re building on the shoulders of giants. Instead of 250,000 games competing for top-level visibility, you have a genre of games with a leading game in it and a mod ecosystem around it. It’s a lot more opportunity for developers to be bigger fish in a smaller pond.

GamesBeat: Maybe 100 thriving sub-markets, then.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

Sweeney: It’s the only plausible way that a million developers can earn a living in the game industry. There’s a path to that, but there’s not a path to 250,000 games all somehow being recognized by users.

GamesBeat: As tech gets more powerful, making games on top of that tech can get harder and harder. How do you keep tools simple and making games more accessible?

Sweeney: Einstein had an axiom on that. He said, “Keep it as simple as possible, but not any simpler than that.” Our experience building games is that it’s easy to start building a game now. It’s an order of magnitude harder than that to ship a game. It’s three orders of magnitude harder to ship a game that’s commercially successful.

What we do in all of our efforts is optimize for the ultimate outcome of success. Building tools that favor being able to build and maintain over a life cycle, scale, debug, ship, and maintain large-scale games. This necessarily means tools that have a steeper learning curve, but it also means tools that are able to reach these high magnitudes of success with a higher probability.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1822078,"post_type":"exclusive","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,games,","session":"A"}']

With Unreal there are multiple games that have made more than a billion dollars in their franchise lifetime with the engine. It comes from this high-end console background and it’s worked its way down to be more usable. It’s an interesting contrast to coming from the other direction.

GamesBeat: What’s your update on the game engine wars? Unreal comes from triple-A on down while Unity has bubbled up from the bottom. What’s the competitive situation like?

Sweeney: As far as wars go, this is a pretty gentlemanly war. There are two major engines used widely now, Unity and Unreal. They’re both thriving. Since we made Unreal free at GDC this year, about seven months ago, a million users have chosen Unreal. These are serious developers building serious games. We’ve been stunned by the growth. John Riccitiello recently announced two million monthly active users with Unity.

That’s amazing, that the game industry has progressed to this point compared to a few years ago, when it was this exclusive club of super-elite developers. It’s a much larger market now. If there’s a war happening, it’s between the game developers who are trying to get their games noticed and reach their audience. It’s far harder for them and their businesses than it is for us at Epic and Unity as engine providers. It’s on us to be able to provide them with the tools they need to be able to achieve those levels of success.

GamesBeat: Is your approach very different from Unity’s?

Sweeney: The first three generations of Unreal had their own scripting language, UnrealScript, which I designed. It was a great way to get started with a small engine. With a bit of coding you could get a project up and running quickly. But what we found over the years was that it didn’t scale up to a larger system.

Let’s face it. We have a C++ operating system at the foundation. We have a C++ game engine built on top of that. Unreal is written in C++. Unity’s written in C++. If you’re building your game in C++ then you have access to the entire technology stack, completely unfettered. The benefits of that, having full engine source – go to unrealengine.com and you can download the full Unreal source nowadays – having the full ability to debug and understand the entire technology stack is really important once your project reaches a certain scale and complexity.

At times you need to dig in and figure out what’s really happening there. It’s a more expert level of development than the sandbox scripting language approach that Unity has now and we had in the previous generation, but we feel that at a certain point it is necessary in scaling. I made the decision to kill UnrealScript, my pet programming language, and move to a pure C++ engine.

GamesBeat: Do I have to be geekier, then, to make games on Unreal?

Sweeney: The neat thing about Unreal is that there’s also this visual programming system called Blueprints. Without writing a single line of code, you can build a small indie-scale project. A number of games have been shipped without any programming whatsoever in Unreal.

But for programmers—To move from a scripting language to C++ is a learning curve. It’s a rite of passage in the industry. But there is a magic to it. By writing code in that manner, you have complete access to the entire operating system and game engine. You have code that can run 20 times faster than an interpretive scripting language. When you’re aiming at higher-end experiences – games with lots of objects, VR and AR where you’re trying to run at 90 or 120 frames per second to keep up with the hardware – it’s a matter of life or death for your project.

GamesBeat: We had Sinjin Bain earlier talking about how game engines need to be brought into the modern age, how cloud computing and other factors need to be applied to game engines. Does that resonate?

Sweeney: If you look at all the major successful games, they have a large component running in the cloud, running game logic and persistent stuff. They have a graphics engine running on your device, be it PC or mobile. For the foreseeable future, we’re going to see that architecture. We’ve moved an awful lot to the cloud. We still need to run a lot on the client, though.

There’s been a theory that video streaming will power gaming “soon,” in just a few years. It’s always been just a few years. With AR and VR, you’re pushing toward a more mobile form factor and a much lower latency requirement for games. It’s pushing the possibility of cloud rendering out even further. So there’s always going to be a client-server separation there. It’s important for the game engine to be pragmatic about that tradeoff.

GamesBeat: As far as your graphics experience goes, when you look at a modern game with motion capture and face capture and so forth these days, what do you notice that can still be improved?

Sweeney: The way that high-end triple-A games solve this problem is by brute force application of tens of millions of dollars to motion capture, facial animation, and acting. It’s a process that works well, if you look at games like Call of Duty and what’s being done with massive budgets. The problem is that it’s woefully uneconomical for most developers.

The real challenge in that area, in simulating more realistic humans and other game systems, is not just how to reach the maximum level of quality that’s theoretically attainable within the engine, but how to make it as economical as possible to do that. Everybody is always going to be under budgetary pressure to create experiences.

All of the efforts we’re undertaking in that area are to make it as straightforward as possible to make human animations – creating reusable libraries of animation, reusable skeletons – so that with as little work as possible you can take an existing human, modify it to do what you want, and have it largely work.

GamesBeat: I love the faces in Sony’s Until Dawn from this summer, but as soon as the people move, the motion of the bodies tends to give it away. It’s not just getting the faces right.

Sweeney: Yeah. There you get into the problem of simulating human intelligence. Having a conversation with a game character is going to result in them having to do what a human would do in that situation. It’s a difficult problem and nobody knows the answer to it yet. It’s not just rendering, where if you get enough computing power, we know the laws of physics, and we can simply simulate the flow of light. With intelligence we don’t know the algorithms yet.

You can talk about neural nets and deep learning, but nobody knows how to apply this yet to anything more than just a brute force algorithm. It’s one of the major problems we’re going to face over the next decade.

GamesBeat: Are you optimistic that we’ll eventually get to simulating reality, simulating believable humans?

Sweeney: That’s the hardest problem we face in the industry. I do believe that within the next decade, everybody’s smartphones, desktop computers, televisions, all these computing devices will be replaced with augmented reality devices that are as convenient as glasses. That will be a fairly natural, incremental transition over that time period. But I feel like simulating human intelligence is more than a decade out. It’s a problem we’re likely to be grappling with for the rest of our lifetimes.