How Sledgehammer tricks you into believing the visual realism of Call of Duty: Advanced Warfare (interview)

Making human faces believable has been a goal of video game developers for a long time. So it may not surprise you that the makers of the new Call of Duty: Advanced Warfare — a game in a franchise that has virtually unlimited development budgets — have tried to deliver on this promise in the latest installment of the multibillion-dollar modern combat series.

Sledgehammer Games, the developer of the new Call of Duty that debuts Nov. 4, wants you to do a double-take when you look at the human faces. The studio tried to do this by pushing technologies such as high dynamic range, physical-based shading, wrinkle maps, performance capture, and physical-based lighting. All of these techniques add subtle features that make animated human skin look and move in a more realistic way.

On a recent visit to Sledgehammer’s headquarters, I sat down with art director Joe Salud and Aaron Halon, the director of product development, to talk about the realism of the faces as well as other features of the game, like real-world environment lighting. The team went out, for instance, to the area around the Golden Gate Bridge in San Francisco to capture the natural lighting of the greenery, the ocean, and the sky.

“It took time for our team [to do this],” Halon said. “It was a big shift.”

Here’s an edited transcript of our interview.

GamesBeat: You’ve noted before that people take a look at your characters and do a double-take. They wonder if they’re real or animated. Is it real, or is it a game? Is that part of the vision for PlayStation 4 and Xbox One development?

Joe Salud: Yeah, that is. We have a tack that we like to call “physical-based” here. We’ve done tons of research on how to achieve that. There are three basic pillars that make that reality. You have to have HDR, you have to have this thing called physical-based shading, and you have to have something else called physical-based lighting.

To make a long story short, what we do is, everything is based on physical values. We let the renderer do all the calculations as to how shiny something is, how matte something is. They’re all related to each other. It’s using less of the individual artist’s eye to decide on the granular details and more of the engine procedurally doing a lot of that.

GamesBeat: Was there something in your background that put you on this path toward photorealism, or whatever you’d call the endgame here?

Salud: There are two approaches I’ve seen in games. They go more stylized or more real. What we’re trying to do from an art direction standpoint is to sell this crazy world. We’re trying to cross the line of what’s believable and realistic. We’re trying to immerse you more. One of the visual tools for doing that is fooling your eye into thinking you’re there. Giving you at least that impression. If that impression comes through, you’re not going to be concentrating on the graphics so much. You’ll just enjoy the experience. You’ll feel like you’re really there.

It helps you buy some of what we’re trying to sell here. These advanced, futuristic vehicles, what could make them more believable than to see them in a world where they look realistic?

GamesBeat: How does full performance capture play a role in this, as opposed to just face capture or motion capture?

Salud: To touch on it from a high level, performance capture’s been huge. We’ve done a couple of layers that are different from previous games. You capture the performance, obviously, from the face, and then on top of that we have what we call wrinkle maps. Depending on the musculature or the muscle movement, the skin also moves on top of that. It’s not just geometry moving. It’s geometry simulating muscle performance and the skin on top of that. We also have a skin shader that receives the light on a scientific level.

Aaron Halon: One of the interesting things about realism and environment art from a production standpoint, which was new for me when overseeing the team and the artists and designers—you talk about going out and capturing facial. But even for the environment, we were capturing real-world lighting with real-world materials. That was a big part of this physical base.

For Collapse, it’s obviously there in our back yard. We had the art team go out to the Golden Gate Bridge and do a lot of color grading tests. For other, more exotic locations, we would find proxies. We’d go on a ship or something like that for things like the later part of the San Francisco levels.

Salud: It was very important for us not to just say, “Hey, to our eye, what color is this?” We have tools that can take the materials out and measure its physical response to light. We would measure the sky and capture the sky and say, “At these points in the sky, we have these physical properties.” We would do that to everything, and we would re-create it.

Halon: It took time for our team. It was a big shift. Imagine you hire a lot of senior artists who’ve been in the games industry for a long time, and they’re used to building this in a certain way, painting textures to get things to look realistic. We said, “Yeah, we want you to do this same thing, but no more painting and tweaking textures.” For a while they said, “Wait, what do you mean? That’s what we’ve always done.” No, we’re going to let the lighting engine dictate that now and control it in a different way.

Some people were not quite comfortable with that for a while. Eventually, though, they started seeing the results and got more comfortable with how to build for that.

GamesBeat: Would it be similar to what the Pixar animation studio did with Monsters University? They brought up global illumination a lot — where the sun is, that’s your light source, and you’re not going to paint shadows on things.

Salud: It is very similar to that, yes. Global illumination is just one part of it. There’s how the global illumination responds to the surfaces. Before what would happen is you had 10 artists, and they’d all work on making this room here. One artist would work on the chair. One would work on this or that. They all see the world a little different. Then, when you turn the lighting on, the room looks almost right, but not quite. It’s because each artist has their own interpretation. If you let the renderer do that, then you eliminate all that interpretation.

Halon: There are efficiencies later on that are exciting. Figuring out how much time we’ve spent with material artists compared to—we can put that time into other details in the environment. It took a while to get there, but now we’re figuring out how to make that more efficient.

Salud: The efficiency that came out of it—before, when we would make this object, we would tune the texture to work in this very specific lighting. But now, since it’s physical-based, it doesn’t matter if it’s in this room or the other room or a room with green lighting. It should all work.

GamesBeat: What was it like joining Sledgehammer and being part of this new studio?

Halon: It’s definitely exciting for us. Call of Duty, for me personally, was something I was really into. I’d been working in games for a while before that, but playing a lot of Call of Duty. That opportunity jumped it right to the top. And also the opportunity to work with Mike and Glen, who I’d worked with in the past.

GamesBeat: You worked on Dead Space?

Salud: Yeah, we were both on Dead Space. It was also just exciting to develop the studio from the ground up. We were in a very small conference room, just coming up with ideas. We were used to EA before, with offices and cubicles. We didn’t have that back then. We started very small, just an agile little team. It was fun.

GamesBeat: Building a large team seems pretty tough to do at the same time as you’re making a game. I don’t know what the average number of people making the game was, but if it was very small at first, it’s very large here at the end.

Halon: The recruiting team—it really talks to the core values. We wanted to get like-minded people, people who cared about the franchise. Some people might be great at these things, but if they didn’t have that desire to come in and tackle the big challenges — how do we come in and work on a franchise that’s known so well by so many people and carry that forward? That was important in growing the team and making sure that the leadership and directors all shared that passion.

GamesBeat: Did you approach the project with certain priorities in your role, things you wanted to do?

Halon: For Advanced Warfare in particular, we rallied around this concept of the advanced soldier in the game itself. More in terms of hiring, there’s a point where you start fully understanding what your game is, and then you can start tailoring that to hiring. Early on, you may be a little more general, still focused on some high-level goals. We really want to focus on the top 10 percent of the industry, people who have pedigrees, people who want to come do something great and have the same goals. How are we gonna make this game look as realistic as possible? How are we going to make it play better, faster, more fun than before? As we started honing the pillars of the game, we tailored that to interviewing and making sure that those things all lined up.

GamesBeat: What are some of the pluses that make it easier and minuses that make it harder to do something like this? It sounds like building the team in the process of making the game makes it harder, but then you’re building Call of Duty, which is more familiar.

Halon: For sure. Right away, people can glom on to some of these goals. That’s good, instead of having to introduce something brand new. Some of the challenges with the expectations—There’s a lot of pressure and stress that comes with that. We all know this is a huge deal for us. But again, as the team grew we started getting a lot of momentum – the levels we were starting to produce, the physical lighting coming on, certain Advanced Warfare mechanics coming online. When the team itself is saying, “This is really exciting. We’re starting to have fun with it,” that confidence snowballs.

Salud: One of the biggest challenges we faced on this project in particular was the first year of preproduction. Just trying to figure out how to make this game different, how to meet these expectations that the fans have for us. What is the design? And on top of that, the new technical things we were going to introduce. How do we accomplish this technically?

The team had to go through a metamorphosis, so to speak. We’d been making games in a very specific way before. Next-gen comes along and then it’s all different. People are going to challenge you on what next-gen means. That’s a good thing, because it inspires a healthy dialogue. It wasn’t just, “Hey, let’s make the same game we made before.” We said, “Let’s break all the rules. Let’s walk over the line.” That was a challenge.

From a team dynamic standpoint, it could be a frustrating challenge. For the first year we’re saying, “Where are we going?” We were trailblazing. It was exciting, but at the same time very frustrating.

GamesBeat: As far as the new consoles, did you consider it an easier transition than the past? Using x86 technology was supposed to make things simpler.

Halon: There’s something to that. But every transition is unique. A lot of the team had been through a transition before, but for some people, this was their first. Globally, it had pluses and minuses as well, and expectations. It takes us a while to fully understand what the hardware is capable of. We might have set some targets and had to adjust.

Salud: We did use our previous experiences from other generational transitions to anticipate what this might be. But we’re not fortune tellers. We probably got a certain percentage of our assumptions correct and some a bit off the mark.

GamesBeat: What about the question of making what you do different from Modern Warfare and Black Ops?

Salud: From the art standpoint, I wanted to have a completely different look. I wanted to distinguish us as—the fans out there could see that this is clearly Advanced Warfare stylistically, as opposed to Black Ops. A lot of that had to do with the rendering, with the color palettes. The design of our characters and the design of our world, we tried to differentiate those.

Halon: On the design side, we wanted to push the bounds of things like core movement, the stuff that hasn’t change in a while. But we also wanted to maintain it so that the combat felt really solid and crisp, what the franchise is known for. Across the board — single-player, multiplayer, co-op — everything in there is going to be changing the way that you approach combat interactions. As that started to develop, we layered in new kinds of weapons, new grenades, things that would impact that second-to-second choice-making when you’re in combat.

GamesBeat: I saw some of the game. I can’t help thinking there’s some Dead Space influence in there.

Halon: The narrative has been really important. As storytellers, we’re certainly influenced by things we’ve done in the past, but also by what’s been going on now. We’re all inspired by a lot of great TV shows out there. We look at pacing and character and how much they put into that stuff. We wanted to make sure that’s a part of the campaign experience.

GamesBeat: How much of the original vision did you hit?

Halon: Games will certainly diverge. You need to adjust to things. I’d say that certainly on our key pillars – the advanced soldier – we’ve been very true to that concept across the board. It’s always fun, as we start to wind down, to look back at some of our early design documents and design goals to see how they line up. I don’t have that all lined up today, but it feels pretty solid.

One thing we do well here is capture a lot of our early development. Most of that will never be seen outside of Sledgehammer, but we do a lot of pre-visualization, even within levels. Early levels being blocked out, we have videos of that, to kind of capture the playthroughs.

Salud: It looks like a time-lapse video.

Halon: As a post-mortem, when we start working on our next thing, it’s good to look at some of that and look at where we ended and try to analyze. How can our process improve? How close were we to our convictions and our goals from early on?

GamesBeat: I remember in Call of Duty 3 in 2006, when they’d have a cutscene where you’d be sitting on a tank while someone was talking to you. You’re looking ahead and their back is to you. You could just sort of see their head move — OK, that’s an indication that they’re talking to me. That was one of the tricks they used at the time, because they couldn’t do faces and lip movements well. Do you think you’ve gotten to a complete photorealism, or are there still some tricks you have to use to convince people?

Salud: We follow a lot of similar tricks they use in the film industry. A lot of it has to do with the camera. We’ve changed our camera to behave like a real camera. It has things like exposure and depth of field and bouquet. We try to add imperfections, because in the renderer, since it’s a computer, everything it does is going to be naturally perfect. If it draws a line, it’s going to be the most perfect line you ever saw. What we have to do to make it feel more realistic is add imperfections.

The camera does a lot of that. We’ll overexpose or underexpose the shot. As far as tricks I like to use, I like to tap in to things that are already programmed into people’s heads – things they see in the movies, or when they search in Google images. What do those images look like? They’re all shot with iPhones and things like that. There’s a particular color response in things like that. People feel like that looks real. That’s what we try to do. There’s still some development time left. That’s what’s left to do.

GamesBeat: Things like how many moments you linger on somebody’s face, that might make a difference between whether people think it looks real or notice something fake.

Salud: No, we’re going to have moments where you just look at someone’s face for a really long time, while they’re talking to you.

Halon: If anything, we’re trying to balance the pacing of that sort of thing with gameplay. That’s usually the bigger concern.

Salud: We’re showing off our faces now. We’re not obscuring anything. We’re putting the camera right there. We have these long back and forth dailies where all we talk about is the face — about the eyeballs, the hair, lighting on the face.

GamesBeat: Do you feel the horsepower is there to do all this? Or do you wish for more from future hardware?

Salud: Oh, there’s always going to be more. You give me a 50-gallon tank of whatever, I’ll still want more. We can always figure out a way to fill it up. Maybe we can only afford two guys who look photo-real — well, then we’re going to want four.

GamesBeat: What about the speed of things, too, the moving objects on the screen? In what we saw today, there seemed to be a ton of things happening, with the drone swarm and so on. Is that also affected by how much hardware power you have?

Halon: It’s certainly tied to that. There are still lots of things that we’re used to doing to get our game running at 60 [frames per second]. That’s always been a key pillar of the franchise. There’s a lot of tricks and tactics we’ll use to do that. But at the same time, we’re taking advantage of all the power we have. There are times when we’ll blow that out of the water — when we’ll say, “OK, we need to get this look, this effect, but how do we do it in a way that’s more optimal?” LOD and things like that are happening behind the scenes. There’s still a lot of trickery going on to make sure we run perfectly.

Salud: Overall, we have what we need to convey the impression that we want to. Whether it’s a vista over an entire city, or–

Halon: Like your example before, we used to have meetings where we’d say, “Yeah, we’ll just do that behind-the-head thing and some [voice-over].” Now we’re not needing to rely on those tricks as much. Now it’s more just about time. We can do it, but we have to make sure we’re spending our time in the right places.

Salud: The bottleneck now is scaling, manpower. How much work can we do with the people that we have?

GamesBeat: What about the futuristic aspect of it? Some things you can performance capture because they already exist, but there are lots of things depicted in the game that don’t exist yet. How do you visualize that?

Salud: Everything goes back to what’s relatable. The design of it, there’s a couple of layers in there, but it boils down to everything being based on function. What’s it supposed to do? Then we think about, okay, what technology is currently out there? Then we extrapolate from that. Especially for the exo, we’ve talked to military advisors. We’ve hired concept designers who have worked on things like this before. One of the guys contracted for Boston Dynamics. There’s knowledge in there that motivates the designs.

On top of that, let’s say you have the knowledge to create a cool design. Then you have to package it for the fans. It might look so foreign that people just say, “I don’t even know what that is.” You have to look for elements that are relatable to the players and add that back in there. Then people say, “I can identify with that. It’s just more advanced. It’s cool.”

GamesBeat: I’m already looking at the Golden Gate Bridge and thinking, “That part’s familiar, but these ships floating past it are way out of the future.”

Salud: If you look at the new military watercraft out there, they’re shaped differently now. They’re a lot more slender and angular, stealthy. They have these really long noses up front. We were extrapolating from that. Have you seen those hydrofoil ships, the ones that start to raise out of the water? We mixed those ideas together. When people see them, they still look like ships. It’s not like a spaceship on the water. But it’s something different and familiar at the same time.

GamesBeat: I took a tour of DreamWorks this summer, talking about How To Train Your Dragon 2. They have these drawing tools now where they just draw with a stylus on the screen, and you see it immediately show up in the animation system. There’s no gap in between drawing something and seeing it on the screen. Are you guys reaching that point? Does the artist still have to work through another layer?

Salud: We have those kinds of stylus setups, where you can paint on the screen. The concept artists use those. It’s a preference thing. Some people like to paint directly on the screen, because they feel like it’s more interactive. Some artists feel like it puts their hand in the way, so they can’t see. It’s just a question of how people like to work.

Halon: We still have a concept artist who likes to draw on paper and scan it in. It’s all about the end result.

Salud: Some things in our pipeline are in real time. The materials are pretty realtime. Some of the animation elements are realtime. When we’re drawing, that happens in real time, obviously.

GamesBeat: What do you guys think of the sort of four-people-in-a-circle pod office environment here? It’s a lot different from a traditional cubicle setup.

Salud: It’s a big deal. It helps us a lot. It’s very conducive to collaboration. That’s the way our studio likes to operate. We like to talk to each other, for better or for worse. If you have walls up, that’s one more barrier for someone talking to the person next to them. Our studio is based on iteration. For iteration, you need communication. If you put walls up, if it even mitigates that by 10 percent, we don’t want that.

Halon: A lot of times we would have departments — like, the audio team has its own rooms, and that’s important for their craft. But within the pods, we’re purposely putting the animator next to the designer who’s next to the key artist. They’re all within earshot, so that when things come up — a director might be there dealing with an issue. All those other groups, it’s important that they’re part of the conversation. In the past, when we didn’t have that set up, there’s just a lag in communication. Now things happen a lot more naturally when everyone’s included in that conversation.