The Talos Principle underwent 15,000 hours of playtesting — but not by humans (interview)

In Croteam’s new philosophical puzzle-adventure title, it apparently takes a robot to fully test one in-game.

The developers of the Serious Sam shooter series have turned their talents to The Talos Principle, a game that puts you in the metal body of a robot with a seemingly human consciousness, attempting to determine your role and the fate of the human race.

It’s fitting, in this game that makes you a human playing a self-aware robot, that almost all the playtesting — some 15,000 hours’ worth — was done by a bot pretending to be human. It did so much testing that the developers started referring to it as “him” and gave it a pet name: Bot.

(OK, we didn’t say it was a creative pet name.)

Bot had a lot to test. As you solve the puzzles, you’ll collect information to figure out your current role and what happened to the human race. The voice of a self-proclaimed deity guides you, though his motivations are suspect. The game takes easily north of 20 hours, with completists clocking in around 30.

You move and stack and manipulate objects — electromagnetic jammers, crystalline refractors, fans, computer terminals — in an open world of beautiful, deserted settings.

If you arrange and use them correctly, you’ll unlock gates, avoid roving mines, refract lasers, read and listen to snippets of information, and get sigils (which look suspiciously like giant Tetris pieces). Each collection of pieces, when assembled properly, moves you along closer to your goal of self-discovery.

The Talos Principle releases for $40 from developer Croteam and publisher Devolver Digital on Dec. 11 for PC, Mac, and Linux. PlayStation 4 and Android versions will launch early next year. A free tech demo will launch Thursday on Steam.

I played through about 40 percent of the game on PC and spoke with Alen Ladavac, the chief technology officer at Croteam, about its construction and testing.

GamesBeat: What inspired the game’s story?

Alen Ladavac: Most of us are science- and tech-loving folks, often debating about visions of the future and how humankind will cope with rapid advances in technology.

We feel the story dances with both the glorious past of humankind and some issues that will become important in the near future. It means a lot to us, and we hope that it will make people think about life and what it means to be a human.

GamesBeat: What led you to do a puzzle game, so different from your previous titles?

Ladavac: We didn’t plan it! It happened spontaneously. The idea to do a different [intellectual property] had been cooking for a long time. As we worked on the next Serious Sam game, our team experimented with some new gameplay involving devices. Two team members created a series of test levels for our teammates to sample.

I created levels of complexity that fit into the Serious Sam universe, but my colleague didn’t stop there. He also created several very demanding levels to test the maximum complexity that could be achieved within our new gameplay concepts.

When the team tested the “puzzles,” everybody loved it. Then we started brainstorming about possible game structure, new mechanics, and story. We continued to discuss it, and soon we saw a bigger picture — and we really liked where it was leading us.

GamesBeat: What led you to build the testing bot?

Ladavac: We discussed the possibility way back while we were working on Serious Sam HD, but we couldn’t devote enough time to implement it then.

In the period between the release of Serious Sam 3: BFE on PC and the start of The Talos Principle, while we were porting Serious Sam 3 to Xbox 360 and PlayStation 3, Nathan Brown joined our team. Since he had significant previous experience on bot programming, he took over the bot work, and pretty soon had a working bot for Serious Sam 3. But the process was still partially manual.

When we started working on The Talos Principle, it became obvious early on that the puzzle gameplay is susceptible to accidental errors made while tuning or decorating levels, which broke it in such a way that you couldn’t finish the game.

Since the game is pretty big, it would have taken an enormous effort to retest the entire game after each build, especially if humans had to test it. So the idea of the bot was reborn.

GamesBeat: How does it work?

Ladavac: It works by having a human player play each puzzle once while the bot “watches” and records the crucial steps. It doesn’t record precise movements, but just the logic of the solution — like “take this box, put in on that pressure plate, then flip that switch, then connect this power source to that door, [and so on].”

The bot can then later repeat the logic by dynamically handling eventual changes in the level geometry, obstacles, and placement of puzzle objects.

GamesBeat: What does it test?

Ladavac:  The bot replicates the gameplay played by human and uses game mechanics to solve puzzles, just like a human player would. When the bot notices a problem, it automatically reports a bug using our in-game bug reporting system, then uses cheats to move on and continues testing further.

Thus, in one “playthrough,” the bot can report several possibly game-breaking bugs and give us a report of overall health of the game as it is in that moment.

GamesBeat: How long does the bot take to finish the game?

Ladavac: The bot plays in fast-forward mode: It doesn’t respect real-time synchronizations but plays as fast as the CPU can calculate physics and A.I. It used to take him — yeah, we started referring to it as “he” — about 60 minutes, which we later optimized to 30 minutes by optimizing the A.I, to finish the complete game with all the secrets. So we can get several complete playthroughs of the game per day.

GamesBeat: How many games has it played so far?

Ladavac: We estimate that it has already performed about 15,000 hours’ worth of playtesting, or about 90 work months.

GamesBeat: Are you doing traditional human testing as well? How many hours have those players logged?

Ladavac: Yes, but those testers concentrate on other things now, like visuals, exploits, and narrative. Some of the testers and developers have logged a few hundreds of hours in the game, in the Steam version alone, and their input is very valuable to us. There are limits to what the bot can do and things only real testers can report.

GamesBeat: Was there anything that surprised you about what the bot unearthed?

Ladavac: Since Bot is doing everything that a player would do, there’s not a lot of surprises in his reports. He basically unearthed who in our team is clumsiest since he acts like a tattletale each time he plays through the game. [Ladavac smiles.]

GamesBeat: What did the bot help you accomplish?

Ladavac: The most important thing Bot saved us was time. During development, we often had to ask things like, “If I do this fix or add a feature now, when can I be sure that it didn’t break something?” Previously, the answer was, “If we start testing now, we will know in a few days.”

So we often had to do changes blind and batch them — and we only learned if something was broken when the next batch was tested. That led to a lot of unpredictability and uncertainty in the project, especially around crucial deadlines.

Now we can just submit the change and have the answer back in an hour or two after the build is done and tested. If it is critical, the person who made the change can run the bot on his or her own machine, and in half an hour, they will know if there are any problems with it. This is something that is just impossible for a human to do, no matter how much you pay them.

GamesBeat: Any chance you’d use it again for future games?

Ladavac: Absolutely. We will certainly use it for all future titles.