GamesBeat: How long does the bot take to finish the game?
Ladavac: The bot plays in fast-forward mode: It doesn’t respect real-time synchronizations but plays as fast as the CPU can calculate physics and A.I. It used to take him — yeah, we started referring to it as “he” — about 60 minutes, which we later optimized to 30 minutes by optimizing the A.I, to finish the complete game with all the secrets. So we can get several complete playthroughs of the game per day.
GamesBeat: How many games has it played so far?
Ladavac: We estimate that it has already performed about 15,000 hours’ worth of playtesting, or about 90 work months.
Above: With the bot busy testing gameplay, humans tested beautiful visuals like these.
GamesBeat: Are you doing traditional human testing as well? How many hours have those players logged?
Ladavac: Yes, but those testers concentrate on other things now, like visuals, exploits, and narrative. Some of the testers and developers have logged a few hundreds of hours in the game, in the Steam version alone, and their input is very valuable to us. There are limits to what the bot can do and things only real testers can report.
GamesBeat: Was there anything that surprised you about what the bot unearthed?
Ladavac: Since Bot is doing everything that a player would do, there’s not a lot of surprises in his reports. He basically unearthed who in our team is clumsiest since he acts like a tattletale each time he plays through the game. [Ladavac smiles.]
Above: So many worlds to explore, so many ruins.
GamesBeat: What did the bot help you accomplish?
Ladavac: The most important thing Bot saved us was time. During development, we often had to ask things like, “If I do this fix or add a feature now, when can I be sure that it didn’t break something?” Previously, the answer was, “If we start testing now, we will know in a few days.”
So we often had to do changes blind and batch them — and we only learned if something was broken when the next batch was tested. That led to a lot of unpredictability and uncertainty in the project, especially around crucial deadlines.
Now we can just submit the change and have the answer back in an hour or two after the build is done and tested. If it is critical, the person who made the change can run the bot on his or her own machine, and in half an hour, they will know if there are any problems with it. This is something that is just impossible for a human to do, no matter how much you pay them.
GamesBeat: Any chance you’d use it again for future games?
Ladavac: Absolutely. We will certainly use it for all future titles.