Marvel Puzzle Quest’s road to the mythical $1 ARPDAU, Part 7: Event Analytics

Albert Reed is the CEO and cofounder of Demiurge Studios. Follow the entire series here.

Here at Demiurge Studios, we do our best to run lean and flat teams. To keep everything running efficiently, we do our best to push the analytics data into the hands of the team members who are making changes on a day-to-day basis. Our partners at D3Publisher do the same on the UA side — putting the analytics data and tools directly into the hands of the people making the ad-buy decisions.

In part 3, we described our design for in-game events and how they’re tied into the economy. While we see surges in monetization at the start of events, it’s been very difficult to say with any precision exactly how good an event was for our players. We recently made our event analytics quite a bit more sophisticated, and they’ve yielded some interesting surprises to the team. In Part 7, we’ll describe the metrics we’re using and go through a couple of examples of how we’re applying them.

Designing metrics

Because events are tied into all of the game’s systems, it’s not really possible to pin down a KPI (Key Performance Indicator) that tell us if an event is good or bad. Additionally, events don’t take place in a vacuum. Their performance is muddled by the player-base’s excitement over the featured character, external factors like weekends, holidays and marketing promotions as well as overlap with other, potentially more popular events. At any given moment, we’re running at least two in-game events — enough to satiate our players’ appetite for the game. As a result, we observe that running too many events doesn’t actually increase engagement.

As a result of this complexity, we opted to pick a handful of numbers that not only attempted to pin down the event but that actually paint a picture for our designers of what what happening under the covers. We aimed to pick measurements that accurately described things that the live team could impact and avoid so-called “vanity metrics.” Here’s what we instrumented first:

• Retention: This metric is exactly the same as the standard daily retention most developers use to measure the stickiness of their applications. We look at the number of players that completed a mission in the event and then proceeded to complete a mission in the event 1 day, 2 days, etc., later.
• Currency Spend Attributable to Event: Each event has a custom card pack as well as featured characters. Players also use consumables like health packs and boosts within an event. We measure the spending by players on these items.
• Win/Loss: As a way to measure how challenging the event is for players and how well our matchmaking algorithms are working, we look at the ratio of wins to losses.
• Raw Number of Users Engaged: We want some metric for assessing how much of our player base is engaging with the event. This number is easily skewed by the number of active players in the time period, but we’re a bit hamstrung by the tools right now so we look at this value which is arguably invalidated by outside factors.
• Player and Alliance Point Distributions: In all of our events, players earn event points that unlock rewards. As described in part 3, we work hard to keep our late-joiners and lower level players engaged with events even if they can’t be completely competitive. By looking at the distribution of points in an event, we can assess the effectiveness of those tools.
• Missions Played Distribution: Number of missions played by users is one of those things that shakes out to be a power curve. How much an event pushes this effect or combats it can be described by looking at the distribution of missions completed by players.

Results example: Comparing two Versus events

Let’s look at what the numbers tell us about two PvP events that ran back-to-back:

• PvP event featuring the Dark Avengers starting Friday, April 11 to Sunday, April 13.
• PvP event featuring Captain America from Monday, April 14 to Wednesday April 16.

Neither of these events used brand-new characters. In my opinion, the Dark Avengers tournament should have been way more engaging — players need to compose a team of some of the less commonly used characters such as Magneto, Daken, and Venom, and that encourages new and interesting strategies. It also took place on a weekend when our engagement is higher in absolute terms. Totally wrong:

• Number of Users Engaged: The Captain America event had 50 percent more players
• Retention: Measuring “D1” retention for a three-day event can skew oddly. Users who join on the third day are not able to play on the final day but they are factored into our calculations. The Dark Avengers event had 41 percent D1 retention and Captain America and 45 percent D1 retention.
• Monetization: Players spent 50 percent more hero points in the Captain America event. Most of that was in comic backs.

Here’s how the distribution for each event looks over time.

Above: Hard Currency spent in the Captain America Event.

Image Credit: Demiurge Studios

Above: Hard Currency spent in the Dark Avengers Event,

Image Credit: Demiurge Studios

Results example: ‘Heroic Mode’ vs. ‘The Hunt’

One interesting angle of analysis is to compare PvE events that are very different in structure to see if we can learn anything about which events appeal to which types of players. In this example, we’ll look at two very different events:

• Heroic Chapter 3, Easy Mode, Run 1: “Heroic” events are multiphase events in which we take some of our single-player content and apply event mechanics to it. There is an “Easy” and a “Hard” version of these and each of those run twice back-to-back for 4.5 days each.
• The Hunt: This is one of our big in-game “episodes”. We’ve run it several times before. It lasts for 10 days and has a “Parent” event with long-term rewards. During the event, several S.H.I.E.L.D. bases are being attacked and players must defend them. These bases are under attack for 2-3 days each and have their own rewards. As of this writing, the event isn’t over so we’re not able to look at the points distrubtion.

Number of missions played

The first striking thing about these two events that we noticed is a huge difference in the shape of the distribution of missions played:

Above: Distribution of number of missions played in Heroic Chapter 3.

Image Credit: Demiurge Studios

Above: Distribution of number of missions played in The Hunt.

Image Credit: Demiurge Studios

We think the curve depicted by The Hunt represents a significantly more healthy game. Lots of our players are engaging with the event a lot rather than the curve shown by Heroic mode which says to us that a few players are playing much more than the majority.

Retention

Retention in The Hunt is also significantly better. If we sample users just from the first four days of The Hunt and the full run of Heroic Chapter 3, we can get a more apples-to-apples comparison. D1 retention for The Hunt was 76 percent compared to 62 percent for the Heroic Mode. Both modes had roughly the same number of participants.

For Part 8 …

Since we started this series, the card store has received another substantial overhaul. In part 8, we’ll cover the store changes and also close out the series with a discussion of how console-centric studios such as ourselves are navigating transitions to mobile and free-to-play.

As cofounder and CEO, Albert Reed has held the reins for over a decade at Demiurge Studios, a game development house focused on innovative, high-quality games for all platforms. Find him on Twitter at @almnop.