Always Be Testing: Why Groupon lives by this mantra

As mobile becomes more and more ubiquitous in e-commerce, there’s one simple rule: If your app is not evolving, neither is your business. At Groupon, we live by the ABT mantra — Always Be Testing — to ensure we’re putting the very best product in front of our customers. Over the last six years, we’ve grown from an email-based business to having more than half of our transactions on mobile, all on an app that’s been downloaded more than 110 million times around the world.

We approached mobile from the beginning as a way to experiment with new products and new types of engagement models with consumers. Consistent and iterative mobile testing is the crux of how we keep the app fresh and effective. We launch 10-15 experiments for each mobile release and use a rigorous A/B testing model to help us evolve the app.

Buy or build

Today there are a good number of tracking and A/B test tools for mobile, but three years ago when we started to experiment, there were no standard tools. We had to build our own mobile tracking tool from the ground up. It’s called NST which stands for Non-Sucky-Tracking.

Building is clearly more labor intensive, but it’s allowed us to control our own testing destiny and create a process that’s effective at scale and tailored specifically to our own app. For those with less complex testing needs, an off-the-shelf testing tool is perfectly workable, particularly in the early stages.

The important thing is to test, track, and evolve your app out of the gate.

Measuring success

When we define experiments, we use the “Define Success, Measure and Iterate” approach.

For each test we define success metrics — increased engagement or conversion are our primary desired outcomes. We collect usage data in the app and upload it to our Hadoop cluster and then process the data and transfer it to our Enterprise Data Warehouse. Finally, we use Tableau as our visualization tool for each experiment.

This process allows us to look at experiment results every day and make a decision on which test to ramp up or down. The beauty of operating these tests at Groupon’s scale is that we can get results and learn much more quickly — often in a week or less — and iterate fast on next releases.

Fast iteration at scale gives you a true competitive advantage. That is why we invested from the beginning in tracking and A/B test tools. You should, too. Buy it or build it, but Always Be Testing.

Sri Viswanath, SVP of Engineering and Operations at Groupon, is joining other senior execs at Mobile Summit today and participating in a boardroom session on personalized marketing.