How this mobile app scales to 1M concurrent users and 900M daily API calls

Mobile user acquisition is generally expensive, hard, and time-consuming. But what do you do when it succeeds beyond all your fondest dreams of success?

You scale, scale, and re-scale your service.

And you use data to understand your users and improve your service.

Above: Grindr CTO Lukas Sliwka

That’s exactly what Grindr CTO Lukas Sliwka has done over the past several years, building a top-ranked dating app for gay men that typically manages over two million daily users on average and regularly peaks at over one million concurrent users. Every day, the service also manages over 900 million API calls, 85 million chats, 300,000 profile picture uploads, and 10,000 geospatial lookups to deliver local and contextual services

That’s not only a lot of data Sliwka has to manage and pump out; it’s a boatload of computation, as each of those users requires — and gets — a fully personalized user experience.

I asked Sliwka how he manages the massive scale.

VentureBeat: 2,000,000 daily active users sounds pretty big. How did you scale?

Sliwka: Like many fast-growing mobile businesses, we needed to manage competing forces between time-to-market vs. code quality and technical debt. The key to carrying such a heavy usage load was to strike the right balance between growing extremely fast while still implementing solid and lean engineering practices that force high-quality code and robust architecture.

Moving too fast without paying attention to quality can easily create brittle systems that accumulate more and more technical debt, bringing development to a grinding standstill. However, too much attention to quality can easily result in over-engineering and missed market opportunities. The secret sauce is investing early on in automation, continuous delivery, environment provisioning, deployment pipelines, and quality assurance.

Why is this important? A high degree of automation enables lean teams to execute quickly while yielding high engineering integrity.

In the early first iterations of Grindr, we focused too much on time-to-market, which resulted in multiple full platform rewrites (accumulated technical debt) as well as a platform that made scaling over time extremely difficult. Our latest generation Grindr platform included a major transition from Ruby-on-Rails to Scala and Java. We also successfully automated 95 percent of our regression and integration testing, and implemented Agile XP practices like BDD (behavior-driven development) and TDD (test-driven development) with over 90 percent of code coverage.

VentureBeat: You have supported 1,000,000 concurrent users? How?

Sliwka: Not easy. The answer is in two parts. First, we implemented a highly elastic and cloud-based architecture that “breathes” with traffic patterns. Secondly, we also partnered with companies who specialize in infrastructure as a service (IaaS) and software as a service (SaaS) at scale to extend our engineering team with their expertise.

This allowed our engineers to focus on real problems that have not yet been solved, while freeing resource cost and management which would have otherwise been invested in hiring and retaining experts in areas like high throughput/high concurrency caching.

VentureBeat: Which technologies are you currently using to build Grindr’s stack, and what are they used for?

Sliwka: We use a variety of cloud and open source technologies. Before I joined the team, the Grindr platform was based primarily on Ruby-on-Rails and in-house solutions that were very difficult to manage. With the release of Grindr’s latest generation platform, our stack includes a variety of open source and cloud solutions such as:

• Amazon Web Services for scalable, available, and cost-effective infrastructure
• MongooseIM stack by Erlang Solutions
• Akka for non blocking I/O
• Apache ZooKeeper for configuration management
• Amazon Elastic Beanstalk for application packaging
• cloudAMQP Rabbit MQ for back-end messaging
• Redis Labs for highly scalable caching
• Sauce Labs and Appium for functional testing
• Treasure Data for data capture, ingestion, and encryption
• Cloudflare for latency optimization
• Cloudability for AWS cost and utilization optimization
• Localytics for targeted in-app marketing messaging
• Pivotal Labs for agile XP coaching

VentureBeat: Do you use some form of engagement/push messaging/mobile marketing automation solution? How does it help you?

Sliwka: We built our own Grindr Push Notification System (GPNs) to integrate faster with our internal systems, but we also employ Localytics for targeted in-app marketing messaging. A core part of our engineering philosophy focuses our engineering talent to solve big problems.

VentureBeat: Has it improved day 3, day 7, and day 30 retention?

Sliwka: I love this question because we are investing in solving this challenge right now.

To date, our growth has been purely organic. Grindr has only recently built a marketing department (2016) and we are forming a full stack and data-driven engineering growth team that will be solely focused on improving retention and conversions.

VentureBeat: How do you segment your users? How has that benefited you?

Sliwka: Oh yes. We use machine learning to analyze user behavior with our application and then transform these datasets into persona sets that represent our userbase. This allows us to both create more meaningful experiences for our users and accurately prioritize feature roll-outs based on larger persona sets.

VentureBeat: How do you use location and geospatial data?

Sliwka: We see a strong correlation between organic growth and end-user response time.

Organic growth of our app is strongly tied to latency (response time) that our users experience around the world. I suggest to any mobile app developer to start architecting their mobile application with infrastructure latency in mind to get as close to your users as possible.

Also, geospatial data is an opportunity for end users to discover. Users want to discover the world around them, and geospatial data enables all of us to better connect to the environment around us. We use this principle to develop features that connect end users to the community, both within our app and outside of it.

Finally, there are both passive and proactive users. People use our app throughout their day-to-day routine. By providing a better matching algorithm, we can analyze exactly what kind of places they’re most interested in. For example, someone spending a lot of time in a gym may be interested in people who are also into fitness. The key is that we are determining these behavior patterns passively without disrupting our users.

VentureBeat: Do you do ASO? And are you working on new app store optimization techniques, like making your app indexable?

Sliwka: Yes, our marketing team is always working to improve visibility to our mobile apps in app stores, as well as to help us rank highly in each app store’s search results and top charts rankings. Based on various testing, we’ve learned that investing in activities that will improve our ranking in search results and top charts will drive more awareness and downloads, as well as help build on the growth we have driven to-date.

App meta-data is also a key area of focus for us in order to maximize search visibility and increase ranking among more competitive keywords. App indexing has a couple of benefits, and we take advantage of our position on key search terms that yield placements on the first page of search results by displaying an install button and autocomplete search suggestions. This increases search-to-install conversions by cutting out searching for our app directly in the app store. More involved deep-linking and app streaming previews are on the horizon, which are important to help users with high-quality decision support.

VentureBeat: If you were to start all over again from scratch, what would you do differently?

Sliwka: One of the biggest challenges that we faced early on is still one that we face today — getting talent. I would start growing the team much sooner by partnering with nearshore dev shops in Argentina and Europe. We spent too much time trying to build our team by competing for people in our local market, which were otherwise only practical to onboard as our platform and company became more mature.