The inside story of how Fluenty made their smart reply function

Fluenty is a chatbot that makes you sound like James Bond is writing your auto replies. The team began building its Smart Reply systems in 2014. By the summer of 2015, the app Fluenty was able to respond to SMS message, paving the path toward support for a total of four external messaging platforms. And so far we’ve seen 60,000 downloads and 13,000 active users.

Learning about deep learning

Our team first observed the advances of deep recurrent neural network technology while working as engineers for major search engines Naver and Daum in South Korea. This technology allows devices to interpret the meanings of phrases and learn from previous entries by deciding what information to remember and (like our aging selves) what to forget.

We were fascinated by this technology because we felt that it could revolutionize the world of mobile apps. We envisioned a world where chatbots could respond to messages in a humanlike manner after processing enough conversational data. At the time, however, the technology was far from perfect and our vision far from complete.

Our team realized early on that we had to create bots that were able to “think” for themselves and generate relevant responses to incoming queries in order for our vision to become a reality. This was a difficult task to say the least, but we were motivated by our love of technology. For this reason we studied recurrent neural networks (RNN) and long short term memory (LSTM) systems so that we could address the challenges associated with natural language technology systems.

The rise of smartwatches

While two of us were trying to figure out the best way to make chatbots more useful, our third cofounder, Sungjae Hwang, was designing concepts for smartwatches. He was also developing app support for his creations.

In 2014, he completed roughly 10 designs, almost a year before the release of the Apple Watch on April 24, 2015. While working on his project he realized that a major weakness of smartwatches is their small input interface. Smartwatches delivered lots of notifications from their connected phones, but had little ability to interact with the messages it received.

After Apple announced their smartwatch, we began to shift the direction of our research. We saw the positive attitudes the general public had toward the Apple Watch and decided to focus on creating apps that supported smartwatch devices. We figured that a chatbot optimized to generate human-sounding replies would make a killer app for communicating using a smartwatch.

Above: Sungjae Hwang’s smartwatches circa 2014

Image Credit: Fluenty

Prediction experiments

Thanks to Google Inbox’s Smart Reply function, many people have become familiar with automatically generated reply systems. In 2014, however, such technology was incomplete. Most people weren’t even aware of just how repetitive most of their daily interactions were.

We tested that predictability by conducting a test. Our test sample included 50 heavy WhatsApp users, ages 20 and 30. We asked them to give us their chat logs from conversations with people they frequently talk to. Miraculously, they agreed.

Our team’s next step was to read the dumped chats line-by-line and to label each chat according to how predictable the responses were. For instance, when we saw “How are you?”, we could easily assume that the replies would be something like “I’m fine” or “Great!”. However, when we saw messages like “Our group one lol”, we did not have enough context to predict what the receiver would type next.

The results of our research were pretty impressive. Of all the chats, more than 30 percent of messages were predictable, with a limited set of corresponding responses. Now it was time for us to use what we learned to make an app capable of consistently predicting replies.

Building smart reply functionality

We are not Facebook Messenger or WhatsApp. When we started, we did not have a massive amount of conversational data to train our model. Instead, we decided to collect public conversational data on social media, gathering 200 million conversation pairs.

As Voxable cofounder Matt Buck wrote, “Until recently, building advanced natural language processing into a product was limited to only the largest companies.” In the past, few companies could afford to hire researchers and engineers to create reliable test models. Fortunately for us, our previous engineering experience allowed us to develop our own models instead of hiring outside firms.

We developed vector embedding and deep recurrent neural network models, and then we fine-tuned them to enhance the relevancy of suggested replies and reduce the system’s calculation time. Afterward, we began developing our Android app.

Testing the smartwatch app

Soon after we released our open beta version Android app in August 2015, Google Play featured us in its Android Wear section. After a month, our smartwatch app had been downloaded 3,000 times. Soon we began to monitor the habits (not conversations) of our users and consider their feedback. After looking over our findings, we arrived at the following conclusions:

• Users found the “suggested replies” feature helpful when using their smartwatch.
• Users had problem with delayed notifications. Depending on the complexity of the calculation and the user’s network connection status it could take up to seven seconds for the app to deliver notifications.
• More third-party messengers should be supported by our app. Many users said that they liked Talkey (the original name of Fluenty), but they felt limited by the app because they did not use SMS often.
• Users wanted more “personalized” responses.
• The app’s layout needed to be improved.
• We also received requests to build a smartphone version of the Talkey app.

Following the users’ feedback, we enhanced our model to complete all processes faster. Now the smartwatch app is able to send and receive messages in less than 50 milliseconds at most. Our team was skeptical when our users asked for a mobile version, but we created one anyway because the modifications were easy and we had nothing to lose.

After releasing the mobile version of Talkey, our Google Analytics data revealed that users who used smartwatches sent replies on their mobile phone more than 80 percent of the time. I was shocked. We thought the easy-to-use keyboards of modern smartphones made an automatic message generator unnecessary.

What lies ahead

Since having that eureka moment, we have focused on developing support for the smartphone version of our app. In order to reflect our new vision, we changed the app’s name from Talkey to Fluenty, believing that this product could be bigger than a keyboard app. We envision our product improving the flow of communication between humans and machines in lots of applications.

Most recently, we have added support for Kakao, Facebook Messenger, WhatsApp, and Google Hangouts. We are also developing more support for other applications as well. We believe the Fluenty app has the potential to become a predictive Smart Notification tool that will change how people think about and approach digital communication.

We are currently developing ways of implementing our API technology within customer service automation and cars. We want our services to help customer service representatives answer routine questions with a single click and allow drivers to send messages and search address while keeping their eyes on the road.

We are also experimenting with incorporating Fluenty to assistive devices and VR headsets, so people can send messages by looking at the response item they would like to send. With the current advancement of API and  DNN technology, we believe that anything is possible. Fluenty plans to take the use of API technology to the next level and beyond.