Behind Cortana: How Microsoft aims to stand out with its personal assistant

SAN FRANCISCO — You might not know when you use a Windows Phone with Microsoft’s new Cortana personal digital assistant, but lots and lots of computing is happening silently to make it all work.

After listening to you speak into one of these Cortana-equipped phones, Microsoft is matching your words to the information you’ve given the phone, as well as the knowledge graph Microsoft has accumulated from the Internet.

It’s figuring out which app you want to use. And it’s guessing at the action you want to take.

But you’d never know that great computational effort and lots of processes go into operating Cortana at scale.

“We’ve done quite a bit of work in order to make the experience of interacting with Cortan and the experience of interacting with search as seamless as possible,” Mike Calcagno, partner development manager at Microsoft’s Bing division, said during an engineering talk about Cortana at Microsoft’s Build conference.

The revelation of Cortana was arguably the biggest news to come out of Microsoft’s Build conference yesterday, alongside the teasing of Windows Phone 8.1, new Nokia phones, and updates to Internet Explorer.

Cortana is a big step for Microsoft because it brings the technology giant to feature parity with Google, which runs Google Now on Android phones, and Apple, with its Siri assistant on iOS phones. And it gives Microsoft a touch of the OS in the recent movie “Her”, with its ability to carry on a back-and-forth conversation, its way of verifying if information is correct, and its sense of humor.

Today Cortana’s speech-recognition capabilities shine when you check the weather and when you searching for restaurants, but surely Microsoft will strengthen Cortana in the months and years to come.

And just as Microsoft is making it possible for developers to write Windows applications for multiple devices, it’s surely thinking about how to incorporate Cortana into more Microsoft services, right as Google is trying to push Google Now onto desktops and Apple sees cars as logical places for Siri to run.

A Windows Phone won’t come out and tell you all that it’s doing in order to make its digital assistant work, but it’s doing a lot — in some ways, more than Google Now and Siri. It taps technologies like natural-language understanding, entity understanding, stream processing, conversation management, inference systems, and Microsoft’s extensive Bing knowledge base.

Of course, Cortana also performs speech recognition, drawing on artificial intelligence technology known as deep neural networks, which mimic the brain and get smarter over time.

“We at Bing have built a tremendous set of assets and a tremendous set of technological capabilities,” Calcagno said.

“What was missing in terms of making Cortana come to life was really an experience that would actually bring that stuff together right in a cohesive way, such that whole system actually added up something that users could relate to.”

That challenge included devising with new methods of visualizing information — like weather — and changing presentation based on what users ask for so Cortana can give you better answers as it gets to know you, Bing’s Annie Wang said.

But while it makes sense for Microsoft to take its extensive data center infrastructure to perform herculean tasks and spread the work across armies of servers, Microsoft believes in the power of each individual smartphone.

Some of the data Cortana needs resides on the phones themselves, while other data surely is out somewhere else. Computations for Cortana actually use both the proveribal client (a phone) and the server (the cloud).

Another advantage of using the phone? It means answers get served up faster, because less information has to travel from the phone to a data center and back.

“Everything we can do on the device, we should do on the device,” Rob Chambers, a principal group program manager at Bing, said in a conversation with VentureBeat after the technical talk.

Indeed, he said, Cortana can perform some functions without using the Internet. This distinguishes it from Google’s and Apple’s assistants.

But at the end of the day, you don’t need to know Cortana’s complexities. It just works — all in the name of making you more productive, at first through Windows phones and gradually across more devices.

“We approached the project with, really, ‘Let’s build it as a more contextual, more personal version of search, which is geared toward a particular domain which is not knowledge, but really task completion,'” Calcagno said.