Cortana on Windows 10: Your PC gets a personal assistant

Windows 10 launches today with a lot of new features (read our review). One of the most important, unique, and frankly intriguing features is Cortana, Microsoft’s digital assistant.

To be clear, Cortana isn’t new. She first arrived with Windows Phone 8.1 in April 2014. Yet aside from her birthday, today is arguably her most important day yet. Cortana has arrived on the PC, the market Microsoft has dominated for decades.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

If you look at the desktop landscape right now, operating systems don’t really have too many differentiating features. Apple has yet to port Siri to OS X, leaving the door open for Cortana to be Windows 10’s killer feature.

So the question is: How good is Cortana? It’s not a simple question to answer because of Cortana’s nature: She’s a personal assistant, and that means she takes some time to set up and get used to, she is continuously improving, and she is constantly learning more about you.

In other words, you’re not going to immediately love or hate Cortana. You will either end up using her more and more, and she will grow on you, or you’ll find yourself not using her at all.

This is an overview of Windows 10’s most personal feature. Microsoft is betting a lot on Cortana, and we had the opportunity to talk to Susan Hendrich, principal program manager lead on the Windows team, to learn more about what exactly Microsoft’s intelligent assistant can (and can’t) do.

Customizability

Cortana is a big deal, but don’t just take my word for it. You can tell Microsoft is excited about Cortana given where the company has placed her: When you first launch Windows 10, Cortana is available right there, in the bottom-left hand corner.

She’s right beside the Start Menu, the most trafficked area of Windows. She’s also heavily integrated in Microsoft Edge, the company’s new browser that will finally kill IE. When you first open her up, you’ll go through a quick interview process.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

After that, you can change what she knows about you by adding and editing entries in her notebook (including stocks, news, sport scores, travel information, weather, food preferences, and so on). Every time you open Cortana again, she will give you a greeting and list the most relevant information for your day.

For example, if you update your notebook to say that you like French food, Cortana’s restaurant recommendations will immediately update to reflect this new fact she has learned about you. This is pretty basic, since you’re explicitly adding this customization.

But Cortana is also capable of figuring out that you might like to track something without you explicitly telling her. If you search for a stock often enough, or a band, or a company, she may ask if you want her to keep a tab on those for you.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

There’s quite a bit of room for improvement though. Cortana can figure out that you’re interested in a movie and recommend show times if you’re near a theatre that’s playing it. But she can’t yet stop reminding you about that movie without you explicitly telling her: If you sat in a theatre for two hours after she recommended going to a specific cinema, if you bought tickets to the movie and they’re sitting in your email account, or if you shared your thoughts about the movie on Facebook and Twitter. Cortana is smart, but she’s not that smart, yet.

Like any personal assistant, Cortana can answer your questions, book appointments, set alarms, keep track of your meetings, and remind you about upcoming events. Reminders are probably the most useful feature of them all.

These can be related to people, to location, or even time-based. For example, you can get Cortana to tell you to buy groceries when you’re done working.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

Individual notifications can be snoozed (ping me later) or dismissed (don’t ping me again). You can also turn all notifications on or off – but, you can’t turn specific types of notifications on or off.

Reminders can fire on any platform, meaning if you have Cortana on a Windows 10 PC and on an Android phone, you’ll get multiple notifications for the same thing. That said, if you dismiss the notification on one platform, it won’t fire on the other. You have to be quick!

You can also set duplicate reminders. If you set a reminder to go to the dentist tomorrow, Cortana isn’t smart enough yet to tell you that setting a second one would be redundant. Instead, she’ll happily add it and send you two notifications the next day for the exact same thing.

Voice

You can type your requests to Cortana and she can display her answers as text back to you. But Cortana really shines when you speak to her and she speaks back.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

To speak to her, you’ll need a decent microphone. You’ll also need to be in a quiet place, especially if you want to be able to query her by saying “Hey Cortana.”

“This type of technology is specifically created so that you can’t say ‘hey Cortana’ and my computer will respond to you,” Hendrich told VentureBeat. “Things like I’m speaking while you’re speaking pretty loudly in the background, that’s still going to be confusing to Cortana.”

In Cortana’s settings, there’s an option for her to learn your voice. Still, even if you go through the training exercise, she still may have difficulty understanding what you’re saying if you work in a loud environment.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1775758,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"business,mobile,","session":"D"}']

“Not because of a voice identity recognition issue but because it’s going to be difficult for her to do that speech recognition with those different layering of sounds in the background,” Hendrich explained. “And it really just depends on how much sound is in the background in terms of whether she gets confused by it or not.”

If you play music in the background, that’s usually okay for Cortana to disregard. The same goes for some quiet background noise. Once you throw other speech into the equation, it can get messy.

Yet Microsoft’s biggest focus in terms of voice recognition is “to invest in understanding different accents,” Hendrich emphasized. This makes sense, as comprehension full stop is more important than interpretation over various noises.

The more people use Cortana, the better she will get at understanding what people are saying. Speech recognition is “really just about getting more data” and Hendrich is confident Cortana will improve as Windows 10 rolls out across the world.

In the U.S., Cortana is voiced by American actress Jen Taylor. If you’re a Halo fan, you’ll recognize her as the same voice as Cortana in the Halo games (pictured at the top).

In many cases, Cortana’s voice is not digitized at all. It’s really Taylor’s voice or whoever else owns Cortana’s voice in a given country. If, however, there is no time to get a new answer recorded, Microsoft can have Cortana read out her response using her digitally-generated voice font, which is based on the actor’s or actress’s original voice.

You won’t be able to hear a huge difference between these two “voices” unless you listen carefully: The digitally-generated version has less pitch variation, fewer ups and downs.

If you want to hear the difference, ask general questions like “what is the square root of 1337?” and compare the answers to queries like “who is your daddy?” that have specific responses. Again, it won’t be too obvious unless you’re specifically looking for it.

Oh, and because Cortana is constantly getting smarter, she’s also getting new answers. We asked Hendrich how often Taylor is at Microsoft and were told she “comes in a couple times a month for new recordings.” That’s pretty sweet.

Personality

To say Cortana is a simple program would be a massive understatement. Hendrich says Cortana has a 24-page personality brief.

Furthermore, Cortana has an army of writers constantly adding new responses to her arsenal. Microsoft spent quite some time trying to figure out just what kind of humor Cortana should have, and ended up settling on witty humor rather than slapstick-funny humor.

Hendrich says Cortana’s writing team is not composed of purely technical writers like Microsoft would have used in the past, and there are individuals dedicated to each and every market where Cortana is available. The group includes playwrights, novelists, and overall “extremely creative” writers.

“That’s what we’ve been really focusing on really deeply over the last year: Making sure we have an extremely culturally-relatable Cortana in each of the target countries,” Hendrich told VentureBeat. Sometimes, the answers get very specific.

The writer managing Cortana in Canada, for example, told Microsoft that almost every kid wants to be a Zamboni driver. So if you ask Canadian Cortana what her favourite car is, she’ll tell you it’s a Zamboni.

(Note: I grew up in Toronto and I have never heard of anyone in Canada wanting to be a Zamboni driver. But that’s the tricky part of creating a digital assistant: Regardless of how many custom responses you add, you can’t please everyone.)

Indeed, the editorial team also makes decisions that impact Cortana in ways that may not make sense for everyone. For example, Cortana can suggest for you to set an alarm, and the default is set for 5:45 a.m. That’s not Microsoft using data to pick an average wake up time: Hendrich said that the person who manages these writers wakes up that early, and he chose that insane time as the default suggestion.

We already talked about Cortana’s voice, and how it’s Jen Taylor in the U.S. For many of the other countries, Microsoft went in and did user studies asking what people want in a personal assistant when it comes to age, gender, qualities, and so on.

For example, Hendrich said people in China told Microsoft they want a voice that sounds like it’s smiling and acts like an older sister. In the U.K., people said they wanted someone who sounds “more efficient.”

Voice aside, Microsoft also spent a lot of time crafting how Cortana looks. Even though she’s a circle, she still reacts: when Cortana is sorry, she looks sad (almost as if her shoulders are slumped forward) and when Cortana is excited, she’s jumping up and down (like when an alarm goes off and you really have to get moving).

“For the visualization, we’ve specifically chosen not to go with a facial representation and for now we’re going with a circle,” Hendrich explained. “Going with a facial visualization has a set of interesting experience complexities that we’ve been thinking about, but we really feel like for now, we can communicate emotion and have microexpressions through a circle.”

When I pointed out she said “for now” multiple times, Hendrich noted that in China, Cortana won’t be a circle. Instead, the company’s personal assistant is a cute little character with blinking eyes. Microsoft is experimenting in China because it feels like the culture is different enough there to warrant a drastic new look.

What’s really interesting about Cortana is that she doesn’t change based on your physical location, but rather your settings. “If a user travels to another country they will still have the Cortana from their original country,” a Microsoft spokesperson explained. “If they ask for local suggestions, e.g. restaurants, they will get whatever their Cortana knows, which might be slightly incomplete because their Cortana is not optimized for that particular country.”

In other words, if you want the Cortana in France, all you have to do is change some settings. Specifically, change your region, display language, and speech language to France. As long as Cortana is available in the country you pick, and that country matches across all three settings, you’ll be good to go.

Microsoft has not made any indication it will be blocking language or regional changes to get Cortana. So if you tire of your version of Cortana, you should be able to just swap her out.

Freshness

A lot of Cortana’s answers are powered by Bing, and so she’s constantly getting new information and features. In fact, this is probably her biggest advantage: Bing is constantly learning about the world, and Cortana can tap into that.

“Cortana and Bing share so many different technology stacks, for example speech recognition and instant answers,” Hendrich told us. “Any time Bing makes improvements in those areas, Cortana’s going to instantly see that improvement as well. One of things that’s so great in having that close relationship with Bing: Truly and utterly Cortana is improving weekly.”

In short, the two are very closely related. That said, the “instantly” part isn’t exactly true.

The writing team has the ability to add new responses based on Cortana’s logs, which can highlight trending questions. In other words, Microsoft can see what people are asking Cortana, and which queries are not getting suitable responses.

In fact, one of Cortana’s newest features was built thanks to the logs. Many were asking her “What’s up?” and Microsoft decided to take the opportunity and build a “random fact of the day” answer. Now, the writing team writes a new little tidbit for every single day.

If something very important is happening and users are asking about it, the Cortana team can add new answers to questions “really quickly,” Hendrich says. In fact, going forward Microsoft plans to introduce capabilities on Bing and Cortana simultaneously, so that both products can learn from each other. Until now, capabilities started on Bing and the Windows team would decide whether to bring them to Cortana (which would require some extra work, like language generation).

Final thoughts

Cortana is not the first digital assistant, and she won’t be the last. Yet she is the first digital assistant on the PC that matters. She is going to crush all the poor attempts that have been made so far, and she will also pave the road for alternatives.

Digital assistants are not a new idea by any stretch, but now they’re finally becoming powerful enough to actually be useful. If Cortana succeeds on the PC, you can expect she will soon see competition.

The real test for Cortana will of course be how much time and resources Microsoft dedicates to keeping her fresh. If she can gain capabilities as quickly as Google Now and Siri, Microsoft will be lending a helping hand to hundreds of millions of desktop users.