To really get smart, Apple’s Siri may need a human touch

In a recent study by Butler University, Apple’s iOS personal assistant Siri scored only 16.3 on an intelligence index out of 100 and ranked below Bing, Yahoo, Google, WolframAlpha … and answers engine ChaCha.

“If the metric is providing users with a high quality answer within seconds, Siri’s at the bottom of the list,” ChaCha CEO Scott Jones told me recently. “From my perspective, Siri needs to be able to answer the kind of questions that we ask when we’re driving or in the mall … the kinds of questions do people really ask, as opposed to giving you search results, just like a search engine.”

Apple recently updated Siri, as it announced at its recent WWDC conference, adding integrations with Wikipedia, switching the default search engine to Bing, and enabling more control of your phone. And an earlier study by Piper Jaffray found Siri capable of understanding and answering much more. But still, Jones says, Siri can’t answer the kinds of questions people would normally ask others.

And he says it may never get there.

“Speech recognition is easy,” Jones says. “The hard problem is figuring out what is really being asked for … and humans do that really well.”

That’s why ChaCha, which has answered 2.5 billion questions in seven years of operation, uses humans for some of its search results. The service uses algorithms as well — 80 percent of its answers are automated — but it also uses real human beings to answer some of the tougher, more qualitative questions. Those 80 percent include things like sports scores, train schedules, and other data that anyone can buy feeds for.

But that automated 80 percent also includes answers from real-live humans.

“Maybe 60 percent of the 80 percent are answers that were at one point provided by a human but can be reserved,” Jones says. “We now have 129 million question-answer pairs in our database.”

Those are answers to questions like what’s the best Greek restaurant in Soho, how to cook a turkey, or what’s the best local spot for Thai food. ChaCha crowdsources those answers, stores them, and serves them as needed.

“We find other people who have passion points around the thing you’re asking about,” Jones told me. “It’s a scale problem to do that right, and the magic that ChaCha brings to the equation is the database and automation so that they get a rewarding experience nine times out of 10 so they come back.”

If there are no answers, ChaCha defaults back to a human guide who can help find the needed answer, which may take a bit of time. But the answer will be eventually stored in the database as one of those millions of question-answer pairs so that over time, the service’s real-time answers deliver an increasingly better experience.

That database of questions and answers means that ChaCha scored higher than any of the other services in the study. ChaCha sponsored the study, so it’s worth taking with a grain of salt, but it was an independent study by Butler University. The real problem with the study lies in the question source, which includes both ChaCha and Iris, the Android equivalent of Siri. Study questions were drawn from real-life questions submitted by real people, which is good, but that some of them came from ChaCha means that they were likely to be included in ChaCha’s question-answer database.

That’s kind of like cheating on the test, I would think.

On the other hand, they are real-world questions, the kind that we ask and Siri doesn’t answer well — like whether Target is open right now. Siri doesn’t know and won’t guess, but ChaCha checks your location, the time, and answers: yes. That’s the kind of thing that ChaCha, which Jones calls a “hybrid intelligence” that blends machine learning, data, and human knowledge, can answer but Siri cannot. And, he says, that helps ChaCha be able to answer “everything” … a tall order.

ChaCha’s not there yet, of course, but Jones also says that no one — including Google — is good enough yet.

“If you’re not at nine times out of 10, consumers won’t think of you as the place to go. We’re still not good enough.”