More on Powerset, the secretive search engine

(Update: See our subsequent funding story here)

We promised to bring you more on Powerset, that secretive company that wants to better Google with a new kind of search.

Powerset is going after the holy grail. It is called “natural language” search, or understanding language as it is actually spoken — and that is something that has defied everyone until now, even the Google guys.

Take, for example, if you type “Books by Children” into Google’s search box. Google essentially drops the word “by” and looks for all the pages that are relevant to “books” and “children.” That’s because the English language is so idiomatic that no engine has been able to understand meaning within sentences. Some companies, most famously Ask Jeeves, have tried. You prompted Ask Jeeves’ engine with a question ending with question mark, but as soon as your question got remotely complex, Ask Jeeves broke down • because its engine could only answer specific questions its engineers had prepped it for.

Google, while acknowledging that natural language is a big goal, hasn’t made very big advances in the area. This makes sense, because people have become trained to use “a grunting pidgin language,” as Powerset’s Barney Pell (pictured here) puts it. Pell calls this “Keyword-ese.” Many search engines recognize some advanced query syntaxes — for example to find Web pages that don’t carry certain words, or that have two words within a certain number of words of each other, and so on.

But people have a hard time remembering these advanced syntaxes, and each search engine has a different syntax. Finally, Google’s core engine has been built around this keyword-ese language, and it is hard to change all of the layers that have been built around it.

Which is where Pell and co-founder Steve Newcomb come in. We talked with them at their offices in Palo Alto. Pell is an enthusiastic guy, and has a rich career developing intelligent systems, but also trying to make them work in the market. Pell has just posted about the start-up on his blog here. Newcomb is the operations guys, previously having worked at voice recognition company Promptu.

Search is so crucial in our lives, Pell says, it is like oxygen. “It is a metabolic function,” Pell says. And yet, search is surprisingly underdeveloped.

Powerset is trying to solve the natural language problem, by making its core engine understand concepts of time, place, sentiment and other intent. But Pell and Newcomb stop short of going into the details of their computational linguistics approach, saying it is sensitive. They are also giving no dates about when it will be released.

They insist, however, that it is a radical improvement. So when Craig Silverstein, first employee of Google said it will take many years to get a computer to a point to understand exactly what people are searching for, Powerset thinks differently: “It is not a long way away,” says Pell. “This is not a change of some technology out on the periphery,” adds Newcomb, “we’re changing the core of the engine.”

He says such a transformation hasn’t happened in eight years, since Google invented “page rank,” a concept that ranks a page higher in relevance depending on how many people are linking to it. So while Google has bought companies like Applied Semantics to help find “themes” on web pages, Newcomb shakes his head, and says Powerset wants to do much more. “We’re switching the core out,” he says, adding that when you do that, you’re also going to fundamentally change the image, video, blog and all other searches that Google is doing. Google may be caught in an “innovators dilemma,” co-founder Steve Newcomb says, because it can’t turn on dime.

So, is this so much hype?

They point to the credibility of their investors, who include PayPal co-founder Peter Thiel, analyst Esther Dyson, Reid Hoffman and the Amidzad fund.

And when you talk with Pell, you get a sense for why he may be as prepared as anyone to do this. While an undergraduate at Stanford, he spent much of his time working at SRI’s “natural language group.” There, back in 1988, he began working on the problem that restricts search engines from relying on natural language: In natural language systems, you have to teach the system every single word. If the system, doesn’t know a word, it crashes. So Barney went to work taking the words the system did know, and bridged the gaps formed by the words it didn’t know, which he said created better results. Later he built a language engine that talked with a office processing system — this way, a company could ask the system say, “What were the top five products ordered over the past week?” and the system could spit the results back in the form of a table. His PhD was in machine learning and games.

Then he worked at NASA, and architected the artificial intelligence system that was embedded in the $200 million Deep Space 1 mission. For a week, his system operated the mission autonomously. In 1999, he left and joined the internet revolution, working with Stockmaster.com, and then at Whizbang Labs, which used machine learning and statistical natural language processing to build advanced search applications. His company built Flipdog, a site that extracted job listing information from millions of sites on the Web • which was sold to Monster.com.

Later, exhausted by the post-bubble era, Pell returned to NASA and managed an 80-person operating division deploying information technology for NASA’s missions. When the market recovered, Pell did a brief stint at Silicon Valley venture firm Mayfield, which he left to start Powerset.

On the one hand, Pell and Newcomb have detractors. “We do have a lot of skeptics,” Pell said. But he and Newcomb also believe they are on to something: “We show the demo to people, and their jaws drop, and they say ‘Holy (expletive),'” says Newcomb. “They say: ‘I’ll never use the old search engines again!'”

Update: See search expert Danny Sullivan’s scathing critique of Powerset ambitions.