Real-time search startups blow away Google, Yahoo, and Bing at SMX West

This week at Search Marketing Expo (SMX) West in Santa Clara, I sat in on two panels on realtime search: one with representatives from Google, Yahoo, and Bing, and the other with four real-time search startups: Collecta, CrowdEye, OneRiot, and Topsy. It was readily apparent that real-time search is a very different problem to solve if you’re a major search engine than if you’re a start-up focusing on ways for users to get value out of it.

I admit, when I first heard about real-time search, I thought “who cares?” So I get to see a stream of unedited tweets and status updates going by? How is this going to be useful?

That’s exactly the question that the major search engines had to ponder while embarking on the project of bringing the live web to their search engine results pages. Which queries to show results on? Which sources to include? How to filter out spam? How to determine relevancy? Dylan Casey, Product Manager at Google said there were “huge debates” internally about the usefulness of real-time results. Sean Suchter, General Manager of Microsoft’s Search Technology Center, Silicon Valley, said, “our goal is to get user value out of it.” Similarly, the presentations from each panelist mostly rehashed the value and potential use cases for real time search, and discussed the technical approaches to dealing with a the massive amount of real-time data.

None of the major engines are focusing on how to monetize real-time search results, at least not yet. Casey mentioned that at Google there is a “solid separation between church and state,” i.e., product development and monetization.

Monetization is an angle that one real-time search startup, OneRiot, has tackled with some success. In the following conference session, Tobias Peggs, General Manager at OneRiot, explained how with RiotWise, OneRiot is showing ads from the Huffington Post for articles that match real-time search queries. OneRiot also has a wide distribution network, saying that 97% of search volume comes through partner sites via its API.

Distribution is a big issue for real-time search startups, especially now that the major engines are involved. After seeing these presentations at SMX, I don’t agree that Google has crushed real-time search for startups. What the real-time startups are doing is more sophisticated in some respects than Google or Bing. And Twitter just gave them a push by allowing access to its full firehose of data. Each of the startups presenting had a different, and relevant approach to real-time search. They are not expecting to compete with Google — Gerry Campbell, CEO of Collecta, said, “it’s foolish” for any real-time search company to believe it can become a destination site and not be focused on finding distribution partners. Collecta is hoping that bloggers themselves become distribution partners by offering a customizable widget.

Ken Moss, co-founder of CrowdEye and a 20-year Microsoft veteran said “startups can create and explore opportunities that no one else would have done.” Indeed, CrowdEye, which currently only sources from Twitter, uses its CrowdRank algorithms to display tweets and links by relevance. It also allows users to filter results by time span and offers a chart of hourly tweet volume over the past 3 days.

Filtering is taken to another level by Topsy and Collecta. Collecta allows filtering by source: status updates, photos, blog posts and articles, blog comments, and videos. Topsy not only filters by source — web, photos and tweets — but also by time and relevance. Because it indexes data, it allows you to see popular tweets from the past, such as the Twitpic of the plane crash in the Hudson River over a year ago.

The full range of use cases for real-time search has not yet emerged, and I’m betting that one of these startups will see it before Google or Bing does. To quote Ken Moss, “it’s super early to tell what’s going to be successful and in what way.”