Real-time search: How a whole ecosystem bit the dust

Yet another real-time search engine has called it a day, causing many to ponder how a white-hot industry fizzled so entirely in just two years.

Last week, startup Wowd announced its team was “moving on.” Many read this announcement as one of the darker stories of the entrepreneurial world: the kind of failure that entails giving investors their money back, open-sourcing the code and returning to corporate life.

In this case, however, this young company was just one of a string of real-time search engines, once a sexy and competitive field, to pivot to another sector or close shop entirely.

“From the business side,” said OneRiot CEO Tobias Peggs, who was one of the entrepreneurs working in this field, “assuming you have world class engineers to build the product (which we did), you still need great business development and a lot of money to make a go of it. And even then, it’s hard.”

Yesteryear’s crop of seemingly healthy and promising real-time search companies first sprang up to fill a very real need. In 2008, a socially connected web of real-time communication was exploding, and users needed a way to parse and search that information. In response, highly innovative companies such as Summize, OneRiot, Collecta, Wowd, Ellerdale, Crowdeye and a few others began presenting a range of options for making sense of real-time data.

Gerry Campbell, who served as an advisor and investor for Summize and, later, CEO of Collecta, spoke to VentureBeat, saying, “The Twitter firehose as a dataset sparked people, then people wanted the Foursquare firehose for geodata, the Digg firehose for links. But it was Twitter driving it, and people wanted to see more than a stream.”

Summize was acquired by Twitter; its product lives on as Twitter’s search feature. (And insofar as people need real-time search, Twitter’s own Summize-based search tool works pretty well.) However, this company had one of very few success stories in the real-time search category.

In December 2009, Google announced a partnership with Twitter that would allow it to enter the real-time search space. Google’s real-time search features meant more than just stiff competition for the startups in the category; it was a death knell.

“If you back up three years ago,” said Campbell, “a search index was deemed pretty fresh if it was surfacing things daily. There was no social data flowing in, and the expectation for timely content was not there. Then Google went off on their own and indexed everything faster.”

And it wasn’t just Google getting into bed with Twitter. As OneRiot CEO Tobias Peggs told VentureBeat, all the major search players were hopping on the bandwagon, putting even more pressure on the startups in that space.

“Around that time,” Peggs recalled, “Yahoo did the search deal with Bing, which brought an abrupt end to our own fledgling partnership. By then, both Google and Bing had licensed the Twitter firehose and were busy building their own flavors of realtime search, so the opportunity to partner or distribute there was limited.

“Basically, it was going to be tough to build scale without one of those three in our camp, so we had to pivot.”

As a result of all the interest and experimentation from the three major corporate search engines, the real-time search space began to fall apart. OneRiot pivoted toward the end of 2010 and is now an ad network of sorts. Collecta had plenty of runway (it had raised a significant round of funding mid-2010) when its team realized the search angle wasn’t working. Rather than pivoting, the Collecta team simply shuttered the web app in January 2011.

And Wowd, which had an interesting, people-powered and distributed approach at its launch in 2009, first pivoted to a news-filtering service early in 2010. When that approach didn’t work, the company closed its doors, dismissed its team, and distributed its assets. The startup had raised a total of $5 million from DFJ, KPG and Stanford University. Seven of its engineers are headed to Facebook, and a few others are working on a mobile social search app.

Ironically, even Google’s real-time search features were shut down a couple weeks ago, when the company’s agreement with Twitter expired and the search giant’s latest social experiment, Google+, was launching. The company said it planned to relaunch real-time search after a reconfiguration that would incorporate Google+ results.

Still, the concept of real-time search is more than just a ripple in the larger tech startup deadpool.

Campbell said, “Here’s the reason the industry never happened: it turns out that social discovery scratched that itch well enough. But I think in a couple years, really timely information is going to make a comeback.”

And even now, real-time technology isn’t dead as much as it is — to most end users, anyhow — invisible.

“It’s everywhere,” said Peggs. “But now it’s a feature of fully fledged search, rather than a standalone product, which is what users want.”

Both Peggs and Campbell said they see a future for interesting uses of real-time tech in common social platforms such as Flipboard and StumbleUpon.

“I think real-time tech is part of things [as opposed to the whole],” said Campbell. “But real-time search on its own needs a context.”

And as the social web continues to infiltrate online commerce and advertising, the context for real-time technology is readily apparent.

As Peggs pointed out, “If you think about what we did in realtime search, we found the content that was resonating with users on the social web right now, did some sophisticated processing of that data, and surfaced up a single piece of content… We had a terrific team and really smart technology, so we thought, ‘How can we deploy both into a new field, fast?’ Here we are, 12 months later, as the leading socially targeted mobile advertising company. It was a good decision to pivot.”

Finally, Campbell observed that even as real-time becomes a feature rather than a stand-alone product in itself, technologies are evolving to handle low-latency, concurrent events.

“We created everything in XMPP because it was the fastest language to index with,” he said. “But other languages are far more prepared for timely events. Nothing still is quite as fast as true real-time. That capability is still not available, but things feel fast enough.”

Any time a startup tackles a huge, low-level, complicated technology problem — one that involves top-shelf (read: expensive) engineers and massive amounts of R&D time — it runs a larger risk than other startups with a simple consumer problem and an obvious business model. An overpowered mammoth like Google could step into the field and effectively quash any hopes of funding, acquisition or graceful exit. Or a faster technology could emerge and accomplish the same tasks with a lot less brainpower — that’s the very nature of tech, after all.

Inevitably, huge ambitions will often lead to pivots and retreats. In the case of real-time search, we saw plenty of both. But the losses of individual companies (and the individuals who built them) has not been a net loss for the wider industry. The so-called death of real-time is merely its transition into ubiquity.

Sadly for most of the optimistic real-time search entrepreneurs of 2008, that transition didn’t pan out into business success.

Image courtesy of chidorian.