Twitter’s earnings leak is latest sign that ‘good’ bots are going too far

When Twitter’s disappointing earnings report was discovered and prematurely released by a financial intelligence firm on April 28, causing Twitter shares to tumble, it was more than a just a big oops for the social media giant. The affair has thrown a spotlight on the shadowy world of bots – computer programs that crawl the web and perform tasks at a volume and speed a human could never match – and the legal and ethical grey areas they increasingly operate in.

According to multiple news reports, this is how the results got out: NASDAQ OMX Group’s Shareholder.com accidentally posted the earnings release on Twitter’s publicly available investor-relations page for 45 seconds about an hour before the results were supposed to go public. Selerity, a New York company that offers a “real-time news and event detection platform” to investors, found the release – presumably through the use of automated web-scraping bots – and tweeted the information.

“No leak, no hack,” the company tweeted.

That may be true, but does that make it right?

For those of us in the bot-detection business, the landscape used to be fairly polarized between bad bots and good bots. Bad bots are often associated with cybercriminals stealing data, identities, and intellectual property or initiating denial-of-service attacks. Good bots include Googlebot, Google’s web crawling bot that discovers new and updated pages to be added to the Google index.

When my company identified, tracked, and cataloged 8 billion bots for our 2014 report on the bot landscape, I was surprised by the number originating from financial services companies. In fact, we found that financial services led all industries in bot traffic.

Many of these firms are running their own automated reconnaissance on publicly traded companies – without those companies’ consent, of course — to gain insights that may not yet be public. Some rely on a growing number of startups like Selerity, 80legs, Connotate, Dataminr, and Kapow that offer web-crawling services.

And we’re talking about more than the sort of news-harvesting that happened with Twitter, where a bot found information that a layperson just browsing around would not have noticed. For example, we’re seeing bots being used to track retailers’ inventory levels and pricing information in an attempt to forecast sales.

And therein lies the murky legal and ethical territory. If a financial services company can use high-powered computers and bots to get information that’s not accessible to the average person, do they gain an unfair advantage? Should it be considered insider trading and thus illegal? Has Selerity and its ilk built a better mousetrap, or are they merely Internet peeping Toms?

At the very least, the release of Twitter’s results should be a defining moment, prompting extra vigilance by companies about what they put on their websites – even tucked-away pages that aren’t obvious to the public. Today’s bots leave no margin for error, and companies must treat all sensitive information with the same care that the federal government does when it locks monthly job figures in a safe before releasing them.

The episode also should spur not only soul-searching by companies in financial services and other industries that are engaged in these practices but also a long, hard look by regulators and lawmakers.

In my view, a bot crosses over to the dark side simply if it piggybacks on a website without providing any value back. The dictionary defines “parasite” as an organism that lives in or on another organism and benefits by deriving nutrients at the host’s expense.” The same can be said of too many bots, which even if they don’t aim for nefarious extremes, such as stealing data, can still rob bandwidth or, as in the Twitter situation, perform the online equivalent of snatching a piece of paper out of someone’s hand.

The courts already have weighed in on some of the principles at issue here.

In Associated Press v. Meltwater, AP claimed its copyrights were infringed when Meltwater, an electronic news clipping service, included excerpts of AP stories in search results for its clients seeking news coverage based on particular keywords. In March 2013, the U.S. District Court for the Southern District of New York sided with AP, ruling that Meltwater was using AP’s resources and not providing any value back.

Fair use or free ride? That’s the question that should cut right to the heart of whether a bot is a good bot or a bad one.

Bots are on the increase for the simple reason that they have become a reliable and effective weapon for legitimate businesses and organized criminal hackers alike. The Twitter miscue has demonstrated that it’s become too hard to tell the good guys and the bad guys apart.

The business world, technology community, and government leaders need to carefully examine the issue.

Rami Essaid is CEO and cofounder of Distil Networks, a bot detection and mitigation company.