Why ISPs are hijacking your search traffic & how they profit from it

A handful of Internet service providers (ISPs) in the U.S. are redirecting search traffic around specific keywords to brands’ websites, presumably for affiliate marketing revenue.

A study released today by a UC Berkeley research group revealed that for some Internet users on some ISPs, using a search engine and typing in a word such as “apple” or “bloomingdales” would redirect the user to websites for Apple or Bloomingdale’s rather than to a page or search results about the keyword in question.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

The Berkeley project, called Netalyzr, was created to measure DNS behavior. However, over the past few months, the Netalyzr team noticed some unexplained and unexpected redirections across at least 12 ISPs in the United States.

In a blog post on the findings, the team wrote, “The affected ISPs use services provided by a company called Paxfire to monetize certain web search requests. Paxfire’s main line of business is DNS-error traffic monetization, i.e., the practice of presenting advertisements and search results to users who mistyped a website’s address in their browser.

“In addition, some ISPs employ an optional, unadvertised Paxfire feature that redirects the entire stream of affected customers’ web search requests to Bing, Google and Yahoo via HTTP proxies operated by Paxfire.”

Following the money

The Electronic Frontier Foundation helped the Netalyzr team investigate the matter. As EFF senior staff technologist Peter Eckersley told VentureBeat, “They knew the general category of false DNS responses might be possible and worth checking for, while the details that emerged about Paxfire and what it was actually up to were a bit more surprising.”

“We knew that some forms of malware would change DNS results locally on a victim’s computer, so it made sense to look for such meddling,” said Vern Paxson, one of the Berkeley researchers.

The research team found that around 170 specific, brand-related keywords would trigger interference by the HTTP proxies, causing users to be redirected to affiliate marketing landing pages. “We don’t have a comprehensive list [of keywords], just a bunch of terms we tried, such as the names of popular web sites,” said Paxson. Although the team was testing only for single-word search terms, Paxson also said, “It’s possible that other searches are redirected too, but we haven’t tried that.”

Through the redirection process, the researchers wrote, “The ISPs and Paxfire presumably earn commission payments for the redirected flows.”

Some of the ISPs involved are, according to data presented by multiple organizations involved in the investigation, Cavalier, Cincinnati Bell, Cogent, DirecPC, Frontier, Fuse, Hughes, IBBS, Insight Broadband, Megapath, Paetec, RCN, Wide Open West and XO Communication. Charter and Iowa Telecom claim to have recently stopped doing DNS redirects.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

While it’s likely that ISPs had at least some knowledge of at least some of the DNS redirection, if not search traffic redirection, it’s less likely that the brands themselves were involved in the scheme. “There is probably a chain of several intermediaries in these affiliate marketing programs between the brand itself and Paxfire,” said Eckersley.

“We would find it surprising that so many brandholders would agree to this sort of redirection, so we expect that they are not complicit,” said Paxson.

In other words, it’s difficult to say at the outset where the buck stops in this scheme and whose hands are in the cookie jar. What we do know is that many of the ISPs involved are claiming a lack of knowledge about the search redirects and pointing to third-party vendors as the real villains in the scenario.

A Charter representative told VentureBeat today that when search traffic redirects were occurring across that ISP, “We were not aware of it. It was a third party, and in a sit-down with the vendor, we said, ‘You need to be more careful about putting us into this mix… Charter doesn’t think this practice is acceptable.”

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

Steven Crosby of Frontier Communications Corportation told VentureBeat, “In terms of Frontier’s practices, we do not hijack any search traffic. We have clear business rules in our legal agreement with Paxfire that allows them to monetize URL address bar errors (e.g., ‘www.abc.cmo’ instead of ‘www.abc.com’ or typing an actual word like ‘PC’ into the address bar). Paxfire is not allowed to touch any search traffic that originates directly from toolbars or search bars.”

While the Charter rep was not able to name the exact vendor involved, Paxfire is just one of many Internet marketing companies that are using technical architectures for commercial and marketing purposes. These firms, which include companies like Barefruit and Golog, engage in murky practices such as search redirects, practices that violate our expectations of how the web should work and that rob us of any trust we might have in our ISPs.

If you use one of the affected ISPs, the EFF recommends running a Netalyzr test and installing a browser plugin such as HTTPS Everywhere to use HTTPS for all your web browsing “With HTTPS, attempts by the ISP or a company like Paxfire to alter the results would cause a certificate warning,” said Eckersley.

Google has also recommended using Google Public DNS and is beta-testing encrypted web search for users who want to better protect their search traffic.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

The problem with Paxfire

“I’m not an expert on affiliate marketing programs, so I can’t comment on whether anything that Paxfire is doing might be a violation of the rules or norms of that business sector,” said Eckersley. But he did say that the marketing company “has no business” granting itself access to the keywords people are using to navigate the Internet.

“If my search engine is untrustworthy or not returning the results I was actually looking for, I can go and pick a different search engine. But if Paxfire has snuck out onto the network and secretly replaced all my choices of search engine with itself, I no longer get to go elsewhere for my searches.”

And when Paxfire’s proxies malfunction, any search attempts return an error message. “Users will often blame the search engine for that, when in fact it’s the fault of the company that’s secretly hijacking them,” said Eckersley.

In the end, said the EFF spokesman, it all comes back to net neutrality and how the lack of neutrality fundamentally degrades the reliability of the Internet. “Programmers assume that when they send data from A to B over the network it will arrive as it was sent. But if in fact the data is transformed by a series of companies that are trying to find ways to make a quick buck, things become more complicated, unpredictable, and fragile.”

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

The Frontier fiasco

In the ongoing quest to put a stop to deceptive Internet marketing practices, it’s hard to tell exactly where to lay the blame for search redirection and the responsibility for ending it. But Google took the issue upon itself when users were complaining about redirects.

Google’s security teams had been aware of DNS-based traffic interference from ISPs for months, at the very least. Google security engineer Damian Menscher wrote in response to user issues with Frontier back in March, “At Google, we are following this very closely, and trying to get Frontier to fix the issue. The root of the problem is that Frontier is intercepting some traffic, so when you try to use Google your search actually goes through a Frontier server first.”

At that time, entrepreneur and investor Andrew Payne noticed the redirection happening in his own searches. He wrote, “ISPs have redirected DNS queries for a while, but have mostly focused on typos and misspellings. I’ve never seen an example of an ISP actually hijacking a user’s Google search and inserting their own results, and that seems pretty egregious to me.” Menscher recommends users contact Frontier directly about the practice.

With folks like Payne making waves online, Frontier responded directly. Maggie Wilderotter, the ISP’s CEO, told Payne a story similar to the one we heard from Charter today: “that this had been done by one of their vendors in violation of Frontier’s business rules and it’s been shut down,” as Payne wrote. Around May 2011, Payne said Frontier had stopped redirecting Google search traffic, as far as he could tell.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":316810,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"security,","session":"C"}']

According to the EFF, Google has repeatedly put pressure on ISPs to stop DNS-based redirects and has been at least somewhat successful. However, the EFF notes that Yahoo and Bing search engines are still particularly susceptible to redirects.

“This is why the ISPs that were proxying Google stopped in the past couple of months,” wrote Berkeley researcher Nicholas Weaver in a Slashdot thread today. “Google’s abuse-detection threw up a CAPTCHA on the queries, and then Google posted about it.”

Evidently, the combined noise from the web and pressure from the search engine were enough to put a stop to search redirection in some cases. A Google spokesperson confirmed, “We aren’t aware of any DNS providers that are currently doing this hijacking for searches intended for Google.”

Hopefully, continued pressure and the watchful eyes of the media, Berkeley researchers and advocacy groups like the EFF will help to end the practice of search redirects.

Image courtesy of Magic Glasses.