Google is killing the keyword, unless you pay (here's what to do about it)

Seth Besmertnik is CEO and cofounder of SEO technology firm Conductor.

In October 2011, Google announced it was going to start blocking valuable data about which keywords consumers use to discover your content. By encrypting all searches, Google would instead dump visits from natural search into the nebulous “not provided” category in web analytics software.

At the time, the company said it would impact less than 10% of search traffic.

Fast forward 24 months, and web marketing has been completely turned upside down. When search marketers opened their Web analytics packages at the end of September, they were greeted by a rude surprise: that 10% number is much closer to 100%.

To validate the extent of the phenomenon, the Conductor Research Team analyzed the search traffic from three major retail web sites, more than 11 million visits in all. The analysis showed that the phenomenon is indeed real. For site one, keywords for 69 percent of Google search traffic was “not provided”; for site two, the figure was 78 percent; and for site three it was 80 percent.

Analysis of the growth trends showed that at the current rate, all three sites will reach 100 percent “not provided” within six weeks:

Google’s statement on why it is hiding keyword data is that it is for extra protection — unless you click on a paid search ad, in which case you are slightly less extra protected.

The fact is, the loss of organic search keyword data creates a very real problem for marketers. Search marketers use keyword data to see what terms searchers use to arrive at their site.  With this knowledge, marketers can adjust search strategies, develop content, optimize pages, and generally understand what is and is not working for them.

The loss of this information means marketers are suddenly, painfully, flying blind.

Editor’s note: Our upcoming DataBeat conference, Dec. 4-5 in Redwood City, will focus on the most compelling opportunities for businesses in the area of big data analytics and beyond. Register today!

Creating great content that your customers will love was challenging enough before. Without the ability to read the interests of your customers through their search terms, it has become far more challenging. While the loss is a blow to marketers’ online insight, it is equally detrimental to searchers. This is because knowing how visitors are arriving at their website means marketers can tailor content to ensure visitors are seeing the most relevant information possible.

For example, say a furniture retailer sees a page about “modern style dining room tables” is receiving a lot of visitors for “old fashioned dining room tables” queries. Closer analysis shows a high percentage of visitors are bouncing, leaving the page almost as soon as they arrive and see that the page does not contain the information they are looking for. Knowing this means the site owner can create a page with relevant content for “old fashioned dining room tables” and take steps to optimize the page to appear in the search results for the query, thus improving the searcher experience.

This kind of insight becomes far more complicated to extricate after Google’s recent change. In the end, both marketers and online users suffer.

Why, Google?  Why?

Google’s publicly-stated reason for the change, framing it as a privacy issue intended to protect users, seems disingenuous at best.

Some have speculated that the sudden, unannounced increase to near 100 percent “not provided” is motivated by an attempt by Google to distance itself from its recent ties to NSA spying efforts. Whatever the motivation, the statement that it is for extra protection is hard for search marketers to swallow.

First, because no personally identifiable information is passed to the web site, the site owner does not see that “Joe Smith visited your web site via a search for ‘dining room tables.’” They see that “a visitor visited the site via a search for ‘dining room tables.’”

Second, and perhaps most crucially (and what has marketers up in arms ready to march on Mountain View) despite Google’s claim that it is merely looking out for the wellbeing of its users by hiding keyword data, Google doesn’t hide the data if you pay for it. That is, if marketers place paid ads in Google Adwords for their keywords, Google will pass along visitor keyword data.

Some have gone so far as to accuse Google of extortionist behavior — backing marketers into a corner by taking away keyword data they’ve had for years and forcing them to pay Google big bucks to get it back.

If Google was genuinely concerned about giving users extra protection, it would hide the keyword data for both paid and natural search visits, even if it meant turning away a bit of extra revenue above and beyond the $50 billion in revenue it saw in 2013. Now that move would have people agreeing that Google really does live by its stated motto, “Don’t Be Evil.”

Where Do We Go From Here?

Whatever you think of Google’s behavior, as online marketers, we’ve still got revenue and traffic targets to hit. The CFO doesn’t really care that Google isn’t playing nicely.

Luckily there are a number of ways to gain insight into the online landscape. No one tactic or collection of tactics will bring us all the way back to where we were, but with some tap dancing and good old fashioned elbow grease we can get most of the way there.

• Page Level Insight
  Although we are used to looking at our online search performance on the keyword level, now that we no longer have keyword data, we can start looking more closely at page level data.  This means analyzing which pages are receiving visits from Google search to get a good idea of the kind of visitors you are receiving.
• Alternative Search Engines
  Although Google no longer passes keyword data, other search engines such as Yahoo and Bing still do. Mine keyword data from other search engines to determine how visitors are coming to your site. You can even derive an estimate of Google search traffic by calculating the distribution of search traffic by search engine historically coming to your site and multiplying by market share. For example, if you calculate that Google usually drives 70 percent of natural search visits and Yahoo and Bing drive 30 percent, allocate seven visits to Google for every three that come in through Bing and Yahoo.
• Rank Tracking Data
  Prior to the day the keywords went dark, some in the industry had suggested that rank tracking had become irrelevant as a key metric, taking a backseat to other, bottom-of-the-funnel metrics. While that point was debatable then, now, with the loss of keyword traffic data, it is clear that rank tracking data has become more important than ever as a top-of-the-funnel indicator. Take a second look at whether you are getting everything you can out of your keyword rank data. Consider segmenting keywords and looking at your search visibility from other angles. Some third-party analytics vendors have created algorithms that leverage rank data, clickthrough curves (the rate at which users click on search results by search position), and other data inputs to recoup keyword traffic metrics.
• Invest Some Money in Paid Search
  To fill in some of the insight gaps that were lost, it may make sense to invest some budget in Google Adwords, which still includes keyword data and will help you determine the content that is, and is not, resonating.

Look Forward, Not Back

At the end of the day, as much as Google’s behavior leaves a bit of a bad taste in our mouths, and we know that both marketers and searchers are worse off after the change, the fact is, if you are an online marketer, you are playing in Google’s sandbox.

No matter how much we might not like how we got here, we’ve got to find a way to operate in Google’s search results and recoup as much insight as we possibly can to make our search efforts successful. Do your best to set your distaste aside and resolve to investigate how the tactics above can help you succeed in search.

Seth Besmertnik is CEO and cofounder of SEO technology firm Conductor. Follow him on Twitter @besmertnik.