Hands-on with Google Now on Tap in Android Marshmallow: It’s a start

Google today issued the final release of Android Marshmallow, the newest version of its mobile operating system, after it was first unveiled back in May. Perhaps the biggest feature in Marshmallow is Google Now on Tap. It quickly gives you information from Google search, along with shortcuts to relevant content in multiple apps, based on what’s on the device’s screen.

If you have an Android Nexus device, like a Nexus 5, Nexus 6, Nexus 9, or a Nexus Player, you should be able to try the new feature now, after downloading the Android Marshmallow over-the-air (OTA) update. (To get the update, open the Settings app, select “About tablet,” and hit “System updates.”) If it’s not available to you yet, you can manually flash the Android Marshmallow factory image onto your Nexus device. Marshmallow will be rolling out to other Android devices later.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

Once Marshmallow is installed on your Android phone or tablet, you can access Google Now on Tap from any app by holding down the home button (the circle at the bottom of the screen) for a couple of seconds. A white line appears around the edge of the screen, visually indicating that Google is searching for, and recognizing, entities currently being displayed. And then, poof — you’ll see results above a white bar at the bottom, with Google’s new “G” logo. The tool can bring up information about people, places, and movies, among other things.

I just got Now on Tap running on my Nexus 9, and I’ve been taking it for a spin. Here are my initial impressions.

It just might teach you something

When you’re looking at tweets in the Twitter app, where information on a bunch of subjects is brought together in the timeline, it’s fun to hit up Now on Tap and see what turns up. You could very well find yourself learning something new.

And while you’re reading an article that talks about a specific location, it’s nice to be able to quickly map it just by hitting its Google Maps icon in Now on Tap.

Without this feature, you would have to leave the current app (Chrome in this case) and open up Google Maps to search for the place in question (not to mention going back to the app just to make sure you’re spelling the name right).

In this case, Google Maps automatically shows me a map of a Portland neighborhood cited in an Oregonian article, and it contains photos, a place description, and a button for accessing directions.

And if you’re scanning articles in an app like Feedly, and you see a reference to a music artist, you can find links to songs, albums, and concerts that would normally pop up in a card on Google Knowledge Graph if you were Googling on desktop.

That’s not bad. Integrations with music streaming apps installed on the device, like Rdio and Spotify, would be excellent additions.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

If you want information about the music artists playing in an app, Now on Tap has links to social profiles, results of searches from Google News, Google Search, and so on. This isn’t huge, but it’s a start.

It won’t recognize words in your photos

One thing I was hoping Now on Tap would do is provide deep links based on things that it identifies in images you look at. That’s available in Google Photos (which has excellent an ability to identify objects, and even faces, that appear in images) and Instagram.

But when I pinged Now on Tap while looking at a photo I took at an Asana event last week, a photo where the word “Asana” is plainly visible, I got nothing on Asana the company, and nothing on asana in the yogic sense.

If you’re looking through images with Google search on the Web and you see a picture that you don’t recognize, you might want some contextual information from Now on Tap. But here, too, you’re in for a disappointment.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

I searched for “shake” and found loads of images of drinks, but when I found one that stood out and wanted assistance from Google — what was the name of that hilarious show with the walking and talking shake? — Now on Tap couldn’t give me anything.

The show, in case you don’t know, is “Aqua Teen Hunger Force,” and the character’s name is Master Shake.

In a food app like Yelp, if you’re looking at an unlabeled image of a dish, good luck finding out what it is, because Now on Tap won’t tell you.

Say you want to learn more about a product you see in an app like Pinterest. Now on Tap is not especially useful in this situation, either.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

In one case, I was looking at a photo of a jacket that the clothing company Eddie Bauer itself had posted on Pinterest, with a link to buy the jacket online. While I was looking at the jacket, Now on Tap provided no results. When I scrolled down, I did receive a link to a Google search and a Google image search for “Twill Shirt Jacket,” a term extracted from the product description, but I didn’t get links to the same product on different sites, nor did I get a display of visually similar products.

Given Google’s formidable artificial intelligence talent pool and its existing technology, the lack of image recognition in Now on Tap seems like a missed opportunity.

Not exactly speedy (or always right)

It’s not a speed demon — usually it takes a few seconds to bring back results.

And as it’s new software, it doesn’t function perfectly, either. Once, when I pinged Now on Tap, I received an opaque error message: “Something went wrong. Try again.” What’s far more common is the “Nothing on tap” message with a sad face. That’s what you see when, I guess, Google isn’t confident that it has something relevant to offer you.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

There might even be blatant factual errors. While I was zoomed in on a part of Vancouver, Washington, just across the Columbia River from Portland, Oregon, Now on Tap said that the Vancouver I was looking at was a “city in British Columbia, Canada.” Sorry, Google, but Vancouver, B.C. is five hours north of Vancouver, Washington.

More than once, when I wanted information on something mentioned onscreen, there were no Now on Tap cards available, and instead I was offered context about something else, which is frustrating. (Note that there is a “Send feedback” option available when you tap the three dots above the top Now on Tap card.)

In one case, I wanted to see what context I could get about a restaurant that was mentioned in a comment on a card in a Trello board. But rather than giving me, say, the location and the menu for the restaurant, Now on Tap showed me information about several things mentioned in other cards on the Trello board. Not cool. It would be better to constrain the Now on Tap search to the text in the individual card I’m looking at, but that function doesn’t seem to be programmed into the current version of Marshmallow.

When Now on Tap does bring up cards on the things you actually want to know more about, every single social media profile won’t always show up for a given entity. And if it’s an app that’s not installed on your device, Now on Tap won’t necessarily provide a deep link to the app in the Google Play Store.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":1816033,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,dev,mobile,","session":"D"}']

That was the case when I was looking for more information on Airbnb while reading an article about a recent acquisition.

I went to the Play Store and installed Airbnb, and then when I ran the same Now on Tap search, the Airbnb app appeared inside the card about Airbnb. That was reassuring.

If you’ve starting using Now on Tap, I’d like to know what you think about it. Is it helpful? Is it frustrating? So far, I think it’s a little bit of both.