When you should break your product

I’m a tough critic when it comes to products. I usually compare products to what I know technology is capable of, not what the competition is doing. Every product can be made better.

That’s an extremely high standard. The products I have launched wouldn’t meet it. But the goal of the criticism is to get people to think bigger and deeper about problems, even if those features or improvements come a few releases down the road.

But sometimes you should break your product to improve it for the long term.

I’ve been testing the new Google Shopping Express service, a same-day delivery service in the San Francisco area. You can go online, find things to buy, and select a delivery window, and a courier brings it to your house later that day or the next day. (Disclosure: I own stock in Google.) It’s an immensely complex problem. You’re testing a new market, dealing with partners, trying to figure out what people want to buy, trying to figure out the right price point and more.

The service gets a lot of things wrong, but two of those things I look at and say, “It’s not the best consumer experience, but it provides valuable input.” These are things I would leave deliberately broken to capture data on user expectations.

The first is what you can buy. You can enter anything in the search box. GSE offers little guidance as to what you can buy other than a list of stores and a few categories. Can I buy eggs? Nope. (Although L’eggs pantyhose comes up.) Jalapenos? Not fresh ones. As a user, this is a bit frustrating. From an optimal user experience standpoint, you might want to have a walkthrough that lists the types of products the service offers. But not having that is a good way to capture data. If you put up a big sign that says “We don’t sell eggs,” people won’t search for eggs. But leave that out and you can get a good sense of customer intent. These are people who signed up for your product and have an interest in using it.

I’ve found that some of the best consumer research data can be found in “no results found” logs. By looking at the list of things people were looking for and couldn’t find, you might discover new product categories you didn’t think of. Or determine that there is enough demand for things like fresh produce that it makes sense to implement the extra steps necessary to deliver fresh produce. I usually take a sample of the logs and bucket them into categories like: good idea, not enough demand, not practical. There’s also a bucket of things that should have come up (because we offer them) but didn’t. (This can help identify problems in the search algorithms.)

Another thing that annoyed me was that I would put things in my shopping cart and then only at the very end find out there weren’t any more delivery times available today; the soonest I could get my product was tomorrow. From a consumer experience standpoint, you want that information upfront, before people have invested all that time building out a shopping cart you have no ability to execute on. But letting people put things in their cart gives you a sense of what types of things people want when.

As the product gets further along and fully baked, these experience can be shifted towards the consumer-friendly end of the scale.

Jess Lee, CEO of social commerce company Polyvore, takes this a step further with a technique called “fake doors.” These are hooks for features that aren’t built. It’s just a test to see how many people would be interested in that feature if it existed.

One of the biggest challenges in market research is that people rarely do what they tell researchers they would do. As a result, you could spend a lot of time and resources building features that never get used. By leaving certain parts of your product broken, you can do much better research with actual customers.

Rakesh Agrawal is a consultant focused on the intersection of local, social, mobile and payments. He is a principal analyst at reDesign mobile. Previously, he launched local, mobile and search products for Microsoft, Aol and washingtonpost.com. He blogs at http://redesignmobile.com and tweets at @rakeshlobster.