A Guerrilla Usability Test on AmazonFresh

Amazon has been dazzling customers for years with its core one-stop shopping platform. They make just about everything available and have become the first visit (online or off) for many consumers. Yet, AmazonFresh, has had difficulty reaching the same level of traction. This is puzzling. After all, who doesn’t want groceries delivered to their door along with their other Amazon orders?
To examine why users were not flocking to AmazonFresh, I conducted a guerrilla usability study to identify its pain points.

Main Questions

The research addressed these questions:
  • Can users search, browse, and shop for grocery items?
  • Can users remove items?
  • Can users read ratings and reviews?
The research did not address the on-boarding or the check-out process.
I recruited 6 participants by walking into various coffee shops in San Francisco. I asked existing Amazon users to complete three tasks on AmazonFresh. I asked all participants to:
  • Place items into their shopping cart.
  • Remove an item from their shopping cart.
  • Read a review.
Sounds easy, right? Wait until you see what I found out.

Guerrilla Test Debrief:

After the interview sessions, I sifted through my collection of videos and field notes searching for themes.

1. Assigned user quotes to the relevant pain points of different tasks. (As we all know, some people stray from the task.) 2. Dumped all of the information onto sticky notes 3. Clustered them into categories.

1. Placed high-level pain points onto yellow Post-Its

2. Prioritized pain points based on importance to (a) users and (b) AmazonFresh

Three critical findings emerged that affects AmazonFresh’s user experience:
  • Customers could not find what they were looking for.
  • The ratings and reviews is frustrating.
  • Is it fresh?
Let’s start digging into what the pain points were.

3 Critical Findings

Problem #1: The search results confused users
By far, the biggest cause of confusion and frustration was the search. Four out of the six users, either did not find what they were looking for or they got some strange search results.
The quotes I used on the right side of this image are what people actually said out loud when I was watching them perform a search.
For instance, a user looked for gin. The result? the search found matches for gin and Gin Tama, a Japanese manga (comic) series.
AmazonFresh offered irrelevant options. The search results confused users because the site misinterpreted their keywords. Some of these items just are not groceries (books should stay in the regular Amazon site.) The site offered too much irrelevant information to find effective results.
Recommended Design Improvement: Use Amazon.com’s established search patterns
Since 1994, Amazon.com has refined their search engine to:
  • process customer intent;
  • bound search results through contextual categories; and
  • leverage faceted search
Every interviewee uses Amazon.com on a regular basis and loves it. My recommendation is that AmazonFresh use Amazon.com’s established search patterns. Here’s an example:
As you type in the item you are looking for, have the search bar auto-suggest the item and category.
The idea here is to use information architecture and a controlled vocabulary to generate relevant search results.
Problem #2: Ratings and reviews experience is frustrating
Five out of the six users found the AmazonFresh’s ratings and reviews section disappointing. This is because they expected an experience that was similar to Amazon.com.
Let’s take a look at the page for Napa Valley’s Organic Olive Oil.
The quotes I used on the right side of this image are what people actually said out loud when I was watching them read ratings and reviews.
Recommended Design Improvement: Clarify ratings from reviews. Add a “write a review” call-to-action above the fold.

People want to trust that AmazonFresh is delivering fresh groceries (i.e. boxes of crisp, uncrushed cereal or milk that isn’t uncomfortably near its expiration date.)
To help build that trust, let users read reviews and ratings in an expandable section that is above-the-fold. This area can also be distinctly broken down into ratings and reviews as separate sections. Users wont’t have to scroll down to the end of the page to get an idea of the product’s quality.
Here’s what I propose:
The quotes I used on the right side of this image are what people actually said out loud when I was watching them read ratings and reviews.
By clarifying ratings and reviews, users can now decide if the product is worth buying or not. Also, by adding a call-to-action above-the-fold, users are encouraged to leave a review for products that have no reviews. This helps Amazon drive more trust and confidence from the community.
Problem #3: Is it fresh? What does it look like?
Grocery shopping is tactile and visceral. Users like to look at the meat and produce when they buy it. Everyone knocks on the melon or sniffs the pear before they buy it.
When one user was looking at milk, she asked, “How do I know it’s fresh? What’s the expiration date?” Similarly, another user had a hard time buying grated cheese. He clicked around, fixated on the photos: “I wish there were more photos…Is the cheese long grated or short grated?”
What’s the use a package of grated cheese image when you can’t even see the cheese
In a grocery store you can use your senses to figure out if you want to buy that basket of strawberries. Is it fresh? Is it ripe? While we’re probably years away from widespread “smell-o’vision,” this means that online marketplaces for food have to provide this information to users through picture and text. They also have to provide some indicator of freshness (AmazonFresh’s name notwithstanding).
Recommended Design Improvements: Present high-resolution photos & relevant product information
Transforming a tactile experience into a digital one is not easy. Granted, Amazon has done it before by introducing the Kindle. It took years of R&D. In AmazonFresh’s case, it will take time for people to get comfortable with online grocery shopping. To help users break down the psychological barrier, here are two ideas:
  • Larger and higher quality photos that present food in its best light instead of the sterile images that AmazonFresh produces now. GoodEggs does a good job of this. Take note Amazon!
  • Relevant product information: Long or short grated cheese, expiration date, etc.
Look you can see the cheese! And the photo is aesthetically pleasing too.

Putting it all together:

To fix the broken search and review sections, use the established patterns from Amazon.com. Users anticipate information and the experience to be the same. Let’s make the user experience between Amazon.com and AmazonFresh consistent for search and review.
However, AmazonFresh may have to depart from Amazon.com’s user experience with regard to photos. Rather than presenting the small, white-background photos that it does with other products, AmazonFresh should take a page from other successful food marketplaces and present larger photos with lighting and background that highlights the food’s desirability.
AmazonFresh is still in its infancy. It has a long way to go, but if the organization can make a few incremental changes to the search, the ratings sections, the pictures and the information it provides, it can vastly improve the user experience. I, for one, may start actually shopping on it.
I am not affiliated with AmazonFresh. I love the idea that you can summon food with the click of a button. I studied under Kate Rutter & Laura Klein @Tradecraft in San Francisco. Big thanks to Devon Edwards, a fellow Tradecrafter for going above and beyond.

This piece is also posted on Medium.