ZAPPOS USABILITY TESTING

Zappos is an online shoe store. As frequent online shoppers, my research partner and I were curious what search features enable users to find shoes they want and then after that, what information helps them decide which shoes to put in their cart. 

Project type
Usability test

My Role
UX Researcher

RESEARCH QUESTIONS

1. Which search features enable users to find the shoe they want?
2. Which information helps users decide which shoes to put in their cart?

METHODS

We recruited a convenience sample of five female graduate students to perform four randomized search tasks.
We watched and recorded their search behavior using screen sharing.
Subjects could add as many items to the cart as they wished and were asked to pick only one pair of shoes at the end of each task. 

The four tasks were:

• Familiar and open-ended: find winter boots for the Michigan winter

• Unfamiliar and open-ended: steel-toed boots

• Familiar and closed-ended search: black party shoes with a 1.25 inch heel

• Unfamiliar and closed-ended: pink sneakers for a young girl

FINDINGS

The search features that enable users to find the shoe they want.

Participants' Total Search Feature Usage

• Filters were the search feature that our participants used most frequently to find shoes. All 5 subjects used filters.

• The Category Menu was used frequently, and was a frequent starting place

• The Search Box was only used when the participants could not find the shoes they wanted with filters or the category menu.

• Sort was rarely used

The information feature that helps users decide which shoes to put in their cart.

Participants information feature usage

• The pictures were the most important.

• Customer created pictures were deemed more reliable than seller created pictures.

• The customer created fit surveys and reviews also helped customers make purchasing decisions.

The information that other customers posted on the Zappos website was by far the most important information that the subjects of our study used to make a purchasing decision. While the pictures of shoes provided by Zappos on the individual product page were used by all the subjects in the study, the reviews that contained customer images were deemed more reliable. In both cases where customer images where present, they took preeminence over the images on the product page. The two participants whose searches yielded customer images within the customer reviews said that the customer images were more important than the company’s images in their decision to add shoes to their cart because they were more accurate in showing what the product looked like.

“The website will show the best possible image of the product and that’s not what the products actually look like in real life.”

–Participant 2

The customer reviews and the fit survey were the used by three out of five participants, therefore this information that was of secondary importance in determining which shoes the participants would add to their carts. Since the reviews and the fit survey are user generated, we see again that the information generated by previous customers is the most important.

RECOMMENDATIONS

Recommendation #1

During our study the location of the heel filters on the webpage changed after our participants selected a single heel height. In order to click on another box in the heel filters section, they had to look up and down the page to locate the heel filters section once again. Like our study’s participants, Zappos shoppers might want a variety of heel heights in their search results. Similarly, shoppers might become annoyed that the filter location changes before they have selected all the heights they desire. Therefore, the filters shouldn’t move after a customer clicks on a single box within a filter group.

Recommendation #2

Our study showed that customer-created images were deemed the most reliable. It also showed that images, especially customer-created images were the most salient information on the website. Therefore, Zappos should encourage, or even incentivize, customers to post pictures of shoes they have purchased and enjoyed.

Recommendation #3

Finally, participants were disconcerted by the way the filters and search terms eliminated items. For example:

When a participant entered several search terms into the search bar and/or clicked multiple filters, for example “winter women leather”

They got 2309 items.

Then let’s say they click the first brand on the page, which is: 1.STATE

Now there are 10 items.

But as our usability study showed, our participants wanted between 20-30 items in their search results, so they deselected the brand 1.STATE to increase their search results.

Now there are 1941 items.

Where did the other 368 items go?

This experience bothered several of our participants. Their mental models said that there should be 2309 results and not 1941. They felt that the addition and subtraction of the filters should mirror the addition and subtraction of the search results. They also didn’t like the idea that they were missing out on the 368 items that had disappears. Zappos should change the filters so that the addition and subtraction of the filters mirrors the addition and subtraction of the search results.

THINGS LEARNED

In a future study, I will recruit people who actually need or want to buy shoes rather than recruit a convenience sample.

I will create a pilot test script so that every participant receives the same information when they start the usability test, and they are not affected by receiving the information in different ways or at different times during the pilot test. I thought I knew the test well enough that I didn’t need a script, but I was wrong. The organization of the information I delivered was not consistent and some critical information was not delivered until the middle of the first task.

The tasks did not include a size criterion; while this led participants to become more immersed in their shoe searches it also caused problems when they were looking for a shoe for their niece-who did not have a defined shoe size. Additionally, some participants with larger feet had difficulty finding shoes in their size. For a future study, I will recruit participants who had common shoes sizes OR I will create tasks that yield appropriate search results for the participants I’ve recruited.

In the future research, my pilot test schedule will have extra time in between participants. When the first participant of the day took longer than we expected, the second participant’s time was cut short. We missed a great opportunity to learn more about our participants’ search behavior when we made appointments one right after the other.

DISCOVERIES AND SURPRISES

These are outlier findings, that might be of interest:

• They used as many filtered as it took to get to that quantity of results.

• Participants wanted to narrow the search results to 20-30 items.

• Customer-created information had more credibility than seller-created information.

• Participants were price sensitive even though they were instructed to ignore budget limitations.

• All the participants created their own imaginary niece who had different preferences than they did.

Click here to download my pilot study