Analysis, part 2

I’m continuing my high-level analysis with a look at which scenarios caused polarized reactions, and by segmenting the respondents based on the extreme scores they gave to different scenarios.

Polarizing scenarios

Some scenarios had a lot of agreement (most people gave the same score), whereas other scenarios split the population.

• The 50^th movie scenario was not very polarizing. Fifty-four out of 138 people (39% of the respondents) gave it a 3. Twenty-three percent of the 138 people gave it between 0-2, and 37% of the 138 people gave it between 4-6.
  • The prevailing attitudes regarding the movie theater scores scenario were that it’s creepy, and that it’s not enough (the employee doesn’t offer a free popcorn or ticket).

• The Home Depot paint scenario has an interesting polarity graph as well:

• Two scenarios had only 8 or 9 respondents that gave less than a 3. However, the answers ranging from “mediocre” (a 3) to positive (a 4-6) were split, with no one numerical answer receiving the majority of respondents.
  • The attitudes from the free-text responses show that the airport extra info scenario is overwhelmingly found as useful (69 respondents out of 135 said something to this effect), with 22 respondents indicating that this extra information is boring or not special in their opinions.

• For the semi-frequent Starbucks guest, the respondents indicated that the personalized service (utilitarian service orientation) is useful (42 out of 116 said this) and that they like the personalization (49 out of 116 said this). Only 8 said it is creepy, compared to 53 out of 117 respondents on the hotel Starbucks.

• A couple of scenarios were more polarizing. On these two, the crowd couldn’t make up its mind:
  • On the Target cereal sale scenario, respondents were split between saying it’s useful and creepy. Out of 154 respondents, 44 said it was useful, and 44 said it was creepy. (Ten people said both.) Additionally, 34 people said they liked the personalization aspect:

• The hotel Starbucks scenario is especially interesting to me because it very much probes the limits of creepiness and corporate chain store behavior. It’s also useful to compare the data with the data from the similar scenario, where a semi-regular visits his local Starbucks. In that scenario, only 8 people found it creepy, with most people remarking positively on the usefulness and the personalization factors. In the hotel Starbucks scenario, 53 out of 117 responses mentioned the creepiness, with only 32 saying they liked the personalization, and only 22 finding it useful:

 

• Interestingly, from the employee perspective, two scenarios didn’t get any scores below 3. That would be the hotel information scenario.

• One surprise is that in the Target cereal sale scenario, 11 out of 44 respondents said that the scenario made them uncomfortable. (the creepiness factor.) That was the largest number of respondents who said an employee-POV scenario made them uncomfortable. The employee-POV scenario that was second-highest in terms of making respondents uncomfortable was the makeup brand scenario where the employee gives the customer a free sample of the item that goes with the products she already has at home.

What does it all mean?

• I think that the fact that the variers so disliked the employee-POV local coffee shop scenario is a sign that the variers are a group of people that care a lot about the “human touch.” (I say this because the variers rated that scenario from the customer POV as a 4.2, same as the overall average. Therefore, I wouldn’t claim that the variers are upset by the creepiness factor; rather, they have a stronger affinity for “authenticity.”)
  • I think they may also be a more utilitarian group. I think on scenarios such as the 50^th movie, they were more likely to give a low number (instead of a 3) because of the uselessness of the personalization. (Also see the next bullet for more on respondents’ reaction to utilitarian benefts.)
• I think part of why the hotel Starbucks scenario fell flat is not only the creepiness factor, but also because the scenario didn’t emphasize the customer’s utilitarian service orientation. (The local Starbucks scenario did emphasize that, and many respondents mentioned the usefulness. In the hotel Starbucks scenario, Bob’s service orientation was not mentioned.) The fact that the respondents overwhelmingly gave it a 3 rather than a 0 or 1 makes me think that they found it more “useless” than “upsetting.”
• People clearly like and feel comfortable with more information. (The Hotel Info and Airport Info scenarios were some of the most highly rated.) This fits in with my interviews and fall-semester research 100%.

Advertisements

Some analysis from the survey!

I’ve put together some high-level analysis based off my online survey:

Highest rated scenarios

Customer:

1. Home Depot worker helps customer avoid buying wrong primer (4.6) and Airport ticketing agent offers extra info to nervous passenger (4.6)
2. Hotel check-in desk worker gives customized info to customer on what’s new since she last stayed here (4.5)
3. Barista recognizes semi-regular employee during his morning coffee pickup (4.2)
   4. Employee version is 4.3

Employee:

1. Airport ticketing agent offers extra info to nervous passenger (5.0) and Hotel check-in desk worker gives customized info to customer on what’s new since she last stayed here (5.0)
2. Home Depot worker helps customer avoid buying wrong primer (4.8)
3. Cosmetic saleswoman notifies customer of product relating to customer’s political beliefs (anti-animal testing) (4.6)
   3. Customer version is 3.7

Regarding Mechanical Turks vs. other respondents

The Turk respondents and the other respondents gave similar respondents to one another.

Respondents with high variance

Respondents whose answers varied more than the average respondent (i.e., not giving all positives, all neutrals, or all negative numerical ratings to the scenarios) were slightly more negative overall than the average respondent.

The scenarios where the variers’ responses were most different from the whole group:

Customer:

1. Ticket seller notifies moviegoer that it’s his 50th movie at this theater, 18.6% decrease. (This is also the worst-rated scenario.)
2. Barista recognizes new, out-of-town customer as if he were a regular, 18.4% decrease. (This is also the 2^nd-worst rated scenario.)
3. Cosmetic saleswoman offers customer a makeup that goes with the customer’s already-purchased makeups at home, 14.7% decrease. (This is also the 3^rd-worst-rated scenario.)

Employee:

1. Target worker tells customer about favorite grocery item on sale, 7% decrease. (This is by far the biggest gap among varier responses; the next most varier-disliked scenario is 11.0% less liked by variers—see below.)
2. Barista recognizes semi-regular employee during his morning coffee pickup, 11.0% decrease. (Note that from the customer perspective, this scenario received third place.)
   3. Note: for the above two scenarios, the varier population gave the scenarios an average score below 3.0—these are the only two scenarios that got a below 3.0 avg. score from a population.
3. Cosmetic saleswoman notifies customer of product relating to customer’s political beliefs (anti-animal testing), 9.3% decrease. (Note that this scenario, also in employee version, was third-most-liked by the overall group, but got middling scores from the whole group for its customer version.)

Southwest Airlines providing live entertainment on surprise flights

Somehow, I think that if I’d tested this concept in my online test, it would have scored really poorly:

Over the past several years, the quirky carrier has surprised passengers with unannounced in-flight musical sets by middle-of-the-road rock bands, an unannounced in-flight fashion show, and even an unannounced in-flight wedding. Now comes the latest volley in Southwest’s campaign to inflict glee on your customer experience, whether you want it or not: book readings.

The rest is here, from Slate.

Analyzing the survey data

(Firstly, thanks to everyone who took the survey. I had over 200 responses.)

I’m now analyzing the data that came back from my survey. It’s a lot to sort through–twenty questions times over 200 respondents can’t be written out onto post-it notes.

There’s a taste of what my desktop looks like right now.

I’ve got a little preliminary analysis. Each scenario had a question asking the survey-taker to rate the concept from 0 to 6, with 0 being “much less desirable” and 6 being “much more desirable.” (A neutral response therefore would be 3.)

Each scenario had a customer point of view version and an employee point of view version. The customer point of view scenarios asked,

Please think of a time when you recently shopped in a store like this. How does the above scenario compare to your own experience?

and the employee point of view scenarios asked,

How well do the tools this salesperson uses help to make them better at their job?

Here’s how they were rated, on average:

They’re in order with the highest-rated customer point of view scenario on top. (A full list of the scenarios is here, or you’re welcome to click through and answer the survey.)

The survey is up and ready for responses

Finally, approval from the IRB to distribute my survey. It’s here. Please take the survey! Here’s the link again!

A preview:

Hard at work!

Depicting service design scenarios

What’s the best illustration style for storyboards?

Tools of the storyboard trade.

Lately, I’ve been struggling with how to depict service design scenarios, so that I can online-test them. I started by writing each scenario from the perspective of the customer, as well as the employee, then drew rough draft story boards for each:

Scenario 1, customer perspective:

Whenever she’s at Target, Ann checks to see if the Cascadian Farm Cinnamon Raisin Granola is on sale. Whenever it is, she buys one or two boxes. It’s her favorite cereal, but she feels that at the regular price it’s too expensive for her to justify as an everyday purchase.

This afternoon, Ann is at Target browsing for clothing. She’s looking at a rack of tights and socks when a Target employee walks by. He stops and says to Ann, “Hey there – just wanted to let you know that we are having a sale on Cascadian Farm stuff, like cereal and granola bars!”

Ann is happy to hear this, and when she’s done looking at clothing, she goes over to the grocery section to get a box of the cereal.

Scenario 1, employee perspective:

Jay is an employee at Target. Today he is tidying up the displays in the clothing department. He carries with him a mobile device that was given to him by the Target manager – the device has information that he can give to customers to help them have a better shopping experience.

He glances at the mobile device as he walks by the socks display. It shows that a customer who is currently browsing the socks display is a big fan and frequent buyer of Cascadian Farm Cinnamon Raisin Granola, but only when it is on sale. It prompts him to mention the current sale to her.

Jay says to the customer, “Hey there – just wanted to let you know that we are having a sale on Cascadian Farm stuff, like cereal and granola bars!” The customer smiles back.

What I’m struggling with is refining the drawings for my online test. I don’t want to make them too “finished” or “polished,” because to do so would not invite criticism as readily, and I want honest criticism of my concepts. (Furthermore, I want to de-emphasize the aesthetics here, and instead focus people’s attention on the service concepts.)

My first attempt at a more refined storyboard is all wrong:

Scenario 1, customer perspective

Scenario 1, employee perspective

I think they’re too busy and don’t fully emphasize the service.

What direction should I go in?

**Update: A classmate suggested a gray background to increase the attention on the characters: