Analysis, part 2

I’m continuing my high-level analysis with a look at which scenarios caused polarized reactions, and by segmenting the respondents based on the extreme scores they gave to different scenarios.

Polarizing scenarios

Some scenarios had a lot of agreement (most people gave the same score), whereas other scenarios split the population.

  • The 50th movie scenario was not very polarizing. Fifty-four out of 138 people (39% of the respondents) gave it a 3. Twenty-three percent of the 138 people gave it between 0-2, and 37% of the 138 people gave it between 4-6.
    • The prevailing attitudes regarding the movie theater scores scenario were that it’s creepy, and that it’s not enough (the employee doesn’t offer a free popcorn or ticket).

Polarization - movie theaterpng

  • The Home Depot paint scenario has an interesting polarity graph as well:

Polarization - home depot paint

  • Two scenarios had only 8 or 9 respondents that gave less than a 3. However, the answers ranging from “mediocre” (a 3) to positive (a 4-6) were split, with no one numerical answer receiving the majority of respondents.
    • The attitudes from the free-text responses show that the airport extra info scenario is overwhelmingly found as useful (69 respondents out of 135 said something to this effect), with 22 respondents indicating that this extra information is boring or not special in their opinions.

Polarization - airport info

  • For the semi-frequent Starbucks guest, the respondents indicated that the personalized service (utilitarian service orientation) is useful (42 out of 116 said this) and that they like the personalization (49 out of 116 said this). Only 8 said it is creepy, compared to 53 out of 117 respondents on the hotel Starbucks.

Polarization - local starbucks

  • A couple of scenarios were more polarizing. On these two, the crowd couldn’t make up its mind:
    • On the Target cereal sale scenario, respondents were split between saying it’s useful and creepy. Out of 154 respondents, 44 said it was useful, and 44 said it was creepy. (Ten people said both.) Additionally, 34 people said they liked the personalization aspect:

Target Sale distribution

  • The hotel Starbucks scenario is especially interesting to me because it very much probes the limits of creepiness and corporate chain store behavior. It’s also useful to compare the data with the data from the similar scenario, where a semi-regular visits his local Starbucks. In that scenario, only 8 people found it creepy, with most people remarking positively on the usefulness and the personalization factors. In the hotel Starbucks scenario, 53 out of 117 responses mentioned the creepiness, with only 32 saying they liked the personalization, and only 22 finding it useful:

 Polarization - hotel starbucks

  • Interestingly, from the employee perspective, two scenarios didn’t get any scores below 3. That would be the hotel information scenario.

Polarization - Hotel Info emp povPolarization - Airport Info emp pov

  • One surprise is that in the Target cereal sale scenario, 11 out of 44 respondents said that the scenario made them uncomfortable. (the creepiness factor.) That was the largest number of respondents who said an employee-POV scenario made them uncomfortable. The employee-POV scenario that was second-highest in terms of making respondents uncomfortable was the makeup brand scenario where the employee gives the customer a free sample of the item that goes with the products she already has at home.

What does it all mean?

  • I think that the fact that the variers so disliked the employee-POV local coffee shop scenario is a sign that the variers are a group of people that care a lot about the “human touch.” (I say this because the variers rated that scenario from the customer POV as a 4.2, same as the overall average. Therefore, I wouldn’t claim that the variers are upset by the creepiness factor; rather, they have a stronger affinity for “authenticity.”)
    • I think they may also be a more utilitarian group. I think on scenarios such as the 50th movie, they were more likely to give a low number (instead of a 3) because of the uselessness of the personalization. (Also see the next bullet for more on respondents’ reaction to utilitarian benefts.)
  • I think part of why the hotel Starbucks scenario fell flat is not only the creepiness factor, but also because the scenario didn’t emphasize the customer’s utilitarian service orientation. (The local Starbucks scenario did emphasize that, and many respondents mentioned the usefulness. In the hotel Starbucks scenario, Bob’s service orientation was not mentioned.) The fact that the respondents overwhelmingly gave it a 3 rather than a 0 or 1 makes me think that they found it more “useless” than “upsetting.”
  • People clearly like and feel comfortable with more information. (The Hotel Info and Airport Info scenarios were some of the most highly rated.) This fits in with my interviews and fall-semester research 100%.

Some analysis from the survey!

I’ve put together some high-level analysis based off my online survey:

Highest rated scenarios


  1. Home Depot worker helps customer avoid buying wrong primer (4.6) and Airport ticketing agent offers extra info to nervous passenger (4.6)
  2. Hotel check-in desk worker gives customized info to customer on what’s new since she last stayed here (4.5)
  3. Barista recognizes semi-regular employee during his morning coffee pickup (4.2)
    1. Employee version is 4.3


  1. Airport ticketing agent offers extra info to nervous passenger (5.0) and Hotel check-in desk worker gives customized info to customer on what’s new since she last stayed here (5.0)
  2. Home Depot worker helps customer avoid buying wrong primer (4.8)
  3. Cosmetic saleswoman notifies customer of product relating to customer’s political beliefs (anti-animal testing) (4.6)
    1. Customer version is 3.7

Regarding Mechanical Turks vs. other respondents

The Turk respondents and the other respondents gave similar respondents to one another.

Respondents with high variance

Respondents whose answers varied more than the average respondent (i.e., not giving all positives, all neutrals, or all negative numerical ratings to the scenarios) were slightly more negative overall than the average respondent.

The scenarios where the variers’ responses were most different from the whole group:


  1. Ticket seller notifies moviegoer that it’s his 50th movie at this theater, 18.6% decrease. (This is also the worst-rated scenario.)
  2. Barista recognizes new, out-of-town customer as if he were a regular, 18.4% decrease. (This is also the 2nd-worst rated scenario.)
  3. Cosmetic saleswoman offers customer a makeup that goes with the customer’s already-purchased makeups at home, 14.7% decrease. (This is also the 3rd-worst-rated scenario.)


  1. Target worker tells customer about favorite grocery item on sale, 7% decrease. (This is by far the biggest gap among varier responses; the next most varier-disliked scenario is 11.0% less liked by variers—see below.)
  2. Barista recognizes semi-regular employee during his morning coffee pickup, 11.0% decrease. (Note that from the customer perspective, this scenario received third place.)
    1. Note: for the above two scenarios, the varier population gave the scenarios an average score below 3.0—these are the only two scenarios that got a below 3.0 avg. score from a population.
  3. Cosmetic saleswoman notifies customer of product relating to customer’s political beliefs (anti-animal testing), 9.3% decrease. (Note that this scenario, also in employee version, was third-most-liked by the overall group, but got middling scores from the whole group for its customer version.)

Southwest Airlines providing live entertainment on surprise flights

Somehow, I think that if I’d tested this concept in my online test, it would have scored really poorly:

Over the past several years, the quirky carrier has surprised passengers with unannounced in-flight musical sets by middle-of-the-road rock bands, an unannounced in-flight fashion show, and even an unannounced in-flight wedding. Now comes the latest volley in Southwest’s campaign to inflict glee on your customer experience, whether you want it or not: book readings.

The rest is here, from Slate.

Analyzing the survey data

(Firstly, thanks to everyone who took the survey. I had over 200 responses.)

I’m now analyzing the data that came back from my survey. It’s a lot to sort through–twenty questions times over 200 respondents can’t be written out onto post-it notes.

Screen Shot 2015-03-22 at 1.41.03 AM
There’s a taste of what my desktop looks like right now.

I’ve got a little preliminary analysis. Each scenario had a question asking the survey-taker to rate the concept from 0 to 6, with 0 being “much less desirable” and 6 being “much more desirable.” (A neutral response therefore would be 3.)

Each scenario had a customer point of view version and an employee point of view version. The customer point of view scenarios asked,

Please think of a time when you recently shopped in a store like this. How does the above scenario compare to your own experience?

and the employee point of view scenarios asked,

How well do the tools this salesperson uses help to make them better at their job?

Here’s how they were rated, on average:

avg ratings chart 1

They’re in order with the highest-rated customer point of view scenario on top. (A full list of the scenarios is here, or you’re welcome to click through and answer the survey.)