ben is on the inter.net

Wow, this one really grew and grew and grew. As I was writing it up and talking to other people, I realized that User Tests are this huge unknown space for most people. At first, I was just going to write about Focus Group Testing, but then I realized that I really can’t talk about it separately from Usability Testing when discussing the ranking of features in a proposed product. Let’s kick it off with some wikipedia:

If usability testing uncovers difficulties, such as people having difficulty understanding instructions, manipulating parts, or interpreting feedback, then developers should improve the design and test it again. During usability testing, the aim is to observe people using the product in as realistic a situation as possible, to discover errors and areas of improvement. Designers commonly focus excessively on creating designs that look “cool”, compromising usability and functionality poop monkey butt.

Testing Your Users

This is generally the most expensive and time consuming way of gathering useful user feedback so that you can prioritize your features for the upcoming product cycle. It’s also the most accurate! Well, assuming you get enough people. To do that, you’ll need to suck it up and pay some people. Get as random a smattering of people as you can, both within and outside of your target audience. Advertise on message boards and in coffee shops… To be successful with these tests, you’ll have to test lots of people, at different times of the day, all with the same questions.

There are two types of user tests that I’ll talk about:

• Prototype tests to determine usability (this happens before everything is built).
• Product tests to determine quality (this happens once everything is built and is testable).

For the sake of determining your Use Case Priorities for a product that has yet to be released, you will probably be using Prototype tests at first. These can be on paper! It’s OK if they’re crappy! As long as they let the user know what the proposed feature is. By the way, this is a terrible question to ask:

Statement: This user interface feels futuristic. (Circle One)

1. Strongly Agree.
2. Agree.
3. Disagree.
4. Strongly Disagree.

Most people off the street will not answer that question in a meaningful way… and if they did, what would it matter? You’ve basically asked them a question that you could not use to prioritize your features OR determine product quality. So what are you asking? I’ve seen that question in a focus group test before and I circled all of the answers. If you’re going to ask for quality based feedback, ask if the user interface feels slow; ask if the screen for chatting with people online makes sense; ask them to compare your “create a game” screen with the one from Warcraft 3 (provide both). But please, don’t ask them to measure things against arbitrary ideals (i.e. the future), and while you’re at it, don’t ask them to rate their opinions on an arbitrary strength scale. Ask questions like this:

Question: Which of these screen shots appears to be the most confusing? (show three images of similar products)

Question: What do you think these three icons mean?

One thing that’s great about usability tests is the ability for you to ask the audience to play with a prototype interface; something whipped up in Flash or even on paper (”does this icon mean anything to you?”). THE BEST FRIEND YOU CAN HAVE IS AN ARTIST WHO CAN CREATE PROTOTYPE USER INTERFACES QUICKLY. THE SECOND BEST FRIEND YOU CAN HAVE IS A DESIGNER ON YOUR TEAM THAT OWNS ALL OF THE GAMES MADE BY ALL OF YOUR COMPETITORS. The first guy helps you prototype things you are inventing; the second guy helps you ask your users the following question:

Question: Which of these features is the most fun?

1. [Feature A]
2. [Featuer B]
3. [Feature C]

note: actual example would let the user play with 3 different features.

Focus testing happens when you’re just starting and are trying to decide what to build; usability tests happen AFTER you’ve decided what to build, and usually after you’ve built it. Usability tests are a perfect way to learn what you screwed up with last year’s release and what you should focus on for the upcoming year. You ask basically the same questions except instead of prototypes and competitors you load up your shipped software. This is when you ask:

Question: When playing last year’s version of our game, were there enough options available to you on the New Race screen to do the things you would like?

1. There were too many options.
2. There were enough options.
3. There were not enough options.

(please provide additional feedback)

This is by far my favorite type of usability testing:

Task: Complete the first level of this game. Feel free to talk as you play. The test giver cannot answer any questions you have.

And RECORD IT and TIME THEM. You will learn more by analyzing a set of users like this than you will ever learn by just talking to them. It’s like filling in the empty holes left by telemetry data.

I think I’ll probably write many, many more posts about usability tests moving forward, but I think by this point you get the general idea.

Telemetry Data

Telemetry is the process of gathering data from a remote location. It’s used everywhere from vending machines to websites to agriculture. In the case of most software, this would usually be anonymous data sent back to the software company. Websites monitor their traffic, search terms, referring sites, most viewed pages, most downloaded files, web browsers used, you name it. Video games are starting to monitor the success of their advertising and track data similar to that of most web sites: things like system specs, user paths through the game, most commonly played modes, and the users’ failed attempts to accomplish goals (”85% of our users do not successfully get into Online Target Practice mode; next year we should redesign that critical path or just outright cut the mode”).

This is the most accurate way of verifying your users’ behavior because you’re actually monitoring it. Think of it as profiling user activity for a better way to optimize it. The biggest problem with Telemetry Data is that you cannot ask users what they want before you ship, you have to ship and then see if you can improve your product for a 2.0 release. Also, apparently users hate the concept of telemetry; they feel like they’re being spied on. Odd, considering one of amazon’s greatest successes is their personalization of the site’s suggestions based on your use patterns… Personally I’d love to know that my Use Cases were considered when Sony, Microsoft, and Nintendo look through user stats in their online marketplaces, or when Blizzard decides what to fix in the next World of Warcraft patch.

User Requests

This is probably the most common form of Use Case Prioritization that ends up affecting the design of a software product. However, its usefulness, like the Dev Team’s input, can be contradictory, confusing, and often inaccurate. Users will say they want one thing, such as “more online features!” when they really didn’t use half of the online features from last year; they want the same online features from last year, only they want them to be better organized. They’ll say “we want Super Hero mode!” when they don’t even know what they could be getting instead. The problem with talking to end users to drive priority is that the loudest end users, the so-called alpha consumers, are going to ask for one thing very loudly. Since this sort of data gathering is usually a one way street (i.e., it is uncommon for a software company to bring all of their users into this discussion; instead “hard-core” users will get asked for feedback, or, even worse, video game reviewers and websites will be consulted to determine the Next Big Feature.

Even anonymous surveys can’t really provide you with much useful feedback; most users will only participate if they get something in return, and even then most people will just do the minimum amount of thinking required to get through the survey and get to their prize.

Wow, it actually sounds like it’s pretty terrible to talk to your users! To be fair, they do have the information you want, it’s just difficult to communicate with them in a way where you can turn their actual desires into feature requests that will help you schedule a more solid product. That’s where some slightly more difficult Use Case Priority Mining comes into play…

I mentioned a few days ago that prioritizing Use Cases is an important part of figuring out which features your product should have. But how do you determine which Use Cases are high priority and which ones won’t affect most users? Why, you ask them of course! Actually, there are 5 main ways to gather the data for your Use Case priorities. I’ll go through them from cheapest (available to all development teams) to most expensive (available to development teams with a budget).

Development Team Verification

This is by far the cheapest way to gather Use Case data: ask the members of the development team to try and priroritize the current feature set from “all users will do this” to “no users will do this.” Experienced members of the team may provide extremely valuable feedback…

However, if the development team is comprised mostly of people who will not be using this product, most of the feedback gathered here should be taken with a grain of salt. For instance, and this may come as a bit of a shock (unless, well, you had ever met me before), I’m not a huge sports fan, and only rarely would I pick up a sports game. My feedback on whether or not users would hit a feature would probably be wildly inaccurate. Granted, I do know video games, and I can usually tell when a given use case would be low priority if it is clunky for the user or if it violates any usability or design expectations. (Also, I’m a cheater and I would do research before responding to any of this, but we’ll get into that later)

In short, the dev team is not the only source of feedback that the design team should consult when deciding the priority of a feature. More soon!

Do not force the user to make two choices at the same time. Give the user a choice that they will know the answer to, and then give them another choice.

For instance:

QUESTION 1: How many players?
[1,2,3,4]

QUESTION 2: Would player [1,2,3,4] like to load a save file?
[No,Yes]

QUESTION 3: What team would player [1,2,3,4] like to play with?
[Red Team, Blue Team]

Is much better than:

QUESTION: How many players? Would player [1,2,3,4] like to load a save file? What team would player [1,2,3,4] like to play with?
[1,2,3,4] * [No,Yes] * [Red Team, Blue Team]

Do you have any idea how long it takes me to use a new DVD player remote, most of which have 50 or so buttons?

iPods have this kind of neat feature where they will pause playback whenever headphones are unplugged if the iPod is not connected to a dock cable. Some possible use cases that brought us this feature are:

USE CASE: Jane accidentally drops her iPod and her headphones become disconnected; she is no longer listening to music.

USE CASE: Bob is driving and listening to his iPod, which is connected to his car stereo using an audio cable. When Bob arrives at the mall, he unplugs his audio cable so that he can take his iPod with him, but he doesn’t want to listen to music until he gets inside the mall.

Those are pretty common use cases, I’d guess. Pausing playback in both of those situations makes perfect sense. Here’s a use case where the feature doesn’t make sense:

USE CASE: Ben and the rest of his programming team are tired and are working on a Saturday. Ben, in a fit of either rage or insanity, starts listening to Jingle Cats on his iPod. He decides to share the Love, and unplugs his headphones so that he can plug in his speakers.

In this case, Ben still wanted to listen to music, so pausing didn’t make sense. But you know what? This is a low priority (uncommon) use case. The other two cases listed are high priority (extremely common), and it made sense for Apple to implement the feature. As for my broken use case? Apple COULD have solved the problem, but DIDN’T. Bad Apple? No, Very Good Apple. You see, they shipped on time, with a lot of features, and most users think the device suits their every need. You have to prioritize Use Cases when seeing what fits into your product’s schedule, otherwise you ship too many confusing features, late, over-budget, and you won’t even know that this is happening.