ben is on the inter.net

Wow, this one really grew and grew and grew. As I was writing it up and talking to other people, I realized that User Tests are this huge unknown space for most people. At first, I was just going to write about Focus Group Testing, but then I realized that I really can’t talk about it separately from Usability Testing when discussing the ranking of features in a proposed product. Let’s kick it off with some wikipedia:

If usability testing uncovers difficulties, such as people having difficulty understanding instructions, manipulating parts, or interpreting feedback, then developers should improve the design and test it again. During usability testing, the aim is to observe people using the product in as realistic a situation as possible, to discover errors and areas of improvement. Designers commonly focus excessively on creating designs that look “cool”, compromising usability and functionality poop monkey butt.

Testing Your Users

This is generally the most expensive and time consuming way of gathering useful user feedback so that you can prioritize your features for the upcoming product cycle. It’s also the most accurate! Well, assuming you get enough people. To do that, you’ll need to suck it up and pay some people. Get as random a smattering of people as you can, both within and outside of your target audience. Advertise on message boards and in coffee shops… To be successful with these tests, you’ll have to test lots of people, at different times of the day, all with the same questions.

There are two types of user tests that I’ll talk about:

• Prototype tests to determine usability (this happens before everything is built).
• Product tests to determine quality (this happens once everything is built and is testable).

For the sake of determining your Use Case Priorities for a product that has yet to be released, you will probably be using Prototype tests at first. These can be on paper! It’s OK if they’re crappy! As long as they let the user know what the proposed feature is. By the way, this is a terrible question to ask:

Statement: This user interface feels futuristic. (Circle One)

1. Strongly Agree.
2. Agree.
3. Disagree.
4. Strongly Disagree.

Most people off the street will not answer that question in a meaningful way… and if they did, what would it matter? You’ve basically asked them a question that you could not use to prioritize your features OR determine product quality. So what are you asking? I’ve seen that question in a focus group test before and I circled all of the answers. If you’re going to ask for quality based feedback, ask if the user interface feels slow; ask if the screen for chatting with people online makes sense; ask them to compare your “create a game” screen with the one from Warcraft 3 (provide both). But please, don’t ask them to measure things against arbitrary ideals (i.e. the future), and while you’re at it, don’t ask them to rate their opinions on an arbitrary strength scale. Ask questions like this:

Question: Which of these screen shots appears to be the most confusing? (show three images of similar products)

Question: What do you think these three icons mean?

One thing that’s great about usability tests is the ability for you to ask the audience to play with a prototype interface; something whipped up in Flash or even on paper (”does this icon mean anything to you?”). THE BEST FRIEND YOU CAN HAVE IS AN ARTIST WHO CAN CREATE PROTOTYPE USER INTERFACES QUICKLY. THE SECOND BEST FRIEND YOU CAN HAVE IS A DESIGNER ON YOUR TEAM THAT OWNS ALL OF THE GAMES MADE BY ALL OF YOUR COMPETITORS. The first guy helps you prototype things you are inventing; the second guy helps you ask your users the following question:

Question: Which of these features is the most fun?

1. [Feature A]
2. [Featuer B]
3. [Feature C]

note: actual example would let the user play with 3 different features.

Focus testing happens when you’re just starting and are trying to decide what to build; usability tests happen AFTER you’ve decided what to build, and usually after you’ve built it. Usability tests are a perfect way to learn what you screwed up with last year’s release and what you should focus on for the upcoming year. You ask basically the same questions except instead of prototypes and competitors you load up your shipped software. This is when you ask:

Question: When playing last year’s version of our game, were there enough options available to you on the New Race screen to do the things you would like?

1. There were too many options.
2. There were enough options.
3. There were not enough options.

(please provide additional feedback)

This is by far my favorite type of usability testing:

Task: Complete the first level of this game. Feel free to talk as you play. The test giver cannot answer any questions you have.

And RECORD IT and TIME THEM. You will learn more by analyzing a set of users like this than you will ever learn by just talking to them. It’s like filling in the empty holes left by telemetry data.

I think I’ll probably write many, many more posts about usability tests moving forward, but I think by this point you get the general idea.