Qualitative vs Quantitative Results in Usability Testing

As regular readers might already know, I am taking a certificate program in User-Centered Design. This quarter I am taking a class on usability testing.

One of the things I've learned in the course is that there seems to be a debate on qualitative vs. quantitative results. For the sake of this post, I'll define them as thus:

Qualitative - descriptive, subjective. Not easy to provide objective measurement.

Quantitative - numerical, objective. In sufficient quantity can provide statistically significant facts.

As you can probably imagine, most people would like to see quantitative results. They are easy to understand and see just how important a particular issue might be (not to mention those cool graphs). However, due to budget and time constraints, it is extremely uncommon for a usability test to have a sufficient number of participants to qualify for statistical significance. Without statistical significance, are quantitative results useful?

I have to say emphatically no. Quantitative results are completely useless within a usability test. The goal of usability testing should be to find flaws in the design (from the users perspective) of the software. These flaws should be tracked and prioritized the same as any other defect within the application (hopefully you use issue/bug tracking software).

Usability testing should focus on qualitative feedback. There are many techniques that you can use to elicit feedback from a user (for example, thinking out loud). Post study questionnaires can also be useful, but the questions should be open ended. You can ask a participant to rate the software, but only to lead them to the next question which should ask why they rated it the way they did (the actual rating is useless without statistical significance).

Furthermore, when reporting the results of usability tests, it is important not to imply any kind of statistical significance. For example, don't say that 20% of the participants failed to complete a task when you only had five participants. For all you know, that participant is the only person on the planet that would fail or perhaps the other four just got lucky. Even mentioning something like 1 out of 5 can be dangerous. Always make sure that people that are reading your report understand the limitations of the data.

Usability Rule #3: Know Thy User

Before you can design an interface that is easy-to-use, you must first know something about the user and how they plan on using the interface. Unless you are building an interface for only yourself, it is unlikely that you will get it right without actually talking to potential users.

So what characteristics are important when designing an interface? Although this list might change based on the interface you are designing, for a line-of-business application, the important characteristics tend to be...

• Frequency of use - How often will the user use this interface. If they will use it every day, you will probably want to provide a very efficient interface. Focus on keyboard entry, allow entering codes instead of selecting from a list, make sure that frequently used fields are grouped together, etc. If they will use it once a month, the interface should be very easy to learn since they will probably have to re-learn it every time they open the interface.
• Duration of use - How long will the user use the interface. If they will only use the interface for a few minutes then you can get away with a less-attractive, less-responsive interface. However, if they use the interface all day long, it had better respond very well and be at least somewhat aesthetically pleasing (would you want to stare at an ugly screen all day?).
• Technical Competence - What kind of computer experience does the user have? Are they a power user? If so, you can provide many advanced bells and whistles (in fact, you probably should). On the other hand, if the user is afraid of turning on the computer, then the interface should be extraordinarily simple or the user doesn't stand a chance.
• Work Environment - What kind of environment will the user be using the interface in? Will they be sitting in a private office with air conditioning and sound protection, free of distractions? If so, then work environment probably won't have much impact on the design. However, if they work on the shop floor of a manufacturing plant with heavy machinery constantly grinding away and a conveyor belt that moves every minute, this could have a significant impact on your design. In this situation for example, you will want to provide an interface that will allow the user to complete the task in less than a minute and does not rely on audio cues.
• Hardware - What kind of hardware will the user be using? Hardware includes the computer and any peripherals (such as keyboard, mouse, monitor, printer, scanner, barcode reader, etc). Many users suffer with older, slower computers (you might be surprised by what even some software companies use). The user might not have much memory on their computer, have their monitor set to a low resolution, use a barcode scanner, or many other hardware considerations.
• Physical Limitations - Does the user have any physical conditions that might limit their use of the interface? This includes eyesight, hearing, and dexterity along with any other physical conditions that might be prevalent amongst your targeted users. Getting an age bracket is probably the most useful information you can gather for this characteristic because that can have a significant impact on their abilities, though work environment could very well play a role here as well (perhaps many users come from years on the shop floor and now have some hearing loss because of it, perhaps they have worked with chemicals and have difficulty controlling the mouse).

As you can imagine, if you are planning on your interface being used by many people, you will probably need to talk to many potential users to get a reasonable understanding of what is common (though if you only talk to a single individual that would still probably be better than having a software developer guess).

So how do you compile this information? Once you've interviewed the users, you can use the information to put together a user profile that includes all of the common and important user characteristics. Persona's have become a common technique for building this profile. A persona is basically a fictional character that you create that typifies the target user. The reason that many people use personas is that it makes it easy to discuss requirements by just mentioning the persona's name.

 Example conversation between two interface designers:

[John] Hey Sally, what do you think of this form I'm designing?

[Sally] Who's it for?

[John] Frank [persona] is going to be using it.

[Sally] It looks good except for that link. I don't think Frank will be able to use it. He's been working with those chemicals for years and would never be able to get the mouse to hover over it. You should make it larger.

As long as both designers agree on the persona, John can immediately see that Sally is right and they can avoid all the discussion usually accompanied by such criticism. Of course, the main power of personas is that John's interface is already mostly correct with no discussion necessary.

One of the best examples of personas I've found on the Internet is from Microsoft. The Dynamics team has put together an excellent list of personas for their product and have made a couple of those personas available on the Internet. Microsoft Dynamics RoleTailored Business Productivity whitepaper

Usability Rule #2: Stick with the K.I.S.S Principle

Keep It Simple, Stupid, or the way I like to phrase it, make the simple things simple and the complex things possible. This is a very common design mistake. Users ask for tons of features and developers work hard at delivering those features, but often times at the expense of a simple to use application.

Of course by simple, I mean "everything should be made as simple as possible, but no simpler" (quote attributed to Albert Einstein). If the interface doesn't make the most common way of performing a task as simple as possible, the interface has failed the user.

An example for good use of this principle is the Remote Desktop connection interface. When you start Remote Desktop, you see a UI with the absolute bare minimum interface for connecting to another computer (figure 1), a single textbox for entering the name of the computer you want to connect to and some buttons to perform what should be easy to understand commands (at least, easy to understand if you know what the application is supposed to do in the first place).

Figure 1: Simple interface for Remote Desktop.

Of course there are a lot of options that you might want to change when connecting to another computer, if you are an advanced user with a special need you can select the Options >> button. This button opens another window (figure 2) that allows you to change the advanced settings of Remote Desktop, you can even save the settings so that you don't need to go through the options dialog again.

Figure 2: Complex interface for Remote Desktop.

The term for this type of interface is progressive disclosure which simply means removing less frequently used interface elements from the primary screen while still providing a means of accessing those elements. This can be in the form of a secondary screen, such as in Remote Desktop, collapsing regions, or providing an Advanced tab within a tabbed interface (this can be seen in the options dialog in the Remote Desktop screenshot, figure 2).

If you want to see how not to design a UI, take a look at Bulk Rename Utility (figure 3). It seems to provide every possible option on the main form, regardless of how often it is used. Bulk Rename Utility is a useful tool intended for computer savvy people who can probably overlook a bad UI if it provides the features they need, but you would never want to put a UI like this in front of a technophobe.

Figure 3: Overly complex interface for Bulk Rename Utility.

So how do you know what the simplest interface is for the user? Asking the users is certainly a good start, though not entirely reliable (the mind has a tendency to assign equal importance to exceptional cases and common ones). Other methods include watching the user actually performing the task and logging feature usage and analyzing the results. Feature logging will probably result in the most comprehensive and accurate view of the current system, but it requires you to have a system in place already and for your users to be willing to share the information that is being logged.

By keeping the interface as simple as possible, the user is able to accomplish the task quicker with higher accuracy, less training, and fewer support calls. By providing a simple interface with the ability to perform more complex tasks you have taken nothing away from the user except a barrier to using the application.

Usability Rule #1: Consistency is More Important Than Correctness

One of my favorite usability rules is "consistency is more important than correctness." Just to be clear, this is not intended to diminish the importance of correctness, it simply means that if you can't be correct everywhere, you should at least be consistent. Consistency can help lower training costs and reduce mistakes.

As an example of this rule, let's take a complex application with many forms. Dates are displayed on many of these forms using a hard coded format of Month/Day/Year. You have been tasked to create a new form. As a good, conscientious developer, you know that dates should be formatted differently based on the culture of the user and that there is a built in function for formatting the date that actually requires less effort on your part than formatting the date yourself.

What should you do?

1. Format the date correctly using the built-in culture sensitive formatting.
2. Use the same hard coded format as every other place in the application.

Of course, the best answer is C, fix all the other places in the application where the date is formatted incorrectly. Unfortunately, that's not an option you. You don't have access to the other source code, there is business logic relying on the date format, nobody has reported it as a problem before, there is not enough time to fix it, we don't have the testing resources to test in different cultures, etc. (I'm sure you have heard plenty of reasons why people don't want to fix incorrect code).

The second best option is B, format the date consistently with all the other dates in the system. If all the dates are formatted wrong and the user is forced to use the software, they will eventually learn how the date is formatted and will expect the date to always be formatted that way. If their culture reverses the month and day (Day/Month/Year) the user may not realize that you are formatting it correctly and read the date incorrectly possibly causing costly mistakes (a customer doesn't get charged for 6 months, the company's domain name expires, a patient doesn't get scheduled for treatment soon enough and dies, etc.).

Of course, data formatting is not the only thing that should be consistent. Here is a list of common things to look for...

• Formatting of data - date, time, elapsed time, currency, numbers, etc
• Theme (colors and fonts) 
• Unit-of-measure - Be consistent where you can and always make sure the unit of measure is clearly marked.
• Captions - If a field is called XXX on one screen, it should always be called that. Same goes with commands (such as buttons, links, menu items, etc) and any other non-data text displayed to the user. This applies to icons as well.
• Shortcuts - Common commands should have the same shortcut across the entire application.
• Messages - Decide on a common style and use it consistently (for example, friendly or professional)
• Margins - Space your controls consistently. Different controls may require different margins, decide up front what those should be and stick with it.
• Alignment - Are captions aligned left? right? top? bottom? Are numbers aligned to the left? right?
• Layout - How do you group controls (group boxes, horizontal rules, tabs)? Where do you place commands that are related to data fields?
• General flow - How does a user open a record? Save it? Delete it? Create a new one? Access related data? View messages? How are errors handled?
• Consistent with other applications the user may be familiar with - If a user is already familiar with another application, you can leverage their skill from that application in your own.

Software Usability

I remember playing bumble-bee soccer (see footnote below) when I was a young kid. I was never around the ball, I was always looking for the position that needed to be filled, usually somewhere between the swarm and the goal. When I look back at most of the activities that I have been involved in, I see the same pattern being repeated, I always look for the most important task that is being neglected. Sometimes that has meant stepping on somebody's toes, other times I have been seen as a savior :).

Within the last couple of years, I have become more and more interested in software usability. Based on my soccer experience, this is probably because the company I currently work for has never concerned themselves with usability (or rather, the CEO and president were the usability experts, period). We never did usability testing, other than a demo to them. In fact, we never even bothered having a review process to make sure areas with similar functionality functioned similarly. Within the last year we were purchased by another company that alleges they are more focused on supporting existing customers than on attracting new ones (the difference as I understand it is more focus on the user and less on a feature checklist).

In my career I have come to the conclusion that most software sucks. In fact, it seems that the quality of software is inversely proportional to the price. Although the field of software usability is not new, it doesn't seem to be getting as much attention as it should (what could be more important than the user interface?). The benefits of creating usable software are well known. For the end user the benefits include increased productivity, reduced training, improved data entry accuracy, and reduced anxiety over fighting with bad software to get the job done. Their are also many benefits to the software company such as increased sales (to both new and existing customers), ability to charge a premium for the software and/or service, and decreased customer support costs. Although there is a cost to usability (you have to commit to it as a company, train developers, testers, and managers, and perform some type of usability studies which should include talking to real customers), based on what I believe to be true (from reading and limited experience), the benefits seem to far outweigh the costs.

I have recently read the book The Design of Everyday Things by Donald Norman. This seems to be the beginners handbook of usability. It lays the groundwork for understanding usability from a designers perspective. Although Donald is an academic (professor of cognitive science at the University of California), the book is well written and understandable even for those without a PHD. The book was originally published in 1988 and was recently reprinted with few modifications. One of the things I enjoyed most when reading his book were the predictions he made, some of which have come true already and others we still have not reached yet. If you are interested in learning about usability, I would recommend starting with this book.

The next book in my queue is The Usability Engineering Lifecycle: A Practitioner's Handbook for User Interface Design (Interactive Technologies) by Deborah Mayhew. This book is a bit larger and seems to go into a lot of detail on how to actually do usability. Of the few pages I've read so far, it seems like a good book :). Of course, I also read many different blogs that frequently include posts on usability. One of my favorites is Jeff Atwood's blog, Coding Horror. Jeff writes often and well and has become a minor celebrity within the software industry for posts such as The Programmer's Bill of Rights. He frequently writes posts on software interface design and usability and has a number of people who comment on his articles. He has also been featured on some popular (for developers anyway) net casts such as .Net Rocks.

Bumble-Bee Soccer - When young children play soccer, it resembles a swarm of bees around the soccer ball.