Striking a Balance: Developing Our Usability Testing Strategy

August 3rd, 2010 by Hillary Elmore

Over the past several months, we’ve been experimenting with different strategies for testing FormAssembly and coming up with the best way to maximize our benefit while keeping the cost reasonable.

The In-Person Usability Test

The first option that came to mind when we decided to start doing usability testing was the in-person test. Conventional wisdom suggests that this is the most effective–but most expensive–method. However, one of the benefits of being in a town with a large university is that there are many students on a limited budget willing to play on a computer for an hour in exchange for a gift certificate to the local bakery.

We found the in-person tests very informative. Interacting with participants and asking about their thoughts in real-time gives us unique insight into the testers’ thought processes that really isn’t available any other way. This led us to make a few changes that we otherwise wouldn’t have known to make. Since we sat right next to the testers, we could see that they only looked at the outline in the Form Builder. For example, almost all of our testers were confused because they didn’t notice the form Preview changing when they added elements to their form, so we realized we needed to make this UI feedback more visible.

There are a few drawbacks to in-person tests. While paying students to do tests isn’t particularly expensive monetarily, the time spent finding participants, setting up, doing the tests, and then analyzing the results adds up quickly. For each test, both of us in the Bloomington office observed, allowing for better recollection (even though we videoed the tests) and for more effective questioning. Analyzing the results from these hour-long tests took about one day per test.

Remote Usability Testing

After our first round of in-person usability testing, we decided to explore the options for remote usability testing. We anticipated were a broader tester base, a smaller time commitment on our part, and the ability to conduct more frequent tests.

We’ve used two different remote usability testing services. The first service we used is The greatest thing about this site was that we received the total number of requested responses (three) within one hour of requesting the test. We received a video and a written summary from each tester. However, we found these testers a bit too efficient; they were clearly advanced web users and seemed to have an easier time completing the task than our in-person testers and, probably, than our FormAssembly users.

We tried when it was new to the market and paid $68 for the three tests.

The second remote testing service we used was This time, I made sure to select the beginner/intermediate option for web users, since the results had been a little too polished. It did take longer to get all three responses from (about 5 hours). Having beginner web users test the application gave us a better approximation for the way most of the FormAssembly users initially experience the application.

We spent $75 for the three tests.

Although the out-of-pocket cost difference between in-person and remote tests is negligible for us, other costs were lower for the remote tests. Since we weren’t moderating the tests, we worked on other tasks while the tests were happening. Analyzing and organizing the data for these 20 minute tests was quicker. Nonetheless, we did get valuable information from the remote tests. We observed a bug that we hadn’t seen with the in-person tests, and got further confirmation of users’ difficulties with the Form Builder’s side tab design. This, combined with our experience during the in-person tests, convinced us to mock up a complete redesign of the Form Builder.

Spur-of-the-Moment Tests

Sometimes you need to test a very specific aspect of your site or application: a color, wording, or image, for example. In these cases, a full-scale usability test is impractical and a little bit of overkill. When we wanted to know what labels would be most effective for the different question types in the Form Builder, we used Amazon’s Mechanical Turk service to poll a number of testers. We found that few people knew the names for different question types (radio button caused particular confusion) but most people understood the image question type. After this test, we increased the size of the images in the Form Builder and saw a reduction in the number of support requests confusing the different question types.

A Little of This, a Little of That

We’ve concluded that for us, both in-person and remote tests are effective usability evaluation methods. We’ve decided to combine early-stage in-person tests with post-release remote tests. Doing in-person tests with parts of the application that haven’t been released gives us the opportunity to test earlier and have more control over the situation. It also allows us to take risks and test more complex aspects that would be difficult to convey to a remote tester. Remote testing, especially with beginner users, gives us a chance to be a fly on the wall when someone first encounters the application. In this way we can improve that experience so that it is easy and productive.

« Previous post | Next Post »

Leave a Reply