The In-Person Usability Test

The first option that came to mind when we decided to start doing usability testing was the in-person test. Conventional wisdom suggests that this is the most effective–but most expensive–method. However, one of the benefits of being in a town with a large university is that there are many students on a limited budget willing to play on a computer for an hour in exchange for a gift certificate to the local bakery.

We found the in-person tests very informative. Interacting with participants and asking about their thoughts in real-time gives us unique insight into the testers’ thought processes that really isn’t available any other way. This led us to make a few changes that we otherwise wouldn’t have known to make. Since we sat right next to the testers, we could see that they only looked at the outline in the Form Builder. For example, almost all of our testers were confused because they didn’t notice the form Preview changing when they added elements to their form, so we realized we needed to make this UI feedback more visible.

There are a few drawbacks to in-person tests. While paying students to do tests isn’t particularly expensive monetarily, the time spent finding participants, setting up, doing the tests, and then analyzing the results adds up quickly. For each test, both of us in the Bloomington office observed, allowing for better recollection (even though we videoed the tests) and for more effective questioning. Analyzing the results from these hour-long tests took about one day per test.

Remote Usability Testing

After our first round of in-person usability testing, we decided to explore the options for remote usability testing. We anticipated were a broader tester base, a smaller time commitment on our part, and the ability to conduct more frequent tests.

We’ve used two different remote usability testing services. The first service we used is UserTesting.com. The greatest thing about this site was that we received the total number of requested responses (three) within one hour of requesting the test. We received a video and a written summary from each tester. However, we found these testers a bit too efficient; they were clearly advanced web users and seemed to have an easier time completing the task than our in-person testers and, probably, than our FormAssembly users.

We tried UserTesting.com when it was new to the market and paid $68 for the three tests.

The second remote testing service we used was TryMyUI.com. This time, I made sure to select the beginner/intermediate option for web users, since the UserTesting.com results had been a little too polished. It did take longer to get all three responses from TryMyUI.com (about 5 hours). Having beginner web users test the application gave us a better approximation for the way most of the FormAssembly users initially experience the application.

We spent $75 for the three tests.

Although the out-of-pocket cost difference between in-person and remote tests is negligible for us, other costs were lower for the remote tests. Since we weren’t moderating the tests, we worked on other tasks while the tests were happening. Analyzing and organizing the data for these 20 minute tests was quicker. Nonetheless, we did get valuable information from the remote tests. We observed a bug that we hadn’t seen with the in-person tests, and got further confirmation of users’ difficulties with the Form Builder’s side tab design. This, combined with our experience during the in-person tests, convinced us to mock up a complete redesign of the Form Builder.

Spur-of-the-Moment Tests

Sometimes you need to test a very specific aspect of your site or application: a color, wording, or image, for example. In these cases, a full-scale usability test is impractical and a little bit of overkill. When we wanted to know what labels would be most effective for the different question types in the Form Builder, we used Amazon’s Mechanical Turk service to poll a number of testers. We found that few people knew the names for different question types (radio button caused particular confusion) but most people understood the image question type. After this test, we increased the size of the images in the Form Builder and saw a reduction in the number of support requests confusing the different question types.

A Little of This, a Little of That

We’ve concluded that for us, both in-person and remote tests are effective usability evaluation methods. We’ve decided to combine early-stage in-person tests with post-release remote tests. Doing in-person tests with parts of the application that haven’t been released gives us the opportunity to test earlier and have more control over the situation. It also allows us to take risks and test more complex aspects that would be difficult to convey to a remote tester. Remote testing, especially with beginner users, gives us a chance to be a fly on the wall when someone first encounters the application. In this way we can improve that experience so that it is easy and productive.

The On-Site edition offers the same great features enjoyed by thousands of our online users, but you can run it on your own server and keep your data safe and private.

FormAssembly On-Site is a reliable and secure solution designed to work on the most widely available configurations: Windows or Unix, IIS or Apache, and with any of the major database vendors. The support for such a wide range of technologies is made possible by the great open-source framework FormAssembly is built upon.

You can learn more about FormAssembly On-Site at http://onsite.formassembly.com

After months of work, the new FormAssembly.com is nearing completion.

This new version is a complete re-write, and since it is hosted on a completely separate platform, we are looking forward to a very smooth transition, with the two versions running concurrently for as long as necessary.

The feedback from the early beta-testers is pretty good. There are also plenty of new (and long requested) features, so I’m confident that our users will be happy with the new version.

I will also look at the traffic and conversion numbers attentively anxiously. In theory, the new site should perform much better, but sometimes what sells is really not what you think, so it will be interesting to compare the numbers. I’ll try to post my findings here.