The In-Person Usability Test

The first option that came to mind when we decided to start doing usability testing was the in-person test. Conventional wisdom suggests that this is the most effective–but most expensive–method. However, one of the benefits of being in a town with a large university is that there are many students on a limited budget willing to play on a computer for an hour in exchange for a gift certificate to the local bakery.

We found the in-person tests very informative. Interacting with participants and asking about their thoughts in real-time gives us unique insight into the testers’ thought processes that really isn’t available any other way. This led us to make a few changes that we otherwise wouldn’t have known to make. Since we sat right next to the testers, we could see that they only looked at the outline in the Form Builder. For example, almost all of our testers were confused because they didn’t notice the form Preview changing when they added elements to their form, so we realized we needed to make this UI feedback more visible.

There are a few drawbacks to in-person tests. While paying students to do tests isn’t particularly expensive monetarily, the time spent finding participants, setting up, doing the tests, and then analyzing the results adds up quickly. For each test, both of us in the Bloomington office observed, allowing for better recollection (even though we videoed the tests) and for more effective questioning. Analyzing the results from these hour-long tests took about one day per test.

Remote Usability Testing

After our first round of in-person usability testing, we decided to explore the options for remote usability testing. We anticipated were a broader tester base, a smaller time commitment on our part, and the ability to conduct more frequent tests.

We’ve used two different remote usability testing services. The first service we used is UserTesting.com. The greatest thing about this site was that we received the total number of requested responses (three) within one hour of requesting the test. We received a video and a written summary from each tester. However, we found these testers a bit too efficient; they were clearly advanced web users and seemed to have an easier time completing the task than our in-person testers and, probably, than our FormAssembly users.

We tried UserTesting.com when it was new to the market and paid $68 for the three tests.

The second remote testing service we used was TryMyUI.com. This time, I made sure to select the beginner/intermediate option for web users, since the UserTesting.com results had been a little too polished. It did take longer to get all three responses from TryMyUI.com (about 5 hours). Having beginner web users test the application gave us a better approximation for the way most of the FormAssembly users initially experience the application.

We spent $75 for the three tests.

Although the out-of-pocket cost difference between in-person and remote tests is negligible for us, other costs were lower for the remote tests. Since we weren’t moderating the tests, we worked on other tasks while the tests were happening. Analyzing and organizing the data for these 20 minute tests was quicker. Nonetheless, we did get valuable information from the remote tests. We observed a bug that we hadn’t seen with the in-person tests, and got further confirmation of users’ difficulties with the Form Builder’s side tab design. This, combined with our experience during the in-person tests, convinced us to mock up a complete redesign of the Form Builder.

Spur-of-the-Moment Tests

Sometimes you need to test a very specific aspect of your site or application: a color, wording, or image, for example. In these cases, a full-scale usability test is impractical and a little bit of overkill. When we wanted to know what labels would be most effective for the different question types in the Form Builder, we used Amazon’s Mechanical Turk service to poll a number of testers. We found that few people knew the names for different question types (radio button caused particular confusion) but most people understood the image question type. After this test, we increased the size of the images in the Form Builder and saw a reduction in the number of support requests confusing the different question types.

A Little of This, a Little of That

We’ve concluded that for us, both in-person and remote tests are effective usability evaluation methods. We’ve decided to combine early-stage in-person tests with post-release remote tests. Doing in-person tests with parts of the application that haven’t been released gives us the opportunity to test earlier and have more control over the situation. It also allows us to take risks and test more complex aspects that would be difficult to convey to a remote tester. Remote testing, especially with beginner users, gives us a chance to be a fly on the wall when someone first encounters the application. In this way we can improve that experience so that it is easy and productive.

Several commenters have pointed out that @importRule would not be that useful since you can already assign more than one CSS class to an element.

However, having (apparently) some time to waste this morning, I thought it would be interesting to implement this pseudo css rule anyway.

Here’s the example page, with a short explanation of the (very simplistic) javascript:

Again, I’m not claiming that this is very useful, but if one wanted to create a javascript css post-processor, that would be a good way to start.

After months of work, the new FormAssembly.com is nearing completion.

This new version is a complete re-write, and since it is hosted on a completely separate platform, we are looking forward to a very smooth transition, with the two versions running concurrently for as long as necessary.

The feedback from the early beta-testers is pretty good. There are also plenty of new (and long requested) features, so I’m confident that our users will be happy with the new version.

I will also look at the traffic and conversion numbers attentively anxiously. In theory, the new site should perform much better, but sometimes what sells is really not what you think, so it will be interesting to compare the numbers. I’ll try to post my findings here.

A simple demonstration is available here, with all the explanations here.

Version 2.1 includes bug fixes and performance improvements. The major changes are:

• Upgraded to latest version of Sarissa (0.9.7.6)
• XSL stylesheet should now set the XSL:output method to “HTML”. This prevents an unnecessary HTML serialization & parsing step and avoids invalid HTML markup altogether, like self-closing <textarea/>. While this change makes 2.1 not backward compatible, this should be the only required change if you are using Freja 2.0.

I also updated the documentation and tutorials, which were beginning to be seriously out-of-date.

The Freja core development team is comprised of Cedric Savarese and Troels Knak-Nielsen. Freja is an Open-Source framework built on top of the Sarissa library by Manos Batsis.

This is still very experimental, but as you can see in the article, there are some clear benefits in terms of accessibility and speed when using XUL for complex UI widgets.

Check out the documentation and source for this experimental library and thanks to all for the feedback.

This time, I was thinking about the interface for the next version of the Form Builder (I know, somehow it’s always work related) and I was wondering if XUL could be of any help here.

A couple days later, I had this working: tabbed panel example.

If you are not using the Firefox browser, you will not see anything interesting, so here’s a screenshot of the second panel, viewed in Firefox (XUL powered).

The first panel in the example uses HTML+Javascript (a.k.a DHTML). It is very basic, there are better DHTML panel widgets of course. The idea is that you could enhance an existing javascript UI library to render widgets with XUL on XUL-compatible browsers…

You can check out the source code if you want to see how it’s done.

Update: As whoughton pointed out, it doesn’t degrades to DHTML as it should in Safari. I will fix that soon, it’s probably not a serious bug.

Update 2: The problem in Safari is caused by its user-agent. To detect whether I should render with XUL or DHTML, I look at the user-agent string. If it contains the word “gecko”, I assume the browser uses the Gecko rendering engine and therefore supports XUL… Well it turns out Safari’s user-agent string includes “Gecko” even though it’s not based on Gecko at all. Weird. Anyway, I’ll just have to fix the detection code.

Update 3. It’s fixed in Safari, thanks to Boris tips.