The In-Person Usability Test

The first option that came to mind when we decided to start doing usability testing was the in-person test. Conventional wisdom suggests that this is the most effective–but most expensive–method. However, one of the benefits of being in a town with a large university is that there are many students on a limited budget willing to play on a computer for an hour in exchange for a gift certificate to the local bakery.

We found the in-person tests very informative. Interacting with participants and asking about their thoughts in real-time gives us unique insight into the testers’ thought processes that really isn’t available any other way. This led us to make a few changes that we otherwise wouldn’t have known to make. Since we sat right next to the testers, we could see that they only looked at the outline in the Form Builder. For example, almost all of our testers were confused because they didn’t notice the form Preview changing when they added elements to their form, so we realized we needed to make this UI feedback more visible.

There are a few drawbacks to in-person tests. While paying students to do tests isn’t particularly expensive monetarily, the time spent finding participants, setting up, doing the tests, and then analyzing the results adds up quickly. For each test, both of us in the Bloomington office observed, allowing for better recollection (even though we videoed the tests) and for more effective questioning. Analyzing the results from these hour-long tests took about one day per test.

Remote Usability Testing

After our first round of in-person usability testing, we decided to explore the options for remote usability testing. We anticipated were a broader tester base, a smaller time commitment on our part, and the ability to conduct more frequent tests.

We’ve used two different remote usability testing services. The first service we used is UserTesting.com. The greatest thing about this site was that we received the total number of requested responses (three) within one hour of requesting the test. We received a video and a written summary from each tester. However, we found these testers a bit too efficient; they were clearly advanced web users and seemed to have an easier time completing the task than our in-person testers and, probably, than our FormAssembly users.

We tried UserTesting.com when it was new to the market and paid $68 for the three tests.

The second remote testing service we used was TryMyUI.com. This time, I made sure to select the beginner/intermediate option for web users, since the UserTesting.com results had been a little too polished. It did take longer to get all three responses from TryMyUI.com (about 5 hours). Having beginner web users test the application gave us a better approximation for the way most of the FormAssembly users initially experience the application.

We spent $75 for the three tests.

Although the out-of-pocket cost difference between in-person and remote tests is negligible for us, other costs were lower for the remote tests. Since we weren’t moderating the tests, we worked on other tasks while the tests were happening. Analyzing and organizing the data for these 20 minute tests was quicker. Nonetheless, we did get valuable information from the remote tests. We observed a bug that we hadn’t seen with the in-person tests, and got further confirmation of users’ difficulties with the Form Builder’s side tab design. This, combined with our experience during the in-person tests, convinced us to mock up a complete redesign of the Form Builder.

Spur-of-the-Moment Tests

Sometimes you need to test a very specific aspect of your site or application: a color, wording, or image, for example. In these cases, a full-scale usability test is impractical and a little bit of overkill. When we wanted to know what labels would be most effective for the different question types in the Form Builder, we used Amazon’s Mechanical Turk service to poll a number of testers. We found that few people knew the names for different question types (radio button caused particular confusion) but most people understood the image question type. After this test, we increased the size of the images in the Form Builder and saw a reduction in the number of support requests confusing the different question types.

A Little of This, a Little of That

We’ve concluded that for us, both in-person and remote tests are effective usability evaluation methods. We’ve decided to combine early-stage in-person tests with post-release remote tests. Doing in-person tests with parts of the application that haven’t been released gives us the opportunity to test earlier and have more control over the situation. It also allows us to take risks and test more complex aspects that would be difficult to convey to a remote tester. Remote testing, especially with beginner users, gives us a chance to be a fly on the wall when someone first encounters the application. In this way we can improve that experience so that it is easy and productive.

I’m not going to criticize this approach but I honestly thought there would be more young entrepreneurs in the Valley interested in emulating companies like Salesforce or 37signals.

It is quite ironic that David Heinemeier Hansson is seen as going against the grain, when his presentation was packed with nothing but grounded advice and a healthy dose of common sense. What’s wrong with the old true and tested way of charging money for your service?

Interestingly, DHH also said that Backback (37signals’ second app after Basecamp), had to be rebranded and relaunched as a “business” application when they realized that the consumer market wasn’t really profitable.

There was maybe one theme missing from the picture though. Building a successful business takes time and hard work. A lot of time and a lot of work, so it was great to meet you all, I have to go back to work now.

While reading State of Denial, I came accross the following quote regarding how a CEO should allocate his or her time.

“…one third of the time on top priorities, one third on executive placement and development and the final third on evaluating the product or results.”

Had I gone through a MBA this could have sound like old news, but to me, it felt like I had just unearthed the secret rule of time management.

Obviously, I don’t have to worry about finding and managing executive people but even a small independant internet company owner needs to spend time on developing personal relationships with its customers, partners and advisors. The rewritten rule could be:

The entrepreneur’s schedule

• 1/3: Top Priorities
• 1/3: Networking, sales and support
• 1/3: Measuring progress

Let’s just say that I think I get the first one right.

Paul Graham, in an interview on TechCrunch, tells founders to “focus on making something great, instead of worrying about how to make money”. This is the kind of statement that generates a fair deal of controversy because it goes against conventional wisdom. We should all know that the best business model is to charge your customers. Should you fail to focus on that, you are bound to repeat the errors of the great dot-com bubble. This is roughly the argument Russel Beattie makes when he proposes a simple test for a “real” business: “do they take my credit card?”.

Before I join ranks with Paul Graham’s supporters, I would like to make a semantic point. A business model is not a revenue model. A revenue model is a way to generate revenues, like charging your customers or advertising. A sound business model includes other important aspects, such as:

• identifying and reaching out to your market.
• creating value for your users.
• building and maintaining a competitive advantage.
• growing your user base.
• and so on…

The revenue model has little to do with the reasons why a business will eventually succeed or fail. It is just one piece of the puzzle. Once you have managed to put all the other pieces in place, this one will fit right in.

When it comes to web applications, Richard White‘s answer is one.

“You only get one shot to impress people; don’t blow it because they won’t coming back next week to see if you’ve improved.” (link)

This is one of the lessons he draws from the demise of his employer, Kiko. Because of a less than perfect first version, nobody seemed to notice the much improved 2.0 release. With a disappointing adoption rate, the founders eventually lost their motivation and called it quits.

While it seems like a strong argument, I don’t think it is actually the right conclusion.

Whatever your target market may be, your pool of potential customers who have never heard of your service is much larger, by several degrees of magnitude, than the people who actually tried it. You may leave the early adopters unimpressed but if you can learn from them and focus on your next users, then you get another chance to make a good first impression.

So how many times can you make a first impression? About 1,043,000,000 times, which is roughly the number of people using the internet these days.

Veer West is a web technology startup founded in June 2006. Veer West is the next logical step for me, Cédric Savarese, after having created a couple of promising web applications.

I guess that makes me an entrepreneur, but really I am just a guy who enjoys designing and coding. A guy well aware of his inexperience and inadequacies when it comes to growing a business.

But I am not one to be deterred by a challenge. To help me keep my focus I thought I would share with you what it is that I intend to achieve with this company.

Goals are goals only if they are well defined, stated and shared. So here we go:

Long Term Goals (2-5 years)

• Veer West will be a small company with a few employees remotely commuting to a virtual office.
• Veer West revenues will come from recurring fees on web-based applications (they call that software as a service these days).

Short Term Goals (January 2007)

• Profitability
• A group of committed advisors on board
• A marketing plan

I will elaborate on these goals in future posts.