Creating a User Interface That Speaks Your User’s Language, was published at Six Revisions on September 27th. Check it out!

The In-Person Usability Test

The first option that came to mind when we decided to start doing usability testing was the in-person test. Conventional wisdom suggests that this is the most effective–but most expensive–method. However, one of the benefits of being in a town with a large university is that there are many students on a limited budget willing to play on a computer for an hour in exchange for a gift certificate to the local bakery.

We found the in-person tests very informative. Interacting with participants and asking about their thoughts in real-time gives us unique insight into the testers’ thought processes that really isn’t available any other way. This led us to make a few changes that we otherwise wouldn’t have known to make. Since we sat right next to the testers, we could see that they only looked at the outline in the Form Builder. For example, almost all of our testers were confused because they didn’t notice the form Preview changing when they added elements to their form, so we realized we needed to make this UI feedback more visible.

There are a few drawbacks to in-person tests. While paying students to do tests isn’t particularly expensive monetarily, the time spent finding participants, setting up, doing the tests, and then analyzing the results adds up quickly. For each test, both of us in the Bloomington office observed, allowing for better recollection (even though we videoed the tests) and for more effective questioning. Analyzing the results from these hour-long tests took about one day per test.

Remote Usability Testing

After our first round of in-person usability testing, we decided to explore the options for remote usability testing. We anticipated were a broader tester base, a smaller time commitment on our part, and the ability to conduct more frequent tests.

We’ve used two different remote usability testing services. The first service we used is UserTesting.com. The greatest thing about this site was that we received the total number of requested responses (three) within one hour of requesting the test. We received a video and a written summary from each tester. However, we found these testers a bit too efficient; they were clearly advanced web users and seemed to have an easier time completing the task than our in-person testers and, probably, than our FormAssembly users.

We tried UserTesting.com when it was new to the market and paid $68 for the three tests.

The second remote testing service we used was TryMyUI.com. This time, I made sure to select the beginner/intermediate option for web users, since the UserTesting.com results had been a little too polished. It did take longer to get all three responses from TryMyUI.com (about 5 hours). Having beginner web users test the application gave us a better approximation for the way most of the FormAssembly users initially experience the application.

We spent $75 for the three tests.

Although the out-of-pocket cost difference between in-person and remote tests is negligible for us, other costs were lower for the remote tests. Since we weren’t moderating the tests, we worked on other tasks while the tests were happening. Analyzing and organizing the data for these 20 minute tests was quicker. Nonetheless, we did get valuable information from the remote tests. We observed a bug that we hadn’t seen with the in-person tests, and got further confirmation of users’ difficulties with the Form Builder’s side tab design. This, combined with our experience during the in-person tests, convinced us to mock up a complete redesign of the Form Builder.

Spur-of-the-Moment Tests

Sometimes you need to test a very specific aspect of your site or application: a color, wording, or image, for example. In these cases, a full-scale usability test is impractical and a little bit of overkill. When we wanted to know what labels would be most effective for the different question types in the Form Builder, we used Amazon’s Mechanical Turk service to poll a number of testers. We found that few people knew the names for different question types (radio button caused particular confusion) but most people understood the image question type. After this test, we increased the size of the images in the Form Builder and saw a reduction in the number of support requests confusing the different question types.

A Little of This, a Little of That

We’ve concluded that for us, both in-person and remote tests are effective usability evaluation methods. We’ve decided to combine early-stage in-person tests with post-release remote tests. Doing in-person tests with parts of the application that haven’t been released gives us the opportunity to test earlier and have more control over the situation. It also allows us to take risks and test more complex aspects that would be difficult to convey to a remote tester. Remote testing, especially with beginner users, gives us a chance to be a fly on the wall when someone first encounters the application. In this way we can improve that experience so that it is easy and productive.

First of all, it’s important that we can access all our documents and applications from anywhere. Thankfully, that’s easily taken care of with online services like Google Apps and Dropbox.

A more serious challenge is how to facilitate social interactions. We need to collaborate effectively, but beyond that, we need to forge and maintain a bond between us. No one should feel isolated because being in the office is not an option. No one should be left out of the loop on important issues. No one should miss a chance to follow and contribute to what other team members do.

Our approach is to rely exclusively on online chat. Yes, it’s a bit awkward when 2 persons in the same office discuss silently on a chat, but the truth is that a conversation in a purely textual form has many benefits. For one, it’s a searchable record of what we’ve discussed and of every decision we’ve made. Discussions also tends to be more to the point, without long-winded arguments or repetitions. It’s not intrusive, so you can easily ignore a topic and save yourself the interruption, and finally, it’s accessible. You can quickly catch up and jump in whenever you feel like it.

Our experience with Campfire

For companies like us, the best role model is 37signals, so we naturally gravitated toward their online chat application, Campfire.

For about a year, we used Campfire for almost everything.  We used it for water-cooler chat,  to share information, to collaborate on customer issues and to discuss projects.

We also have a Basecamp account to manage our internal projects, but we found that we weren’t using it that much. Basecamp is good for sharing information asynchronously, in organized, well thought out bits (this shows that Basecamp was originally designed for client projects), but that’s not really how we work. We need the quick back and forth of a chat. It’s also not practical to have information spread across different applications.

So we stayed on Campfire, but it wasn’t ideal either. Discussions lacked structure, moving too quickly from one topic to the next. Action items would disappear from view, pushed back in the history. (Note that Campfire has the concept of “rooms”, but it didn’t seem practical, so we never really used it.)

We needed some sort of middle ground, so we decided to try Google Wave for a while and see how it went.

Switching to Google Wave

Google Wave is an instant collaboration tool. Discussions can be threaded, so we can cover several topics in parallel and keep them organized. We can create new “waves” as needed and easily follow the activity in  each wave from the “inbox”.  There’s an helpful “next unread” button that makes sure we don’t miss any message, regardless of where it’s posted. Of course, Wave is searchable too.

Each day we create two waves, one for a general chat and one to discuss customer support issues. In the support wave, we can keep each issue in its own thread. This lets us keep track of the work done and of who needs help to troubleshoot an issue.

We also have a weekly wave, where we list our priorities for the week. Everyone can update it if needed with a short status report. Finally, we have specific waves for long-term projects, where we collect our thoughts, post screenshots, discuss mockups and so on.

Google Wave isn’t perfect though. We found that sharing screenshots and mockups was difficult. Wave seems to downsample images, so we would always end up with blurry, unusable mockups. We had to resort to hosting images on dropbox and posting links to Wave. There are a few other irritating quirks, like the odd scroll bars that won’t let you scroll to the end of a wave in one try or the lack of audio notification (there are semi-reliable plugins for that).

Overall, we’re happy with Google Wave. We’ve consolidated two applications in one and it fits us well. When Wave launched, it had this reputation of being a tool nobody knew what it was for.  A sort of email-collaboration-programmable mash-up, with plugins and bots.  Turns out, it simpler than it sounds. The bare bone Wave is just what we need to run our business.

We try to be specific in our job descriptions, but we are in fact quite flexible and open minded. We don’t want to tell you that you must meet some arbitrary criteria, or that you have to apply in any particular way.

In fact, it’s actually quite easy to stand out, if you can avoid the following few mistakes.

1. Not doing any research about us, about what we do, and about what we’re looking for. Some tell tale signs are:

• Sending us a generic cover letter.
• Not being specific about what you like about the job or our company.
• Talking in length about an unrelated technology or experience.

2. Filling your resume with vague and unsubstantiated statements (also known as bulls**t). Things like:

• “I am a good listener with good communication skills.”
• “I have a solid work ethic, and a desire to excel and meet deadlines.”
• “Excellent analytical and problem solving skills with ability to communicate effectively with peers.”‘

3. Showing a lack of discernment by listing every single project you’ve worked on, job you’ve had, course you’ve taken, skill you’ve learned, or software you’ve used. For instance, avoid:

• Listing your knowledge of various OS flavors or trivial software.
• Listing obsolete technologies or languages.
• Listing seminars or workshops you’ve attended.

Here are a few things you can do to help your chances:

• Make sure to show in your cover letter that you’ve read this. For instance, you could tell us about your favorite candy ;-)
• Sign up for our free service and give it a test drive, and then do the same with one of our competitor. The best way to impress us is to suggest improvements.
• Give us objective metrics:  grades,  ranks,  links to publications, contributions to open-source projects, etc..
• Show us something you’ve created in your spare time. Software, writing, furniture, whatever.
• Follow up even if you don’t hear back from us.

Our job openings are listed here. Get in touch with us if you’d like to apply. We’re looking forward to meeting you!

We’re not trying to change the world (just yet).  We’re patient. We take one day at a time. We try to better ourselves, to learn and master our craft.

We believe in opinionated software, but we don’t pretend to know everything our customers need.   We listen to them and if we can make their jobs a little bit easier, then we’re good.

Perhaps our most precious resource is the time our customers give us – time spent learning and using our software. We try to not abuse it.

Results are, in turn, the only thing we demand from ourselves. We work from home, coffee shops, or the office.  Hours and location don’t matter. Efficiency matters. We don’t work hard/play hard. We just tend our work; we get it right, we get it done, and we don’t make it hard.

We have the tools, the technology, the skills, and the freedom to make our job what we would like it to be: that is, rewarding, peaceful, useful, and meaningful. It’s not always like that of course. Our job can sometimes be tedious or stressful, but we’re always on the lookout for better ways to work and get things done.

If this sounds appealing, please join us. We need all the help we can get!

We track a lot of information with Google Analytics. Goal conversion rates, e-commerce revenues, adword campaigns, search keywords, referrers and so on. Collecting information is easy, but trying to make sense of it can be frustrating.

Google wants me to drown in a sea of data

I often find myself wandering through the Google Analytics reports, marveling at the shiny graphs or noting with curiosity that the middle of the month is always slower. While my intellectual curiosity might be satisfied, I’m still left confused, unsure about which variable to tweak and nothing to act on.

I eventually learned that, like Ulysses who tied himself to the mast to resist the alluring chants of Sirens, diving into the reports requires self-discipline and focus. First identify a specific page, then a metric, write it down and do not look at anything else.  Then go and try to improve it.

Another thing I learned is that you can’t just compare the performance of a page before and after a change and decide whether the change was a good idea or not.

Traffic is not a constant. It varies over time, quite significantly. One week the conversion rate is great, and the next one it sucks even though nothing changed on the site. Perhaps it’s a surge of unqualified visitors, or people who get antsy over the latest economic news, or just a random fluke.

I’m not willing to wait days or weeks for a clear pattern to emerge. If my new page is not performing as well, I’d like to  undo my mistake as quickly as possible.

That’s where A/B testing is really useful. It works by randomly splitting your incoming traffic between 2 different versions of a same page. With the appropriate tracking tool, it’s then very easy to see which page performs better.

Google the merciful rewards the faithful

So I finally decided to take a second look at Google’s Website Optimizer, a free A/B testing service.  I wasn’t convinced the first time around. It was rather limited, and I thought that our esteemed visitors would not be swayed one way or another just because we make our marketing copy a bit more clever. After all, we firmly believe in the greatness of the product we’re building,  shouldn’t it just sell by itself?

Well, it turns out that Website Optimizer got a lot better. It’s very easy to set up, you can test pretty much anything and it provides unambiguous results quickly.

No more trying to decipher the reports and understand what’s going on. No more agonizing over a design decision. Should it be a link or a button? Should it say ‘Sign up now’ or ‘Try it for Free’? Just try both and let Google tell you which one works best.

And when the result  is a 300% improvement in conversion rate, I sure feel stupid for not doing it sooner.

I’m not going to criticize this approach but I honestly thought there would be more young entrepreneurs in the Valley interested in emulating companies like Salesforce or 37signals.

It is quite ironic that David Heinemeier Hansson is seen as going against the grain, when his presentation was packed with nothing but grounded advice and a healthy dose of common sense. What’s wrong with the old true and tested way of charging money for your service?

Interestingly, DHH also said that Backback (37signals’ second app after Basecamp), had to be rebranded and relaunched as a “business” application when they realized that the consumer market wasn’t really profitable.

There was maybe one theme missing from the picture though. Building a successful business takes time and hard work. A lot of time and a lot of work, so it was great to meet you all, I have to go back to work now.

The On-Site edition offers the same great features enjoyed by thousands of our online users, but you can run it on your own server and keep your data safe and private.

FormAssembly On-Site is a reliable and secure solution designed to work on the most widely available configurations: Windows or Unix, IIS or Apache, and with any of the major database vendors. The support for such a wide range of technologies is made possible by the great open-source framework FormAssembly is built upon.

You can learn more about FormAssembly On-Site at http://onsite.formassembly.com

Several commenters have pointed out that @importRule would not be that useful since you can already assign more than one CSS class to an element.

However, having (apparently) some time to waste this morning, I thought it would be interesting to implement this pseudo css rule anyway.

Here’s the example page, with a short explanation of the (very simplistic) javascript:

Again, I’m not claiming that this is very useful, but if one wanted to create a javascript css post-processor, that would be a good way to start.