Usability in the cloud: why remote usability testing kicks ass

Truth be told, I am a bit of usability freak. I love it when websites work well – it’s a thing of a beauty. Of course, it’s an art, not a science. No matter how much we learn about the right way to do things (I can wholeheartedly recommend books by Robert Hoekman Jr, Joshua Porter and of course Steve Krug), every project is different. Web apps evolve constantly, presenting us with new usability challenges. It’s pretty easy to get deadlocked in a discussion about how something should be done.  Handily it turns out the solution to this paralysis by analysis is pretty simple: watch some users using your site. Steve Blank, famous for his Customer Development ideas, coined the maxim:

“In a startup no facts exist inside the building, only opinions.”

That rings true for usability discussions too. We’re too close to our products, we’ve lost our objectivity. We need real users to play with our websites and let us know what they think. This used to be intimidating: we’d need to recruit user from some target demographic; get them into an office;  sit them down at a computer; have them “think out loud” – never easy in a room full of strangers; record the whole thing with a camcorder or screen recorder; and finally review what happened and work out where to improve. While the notion that this needs to be difficult or expensive has now largely been dispelled, (hat tip to Steve Krug again) it can be tough for people who aren’t usability pros – developers, designers and business people – to get really good results.

It has got a lot easier, though, in the last few years, thanks to some neat online solutions. Two we’ve used recently on Heybaloo are Usertesting.com and Feedback Army.

User Testing (www.usertesting.com)

Usertesting.com is an online version of traditional hallway testing (find someone; sit them in front of your software; ask them to do some tasks; see how they get on). You write a simple test,with a scenario and a set of tasks to complete, and usertesting.com will find some remote users to complete your tasks and send you a written report letting you know how they got on. Here’s the kicker though – you also get a fully narrated flash video recording of them using your site.

What an awesome idea.

First of all, it’s really cheap. ($29 per user, about £19). Secondly, it’s quick – we received our responses within hours. Thirdly, the results you get are, for the most part, great. It’s all managed through their website, so you can log in anytime and watch the recordings online, and the quality is good enough for me to blow them up to full screen on a 19″ monitor. It’s also really neat to be able to share these videos online, so you can send a link to that colleague who was being obstinate about something – “see? I told you our users wouldn’t understand that”.

There are two more big wins for me too.  When running conventional user testing, you need your users to “think out loud”. People move at frightening speed on the web – we need them to slow down and tell us what they’re thinking. “OK, I don’t really know what that means, OK – this looks promising, I’m gonna click on that…”. While you always ask people to do this,they are often understandably self-conscious, and it sometimes feels a little bit forced. You also need people to use your website in an authentic way. Not so easy when you’re in a room with a bunch of developers watching your every move. Remote testing neatly sidesteps these issues: the tester is probably sitting in his own house, using the site at his own pace. He’s probably on his own, so he’s happy to chat away and let you know what he thinks and what he’s doing the whole time.

Of course, it’s not all perfect. The quality of the results can vary. While some testers gave us great feedback – thoughtfully narrating everything they did – others seemed more interested to talk about our business models and the way our company might expand. The quality of the written reports was not amazing, but that’s a minor gripe – the videos are what you signed up for anyway. You also don’t get too much control over who is testing your site. We were able to request UK users, and you can also request simple demographic information such as male/female, but I’m not too worried by any of this. Remember that in Don’t Make Me Think Steve Krug reveals the dirty secret of usability testing: it actually doesn’t matter too much if testers match your target demographic. Most users tend to find the same problems anyway.

With those minor caveats, usertesting.com comes highly recommended.

Feedback Army (www.feedbackarmy.com)

“$29?”, I hear you cry. “That’s far too expensive. Isn’t there anything available for, I don’t know, maybe $10?”.

Well, yes, it turns out there is. Feedback Army is the quick and dirty approach to usability testing. It’s a supermarket ready meal to the three course lunch of conventional usability testing. You write a series of short questions and for $10 ten remote users will spend a couple of minutes each combing your site looking for the answers.

We ran a test with Feedback Army with some simple questions about our site:

1. What is the site about?
2. How is the content generated?
3. What do you think of the site?
4. Would you use it again?

The answers we received were an interesting mix of first impressions and copy-and-paste from our documentation. The feedback you get is short – one sentence replies in the main – but that’s how it meant to be. You can ask specific questions and get answers very quickly. While Feedback Army can’t be relied upon to find all my usability problems, with well written questions it could have some great applications; testing your information architecture or getting early feedback on new features both spring to mind. I’m sure it would be neat to try this with A/B testing and multivariate testing too.

Roy Fox, Technical Manager, Heybaloo