Experiments in remote user testing
Customers are at the heart of everything we make at Made by Many. We try to ensure we’re always talking to them, showing them our work in progress, and getting their feedback regularly throughout the course of a project.
Generally, we prefer to use in-person interviews as our 'go-to' for customer research. These interviews, with 4-8 people each round, are vital for guiding our direction, validating the decisions we’ve made and informing forthcoming work. However, recently, we’ve been experimenting a lot more with unguided, remote user testing.
For our in-person research, spending time recruiting the right people and having in-depth conversations about their behaviours, problems and motivations is vital; especially at the beginning of a project, as we figure out what the right product is.
Once we have propositions, sketches, prototypes, or are actually building something - we’ll get into the habit of testing with customers on a regular basis. On my current project this means we’re conducting a round of interviews at least once a sprint.
Rapid validation
When we're working on developing a prototype, we often want to move more quickly to validate a direction than in-person interviews can allow. To enable us to make decisions with increased certainty, we’ve been using usertesting.com for rapid testing and validation of a prototype alongside our usual in-person sessions. Usertesting.com allows you to provide a URL to customers then set tasks or questions, which they complete as their screen and audio is recorded. Setting up a test is fairly quick; you set up your screener, add your questions or tasks, then submit. Generally you have the videos back to start synthesising within an hour. (As a side note, Adam recently wrote a brilliant post about how to structure research for efficient synthesis)
The shortened turnaround time of remote testing - compared to the time that it requires to conduct in-person interviews - has allowed us to experiment with what we test and when, allowing us to get much quicker customer feedback, on a more regular basis, about smaller things, which would usually not justify the cost and time commitment of in-person testing. Here's a rough representation of how these turnaround times might affect the structure of a sprint:
How we've been doing it
In reality, we've rarely structured a sprint with just in-person or just remote testing, instead we've been running the two forms of testing alongside each other in various configurations. So far, the main ways we've been using remote testing are:
- After a round of in-person testing, when we’ve made refinements to a prototype based on customer feedback, to provide validation that the changes have solved any issues.
- As a way of increasing certainty about findings from in-person testing by including more people, without having to spend significantly more time doing extra interviews.
- Testing much smaller flows or functionality: our in-person interviews tend to cover around an hour’s worth of material, but being able to do online testing means that it remains efficient to conduct 5–15 minutes tests that sense-check decisions we’ve made about how a feature should be implemented.
- Guided in-person usability interviews are great for understanding in more depth why someone likes or dislikes the way a product works — however, it’s not always the most natural way for someone to interact with something we’ve built. Seeing people use our prototype to complete a task, completely unguided by conversation, on their own device, also helps us to increase confidence that customers in ‘real life’ situations are going to be able to use what we’ve made.
This is a relatively new addition to our customer research toolkit, and whilst it will never replace in-person interviews (and we wouldn’t want it to), it’s always great to have new ways of getting to know and understand customers. Personally, I’m really excited about trying different tools in future too, like UsabilityHub, where the focus seems to be far more on very quick tests of designs.
I’d love to hear from other people who are using a combination of in-person and online testing about how you’re doing it, what tools you’ve been using and anything you’ve discovered in the process. You can give me a shout on Twitter @fionamclaren.
Continue reading
Design
Asking the question: should designers code?
If you asked me “Should designers code?”, I wouldn’t be able to answer you. The question serves as a popular op-ed headline, but drops context in favor fo...
Process
A faster way to paper prototype
I recently created a Keynote template to help with paper prototyping. It has seemed a few others in the Made by Many design team have found it useful so I...
Business Strategy
To Save Twitter, Let It Die Quickly
I felt a momentary pang of sadness when I read the New Yorker piece at the weekend about the impending demise of Twitter in an article titled ‘The End of ...