Stories from the front line of remote user testing

Here at Pollen, like most agencies, we have been adapting to the unique situation we find ourselves in during this COVID-19 pandemic. We’ve been forced to move our internal processes and more of our UX practice online. I have been pleasantly surprised on where we have had successes in this transition, such as with programs like Miro which has been a fantastic tool to facilitate client workshops.

Even though this current situation has changed the way we work, the need for clients to further understand how their products can meet their customers' expectations hasn’t stopped. It’s even more pertinent today to understand customers' behaviours with more services moving online in line with the recent lockdown measures.

With online user testing, we have traditionally carried out our sessions with in-person testing, having the client in the observation room and streaming the test online for a larger audience to access. The last few weeks we have had to expand on this approach and learnt a few things on the way.

I took the time to sit down with our UX Researcher Lorraine Chung (via Google Meet) to get some juicy insights on learnings she took away from her user testing adventures.

You’ve got your participants ready, script sorted and prototypes ready to go… but you can’t hear the participant. Planning ahead for unexpected issues like this could save you a headache or two and a lot of time.

• Ensure their computer/device is properly configured and they are comfortable with the tools. A detailed email beforehand to explain what they are about to undertake is very helpful. This can be aided by booking a little time to have a chat with the participant to help them with the setup.
• For a 45 min session, book for an hour leaving some time to talk through the setup for the session.
• Make sure your phone number is available to the participant. Emails are a great way to communicate, but it is not instantaneous, the few minutes of delay when trying to troubleshoot can be what breaks the camel’s back.

Testing remotely means you get to observe your participants in environments and scenarios that better reflect their actual usage, it also means not testing in the controlled environment of your testing lab.

• The tech setup of the participants will not be as perfect as your own testing lab. You will have no control over what screen size they’re looking at the prototype in or if they’ve recently updated their anti-virus software (we had to abort a testing session on Lookback because of this). Double-check your prototypes in different screen sizes, and have instructions ready when you might need to guide the participants to resize or zoom their browser.
  
• The internet connection speed will always be a problem, even if you both have full speed NBN, you never know if you’ll be caught in a temporary slowdown during your session.

Which brings us to...

The best-laid plans of mice and men often go awry… especially in online user testing. Having a backup to the backup may be the best investment you can make.

• The main platform we use is Lookback as we find it an easy setup for us (note-taking, recording the session etc ) and the end-user. Though when technical issues arise having a backup program to record and run the session such as Zoom.
• While you have one backup, we also suggest having another such as Google Meet.
• If your backup is to use video conferencing software, best to set it as “auto record”, but be sure to ask the participant’s consent right at the start of the session.
• If you have the time, carrying out some dry runs not only to test the session itself but the technology used for it.

Online testing sessions can become less fluid for the facilitator, with interactions becoming stilted even leading to frustration for both parties. The key is to have a more directed session, clearly defining the steps needed to be taken by the participant.

• To ensure the sessions run smoothly, have a clear and simple testing script that you test beforehand with your colleagues, refining it as you go along. The script is much more important in a remote testing setting as it is harder to react in realtime.
• If you are testing with multiple prototypes have a landing page which lists them for users to click from. You can share a Google doc (which can be published as a web page) or send in an email just before the session with listed links to each prototype.

We have seen an increase in participants who may not answer the screener questions honestly, which may be a result of the current unfortunate situation of many people being out of work. This can be an issue if you have strict criteria that participants must meet.

• We suggest once you have confirmed your set of participants, contact them directly to ensure they meet the criteria.
• If you need more specific criteria for participants to meet, add in a few more screener questions when recruiting, but be sure to put the most crucial criteria upfront, we found this helped reduce the number of participants that didn’t fit the bill.
• Have a second set of participants ready to test with as a backup. We recommend Askable as a platform that can facilitate this.

One of the reasons why participants have put up their hand for this is to be paid. We have some ideas about how you can run this smoothly keeping your social distancing intact.

• E-Gift cards are a good option, be sure to consider where the participants can use their newly attained gift cards. More recently we have used Prezzee, which provides an array of gift cards that can be sent via email. They can accommodate a wide range of payments and we suggest creating a corporate account well before the testing session.
• As mentioned previously, Askable is a fantastic platform to not only recruit participants but it manages the incentive payment as well.

We hope our experience has helped to understand how you can run a hassle-free and engaging session for your participants. We are still learning, so if you have any insights we’d love to hear your experience so please share your thoughts. Contact us at the bottom of this page.