Usability testing plan
We started planning our test as early as possible — selecting testing methods, environments and setting up the guideline of participants recruitment and requirement. We wanted to validate our design concept of end-users watching patterns and flow on the streaming app from this testing. The possible testing methods discussed to select the most suitable one for testing.
Possible Methods
● Think-aloud — Understanding participants’ behaviours, goals, thoughts, and motivations
● A/B Testing — Random pick vs Pick by mood
● First-Click testing — Checking participants go down the right path to complete the task
● Interview + SUS — Getting detailed information about a user’s attitudes, desires, and experience
*Note: A/B testing and First-Click testing got abandoned as we iterated our design solution — see Blog 3. for details
Ethics and data protection
The Invitation Letter and Consent Form sent out before testing through IADT email. Consent form included informing recording audio, video, and their screen to get each participant agreement during the test. All the data stored in IADT One drive for data privacy and GDPR.
Test scripts include:
● Consent form
● Instruction
● Warming up questions(Demographic and basic info)
● Main tasks (5 tasks, probing questions and post-task questions)
● SUS
Appendix. Usability testing board / Consent form and Tasks / Instruction / SUS
Test script
Testing script and plan drafted and polished for assessing in ad hoc testing. The test script amended ambiguous task guidelines — remove guidance to make it clear and straightforward. Pilot testing conducted among the members to find task wording for priming, and prototype error.
Final user testing task flow areas -
1. Random Pick
2. Pick by Mood
3. Share and write the review
4. Find the reviews and rate the show
5. Live chat
Recruitment
● 10 Participants / Age: 25–34 / 9 have used RTE Player
Each team member aimed to get 1–2 participants for the test — minimum 5, according to Nielsen(2000)’s advice. We tried to get participants as close as possible to be a realistic user of the product. However, close connections used for recruiting under a restriction of Covid-19. A consent form and a brief instruction sent to a participant to inform an idea of the test and procedure before the testing.
Testing
Conducting test
● Stationery — Introduction letter / Testing Script / Consent Form
● Equipment — Laptop (testing and recording) / Pen and notes
● Interview Techniques — Screen sharing / Think-aloud / Probing question
● Online testing tool — Lookback, Loop 11 and Zoom
Two — in-person and one — remote testing conducted in a moderate setting with Loop 11 in my study. During the session, tried to give as little guidance as possible to prevent priming. Participants also asked to do think-aloud to collect their behaviour and motivations.
Limitation
● A remote usability testing — It limited the observational study of participants’ body language and facial expression
● The desktop testing for the screencasting — the app designed for mobile devices, idly should have tested on mobile. Participants informed for an understanding of interface layout
● Loop 11 — The onboarding page moved to the home screen or other screen automatically when participants finished warming-up questions. Participants had to go back to the onboarding page manually to start the task. It may affect the full step of user flow through the testing
Analysing results
Each testing outcome shows different usability problem areas that depend on experience, usage and understanding of the app. Synthesised qualitative data from observations, open-end questions and quantitative date — success metrics from SUS.
Quantitative data
We used the System Usability Scale (SUS) to measure our test success rate. Overall, we got an 89.2% score shows that most of the new features and solutions well implemented in the interface.
“Better than the current app. Live chat is good but I have to make sure the user set up the profile before using the app. It is a lot of work to do but it’d be worth it, if all working well. “
“…gives a sense of community.”
— from Participants interview
Qualitative data
Our primary solutions — Passive watching behaviour — ‘Random pick’ recognised more than ‘Pick by mood’. However, the word ‘Random’ gave negative influence that they would get unrelated recommendations; participants also prefer to search for something directly.
Key insights
● Profile — 7/10 tendency to set up
● Random Pick — 4/10 first noticed on the home screen
● Pick By Mood — 6/10 used to pick something for watching
● Reviews and rating — Start ratings encourage more feedbacks
● Live chat — 6/10 would use, 8/10 No public chatting
Appendix. SUS analysis board / SUS result /Task analysis board / Key outcome board / Warm Up question / Iteration list & Priority grid
Refinements
The last iteration continued from the testing result and feedback — didn’t overturn our previous findings, only a minor change.
Appendix. Full insights for the final iteration
What would we do differently?
Overall, we all satisfied our testing result, we all agree that there is room to improve our product further if we consider and implement the list below early in the design process:
● UI Audit
● Early and more frequent testing
● Interviews then survey to get better insights for the solution
● Work with more existing users
Reference
Budiu, R. (2016, January 24). Priming and User Interfaces. Nielsen Norman Group. https://www.nngroup.com/articles/priming/
Nielsen, J. (2012, January 15). Thinking Aloud: The #1 Usability Tool. Nielsen Norman Group. https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/
Nielson, N. (2000, March 19). Why You Only Need to Test with 5 Users. from
https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
Schade, A. (2017, April 9). Write Better Qualitative Usability Tasks: Top 10 Mistakes to Avoid. Nielsen Norman Group. https://www.nngroup.com/articles/better-usability-tasks/
T, W. (NA, NA NA). Measuring and Interpreting System Usability Scale (SUS). UIUX Trend. Retrieved 01 15, 2020, from https://uiuxtrend.com/measuring-system-usability-scale-sus/#interpretation
Usability Gov. (NA, NA NA). First Click Testing. Usability.gov. Retrieved 12 30, 2020, from https://www.usability.gov/how-to-and-tools/methods/first-click-testing.html
