Interaction Design Project 4.

Jaesin Yu
5 min readJan 22, 2021

UX/UI redesign case study — RTE Player

Usability Testing & Iteration

Online testing tool -Loop 11 and Heatmap from the testing

Usability testing plan

We started planning our test as early as possible — selecting testing methods, environments and setting up the guideline of participants recruitment and requirement. We wanted to validate our design concept of end-users watching patterns and flow on the streaming app from this testing. The possible testing methods discussed to select the most suitable one for testing.

Possible Methods

Think-aloud — Understanding participants’ behaviours, goals, thoughts, and motivations
A/B Testing — Random pick vs Pick by mood
First-Click testing — Checking participants go down the right path to complete the task
Interview + SUS — Getting detailed information about a user’s attitudes, desires, and experience

*Note: A/B testing and First-Click testing got abandoned as we iterated our design solution — see Blog 3. for details

Ethics and data protection

The Invitation Letter and Consent Form sent out before testing through IADT email. Consent form included informing recording audio, video, and their screen to get each participant agreement during the test. All the data stored in IADT One drive for data privacy and GDPR.

Test scripts include:

● Consent form
● Instruction
● Warming up questions(Demographic and basic info)
● Main tasks (5 tasks, probing questions and post-task questions)
● SUS

Appendix. Usability testing board / Consent form and Tasks / Instruction / SUS

Random Pick VS Pick by Mood icons

Test script

Testing script and plan drafted and polished for assessing in ad hoc testing. The test script amended ambiguous task guidelines — remove guidance to make it clear and straightforward. Pilot testing conducted among the members to find task wording for priming, and prototype error.

Final user testing task flow areas -

1. Random Pick
2. Pick by Mood
3. Share and write the review
4. Find the reviews and rate the show
5. Live chat

Recruitment

● 10 Participants / Age: 25–34 / 9 have used RTE Player

Each team member aimed to get 1–2 participants for the test — minimum 5, according to Nielsen(2000)’s advice. We tried to get participants as close as possible to be a realistic user of the product. However, close connections used for recruiting under a restriction of Covid-19. A consent form and a brief instruction sent to a participant to inform an idea of the test and procedure before the testing.

Testing

Task 3. Rating and review testing flow

Conducting test

Stationery — Introduction letter / Testing Script / Consent Form
Equipment — Laptop (testing and recording) / Pen and notes
Interview Techniques — Screen sharing / Think-aloud / Probing question
Online testing tool — Lookback, Loop 11 and Zoom

Two — in-person and one — remote testing conducted in a moderate setting with Loop 11 in my study. During the session, tried to give as little guidance as possible to prevent priming. Participants also asked to do think-aloud to collect their behaviour and motivations.

Limitation

A remote usability testing — It limited the observational study of participants’ body language and facial expression
The desktop testing for the screencasting — the app designed for mobile devices, idly should have tested on mobile. Participants informed for an understanding of interface layout
Loop 11 — The onboarding page moved to the home screen or other screen automatically when participants finished warming-up questions. Participants had to go back to the onboarding page manually to start the task. It may affect the full step of user flow through the testing

Analysing results

Each testing outcome shows different usability problem areas that depend on experience, usage and understanding of the app. Synthesised qualitative data from observations, open-end questions and quantitative date — success metrics from SUS.

SUS score

Quantitative data

We used the System Usability Scale (SUS) to measure our test success rate. Overall, we got an 89.2% score shows that most of the new features and solutions well implemented in the interface.

“Better than the current app. Live chat is good but I have to make sure the user set up the profile before using the app. It is a lot of work to do but it’d be worth it, if all working well. “

“…gives a sense of community.”

— from Participants interview

Qualitative data

Our primary solutions — Passive watching behaviour — ‘Random pick’ recognised more than ‘Pick by mood’. However, the word ‘Random’ gave negative influence that they would get unrelated recommendations; participants also prefer to search for something directly.

Key insights

Profile — 7/10 tendency to set up
Random Pick — 4/10 first noticed on the home screen
Pick By Mood — 6/10 used to pick something for watching
Reviews and rating — Start ratings encourage more feedbacks
Live chat — 6/10 would use, 8/10 No public chatting

Appendix. SUS analysis board / SUS result /Task analysis board / Key outcome board / Warm Up question / Iteration list & Priority grid

Iteration after testing

Refinements

The last iteration continued from the testing result and feedback — didn’t overturn our previous findings, only a minor change.

Appendix. Full insights for the final iteration

What would we do differently?

Overall, we all satisfied our testing result, we all agree that there is room to improve our product further if we consider and implement the list below early in the design process:

● UI Audit
● Early and more frequent testing
● Interviews then survey to get better insights for the solution
● Work with more existing users

Reference

Budiu, R. (2016, January 24). Priming and User Interfaces. Nielsen Norman Group. https://www.nngroup.com/articles/priming/

Nielsen, J. (2012, January 15). Thinking Aloud: The #1 Usability Tool. Nielsen Norman Group. https://www.nngroup.com/articles/thinking-aloud-the-1-usability-tool/

Nielson, N. (2000, March 19). Why You Only Need to Test with 5 Users. from

https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Schade, A. (2017, April 9). Write Better Qualitative Usability Tasks: Top 10 Mistakes to Avoid. Nielsen Norman Group. https://www.nngroup.com/articles/better-usability-tasks/

T, W. (NA, NA NA). Measuring and Interpreting System Usability Scale (SUS). UIUX Trend. Retrieved 01 15, 2020, from https://uiuxtrend.com/measuring-system-usability-scale-sus/#interpretation

Usability Gov. (NA, NA NA). First Click Testing. Usability.gov. Retrieved 12 30, 2020, from https://www.usability.gov/how-to-and-tools/methods/first-click-testing.html

--

--

Jaesin Yu

Graphic Designer, Textile Designer, MSc UX Design Student