Right, do me a favour. Go on – it’ll only take a second. Look at your hand. Either one, doesn’t matter which. Now, I want you to think about what is good about that hand, and also what’s bad about it. Look at it in real detail – look at the nails, the cuticles, the knuckles. Look at your palm and all the little wrinkles and bumps and freckles. And think about which of those things are good, and which of them are bad.
If you haven’t done it, I really urge you to. It’ll help me make a point in a minute and I know you want to help me do that. If you have – thanks! I hope you didn’t look too weird on the train. I’ll play along and do it too:
Looking at my hand, I can see that there’s a scar on the back of it – that’s bad. My nails are looking pretty good at the moment though, so we’ll chalk that up as a plus, but I’ve been gnawing at the cuticles so they’re a bit red – bad. They’re also not quite the picture of youthfulness they used to be so I’m going to stop looking at them now, I’m getting a bit depressed.
So, here’s my point. 99% of the time, I don’t notice any of those things. That’s because the things I need my hands for the most – holding onto things, typing, throwing gang signs (ok, maybe not) and so on – none of these things are impacted in any way by the condition of my nails and cuticles, nor are they affected by a scar I got ten years ago in a dodgy bar in Staines. However, at some of the places I’ve worked, these would be major issues. The team would insist that, because the user pointed out these flaws, we must fix them before continuing.
I particularly faced this issue once when working on the design of a public service. I was the Lead User Researcher, planning and conducting research while also upskilling other members of the team who were interested in the role. What was great about that team was that everyone made a real effort to attend the user testing, which was conducted in a very nice research lab with a live feed to an observation room. The problem was, they found it difficult to choose their battles.
One of the problems with conducting any user research that the user’s actually aware of is that they feel they must give you their money’s worth. They will pick apart a design, and look much more closely at your website, app or service than they would when using it in real life. This is why, in user research, you’ll so often hear things like,
“It’s alright for me but I think other people wouldn’t like X”
“Someone like my mum would probably struggle with that”
“I think people might have a problem with Y”
Your users are saying these things because they have no other input of their own to add – they are truly grasping at straws so that they don’t seem like a waste of a test. That’s why I always follow up on these remarks by asking if that same problem affects the user you’re speaking to. If it doesn’t, it’s probably not worth noting.
Where it gets more difficult is when the user attending your research session points out something as their own feedback, but you know from experience that they’re only saying this to give you something to write down. This is really only something you notice after conducting or attending a lot of these sessions, and having a good understanding of how users think and behave; that’s why so many UXers come from a background in psychology. And because of that, I can understand when a team without a UX background immediately wants to jump on everything that users have said and start prioritising changes.
With that in mind, here’s some things to consider before you act upon user feedback:
How quickly did they come up with the feedback?
If the user said it straight away, it’s likely very truthful. The longer they took to think of something, the more likely it is that they’re looking for something to say.
How many other users have said the same thing?
If there is a recurring theme amongst users, you can be pretty sure that it’s a genuine issue. That’s why you test with a few people each time; the majority of them will come up with the main issues, and once the problems they talk about all start sounding completely different then you’ve probably got something that’s at least usable.
Does it sound like one of the main issues of the session?
If you’re working in a lean, Agile way then you’re doing lots of testing, and iterating on your product to improve it each time. That’s why I really like what Steve Krug says to his user testing observers – “jot down the three most serious usability problems you noticed in that session” (you can find this in his Instructions for Observers, in the downloads section of his website). Then, after the session, you all get together, agree on what the top few items are to fix, and go back to testing once those are done. Then you sort the next biggest issues, and so on. Doing this means you’ll get to a releasable product much quicker, and you’re not spending a lot of time working on the little things that hardly anyone will notice.
Does the user’s behaviour back up what they’re saying?
This one you’ll need a keener eye for, and sometimes it’ll take some compelling video evidence and UX experience to prove it. As we’ve already established with the “look at your hand” exercise, people can be pretty bad at considering the things that really matter when they’re evaluating something under pressure. A user might say that something seems complicated in comparison to another service, but whizz through the process and find exactly what they’re looking for. They might make a much bigger a deal of a small problem. And this is where it pays to have a UXer or User Researcher evaluating the findings. User behaviour is infinitely more telling than what they say.
One example I’ve seen of this is during a series of sessions where none of the users mentioned an issue with a certain step – but every single one furrowed their brow, paused for a moment, or let out a sigh. Things like this are so much more difficult to pick up on, but they will be the things that make or break your experience.
So what can we conclude from all this, now I’ve told you my million dollar secrets – that are not secrets, nor are they worth a million dollars? The conclusion is that while anyone can run user research, and anyone can interpret the results, if you have brought a User Experience Designer, User Researcher, or something similar on board, please listen to their advice. They’ve seen a whole lot of these sessions, and they know what they’re looking for.
A user pointing out something that’s wrong with your service doesn’t necessarily mean you need to fix it.