User Research is Not Easy

Tara Calihman - July 27, 2015


As promised in my earlier Monitorama write-up, there were a few talks that resonated with me because they are issues that present challenges or topics I want to learn more about.

Chrissie Brodigan, Github’s first researcher, presented one such talk about user research as it relates to releasing new features. As she pointed out, all products have blind spots which is why it’s important to ask questions about user’s workflow. Information like that you can only get when you talk to users actually using your product.

There are tons of monitoring tools out there that tell you what is happening but often, you don’t know why something is happening. The gap from metric to insight can be huge and overwhelming. This also applies to user research - you want to know why people are doing what they are with your product.

To find out, as Chrissie stressed, you need to get out from behind your computer and get in front of people. This is one of the big reasons that we love attending conferences and DevOpsDays events. We get to hear firsthand how our users are interacting with the VictorOps platform and that is invaluable feedback. Chrissie also advised that when you talk with people, look for goals, motivations and workarounds as these will give you insight into your product as well.

Surveys can be a really good tool to discover more about your users. A survey is like a snapshot in time and if you repeat that same survey after some time, it shows change. In order to be effective, the survey needs to be: repeatable, longitudinal and actionable.

Chrissie shared the story of when Github launched Git Large File Storage (GitLFS). They conducted a survey, then decided to do a pre-release of the feature which was limited to only 20. From there, they conducted an early access program that included 13,000 users. This EAP runs as a controlled experiment with half of the users get the feature while the other half act as the control group and do not get the new feature. After six weeks, they sent exit surveys to the different groups and discovered that it’s really hard to get people to change their workflow.

Chrissie finished her talk by defining user research as the process of learning about the source of the problem and the motions behind it. Asking good questions while you go through this process can be really hard but she suggested a few…

“Tell me a story about a time this feature may have helped you….”

“How would this feature help you?”

“You tell me…where would you use this feature?”

Big thanks to Chrissie for sharing her knowledge. If you want to watch her entire presentation, you can see it on the Monitorama Vimeo channel.