UX research for dev tools must not be neglected: our quick-start guide

Essentially everyone can acknowledge the benefit that solid user experience research (UX research) adds to the final product, but how often is this actually carried out to a satisfactory degree?

I’d compare UX search to the practice of meditation. The benefits of meditation have been widely recognized, and most people can readily accept these conclusions, however, actually putting this into practice is another story.

This article is primarily addressed to developer tool managers, founders, and teammates. I’ll lay out the struggles that can prevent these teams from developing a solid UX research habit, and I’ll recommend ways to overcome those.

Let’s be careful when defining UX research

Before we jump in to the actionable steps, let’s lay out a bit of context so we can understand the situation as we face it.

First, it’s crucial for us to differentiate between UX research and customer research. These terms are often used interchangeably, but this can lead to misunderstandings.

Let’s demystify that difference right now:

• Customer research is all about understanding why and what your customers want and need. This research is carried out using surveys, group discussions, and one-on-one talks. It helps us improve our products and keep our customers happy, and it also supports our decisions in terms of how to market our product and support our customers.

• UX research involves learning how people actually use a product in practice and how they feel about it. By doing this research, we can find problems and make our products better. For instance, we can reduce the time it takes for users to complete their tasks, and reduce mistakes that can occur due to the UI design. We want to end up with products that people want to keep using. This can be because they feel effortless to use, or as a result of convenient user flows.

While customer research can inform your product development strategy, UX research is essential for creating a better version of your product in practice. In particular, for teams who are primarily focused on the tech side of things, the confusion between customer research and UX research is essential to clear up. Allowing this misunderstanding to fester can make UX research seem overly complex and not worth the time investment it deserves. Let’s talk about this point a bit more.

Obstacle #1: justifying UX research to the tech-focused crowd

Many dev tool founders were initially software engineers themselves. In my view, people who work with code usually like to work independently, and they might even be hesitant to accept outside opinions; instead, they’ll trust their own judgement first and foremost. This is actually probably just a characteristic of human beings in general, but in terms of making the case for UX research, this can nevertheless present a hurdle to overcome.

“Self-biased” developers could therefore find themselves stuck in a “make-check-adjust” routine, implementing quick product changes instead of collecting and leveraging user feedback to guide their choices.

Beyond that, in general, a programmer’s passion for technology over the users themselves could be an obstacle to UX research progress. The excitement of using a cool API or the latest tech solution could be more attractive than the prospect of understanding user behavior and needs.

Here’s my prescription for this condition:

• First, consider UX research to be part of the software development cycle itself. Undertaking intermediate product prototype evaluation with users should seen kind of like debugging code (in fact, QA has a lot in common with usability testing.)
• Establish your team’s decison-making patterns; you might notice some features are quick and easy to handle, but others seem to undergo endless iterations which can become exhausting, like those involving differing opinions, and sudden pivots. In the latter case, involving users/test participants could provide a path forward here.
• Calculate the costs of internal meetings and interations on tasks you’re not entirely confident in; while this point isn’t strictly related to this obstacle, this is still a worthwhile point to mention here.

Obstacle #2: finding UX research participants

Even if a particular developer tool has a diverse userbase, the fact remains that dev tools also usually have a niche target audience. Seemingly, this limitation could make it challenging to find the right range of participants for UX search purposes.

To guide you as you overcome this challenge, I have 2 do’s and one don’t.

First up, the do’s:

✅ DO: Treat your target candidates like gold. Researchers must focus on building relationships with potential participants, as this is key to gaining valuable insights and feedback that can help your product.

But how to find these ellusive folks? Well, focus on the problems that your product is intended to solve. For instance, you may find target candidates in GitHub issue discussions, or in places where community discussions are ongoing.

Once you do, carefully reach out to these people and encourage them to discuss the problem. After you’ve established a connection, you can gently invite them to participate in some user research: tell enough about your product to pique their interest, and make sure to highlight the features/problems that your product works with in the context of the discussion that you initially connected with them about.

✅ DO: Use recruiting platforms that allow you to screen your audience using advanced beyond just basic demographics. Some of these platforms can also help you handle the research design (like surveys, usability tests, and so on) and participant rewards.

In terms of the aforementioned platforms, I recommend UsabilityHub and Maze for the entire cycle: from recruiting participants to the study and reports, and Prolific.co, Respondent.io, Askable, Userinterviews, Usertesting for matching with research candidates.

❌ DON’T: While you could certainly reach out to your target audience by directly using social media like LinkedIn or Facebook, it’s quite likely that folks won’t appreciate seeing your messages in their inbox. So, I really just don’t recommend using a formal or cold outreach approach for your dev tools UX research goals.

Obstacle #3: knowing what to research

Common products like ecommerce or personal apps usually involve a few workflows, but dev tools are often a kind of multi-tool that users might use in a number of different ways. They often have complex workflows and features that may be difficult to understand or test at first glance.

This difficulty can make designing effective research studies and gathering meaningful insights challenging. For instance, conducting a study aimed at understanding every aspect of user satisfaction within a well-established feature, such as the global navigation UI, can lead to non-actionable items or common conclusions that everyone already knows. On the other hand, conducting user research to improve, let’s say, button colors for dev environment tags in the UI, might appear excessive.

Instead, it’s crucial to ensure that the research study subject is well-balanced, focusing on topics that provide valuable insights while being cost-effective, rather than investing in a thorough research process with a substantial budget allocated for participant engagement on less impactful matters.

Here’s how to know where to go to kick the ball down the field:

• Find your golden middle. Break down your product to its core features, then pick those that are the most “unknown” or questionable.
• Before diving into potential user flow testing, kickstart with user interviews centered around the chosen topic. Briefly step away from your product to explore the wider user experience concerning the matter at hand. Recruit 5-8 people and ask them to describe how they utilize the feature in their work; guide them to navigate through their current workflow while touching on their pain points and jobs to be done.

Obstacle #4: we don’t have time for UX research

Developers must often work within tight deadlines and short iteration timeframes.

Frankly speaking, teams may not have the time to design and conduct lengthy UX research sessions. Indeed, this can make it difficult to gather sufficient data and insights. But the situation is not hopeless by any means.

Here’s what to do:

• Initiate research well in advance of your major features’ development. Aim to complete the research phase before the engineers start working on them. This way, the insights gained from the research can be shared with the team right at the onset of the development phase, ensuring that user-centric considerations are integrated into the development process from the beginning.
• Employ top rapid research methods during the design and development phases: testing prototypes and conducting surveys. These methods are straightforward and provide clear data, like the time taken to complete a user flow. The results can be reviewed alongside the ongoing design and development work.

Obstacle #5: we don’t have UX research experience

Well, actually, maybe you do.

I posit that dev tool developers constantly engage in UX research, it just manifests a little differently than expected.

Think about it: within developer communities, particularly with platforms like GitHub, interaction and feedback are everywhere. If you’ve participated in any discussion like that, you’ve participated in some part of the UX research process.

So, don’t be too overwhelmed, because this may not be such a foreign concept after all. Still, let’s try and devise a straightforward workflow tailored for dev tool developers:

• Organize a knowledge base about your users similar to the documentation for your code; you’re really developing the tool for people, and those people deserve to have a dedicated place in your workspace.
• Learn about UX research principles and methodologies to enhance the user experience of your developer tools. Explore the Complete Beginner’s Guide to UX Research for a broad understanding. Explore multiple guides on user research available on Maze. Additionally, the UX Research Cheat Sheet by NN/g offers a compact overview of research methods. These resources can provide foundational knowledge to improve the user experience of your developer tools.
• Fake it ‘till you make it: pretend your colleague is an agnostic user, and conduct an interview about your product. The results might surprise you, especially in teams where decisions are made by a limited group of people.
• Get used to collecting not only research data but anything you find useful: read comments; explore what people are asking about your product; be curious and fix things that might not be so useful today but could help in future. This mindset is important.

The final obstacle

To bring things back full circle, just like meditation, you can know all the benefits of something, but still fail to put it into active use. The next step is for you to take the knowledge I’ve raised here, and to put it into practice. The path you choose is up to you, but any action taken is an action taken towards conducting real UX research, incorporating those findings into your product, and ending up with a better result than you started with. Good luck!

At Evil Martians, we transform growth-stage startups into unicorns, build developer tools, and create open source products. If you’re ready to engage warp drive, give us a shout!