This article reports the results of a survey, in which 436 UX practitioners completed (or attempted to complete) an online survey in October 2019.
The discovery phase is an important part of the UX-design process, so we aimed to understand how companies perform discoveries on real-world design projects and the barriers to carrying out discoveries effectively.
A reminder from our full article on discovery in UX: A discovery is a preliminary phase in the UX-design process that involves researching the problem space, framing the problems to be solved, and gathering enough evidence and initial direction on what to do next.
Survey Findings
Most UX Practitioners Carry Out Discoveries for Some Projects
75% of the respondents claimed that their organization carries out discoveries. Out of those, only 20% said that their organization performs discoveries for every project, while the majority (60%) claimed that their organization carries out roughly 1 discovery phase for every 2–4 projects.
Many Organizations Forgo Discovery Due to Lack of Time
The top reason for not carrying out discoveries at all in an organization was time pressure to deliver, followed by no management buy-in. Only 7% of the 69 respondents whose organizations did not carry out discoveries believed lack of resources was the biggest factor.
“[The] focus is on delivery — they want to deliver ‘something’, and then fix later.”
It’s worth noting that the desire to deliver quickly may also affect UX professionals’ work in other stages of the design process, particularly in carrying out usability testing or having UX insights incorporated into product development. Rushing into implementation with too little time for UX activities is a sign of an organization’s low UX maturity.
Of course, in reality, several factors dissuade organizations from implementing discoveries, and our respondents explained that it was often a combination of both management buy-in and time pressure.
“In addition to time pressure, there is little CEO buy-in. He tends to feel that we already know the problem that needs to be solved.”
“I think there's also a lack of understanding of their value — the company considers itself very innovative, and there's a race to come up with the next amazing idea.”
Interestingly, respondents that work for organizations that do run some discovery phases in their design projects reported (in a free-text field) the same reasons for why certain projects did not get a discovery phase. Both these findings suggest that lack of buy-in and time are top challenges for running discoveries.
UX Practitioners Who Conduct Discoveries Are More Likely to Report that Their Projects Are Successful
Respondents were asked about the last project they worked on at their organization. Of those who did not carry out a discovery, 52% of respondents said that their last project was successful — barely better than chance! In contrast, 83% of respondents who had carried out a discovery in their last project said that the project was successful. A chi-square test revealed this difference to be highly significant (p<.00001).
Although we can’t be certain if these projects were actually successful and we don’t know whether it was the discovery phase which contributed to the success of the project rather than correlated factors, this result does lend credence to the widely held belief that discoveries lead to better outcomes, because they frame the problem to be solved, set teams in the right direction, and provide them with relevant knowledge to design successful solutions. Further evidence that this is the case was provided when respondents rated their agreement to the following statement: Our discoveries positively impact the products and services we build. 84% of 240 practitioners with experience of carrying out discovery phases agreed with the statement, and only 5% disagreed.
There are two ways of interpreting this chart:
- Performing a discovery cuts the risk of failure by 75% (4 vs. 16 failures).
- Performing a discovery increases the chance of success by 59% (83 vs. 52 successes).
Of course, these two ways of stating the results are two sides of the same coin, but it is remarkable that the most striking impact of discoveries comes from risk reduction. This is actually true of most UX work: doing good UX will often increase product success, but it will almost always insure you against true disasters.
The Average Team Size Is 4
We asked respondents to tell us who was on the team carrying out the discovery. The average number of individuals involved full time on a discovery project is 4 (SD=3.5), with a range from 1 to 36 (which is far too many). The most common roles present on a discovery project were UX or product designer (74%), researcher (60%), and product or project manager (52%). Only 36% of discoveries had a product owner.
Most Discoveries Are Carried Out in Under 2 Weeks
We asked respondents to tell us how long the last discovery they ran was. Obviously, all discoveries are different, but we wanted to know roughly how long the average discovery phase is, across all projects.
I would have expected the data to form more of a bell curve, with few participants running very short or very long discoveries, but instead the chart above is somewhat bimodal. The majority (58%) of discoveries undertaken by respondents were on the short side — at most two weeks. Unless users are easy to access and the problem space is relatively small, two weeks is too short. Just under a third perform longer discoveries (over 7 weeks).
There are many possible reasons why the majority of discoveries are short. One may be related to budget and time constraints, which result in time-limited discoveries. Another reasons might be the popularity of Google Ventures’ Design Sprint, as well as the adoption of continuous product discovery, which means some organizations favor leaner initial discovery phases, while carrying out more discovery-style research throughout the design process.
The problem with slimming down the discovery phase or replacing it with a design sprint is that some teams may miss out on a time of crucial learning, beginning the design process with solution ideas that might not be based on real problems. Survey respondents commented on the brevity of discoveries in their organization:
“In my opinion, the time spent on discoveries in my company is not enough and sometimes discoveries are carried out not to know the problem better, but to confirm [the] initial idea.”
“The UX team believes discovery work can be continuous with 5 or less participants and always presenting a solution first without researching the problem space first, just going off anecdotal evidence on problems. 2–3 people's response to a proposed solution that doesn't address the problems they've actually expressed doesn't seem like best practice.”
Most Common Activities
In the discovery phase, it’s possible to perform many different activities: stakeholder interviews and workshops, analyzing analytics data, field studies. Respondents were asked which activities they had performed in their last discovery. When carrying out exploratory research, it’s best if data is triangulated from numerous sources and through different methods, to ensure that teams are getting a full picture of the problem.
Across all discoveries, the average number of activities performed was 4.6 (SD=2.5). The most common activities carried out were user interviews and stakeholder interviews. Both methods are quick, which would explain their popularity, given that many discoveries are run in less than 2 weeks. The short duration of discoveries might also explain why diary studies are the most unpopular activity.
Biggest Challenges in Carrying Out Discoveries
Many discoveries are far from perfect and are hampered by real-world constraints. Respondents were asked what their biggest challenges were in conducting discoveries effectively. 215 responses were provided via a free-text field; these responses were coded and analyzed. There were many different challenges, so we simply report on the most frequent challenges mentioned by respondents.
Lack of Time and Budget
The most frequent blocker mentioned was limited time (84 comments). Budget (27 comments) was also a factor; note that budget also affects allocated time. Even if the organization plans to do a discovery, the discovery itself is often the first thing to be reduced or removed.
“The intent is there to allow time for discovery but then deadlines and bottom-line revenue is pushed by old mindsets and sadly everything falls back 50 years to system development, and we try our best to [do] some form of discovery.”
In some cases, UX is simply brought in too late, which means UX practitioners have less time to carry out essential discovery work.
“It isn't often built into the project process, or teams come to us too late to do anything. Even if we discover the solution doesn't address a problem, too much development work has been put into it to stop…”
In other cases, respondents spoke about not having enough budget to perform longer discoveries, because time is money, especially for agencies pitching work to clients.
“Demonstrating the value of comprehensive discoveries to [the] public sector. This sector is very accustomed to penny pinching and it can seem like a ‘luxury item’ when they already believe they have identified the problem(s).”
Budget constraints can also affect things like how big the team is and whether there is enough money for incentives or recruiting for user research.
Lack of Buy-in and Hard to Prove Value
Another big theme was lack of buy-in (61 comments). This means UX practitioners are constantly trying to explain the value of the discovery phase, and are sometimes not successful in convincing stakeholders or clients to do a proper discovery due to “legacy ways of thinking,” beliefs that product managers already know their users, that solutions are obvious, or that shipping a Minimum Viable Product (MVP) will provide all required learning.
“Again, selling in a discovery-based project is tough. Whilst we advocate for it and show benefits, some projects still believe [that] shipping [an] MVP will give them the learning they require. Discovery can provide good direction, reducing risk and increasing confidence of the product.”
Many respondents spoke about stakeholders and other team members not understanding what the discovery is. Some specifically mentioned the difficulty in quantifying the ROI of a discovery, which is often needed to gain buy-in from those holding the purse strings.
The Right People Aren’t Always Involved
16 respondents mentioned challenges related to getting the right people on the team or to lead the team.
“Even though every project starts with discovery, it is still a largely inconsistent process and it is not currently led by product managers or designers. Instead, it is led by project managers, engineers, or account managers. We're working to change this, but product managers and designers are still largely left out of discoveries leading to pain later on in the process.”
Some respondents were a team of one, and, thus, responsible for doing discovery alone. Others mentioned that the people with the right skills weren’t involved, which made running the discovery difficult. For example, some teams lacked a person able to competently carry out user research.
“Putting together a multidisciplinary team is a tad difficult when we have limited staff with the right skill sets.”
Lastly, other respondents claimed that people with a lack of understanding were also on the team, and it was hard to collaborate with them or prevent them from jumping to solutions and succumbing to their own biases.
Access to Users
30 respondents mentioned getting access to users was difficult in their discoveries. Several explained that they worked for B2B organizations and were denied access to users or the process to recruit them was very difficult. This problem was exacerbated by lack of time to recruit specialized users. This problem is not unique to the discovery phase, and we hear about it often from our attendees at our UX conferences.
Survey Limitations
We invited UX practitioners to complete the survey through LinkedIn, Twitter, our Alertbox newsletter, and our website. Since we recruited a convenience sample, we cannot claim that it is representative. It’s likely that in our sample, UX practitioners with experience of carrying out discoveries are overrepresented. We tried to minimize this effect by calling for all UX practitioners to complete the survey, regardless of whether their organization does or doesn’t run discovery phases.
Conclusion
While many UX practitioners have succeeded in implementing discovery phases in their organization, most discoveries are still too short, underresourced, and don’t have the full backing of the organization. This is often the case when clients and stakeholders don’t understand their value. This attitude leads to cost-cutting and trimming of what are deemed unnecessary activities. As a result, too many design projects timidly explore already familiar corners of the design space, missing out on building features that satisfy previously unmet user needs. The good news for those who do perform discoveries is that they possess a ‘secret weapon’ that will help them establish a sustained competitive advantage.
As a profession, we’ve done well to find our place in the development process, convincing others that we should test with users. But we need to become better at educating our organization as to why it makes business sense to spend time upfront in discovery, and why getting UX practitioners involved early is important. Imagine how much more successful projects would be if we could make that happen!
Learn More: Discoveries: Building the Right Thing, a full-day course at the UX Conference.
Share this article: