How and Why to Recruit Backup Participants (aka “Floaters”) in User Research

Effective user researchers balance project-schedule constraints with their research budget. Efficient researchers attempt to spend only as much as needed, so the remaining money can be used for more research.

When I run research projects, I try to spend enough money on recruiting and incentivizing participants, to ensure that (1) I find the right kind of users and (2) that they are motivated enough to show up for the sessions (and are also fairly compensated for it). “No-show” participants or last-minute cancelations can thwart the project schedule, make stakeholders lose interest in the research, and waste money.

Consider the expense of each testing slot in a usability study or other UX research activity:

• Amortize the planning expenses across the study participants. So, for example, 20 hours spent planning a 5-participant study amount to 4 hours of expensive UX-specialist salary for each participant slot.
• Look at the costs of using the testing space. For an in-person study, renting lab space that stands empty because of a no-show user is an out-of-pocket cost. Even if you’re testing in, say, a meeting room in your own building, there’s the opportunity cost of having booked the room and prevented other employees from using it.
• Include team members’ time. The biggest expense of a scheduled study slot that goes unused is often the cost of the team members (including yourself) who are waiting for the participant.

Don’t forget the reduced research insights stemming from fewer research participants than the plan called for. If you follow our recommendations for small sample sizes for most qualitative studies, each lost participant represents a fairly substantial proportion of your potential insights, and thus reduces the profitability of your design project.

When to Recruit Floaters

There are some cases when it’s appropriate to throw thriftiness to the wind and recruit extra users, also known as “floaters.” This strategy makes it likely that you will fill in each research slot. Some scenarios when this approach is beneficial include:

• The project schedule allows only a small time window for research. For example, a client may need a usability test and its results within a few days.
• An external factor imposes time limits. For example, equipment or the location for a lab study are rented and only available for a finite period.
• The participant background requires partially screening the participants on site, at the time of the study. For example, in an eyetracking study you may need to calibrate the users’ eyes with the eyetracking system onsite.
• A lot rides on observers seeing a particular research session or set of sessions. For example, if a team had to be convinced to do research and is skeptical, and it finally agrees to observe at a certain time, you need that user to be there at this critical point.

To ensure that you have enough participants when you need them, you can practice “intentional overrecruiting.” Overrecruiting can manifest in a few ways:

• recruit multiple people (a participant and a floater) for each scheduled slot
• recruit one participant for each slot and a floater for every two slots
• add more sessions (and more participants) than needed to the research schedule

One Floater per Participant

The most expensive method, and the one that also gives the most insurance, is the one floater to one user — that is, recruiting two people per slot, even though you’re testing with only one of them.

A research day might look like this:

If the recruited participant shows up, I pay the floater for that time slot and let her go.  

Floaters recruited for just one session are usually paid the same honorarium as participants. So, if the participant’s incentive were $100 per participant, the single-session floater’s incentive will also be $100.

With this approach, the cost for the recruit and incentives per study is twice as much, provided all users arrive for their sessions. For example, if the recruiting and incentive cost is $180 per user and you had 5 users, then the total cost without floaters would be $900 and the cost with floaters would be $1,800.

One Floater Per Two Users

A little less expensive but also a less safe approach is to recruit one floater for every two sessions. With that pattern, a research day might look like this:

In this case, each floater would be available for two sessions. For example, if User1 shows up, the floater would stay available to make sure that User2 also shows up. If User2 does show up, the floater is released at the beginning of the second session. This is probably the best-case scenario for the floater since she basically needed to be available for an hour and a half and got paid for being at the full sessions. 

If User1 doesn’t show up, the floater would take that person‘s place. And, the researcher would hope that User2 is going to show up because there is no floater for that session. If User1 shows up but User2 does not, the floater would take User2’s session. This is probably the worst situation for the floater because she had to wait for one session and then also be a participant. 

Floaters who must be available for multiple time slots are usually paid more. I usually pay double the incentive. (Some people pay more than that because floaters sign up to not only be available for a longer time but participate in the study if needed. Others pay floaters more only if they end up being a test participant.) So, if $100 was the participants’ honorarium, a double-session floater would be paid $200. (Note in the example above, since there is an odd number of sessions, one floater will cover just one session and will be paid $100.) If the research was remote, one might pay floaters less because they are not asked to travel or sit in some space for a period of time; rather they can do other things in their home or office while being on call.

How to Recruit Floaters

The point of having a floater is to ensure that you cover all your research sessions. It’s essential that they show up, whether in person or for a remote study. Thus, when recruiting them, they should not feel as though they are superfluous. I’m always honest with floaters. I recruit them using the same screening criteria used for study participants. Once they are deemed the right target for the study, I explain their role:

We have recruited all the timeslots for the study, but we need to make sure we actually have a person in each session. Since sometimes people have last-minute cancellations, we need to recruit more people to be available in case an already scheduled participant can’t make it.

For one-to-one: So, we would ask you to come (or log in for a remote study) as though you are participating. If the scheduled participant arrives as planned, we will pay you your $100 honorarium and release you.

For one-to-two: So, we would ask you to come (or log in for a remote study) as though you are participating. But we would ask you to be available for two sessions, starting at 8 AM and ending after 10:30AM. If the scheduled participants arrive as planned, we will pay you $200 and release you. But if either user doesn’t arrive, we would ask you to participate in the study.

For a remote research session, I ask floaters to log into the session as they would if they were a user. For in-person sessions, I inform the floaters that there will be a waiting area and wireless and drinks available and they will be asked to stay there until released. If there is no such place, I try to find a café nearby where they can go and wait as needed.

How to Dismiss Floaters

If the situation was explained to floaters when they were recruited, there should be no problem dismissing them. Still, make sure you don’t hurt their feelings or make them feel like they did something wrong and that’s why they are being dismissed. I usually say something like this:

As you know, you were recruited to ensure we had enough participants per session. The original person we scheduled has arrived, so we are all set. Thank you so much for agreeing to be part of our study but we won’t need you this time. Here is (or I will send you) your honorarium of $100 as planned. And I hope we can keep you in mind to participate in future studies. Thank you very much.

Using Floaters to Optimize Participants

Another advantage of recruiting two users for each study time slot is that, if both show up as scheduled, you could choose the one most suited for your study. In that case, you would not designate one person as the “main” participant in advance and the other as the floater.

Picking the most suited participant can be useful in studies:

1. Targeting a particularly difficult recruiting profile or one that is challenging to prescreen — for example, if you are testing a high-end investment product and want to see the investors’ skills before the study
2. Requiring a certain demographic distribution — for example, an even age distribution

If all your early test slots are filled by younger users, you would then prioritize older users for the last slots.

(However, avoid choosing participants based on random criteria — such as which participant is more talkative or seems more pleasant or likeable— because you can easily bias your study results doing that.)

More Research Slots and Cancel as Needed

A little different from scheduling floaters is to schedule extra study sessions. If you’re not on a tight deadline and you (or your team) can pivot and do other work if a user doesn’t show up, this method can be an effective way to ensure you have enough participants, and is usually less expensive than the floater models.

A schedule using this model might look like this:

For unmoderated (no facilitator) remote research sessions, I usually recruit one extra person for every 3 sessions. For example, if I want 5 sessions, I recruit and run the study with 7 people. Once you have enough data, you can cancel the sessions you don’t need (but, of course, still pay those users in earnest). If enough users don’t show up even with the added sessions, you can schedule more sessions.

Often, though, it makes sense to run the extra sessions even if all the needed participants did show up, for two reasons:

1. Unusable data in some of the sessions. After I watch the recordings, I sometimes find some issue with one of the sessions and can’t use all or most of the data. Overrecruiting thus covers not only for no-show participants, but also for sessions with poor data and saves you the cost of relaunching the study and waiting for feedback.
2. Additional insights. Sometimes I need to follow up on something discovered in earlier sessions — if that’s the case, I can take advantage of those later session to change tasks or interview questions and investigate the new questions.

Wouldn’t just Paying a Higher Incentive Eliminate the Need for Overrecruiting?

Research participants driven by the incentive can be more likely to show up if that incentive is larger. So, you could experiment with paying a higher incentive and not recruiting floaters. You would save the recruiting costs of an additional user and could maybe pay a higher incentive to the main participant. But this strategy works only if the factor preventing participation is related to the incentive. It doesn’t account for emergencies or last-minute schedule changes, which do happen, especially when testing users in certain professions, such as system administrators or physicians. (Also, consider whether a very high incentive could bias your study.)

Isn’t Intentional Overrecruiting Wasteful?

What I like least about intentional overrecruiting is that time and money are sacrificed by the team, users, recruiters, and researchers.

Deciding to use these overrecruiting methods is a lot like deciding whether to buy optional insurance. For example, is spending $100 per month on long-term healthcare insurance worth it so that, if you need care in your old age, you have it? Likewise, what is the insurance of having a user at each allotted time worth to your team?

The table below compares the time, recruiting, and incentive costs for a 5-user study using the three overrecruiting models.

Assuming the researcher facilitates five sessions:

• The recruiting and incentive costs are the most ($1800) for the 1-floater-to1-user overrecruit model, and least ($1260) for the more-research-slots model.
• The risk of not having enough users in the study is the lowest with the 1-floater-to-1-user model, and higher for both the 1- floater-per-2-users and more-research-slots models.
• The time before research is complete is the least (one day) with the 1-floater-to-1-user and one-floater-per-2-users models and most (one and-a-half days) with the more-research-slots model, which requires additional sessions and thus time.

In the example above consider the extra cost was $900.

• Is the assurance that you will get the feedback you need from target users within the specified time worth $900 to your team?
• Do you have that in your budget?
• What’s the opportunity cost of that $900 — that is, what else do you need to do and spend money on where the $900 could generate a higher return?

These are the types of questions you should ask yourself when you are thinking about overrecruiting.