The critical incident technique (CIT) is a systematic procedure for obtaining rich, qualitative information about significant incidents from observers with firsthand experience, which in turn helps researchers understand the critical requirements for individuals, processes or systems.

Definition: The critical incident technique (CIT) is a research method in which the research participant is asked to recall and describe a time when a behavior, action, or occurrence impacted (either positively or negatively) a specified outcome (for example, the accomplishment of a given task).

The instance reported by the participant is known as an ‘incident’. In order for the incident to be critical, the participant must be confident that the event had a causal relationship with the outcome (which is the focus of the study). An example of a critical incident question is given below:

Please think of an Agile project that you worked on that was successful. Please describe a specific time when someone did something (or something happened) that positively contributed to the success of that project.

This method was formally introduced to the social sciences by John Flanagan, in a seminal paper published in the Psychological Bulletin in 1954. The technique was developed and finessed in numerous studies undertaken by Flanagan and fellow psychologists from the US Aviation Psychology Program during the second World War, and later by the American Institute for Research and the University of Pittsburgh. Since the publication of Flanagan’s paper, the CIT has become popular in the social sciences and has found applications in human–computer interaction research, because it facilitates the gathering of many detailed ‘incidents’ (behaviors or events). These are useful in understanding critical requirements for roles, systems, and processes. For example, the CIT has been used to highlight characteristics of successful personnel (such as leaders, nurses, doctors, air-traffic controllers), as well as critical requirements for processes (such as training programs and services) and interfaces.

In UX, the CIT is often used in user interviews. However, critical incidents can also be captured using questionnaires, focus groups, or structured diary studies. Flanagan believed that researchers carrying out ethnographic-style research could also document critical incidents, although little has been said on how this approach should be implemented.

Using the critical incident technique is slightly different from asking a standard, example-style question in a user interview. The table below shows some questions that could be given to employees to learn about their experience using an enterprise tool. A critical incident question is compared to other example-style questions.

The participant is asked: Type of question
Tell me about a time where you used the tool in your job. Example question: the participant is asked to provide an example; there is no direction from the researcher as to what kind of example. The answer could be anything that happened to come to the participant’s mind.
Tell me about the last time you used the tool in your job. Specific example question: The participant is asked to describe the most recent time. This is not necessarily a critical incident, just the most recent.
Tell me about a particular time when you used the tool in your job where it helped you to be effective in your work. Critical incident question: the participant is asked to think of a specific incident that was critical to the accomplishment of a task.

Usually, in a critical incident interview, the participant is given time to think of each incident before describing it, as recall can often take time. The interviewer also has carefully scripted followup questions meant to elicit enough factual information about the incident. The interview could look something like this:

For the next series of questions, I’d like you to focus on how you use the tool in your work. Interviewer introduces the focus of the study
  • What are some things you do with the tool?
  • How often do you use the tool?
  • When do you use the tool?

Check tool-use criteria
Please think of, and tell me about, a particular time when you used the tool, and it made you effective in your work.

Critical incidents (positive)
  • What task were you doing at the time?
  • Why did you choose to use the tool?
  • In what way did the tool make you effective?
 Clarification questions
Is there another time you can think of where you used the tool and it helped you to be effective in your work?

Seek out further incidents
Now conversely, please think of, and tell me about, a particular time when you used the tool, and it made you ineffective in your work. Critical incidents (negative)
  • What task were you doing at the time?
  • Why did you choose to use the tool?
  • In what way did the tool make you ineffective?

Clarification questions
Is there another time you can think of where you used the tool, and it made you ineffective in your work?

Seek out further incidents

Generally speaking, when the researcher seeks critical incidents, the participant is asked for events that demonstrate both positive and negative effects on the outcome. These questions are usually separate, as in the example interview above. However, in some cases, the researcher may ask for a positive or negative case at the same time and allow the participant to choose which incident to begin with. When asked separately, it is typical to begin with asking for positive incidents in order to begin constructively.

When the critical incident technique is used in research, each participant could contribute many incidents. It is quite possible that hundreds of incidents (sometimes thousands) are collected through numerous interviews which then need to be coded. When codes are well saturated (e.g., there are many incidents for each code), researchers can be fairly confident that they have documented the core requirements for the object of study. In the enterprise-tool example, these requirements could include ease of access (users need to be able to find and open the tool quickly), responsiveness (the tool needs to respond quickly — for example, autosave shouldn’t slow users down), or nonintrusive updates (updates should not interrupt the user at work).

Pros and Cons of Using the Critical Incident Technique

This method has some advantages, as well as some key disadvantages for usability research.

Pros

  • Quickly uncovers system issues
  • Captures incidents over a long timeframe: Participants can go back as long as they can remember. As a result, incidents could span years. This is an advantage over observational research, which is often time restricted.
  • Captures information about rare or uncommon incidents: When observing users in their domain, key incidents are not always witnessed because they are rare or uncommon. The CIT makes discovery of these incidents possible.
  • Emphasis on more-important issues rather than less-important issues. Most other methods usually collect a preponderance of low-importance issues, simply because they tend to be more numerous. Of course, there’s no guarantee every reported critical incident is actually important, but significant events will likely be easier to recall than minor incidents.
  • Flexible: The CIT can be applied in interviews, focus groups, and surveys.

Cons

  • Relies on memory and pure recall: Memory is fallible, and so details can often be lost, or critical incidents can be forgotten. Recall is also challenging and even stressful for some participants, particularly in a face-to-face setting.
  • Doesn’t represent typical usage: Often, participants recall extreme events, but small usability issues and typical usage are rarely mentioned in CIT interviews.

When deciding whether to use the critical incident technique, consider what your research goal is and whether a usability test or observation in the field will be better suited for achieving it. If using the critical incident technique in your research, ensure that you know what type of incidents you want to learn about. Take the time to compose an interview script and pilot it to check that your questions aren’t too prescriptive, vague, or ambiguous. (Anecdotally, researchers have found that varying the types of words used in the critical incident question can affect the type of incidents recalled, so think carefully on how not to lead participants). Lastly, you will always learn more about how people use existing interfaces and what their pain points are if you carry out observational research, such as contextual inquiry or usability testing.

Summary

The critical incident technique (CIT) is a useful methodology to uncover critical requirements for people, systems, and processes. When using the CIT, ensure that you are clear on the kind of incidents you want to study, prepare an interview script, and pilot it. Complement CIT interviews, focus groups, or surveys with observational research (like contextual inquiry and usability tests) to get an accurate picture of the usability of systems, products, or services.

Reference

Flanagan, J.C. (1954). The Critical Incident Technique. Psychological Bulletin, 51(4), 327-357.