OUP user menu

Determination of health-care teamwork training competencies: a Delphi study

Robyn Clay-Williams, Jeffrey Braithwaite
DOI: http://dx.doi.org/10.1093/intqhc/mzp042 433-440 First published online: 10 November 2009


Objective The purpose of this study was to determine the optimum content of a 1-day classroom-based crew resource management (CRM) course for health-care personnel working in ad hoc teams in complex, time-critical hospital departments such as surgery, intensive care or emergency.

Design A two-round modified Delphi panel. Participants selected teamwork competency components suitable for inclusion in 1 day of training from a list developed via literature review.

Participants Fifteen experts in health care, CRM and training.

Main Outcome Measure Knowledge, skill and attitude competency components for a 1-day CRM course.

Results Of the 110 knowledge, skill and attitude CRM competency components, 40 components were selected by greater than 70% of respondents, whereas the remaining 62 components were selected by fewer than 55% of respondents. These 40 competency components ranged across five competency domains: communication, task management, situational awareness, decision-making and leadership, and provided a consensus on the most critical areas for inclusion in training for health-care personnel.

Conclusions This new competency model is now available for use. Although the sample size was limited, a high degree of consensus was reached after only two rounds. A modified Delphi technique within the context of competencies first refined from the literature was a useful and cost-effective method for determining the content of a 1-day CRM training course for health-care workers.

  • crew resource management
  • teamwork
  • competency-based education
  • needs assessment


Increasingly, health services are delivering aviation crew resource management (CRM) style training to medical staff, with the aim of improving teamwork behaviours and reducing errors. Aviation has become safer over time, and this has been attributed amongst other things to education, so the argument is there is much value in applying the methods to health care [1] in the quest to improve patient safety.

A growing body of published literature supports this approach [211]. CRM training consists of imparting teamwork knowledge, skills and attitudes in a facilitated learning environment [12], and can be delivered in the classroom, or using a simulator.

The majority of present CRM programs are funded commercially or by governments, are usually implemented in large medical centres [10, 13, 14] and are often inaccessible. Many other health-care organizations find it difficult to fund this type of training, and there is a need for structured information on which mix of competencies should be taught, to allow in-house programs to be developed. Although information on general and specific content of CRM training is available in the public domain, there is little in the way of guidelines on how to select from a growing body of material when designing a training program. Publication of CRM competency components selected by experts for the health-care environment will facilitate discussion and help work towards standardization of this type of training.


Research question

The purpose of this study was to determine the optimum content of a 1-day classroom-based CRM course for health-care personnel working in complex, time-critical hospital departments such as surgery, intensive care or emergency. Because team composition in these environments is constantly changing, students should ideally be trained in portable, individual team skills. This is the first phase of a larger study investigating the effectiveness of classroom- and simulator-based CRM courses for health-care workers to improve the teamwork attitudes and behaviours of the participants.

A number of methods are available for selecting course content. Possibilities included conducting a training needs analysis via interview or focus group, engaging an expert panel to develop the curriculum or using a Delphi procedure to survey experts in the field. While alternative methods have strengths and weaknesses, combining approaches can lead to stronger designs. The Delphi technique within the context of a refined, structured, literature-based competency list was chosen as its main advantage is the ability to obtain timely advice and expertise from a wide range of geographically dispersed participants with an economy of effort. Other health-care studies have used this approach; it is a useful method that might be labelled an ‘Evidence-Initiated Delphi Process’. In addition to obtaining consensus on the optimum composition of the CRM course, the panel would also provide avenues for examining alternative content elements, thereby militating against inadvertently omitting important competency components from consideration.

Delphi process

The Delphi process was originally developed by the RAND corporation [15], and has been commonly adopted in health-care research as a method of obtaining agreement among a group of experts [16]. It is a multistage process that collates and synthesizes opinion of individual panellists to form group consensus. Three defining characteristics of the Delphi method are [17]: anonymity of participants, iterative polling rounds interspersed with feedback and statistical analysis of group results. Qualitative and quantitative data are collected and fed back to participants over a number of rounds until consensus is obtained.

The Delphi method has been criticized, most notably by Sackman [18], as not meeting scientific standards in terms of selection of participants, formulation of questions or analysis and interpretation of results. Much of the criticism relates to using Delphi in its original manifestation as a forecasting tool, where broad opinion is solicited about the probability of occurrence of some future event. In this study, rather than formulating a response on a broad topic de novo, seeking initial responses via a series of open-ended questions, participants selected from a large list of competencies distilled from the literature, and seen as valid and applicable to teamwork training for health-care workers. In effect, the respondents contributed to a training needs analysis, whereby they chose the most relevant CRM competency components to prospective trainees. Should the technique succumb to pitfalls, such as group tendency to over-conformity or snap judgements by time pressed respondents, the worst case would be a course that was less efficient, but likely to be still useful and recognizable as CRM training. Although the recommended number of Delphi iterations is often three to four [15], time constraints and the benefit of starting with an extant competency list, meant the process was designed in two rounds following the streamlined RAND-UCLA protocol [19].

Selection of participants

Specialist participants were selected based on their knowledge of CRM principles, familiarity with the health-care environment and experience in education or training. This background would not only give panellists the ability to prioritize the most important CRM components for the target trainees, but also to select a group of components that were both teachable in 1 day and coherent as a course of training. As CRM is relatively new to health care, there were few candidates who met this requirement. Thus it was not possible to sample participants randomly. The majority of candidates were anaesthetists running medical simulation training programs. In order to broaden the sample, participants with expertise in developing CRM training for other industries were included. All except one of these industry experts had experience in translating CRM to health care. Candidates were also asked to recommend others that they thought would be able to contribute, and those nominated with a suitable background were contacted. A final group of 21 experts was enrolled.

Determination of Round 1 knowledge, skills and attitude components

Following an extensive review of the literature on teams and team training in high-reliability industries and in health care, five basic CRM competencies that may be applicable to medical personnel working in surgery, intensive care or emergency departments were determined: communication, task management, situational awareness, decision-making and leadership. These competencies were divided into knowledge, skills and attitudes (KSAs) that were supported by the literature as possible components of a CRM course. Randomizing the statements was seen as counterproductive, as part of the aim was to determine the composition of a cohesive course in addition to prioritizing individual competency components. Competency component statements were formulated as a compromise between optimum length to yield consensus (20–25 words) [20], and minimization of ambiguity. As all participants were familiar with CRM principles and techniques, keywords and phrases were able to be used in many cases to convey complex concepts economically. The list of KSAs provided sufficient material for an estimated 3 to 4 days of intensive classroom facilitated training.

Delphi panel process

In the first Delphi round, participants were given a table listing 102 derived CRM competencies, and asked to select those components that they would include if designing a 1-day course. Space was provided at the bottom of each group to add any extra competencies that participants believed should be included, and at the end of the list provision was made for additional comments.

Following Round 1, eight KSAs nominated by panel members were added to the competency component list, giving a total of 110 competencies for consideration in Round 2. To minimize the possible questionnaire fatigue effect [21], the order of the subsections was reversed, and presented in the order of leadership, decision-making, situational awareness, task management and communication. Within each subsection, the order of the individual competency components was also reversed.

In the second round, participants were provided with collated statistical results from the first round, a summary of the written comments raised by Delphi panel members in Round 1, and investigator responses to any questions. KSAs selected by more than 75% of respondents were highlighted to give participants an idea of the likely composition of the course after the first Delphi round. Respondents were given an opportunity to revise their previous selections based on those results. Initial individual judgements were therefore modified by group information.

Data analysis

Data collected in Rounds 1 and 2 were entered into an Excel spreadsheet. The percentage of respondents who selected each competency component and the percentage of selections made in each of the five competency categories were determined and graphed.

Ethical considerations

Ethics approval for the research was granted by the Hunter New England Area Health Service Human Research Ethics Committee (HNEHREC reference no 08/08/20/5.19) and the University of New South Wales Human Research Ethics Committee (HREC reference nos: 08/HNE/262 and 08274). Although the attempt was made to preserve anonymity of candidates, some of the participants were colleagues and, through correspondence on other matters, became aware of each others’ participation. There were no indications that this distorted results. Strict anonymity was maintained for participant responses.


Characteristics of respondents

Fifteen of 21 invitees returned questionnaires for Round 1, giving a response rate of 71%. The average age of the participants was 47 years (range 35 to >60), and two-thirds were male. The average experiences of respondents were 19.3 years in health care (range 0–30), 10.9 years in CRM (range 2–25), 6.9 years in CRM in health care (range 0–15) and 13.3 years in education or training (range 5–37). Eleven of 21 invitees returned questionnaires for Round 2, giving a response rate of 52%.

Round 1 results

The percentage of respondents that selected each competency component is shown at Annex A. Many of the panellists raised concerns regarding the adequacy of 1 day of training to impart the material. Some panellists believed it was not possible to reduce the training to 1 day and still have a meaningful educational product. These respondents tended to select only knowledge components, and advocated training delivery via a lecture. The remainder elected to preserve the experiential adult learning format and attempted to select across a range of competencies most applicable to the needs of the target group. On average, respondents tended to select more than could be reasonably accomplished in 1 day.

Round 2 results

Competency components selected by participants for inclusion in the course are shown in Table 1. A total of 40 competency components were selected by more than 70% of respondents (Table 1, in boldface). Figure 1 shows a comparison between Rounds 1 and 2 of the percentage of competency components selected for inclusion in the training, and illustrates that fewer selections overall were made in Round 2.

Figure 1

Percentage of competency components selected for inclusion in the training.

View this table:
Table 1

Delphi panel results: CRM's KSAs

% of panel selecting competency
Round 1 panelRound 2 panel
Competency: communication (communicates efficiently and effectively)
  Discusses the importance of open communications in medicine.10091
  Describes call-out and check-back procedures.9291
  Describes strategies to deal with conflict.6936
  Describes a method for asserting a corrective action.8591
  Describes a method for articulating concern about a course of action.100100
  Lists the elements of a brief.549
  Describes situations where a brief would be appropriate to facilitate group performance.6255
  Lists the elements of a debrief.380
  Describes situations where a debrief would be appropriate to facilitate group performance.3818
  Lists the steps in SBAR.6936
  Lists the elements of an effective handover brief.7791
  Discusses the importance of clear written communication in medicine.4618
  Establishes an atmosphere to encourage open communications.85100
  Listens critically and provides feedback to clarify information.8591
  Implements strategies to deal with conflict.6973
  Asserts a corrective action in a firm and respectful manner.8591
  Articulates when concerned about a course of action.8591
  Conducts briefs where appropriate to facilitate group performance.3818
  Conducts debriefs where appropriate to facilitate group performance.159
  Communicates critical information in a succinct manner.85100
  Conducts effective handover briefs.6227
  Speaks up when concerned.92100
  Treats others with respect.7782
  Values the importance of listening.6945
  Values the importance of resolving conflict in the workplace.5436
  Displays a positive attitude towards briefing.469
  Displays a positive attitude towards debriefing.549
  Values the importance of clear written communication.380
  Voices uncertainty, ‘I don't know’.18
 Requests others opinions27
Competency: task management (manages tasks efficiently and effectively)
  Describes contemporary approaches to human error (systems approach).92100
  Describes reason's taxonomy of errors (slips, lapses, mistakes, violations).5418
  Describes a method of minimizing error (avoid, trap, mitigate).5445
  Describes how high and low workload can contribute to error.85100
 Describes how checklists and standardization of procedures can minimize error.100100
  Describes the advantages and disadvantages of automation in regard to error.4618
  Describes the incident and adverse event reporting procedure.389
  Lists the types of incidents or adverse events that should be reported.239
  Manages changing priorities and if necessary, re-focus team members to accommodate the changed priorities. Prioritizes critical tasks.7791
  Maintains vigilance during routine tasks or periods of low workload.3818
  Monitors systems, environment and team members, collects and analyses information to identify potential or actual errors.6955
  Implements strategies and procedures to prevent errors or takes action in the time available to correct errors.6236
  Applies checklists and standard operating procedures to prevent clinical, procedural or communication errors; and identifies committed errors before patient safety is affected.5445
  Allocates sufficient resources and time to complete workload.4618
  Distributes and assigns work thoughtfully; re-allocates functions where required.6955
  Reports incidents and adverse events.5427
  Steps back/hands off.18
  Displays a respect for safe practices.8573
  Displays a positive attitude towards open reporting and a blame-free culture.7791
  Appreciates the ubiquitous nature of error and acknowledges the potential for error in self and others.92100
  Acknowledges that safety is everyone's responsibility.92100
  Applies accepted clinical knowledge.9
Competency: situational awareness (maintains situational awareness)
  Describes the concept of ‘situational awareness’, and how to identify when it has been lost.85100
  Describes the concept of a team ‘shared mental model’.7773
  Lists the steps in identifying work environmental or operational threats that could affect safety of the patient.4618
  Describes a support process to facilitate maintenance of team situational awareness.5427
  Describes a method of providing constructive feedback.3118
  Describes strategies that may be used to maintain individual situational awareness.100100
  Lists the elements of SITREPS and RECAPS.159
  Describes the concept of a ‘mental rehearsal’, and where it may be used to improve performance.5427
  Ensures that all team members have a clear picture of the objective.8591
  Ensures that all team members have role clarity and relevant information to achieve goals.7791
  Identifies work environmental or operational threats that could affect safety of the patient.3118
  Uses support process to facilitate maintenance of team situational awareness.80
  Provides constructive feedback to other team members for improving performance.389
  Asks questions to facilitate understanding of the situation.7773
  Uses strategies to maintain individual situational awareness.85100
  Maintains shared mental models via SITREPS and RECAPS.2318
  Employs mental rehearsals, where appropriate, to facilitate improved performance.150
  Appreciates the importance of maintaining situational awareness.7791
  Appreciates the importance of a team ‘shared mental model’.5445
  Displays a positive attitude towards asking and answering questions.9291
  Displays a positive attitude towards providing and receiving feedback.7782
  Appreciates the importance of SITREPS and RECAPS in maintaining a team ‘shared mental model’.159
  Appreciates the importance of mental rehearsals in improving performance.159
Competency: decision-making (makes appropriate decisions)
  Describes the process of naturalistic decision-making.6936
  Describes the process of analytical decision-making.6236
  Lists the steps in ‘OODA loop’ monitoring.230
  Defines risk in terms of consequence and likelihood.5418
  Describes reason's ‘three bucket method’ of real-time risk assessment.230
  Employs naturalistic decision-making techniques in time-critical situations.3818
  Employs analytical decision-making techniques when time is not a factor.239
  Assesses solutions and risks with other team members. Readily anticipates and shares the information needs of other team members.6955
  Decides on a course of action.7791
  Communicates plans of action and directs team members to clearly specified tasks.77100
  Provides ‘if, then… ’ direction to team members to facilitate future action.6945
  Monitors and assesses progress to ensure a safe outcome; or modifies action when a safe outcome is not assured.6218
  Adopts real-time risk management strategies.230
  Appreciates the importance of making a decision.6255
  Is willing to review decisions in light of new information.92100
  Displays a positive attitude to managing risk.6227
  Understands the concept of ‘Global Review’.9
Competency: leadership (displays effective team skills in position of leader or follower)
  Lists common characteristics of effective leadership.100100
  Lists common characteristics of effective followership.92100
  Lists and describes individual's attitudes that are hazardous to safety.6236
  Lists and describes group attitudes and behaviours that may adversely impact performance.85100
  Lists individual's physiological factors that may adversely impact performance.8591
  Understands [describes] the importance of hands-off role of leader.18
  Leader considers the condition (capability) of other team members to perform duties.7791
  Leader monitors and appraises team members’ performance.6255
  Leader assists other team members in demanding situations. Backs-up and fills in for other team members as required.389
  Leader motivates and encourages other team members.6236
  Self-assesses physiological ability to perform on the job.469
  Cross-monitors actions of other team members.85100
  Steps back/hands off leadership role.27
  Appropriate allocation of roles and responsibilities.36
  Is an active member of the team.6245
  Displays a positive attitude to the importance of giving and receiving support.6945
  Maintains appropriate interpersonal skills.6955
  Displays a positive attitude to the importance of individual and group attitudes in the work environment.6955


The Delphi technique contextualized by a refined literature review and competency listing was a useful method for determining the content of a 1-day CRM training course for health-care workers. It provided a rapid, low-cost alternative to a complex training needs analysis conducted by multiple interviews, focus groups or expert panels. Identification of competency components within each of the five competency domains such as communication, task management, situational awareness, decision-making and leadership provides confirmation of a broad correlation between the outcomes of this study and the content of the major CRM programs implemented in health-care today, where each of these competencies is normally included. By providing a list of ranked competency components, the study will also allow trainers the flexibility to tailor training to the circumstances of their team.

In addition to providing recommendations for content of a 1-day course, the competency component list may provide a greater degree of transparency of the underlying structure of health-care CRM programs in general. The individual competency components can often be inferred by reviewing program material, but they are normally not explicitly listed. The percentage of participants who selected each component may also provide a guide to prioritization for trainers who have less than a full day available for training, or who have identified a specific problem in their team that needs to be addressed. The stated competency component could be used to access public domain training material (e.g. DoD TeamSTEPPS) and provide a flexible and cost-effective path to developing customized training and effective subsequent evaluation.

In addition to fewer selections overall in Round 2, the final selections for each competency component appeared to be more polarized ‘for’ or ‘against’ in Round 2 than in Round 1. Possible explanations for these results include that participants considered the training holistically and pruned their selections to a realistic number of components for 1 day of training; participants were influenced by the 75% line drawn separating components for inclusion or deletion after the first round and responded to that rather than the individual components; or participants succumbed to group conformity whereby the first round effectively summarized the ‘expert correct answer’ to which everyone then agreed. A likely explanation is a combination of the three: most of the panel were conversant with the Delphi process, and, being aware that a decision had to be made after the second round, was helpful in trying to reach maximum consensus as a group.

Polarization of final responses will facilitate selection of competency components for inclusion in the training. Forty competency components were selected by more than 70% of respondents, whereas the remaining components were selected by fewer than 55%. The 40 competency components provided an indication of the most critical areas for inclusion in training for communication, task management, situational awareness, decision-making and leadership competencies for health-care personnel. To minimize the number of required Delphi rounds, it is important to give panel members as much information about the research question as possible; however, provision of too much guidance, such as the 75% cut-off criteria for selection at the end of the first round, may introduce bias against alternate views. Concerns regarding generalizability of the data from a small participant sample size are somewhat mitigated by the high level of agreement reached after only two rounds. This study showed that the recommended three to four rounds may not be necessary to obtain consensus when the presented baseline subject matter is evidence based and panellists are well acquainted with the Delphi method.

A limitation of this study is the large number of variables available for selection. Whilst it was imperative to include all of the major competency components that emerged from the literature review, this has potential to compromise the accuracy of this method. However, fears of questionnaire fatigue appear to be baseless. There were more decision-making and leadership components selected in Round 1, where they were at the end of the survey, than in Round 2 where they were placed at the beginning.


Although health-care CRM training has been widely implemented over the last decade, there is not a large body of evidence linking the training to improved outcomes in patient care. It is requested that trainers who decide to utilize this material consider the importance of training evaluation and its contribution to the evidence base supporting the efficacy of CRM in health care.


This research was partially supported under a University of New South Wales scholarship and by National Health and Medical Research Council (NHMRC) Programme grant 568612.


We would like to acknowledge the assistance of the members of the Delphi panel, who provided their unrecompensed expertise in the form of timely responses to the questionnaires and helpful suggestions concerning this study and the research design.


View Abstract