Jerry W. Hedge
Walter C. Borman
Mary Ann Hanson
Personnel Decisions Research Institutes, Inc.
Minneapolis, MN

Crew resource management (CRM) researchers and practitioners typically take a training approach to improving crew performance. Since the late 1970s, CRM training programs have become an increasingly important part of the commercial aviation industry. Most of the major commercial airlines have either developed their own CRM training programs or adapted existing training programs to suit their own needs (see Prince, Chidester, Bowers, & Cannon-Bowers, 1992). While CRM has existed in "pockets" of the military for years, because of the multitude of commands and locations, a broad-based application has been slower to evolve. The Air Force has recently published an Air Force Instruction (AFI 36-2243) that, for the first time, establishes broad requirements for developing and managing CRM training programs, and requires CRM training for all Air Force aircrew members (Secretary of the Air Force, 1994).

Research has shown that CRM training does change attitudes linked to airline incidents and accidents, including attitudes toward communication and coordination, attitudes toward command responsibility, and recognition of stress effects (e.g., Irwin, 1991). In addition, Helmreich, Foushee, Benson, and Russini (1986) demonstrated a link between CRM attitudes and line flying performance. However, Prince et al. (1992) cite evidence which suggests that the personalities of trainees set limits on training effectiveness and attitude change.

Thus, recent research in the area of CRM points to the importance of interpersonal skills and certain personality traits for effective pilot performance. Training can be viewed as a way to promote awareness of group dynamics, bring about attitude change, and improve interpersonal skills, but it does not change the underlying traits that have been shown to be related to CRM. Selection based on personality traits and/or relevant interpersonal skills could greatly enhance CRM performance, above and beyond CRM training alone.

In the current project, we pursued one particularly promising approach to measuring individual differences in the interpersonal and personality areas, namely, the situational judgment test (SJT). The SJT methodology presents respondents with a series of job-relevant situations and asks them to indicate which of several alternative actions would be most effective and which would be least effective in each situation. This methodology is based on the premise that there are important and often subtle differences between the behavior of effective and ineffective persons as they respond to problems or dilemmas confronted in the course of carrying out their job responsibilities and that such differences are reflected in their responses to similar situations presented in written form.

While there is not a great deal of research on SJTs available, researchers at PDRI and elsewhere have shown that SJTs can be valid predictors of job performance (e.g., Forehand & Guetzkow, 1961; Hanson & Borman, 1990; Motowidlo, Dunnette, & Carter, 1990; Tenopyr, 1969). A relatively new development in SJT research is the use of video to avoid the fidelity problems inherent in presenting situations in a paper-and-pencil format. For example, Dalessio (1994) developed a video-based SJT for insurance agents and showed that scores on this test predicted turnover. Drasgow, Olson-Buchanan, and Moberg (1993) have further reduced the fidelity problem in SJT administration through the use of interactive video administration. As video administration becomes more practical, the usefulness of the SJT technique is likely to be greatly enhanced.

Thus, our objective was to develop and validate an SJT, the Situational Test of Aircrew Response Styles (STARS), as a way to measure the interpersonal problem solving, aircrew management, and decision-making abilities highly relevant to successful performance in the CRM aspects of pilot jobs.

Development of the situational judgment test involved four primary steps: 1) situation generation, 2) response option generation, 3) item review, and 4) response option scaling. Across a ten month period, development workshops were conducted at 22 Air Force sites within the continental United States. For a complete description of the development process see Hedge, Hanson, Borman, Bruskiewicz, and Logan (1996).

Experienced aircrews from C-130 transport units (i.e., basic crews are composed of an aircraft commander, co-pilot, navigator, flight engineer, and loadmaster) in the Air National Guard (ANG) and Air Force Reserve (AFR) served as subject matter experts during the development phase of this research project. In addition, junior Air Force officers and Air Force Academy cadets, with little or no flying experience (novices), participated in the "response option generation" and "response option scaling" phases of test development. In all, 398 individuals (240 experts; 158 novices) participated in the four different types of STARS development workshops.

The outcome of all of the development work was a set of difficult situations targeted toward performance-relevant interpersonal skills, and a representative sampling of the kinds of actions pilots might take in these situations. Each situation included a set of five response options ranging from very effective to relatively ineffective, selected on the basis of comparing expert and novice response data. The effectiveness ratings from the expert sample were also used to develop a scoring key.

The primary objective of this project was to develop a CRM selection test that can help the Air Force, especially AFR and ANG units, identify pilots with superior crew coordination skills. However, from the earliest days of our work in the field, the training applications of this research were apparent. At virtually every unit that we visited to conduct workshops, we were approached by training officers, flight safety officers, or squadron commanders, who were excited about the possibility of using these items as training stimulus materials, whereby junior and senior crewmembers could discuss how to apply good CRM skills to difficult situations. So, one application of the STARS technology, as envisioned by these experienced ANG/AFR managers and trainers, would be as a relatively inexpensive (i.e., low technology) means of CRM training delivery. In other words, a situation or group of situations, are handed out for group discussion.

Naturally, higher technology versions of the same approach can be used as well, incorporating computer-based or videotape-based STARS presentation, thus offering intriguing possibilities at either the individual or group level of application. Therefore, as a final step in our CRM selection project, we developed a videotape depicting STARS situations, whereby the viewer was asked to respond in terms of their choices for the most and least effective response options. To accomplish this, we selected two STARS situations and developed scripts for actors that would depict, in short scenarios, each of the situations.

We elicited the cooperation of a company that conducts CRM training in C-130 simulators. They helped refine the scripts, provided trainers who served as actors, and actually videotaped each of the two scenarios and the associated responses. After completion of the videotaping, a production professional helped us edit the two scenarios, including inserting panels with the five response options presented at the conclusion of each scenario, before the videotaped responses. Viewers watch the evolving scenarios and are then asked to identify the most and least effective response options, which in turn, are acted out on the videotape.

Thus, one of the most compelling applications of the video-based scenarios may be in CRM training. Currently, the highest fidelity CRM training involves placing aircrews in an aircraft simulator and presenting scenarios that require crew actions relevant to CRM. However, simulator time is extremely expensive and often unavailable for CRM training. Video-based training, in which videotaped scenarios are presented and discussions about what actions should be taken are initiated, provides an intermediate level of fidelity at much less cost and increased convenience compared to simulator training. Additionally, the fidelity of the video-based approach is considerably greater than that offered by lecture and other typical classroom approaches.

New FAA rules concerning the amount and type of CRM training commuter airlines are required to deliver may make the regional airline operators primary candidates for such training approaches. Although these new FAA rules stress upgraded pilot training and increased use of flight simulators for CRM training, the use of simulators poses difficulties both in terms of cost and scheduling. STARS scenarios, like the ones we videotaped, may offer some intermediate form of "simulator-based" training.

If the videotaped STARS continues to be well received, future work might include development of additional videotape versions of both military and commercial STARS scenarios. At this point, additional video-based CRM training research and development appears to be the most likely immediate component of our SBIR work that we will pursue beyond Phase II. With an aviation environment of ever-increasing emphasis on crew coordination and communication, including FAA mandates, our simulation-based technology stands an excellent chance of continued application. STARS scenarios, like the ones we videotaped, may offer some intermediate form of "simulator-based" training.

Dalessio, A.T. (1994). Predicting insurance agent turnover using a video-based situational judgment test. Journal of Business and Psychology, 9,23-32.

Drasgow, F., Olson-Buchanan, J.B., & Moberg, P.J. (1993). Development of interactive video assessments. Unpublished manuscript.

Forehand G. A., & Guetzkow, H. (1961). The administrative judgment test as related to descriptions of executive judgment behaviors. Journal of Applied Psychology, 45, 257-261.

Hanson, M. A., & Borman, W. C. (1990, November). A situational judgment test of supervisory knowledge in the U.S. Army. Paper presented at the 32nd Annual Conference of the Military Testing Association, Orange Beach, AL.

Hedge, J.W., Hanson, M.A., Borman, W.C., Bruskiewicz, K.T., & Logan, K.K. (1996, May). Predicting the crew resource management skills of Air Force pilots (Institute Report No. 283). Tampa, FL: Personnel Decisions Research Institutes, Inc.

Helmreich, R. L., Foushee, H. C., Benson, R., & Russini, W. (1986). Cockpit management attitudes: Exploring the attitude-performance linkage. Aviation, Space, and Environmental Medicine, 57, 1198-1200.

Irwin, C.M. (1991). The impact of initial and recurrent cockpit resource management training on attitudes. In R.S. Jensen (Ed.), Proceedings of the Sixth International Symposium on Aviation Psychology (pp. 344-349). Columbus, OH: Ohio State University.

Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative selection procedure: The low-fidelity simulation. Journal of Applied Psychology, 75, 640-647.

Prince, C., Chidester, T. R., Bowers, C., & Cannon-Bowers, J. (1992). Aircrew coordination: Achieving teamwork in the cockpit. In R. Swezey & E. Salas (Eds.), Teams: Their training and performance. Newark, NJ: Ablex.

Secretary of the Air Force. (1994, April). Cockpit/crew resource management (CRM) program. Air Force Instruction (36-2243).

Tenopyr, M. L. (1969). The comparative validity of selected leadership scales relative to success in production management. Personnel Psychology, 22, 77-85.