The Air Force Armstrong Laboratory Human Resources Directorate is a recognized leader in the development and evaluation of advanced technologies for education and training. As such, the Laboratory was requested to help support the development and evaluation of training technologies and content to support the U.S. Air Force Surgeon General’s Mirror Force Training Initiative. The initial effort was aimed at providing improved training to Reserve and Guard Medical Technicians.

The goal of the effort in support of Mirror Force was to develop, implement, and validate a process-oriented approach to requirements identification, instructional design and delivery, and the evaluation of training. Part of this effort was to demonstrate a variety of training delivery options that will provide faster, better, and cheaper training which can be delivered at anytime and anywhere. A key aspect of achieving the goal was demonstrating that there is a return on investment in advanced training technology in terms of manpower and training resource savings to the Air Force operational community while, at the same time, sustaining mission performance effectiveness. Figure 1 illustrates the flow of information from the different parts of this process.

FIGURE 1. Systematic Process for Training Development and Evaluation

The Process

As can be seen from the figure, the process is composed of three major steps. The main focus of this paper is on the first step in the process - the requirements identification, or needs assessment - step. This step includes identifying content areas for training development and delivery. A number of sources were used to gather potential content areas: Existing Stan/Eval documents and Air Force Audit Agency findings, outcomes from focus group discussions with SMEs, existing occupational survey information, career field education and training planning documents, Specialty Training Standards (STSs), Career Development Courses (CDCs), and existing Plans of Instruction (POIs). Once the content areas were identified, customer feedback, obtained with field interviews and surveys, was used to identify the most critical content areas for Medical Technician field performance in both peacetime and wartime terms.

Identifying key constructs for study. Past research has examined the important constructs to consider when identifying and prioritizing tasks as content for training (e.g., Mitchell, Yadrick, & Bennett, 1993). Most recently, Harvey (1991) discussed the importance and the difficulties associated with collecting multiple ratings of tasks from individual respondents. Central to his theme was the concern over the amount of information that was required of each respondent. Specifically, as the number of rating scales increases, the likelihood of capturing redundant information increases significantly as does the likelihood that the respondents will not provide accurate information across the scales of interest (Harvey, 1991; Wilson, 1989). In response to these concerns, this study attempted to reduce the data collection burden in two ways: first by managing the rating process using computer-administered surveys; and second, by developing multiple versions of each survey that included both common and unique constructs to be addressed.

Further research has specifically examined the issue of task-level constructs and their relationship to training priorities. For example, Stacy, Thompson, & Thomson (1977) discussed the importance of percent members performing and task learning difficulty as key constructs. Also, Ruck, Thompson, & Stacy (1987) explored the relationship of task training emphasis ratings for determining training priorities. Finally, Thompson & Ruck, (1978) demonstrated that consequences of poor or inadequate performance were important predictors of safety training priorities.

In our present effort, over 360 content areas related to Medical Technician activities were identified and built into computer-assisted surveys which were sent to the field. Two thousand surveys were sent to Reserve and Guard Medical Technicians respectively, and an additional 1,000 were sent to Active Duty personnel so that a more complete picture of field requirements and recommendations could be obtained. By including all three Service components, a direct comparison of the common and different needs of each was possible. Over 1600 surveys were returned with useable information from the field resulting in an initial response rate of 53%. This is a reasonable response rate for this type of survey.

To help develop training priorities, information about the number of personnel who actually use the content as part of their current job, recommendations from field personnel for the timing of refresher training, and perceived consequences of poor performance of the content were used to help prioritize content for training development. Table 1 summarizes the constructs of interest in this study.

TABLE 1. Scales and Constructs

Table 2 summarizes the STS-level responses for the ten tasks that had the highest cumulative percent members performing them. The table also includes descriptive statistics for each of the tasks across all the scales and versions of the surveys for both 3- and 5-skill level technicians and for 7-level supervisors. Table 3 summarizes similar data for a selection of other medical technician tasks.

TABLE 2. "Top Ten" Percent Members Performing Tasks

Note: % R = percent performing task as part of regular duty; %D = percent performing task during deployments;
% B = percent performing task as part of a backfill for deployed active duty personnel.

TABLE 3. Additional Medical Tasks Sorted by Percent Members Performing

Note: % R = percent performing task as part of regular duty; %D = percent performing task during deployments;
% B = percent performing task as part of a backfill for deployed active duty personnel.

The results of the survey were very successful. By examining cumulative percent members performing the tasks, recommended interval for refresher training, and consequences of poor performance ratings, several critical training areas were identified. These areas included checking blood pressure, using respiratory equipment such as a pulse oximeter, diagnosing respiratory symptoms, and using respiration equipment would benefit from advanced training technology. The following figures depict these results which have been identified in the general categories below. The following figures show selected general categories of content and some of the resulting information obtained for Reserve, Guard, and Active Duty field data collection.

Interrater agreement amongst respondents was found to be quite high for this survey. Medical Technician SMEs assisted in the development of a linkage table between these data and the data obtaining for the most recent occupational survey. This linkage will ensure that data across multiple levels of analysis (e.g., POI, STS, and occupational task-level) can be examined for a variety of manpower, personnel, and training. Analyses of the combined data are underway to examine the impact of differing levels of analysis on the developed priorities for training. In addition, the prioritized areas discussed previously have been used to develop instructional courseware for Air Force personnel.

Use of the Data for Instructional Design and Delivery

The critical content areas identified in the survey were reduced to a series of learning objectives which, in turn, drove the development of interactive courseware (ICW). It should be noted that there is never a single solution to the development and delivery of training. It is always critical to match the objectives and the content of the training with instructional design and delivery technologies that are most appropriate. For our purposes, respiration training was developed and delivered using a simulation-based system known as RIDES. The course has been administered to a number of personnel of varying skill and experience levels.

Evaluation

Preliminary training evaluation results have been encouraging. Comments and data from the field administrations indicate that the content is relevant and of high priority for maintaining unit readiness and effectiveness. In addition, there is considerable interest from all of the participating field units in further training development, delivery, and evaluation efforts. Armstrong Laboratory is presently planning activities for the next steps in our support of the Mirror Force initiative.

The goal of this effort was to determine the important constructs for prioritizing content for training. The study demonstrated the importance of gathering needs assessment information using streamlined and automated methods. In addition, this study provided the prioritized content to courseware developers for use in providing ICW to field personnel. In addition, the use of computer-administered survey methods provided a convenient and manageable method for systematically gathering data for multiple constructs while reducing the data collection burden on individual respondents. To date, the prioritized content has been to develop demonstration ICW for the field and to identify changes to the career field education and training plan and to the CDC training required of all Medical Technicians. Finally, the data have been used to validate tasks and content for recurring training to support readiness and contingency training requirements.

Harvey, R.J. (1991). Job analysis. In M. Dunnette & L. Hough (Eds.). Handbook of industrial and organizational psychology, 2^nd Ed., Vol. 2, pp.71-163. Palo Alto: Consulting Psychologists Press, Inc.

Mitchell, J.L., Yadrick, R.M., & Bennett, W., Jr. (1993). Estimating training requirements from job and training pattern simulations. Military Psychology, 5, 1-20.

Ruck, H.W., Thompson, N.A., & Stacy, W.J. (1987). Task training emphasis for determining training priorities (AFHRL-TP-86-65). Brooks AFB, TX: Manpower and Personnel Division, Air Force Human Resources Laboratory.

Stacy, W.J., Thompson, N.A., & Thomson, D.C. (1977). Occupational task factors for instructional systems development. Proceedings of the 19^th Annual Conference of the Military Testing Association. San Antonio, TX: Air Force Human Resources Laboratory and the USAF Occupational Measurement Center.

Thompson, N.A. & Ruck, H.W. (1978). Methods for determining safety training priorities. Proceedings of the 20th Annual Conference of the Military Testing Association, pp. 242-257.
 

Next Page 

Back to IJOA Home Page