Brenda M. Wenzel and Mary T. Dirnberger
Mei Technology Corporation
San Antonio, TX

The Experimental Advanced Instructional Design Advisor (XAIDA) is an automated instructional development tool being developed by the Air Force Armstrong Laboratory. A cadre of aircraft maintenance instructors and course designers from the 362d Training Squadron, Sheppard Air Force Base were the first to receive formal training with XAIDA in an operational setting. The XAIDA training program was designed to (a) teach participants how to efficiently develop multimedia computer-based maintenance training using XAIDA, (b) evaluate the effectiveness of training, and (c) gather usability data from the participants. We took a multidimensional approach to evaluate the effectiveness of the XAIDA training program (Kraiger, Ford, & Salas, 1993). Learning outcomes were assessed as (a) changes in trainees’ abilities using XAIDA, (b) changes in computer skills, (c) changes in attitudes toward XAIDA, and (d) as the emergence of procedural knowledge structures associated with XAIDA. This paper describes the training design, various methods used to evaluate the training, and results of the evaluation.

The introduction of instructional technology in a training organization does not occur as a single event. Rather it is an iterative process that involves the entire organization (see Figure 1). Important to successfully introducing instructional technology in a training organization is overcoming obstacles that the organization may face when moving from the old way to the new way of doing work.

According to Geoghegan (1994), organizations adopting instructional technology are often faced with obstacles such as (a) failure to accommodate the differences between members that adopt technology early versus later, (b) formation of a clique of technology users, (c) alienation of traditionalists, and (d) failure to provide the users with compelling reasons to adopt the technology.

Figure 1 presents the approach we have taken in introducing the Experimental Advanced Instructional Design Advisor (XAIDA), an automated instructional development tool, to an Air Force technical training organization (Hickey, Spector, & Muraida, 1991; Spector, 1990; Spector, Polson, & Muraida, 1993). We designed the process to not only facilitate introduction of XAIDA but to also serve as a research plan. Data are gathered and analyzed at each stage of the process, as individuals and the organization respond to the new instructional technology. The results feed back to the organization to help meet the information and training needs of individuals who adopt the technology early as well as those who adopt it later. In addition, the results contribute to the software design and development process to continually improve XAIDA’s functionality and graphical user interfaces.

This paper describes results of a training program provided to staff of the 362d Training Squadron, Sheppard Air Force Base, on the use of the XAIDA to develop computer-based maintenance training. XAIDA allows subject matter experts to develop interactive maintenance training courseware by simply providing a description of the system to be maintained. XAIDA training was conducted after the organization was familiar with XAIDA through briefings, demonstrations, and participation in research activities. Prior to training, an implementation plan was devised for XAIDA that worked within current resource constraints.

XAIDA SOFTWARE

XAIDA consists of a program called Develop for entering and editing system descriptions and a program called Deliver for presenting instruction on the system. Develop provides a SME with facilities for describing the part structure of a system, describing physical characteristics and behavior of a system, associating interactive multimedia with the lesson content, and configuring practice exercises. Deliver provides students with an overview of the subject matter, a detailed presentation of lesson material, selective review, and automatically generated practice exercises.

In its full version XAIDA will address four aspects maintenance training: (1) physical characteristics of a system or device, (2) theory of operation, (3) operating and maintenance procedures, and (4) troubleshooting. The XAIDA prototype currently addresses physical characteristics and theory of operation (Hsieh, Halff, & Redfield, 1996). Demonstration courseware developed with the current prototype exists on a basic aircraft hydraulic system, C-141 crew oxygen system, and F-16 egress system.

XAIDA runs under Microsoft Windows 3.1 operating system. The minimum requirement to efficiently run XAIDA is a 486 66MHz computer.

XAIDA offers the instructors developing multimedia CBT unparalleled ease of use and development efficiency. Research on an earlier version of XAIDA has shown that an hour’s worth of interactive instruction could be developed in about 30 hours (Walsh, 1994). The current XAIDA prototype has been specifically designed to cut this ratio even further. To the instructors delivering CBT, XAIDA offers the benefits of multimedia instruction. Instructors can use XAIDA to enhance traditional instruction and to provide dynamic views of systems that are more compelling and comprehensible than that offered by typical conventional training devices. XAIDA offers students the benefits of interactive, adaptive instruction (Wenzel, Richardson, Halff, & Gibson, 1996). XAIDA automatically generates practice environments that track each student’s skills and knowledge and provides practice exercises that meet the student’s ongoing learning needs.

The training evaluated here was designed to enable students to effectively use XAIDA to develop multimedia computer-based training (CBT) that teaches physical characteristics of a system or device.

The training design incorporated practical findings from the training literature (Cannon-Bowers, Tannenbaum, Salas, & Converse, 1991; Carroll, 1984; Kraiger, Ford, & Salas, 1993; Palmiter & Elkerton, 1993). Training was conducted in an open learning setting where interaction was encouraged. A minimalist training approach was employed (Carroll, 1984). Accordingly, we (a) left the amount of reading from manuals to the users discretion, (b) coordinated demonstrations, reading, and hands-on participation (Palmiter & Elkerton, 1993), (c) were prepared to deal with user errors, (d) focused on real tasks and lesson development activities, and (e) kept motivation high by allowing the trainees to choose the topics for their lessons.

1. Assessment of skill levels and attitudes
2. Introduction to XAIDA--What XAIDA is and what XAIDA does
3. Hands-on exposure to XAIDA Deliver using an XAIDA lesson that teaches student-users how to navigate the Deliver interface while actually using Deliver
4. Continue hands-on exposure to Deliver using the hydraulic system lesson
5. Demonstration and discussion of how to use the interfaces in XAIDA Develop
6. Explanation of the structure of an XAIDA lesson and how the user’s expert knowledge fits into that structure
7. Hands-on training recreating the hydraulic system lesson using the tutorial
8. Distribution of remaining paper-based training materials to take home
9. Assessment of skills and attitudes

1. Review and discussion of the steps used to create an XAIDA lesson in Develop
2. Hands-on training developing a lesson on a subject within user’s area of expertise
3. Demonstration and discussion of how to modify the practice exercises
4. Continue to develop XAIDA lessons using new knowledge
5. Presentation and evaluation of trainees’ lessons
6. Assessment of skills, attitudes, and abilities
7. Usability questionnaire completed

Twenty-five Air Force military and civilian personnel from the 362d Training Squadron at Sheppard Air Force Base received XAIDA training in three successive groups. Eighty percent of the trainees were maintenance instructors. Data from two participants in Group 3 were discarded because one participant only observed hands-on training and the other lacked a conceptual understanding of computers.

Three months before training, twenty-three of the then prospective trainees were surveyed to determine the level of computer skills in selected areas pertinent to the use of XAIDA. The percentage of trainees that self-reported having none or poor computer skills are listed in Table 1. In addition, the results of the survey showed that, on average, trainees were very familiar with the Windows environment and Microsoft Word, familiar with Paint Brush and PowerPoint, and unfamiliar with Authorware, a CBT authoring system. We were prepared to assist those trainees who lacked basic computer skills such as navigating the Windows environment, managing directories and files, and creating graphics.

Materials and Equipment

Instructions describing the text, graphics, and other media needed for lesson development were distributed to students before training, and students were encouraged to assemble these materials on a topic that interested them. In this way, we were able to provide practice on XAIDA lesson development in areas relevant to the trainees.

An example lesson on a hydraulic system was created from Air Force paper-based materials. Figures from the paper-based training material were scanned, cleaned up, colored, and saved as bitmap files for use in the example lesson. A step-by-step tutorial was created on the hydraulic system lesson. On the first day of training, trainees followed the tutorial using the multimedia files to recreate the example lesson. The purpose of using the hydraulic system material was to show trainees how they could readily adapt paper-based training materials for use in XAIDA lessons.

Other training materials included drafts of the user’s manuals for Develop and Deliver, and a generic tutorial on using XAIDA Develop.

Three groups of 8-9 trainees attended XAIDA training. Training was held in a computer lab with seven 486 25MHz PCs and two 486 33MHz laptop computers, for the first two groups. Training for the third group was held in a computer lab with nine 486 66MHz PCs. A 486 100MHz laptop computer connected to an In-focus projector was used as the demonstration platform. Training consisted of two days of six-hour training periods. All six days of training were conducted by two staff members from Mei Technology Corporation, Training Technology Division.

The question we asked was whether trainees acquired the ability to develop an XAIDA physical characteristics lesson. Multiple measures of learning outcomes were used to evaluate whether the training objective was achieved (Kraiger, Ford, & Salas ,1993). Skill-based, attitude, and cognitive measures were collected before, during, and after training. Each of these training evaluation measures is discussed next along with the results.

The most direct evidence that the training objective was achieved is the 22 different XAIDA lessons successfully developed during the second day of training. One lesson was lost due to a database error.

Paper-and-pencil measures of self-perceived computer skills, including newly acquired skills using XAIDA,were taken before and after the two-day training period. The results of these self-report data are presented in Figure 2.

Figure 2. Mean Ratings of General Computer Skills Before and After Training
[Note: N = 23. Ratings were made on a four-point scale with labels of "none" (1), "poor" (2), "good" (3), and "expert" (4).] * Signifies a significant difference.

Figure 3. Mean Ratings (N = 23) of XAIDA Develop Proficiency Before, During, and After Training
(Note: Both changes are statistically significant.)

The trainees rated their skill level (i.e., none, poor, good, or expert) using XAIDA Develop and Deliver significantly higher after training than before training. The amount of incidental learning that occurred as a function of XAIDA training is apparent in the significant increases in the self-reported levels of skill involved with general computer use--managing files and directories, typing, and creating graphics. These general computer skills are the basic skills needed to develop multimedia CBT with XAIDA.

Figure 3 presents a more detailed examination of trainees’ perceived level of expertise using XAIDA Develop. Trainees were asked to rate themselves on a 10 point scale where "1" represented a novice user and "10" represented an expert user of XAIDA Develop. Significant increases in trainees’ perceived level of expertise were found at the end of both the first and second training days. On average the trainees rated themselves 3.6 points higher on the 10-point scale after training than before training.

Acceptance and the willingness of personnel to adopt new technology is important to an organization planning to successfully integrate new technology (Geoghegan, 1994). Therefore, an exploration was made of trainee attitudes toward XAIDA. A concerted effort was made to expose all trainees to XAIDA prior to training. Several of the trainees were more familiar with XAIDA than others, due to prior involvement in the XAIDA research and development project. However, for a few trainees their first exposure to XAIDA was the first day of training, due to last minute changes in trainees’ schedules.

Trainees responded to an open-ended item asking for their impressions of XAIDA before training, at the end of training on Day 1, and after training, at the end of Day 2. Trainee responses were coded into seven categories. The coding categories have been found to reflect users concerns when adopting educational innovations (Hall, 1979; Hall, George, & Rutherford, 1986; Bailey & Palsha, 1992). The seven categories and examples of coding criteria follow:

1. Awareness-- have heard of it, interested in the area,
2. Information-- what are the requirements, what are the possible applications,
3. Personal-- time and energy required, how my role will change,
4. Management-- coordination of tasks and people, conflict between interests and responsibilities,
5. Consequences-- how it affects training,
6. Integration --coordinating efforts with others, help others with the innovation, and
7. Refocusing--revise the approach, modify based on use.

Responses were further coded as positive, negative, or neutral statements. The power version of Non-numerical Unstructured Data Indexing Searching and Theory-building, a computer aid for qualitative analysis, was used to code and analyze the data. Results are presented in Figure 4 and Figure 5.

As is indicated in Figure 5, the majority of trainees expressed a positive impression of XAIDA regardless of time in training. Approximately twice as many statements were made concerning their impressions of XAIDA at the end of the first day of training compared to either before or after training (see Figure 4). Before XAIDA training began 64% of the statements made expressed an interest in using XAIDA. Examples of the trainee positive impressions of XAIDA before training from the Information category are: "Ready to learn more about its ability to add to training in the classroom." and "Sounds good! I’m looking forward to learning about and using it." An example of a positive impression before training from the Integration category is: "It will probably be used a lot in the future because the field is asking for more CBT."

The majority of impressions expressed by the trainees at the end of the first day of training remained positive (72%). Trainee attitudes toward XAIDA shifted in the direction of Personal and Integration concerns, after the first day of hands-on training. A quarter of the trainees’ impressions expressed positive attitudes about the consequences of integrating XAIDA, for example-- "I have already come up with ideas and ways to use XAIDA in our classroom."; "Minimizing the computer skills is a great plus, allowing the developer clearer access to the development."; "Very applicable to our courses."; and "I think this type of computer software will be very beneficial in the classroom."

Figure 4. Numbers of Comments in Each Category at Different Times During Training.
(Note: "During" comments were collected at the end of the first day of training.)

Six of the fifty statements made at the end of Day 1 were categorized as negative. One negative statement concerned personal abilities-- "It will take a lot of practice for me to exploit its full potential." and another concerned management-- "I feel this will be another good tool, providing we get the hardware to run it." Four suggestions on how to modify the program were coded in the negative Refocusing category, an example-- "Screen appears busy and can also be confusing to students and new course writers but would improve if taken to full screen."

A shift in trainee attitudes toward XAIDA occurred at the end of training, on Day 2, in the direction of increased Personal and Management concerns (see Figure 4). Trainees spent Day 2 of training developing their own lessons on a topic within their areas of expertise. While developing their own lessons, trainees were faced with the practical problem of describing their knowledge of a system or device in a manner that was acceptable to XAIDA. Problems facing novice users were evidenced in their Personal concerns-- "Once an individual rearranges the thought processes, the lesson development seems to flow smoothly." and "I have figured out how to incorporate a lesson plan into XAIDA." Even in overcoming the challenge of rethinking the material that they teach, the majority of trainee impressions remained positive (60.6%)-- "Still like it. Ready to use it in my course."; "Good program, has a lot of potential will cut down on the amount of repetitive work."; "An excellent medium for the development and presentation of entry and mid-level courses."; and "I believe this system can be used to develop very useful courseware material."

The XAIDA training program successfully taught twenty-three Air Force personnel from the 362d Training Squadron to effectively use XAIDA to develop multimedia CBT. Participation in the two-day training session had a positive impact on trainees’ general computer skills. This type of incidental learning is likely to be found in educational settings where CBT is developed and delivered. Our previous research has shown that the mere exposure of students to CBT increases their willingness to receive instruction from computers. Eighty-six percent of Air Force aircraft maintenance apprentices (n=16) who received two hours of instruction using a stand-alone XAIDA lesson indicted they would choose to receive training from computers. Whereas, only 35% of apprentices (n=14) who received two hours of traditional classroom instruction indicated they would choose to receive training from computers.

Attitudes of an organization’s personnel is one determinant of the receptivity of the organization to new technology. Therefore, the attitude of the trainees toward XAIDA is as important as the new skills they acquired. Geoghegan (1994) notes that early adopters of new technology tend to have a positive attitude toward the new technology, express a willingness to participate and to experiment with the new technology. The trainees were found to possess characteristics of early adopters. The majority of trainees gave XAIDA a positive review. The trainees expressed their ideas on how XAIDA will augment classroom instruction. They also stated their beliefs as to the positive effects that multimedia CBT will have on training in general. In an earlier survey, 90% of 29 instructors scheduled for XAIDA training agreed that they would be willing to use CBT in their classrooms. Moreover, 93% of these same instructors agreed that they would volunteer if instructors were needed to develop CBT lessons. Positive attitudes such as the ones expressed by these trainees will facilitate the successful implementation of XAIDA into the aircraft maintenance training environment.

The 362nd Training Squadron plays an important role in shaping the future of multimedia CBT in aircraft maintenance training. The Air Force personnel who participated in the XAIDA training were also part of a formative evaluation of XAIDA. Valuable usability data collected from the trainees has not been addressed in this paper. However, several of the trainees’ suggestions on how to improve XAIDA have been implemented. For example, at the trainees’ suggestions, the latest version of XAIDA has an expanded capability for utilizing any file type in a lesson. Instructors developing XAIDA lessons in the future will not be restricted to using bmp, avi, or wav files as lesson resources.

The XAIDA Design and Development Team continues to apply the trainees’ input toward improving XAIDA’s graphical user interfaces and XAIDA training programs for future users.

Bailey, D. B.,& Palsha, S. A. (1992). Qualities of stages of concern questionnaire and implications for educational innovations. The Journal of Educational Research, 85, 226-232.

Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Converse, S. A. (1991). Toward an integration of training theory and technique. Human Factors, 33, 281-292.

Carroll, J. M. (1984). Minimalist Training. Datamation, November, 125-136.

Geoghegan, W. H. (1994). Whatever happened to instructional technology? Paper presented at the 22nd Annual Conference of the International Business Schools Computing Association. Baltimore, Maryland.

Hall, G. E. (1979). The concerns-based approach to facilitating change. Educational Horizons, 4, 202-208.

Hall, G. E., George, A. A., & Rutherford, W. L. (1986). Measuring stages of concern about the innovation: A manual for the user of SoC Questionnaire. Austin, TX: Southwest Educational Development Laboratory.

Hickey, A. E., Spector, J. M., & Muraida, D. J. (1991). Design specifications for the advanced instructional design advisor (AIDA) (Vols 1 & 2) (AL-TR-1991-0085). Brooks AFB, TX: Technical Report for the Air Force Armstrong Laboratory, Human Resources Directorate.

Hsieh, P. Y, Halff, H. M., & Redfield, C. L. (1996). Knowledge-based tools for developing computer-based maintenance training. (Manuscript submitted for publication).

Kraiger, J., Ford, K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311-328.

Kraiger, Salas, & Cannon-Bowers, (1995). Measuring knowledge organization as a method of assessing learning during training. Human Factors, 37, 804-816.

Palmiter, S., & Elkerton, J. (1993). Animated Demonstrations for learning procedural computer-based tasks. Human-Computer Interaction, 8, 193-216.

Spector, J. M. (1990). Designing and developing an advanced instructional design advisor. (AFHRL-TP-90-52). Brooks AFB: TX: Technical Paper for the Training Systems Division of the Air Force Human Resources Laboratory.

Spector, J. M., Polson, M. C., & Muraida, D. J., Eds. (1993). Automating Instructional Design: Concepts and Issues. Englewood Cliffs, NJ: Educational Technology Publications.

Walsh, W. J. (1994). Automated authoring: Some preliminary results. Proceeding from the 16th Interservice/Industry Training System and Education Conference. Orlando, FL.

Wenzel, B. M., Richardson, K. L., Halff, H. M., & Gibson, E. E. (1996). Assessing the instructional effectiveness of the Experimental Advanced Instructional Design Advisor (XAIDA). Manuscript in preparation.