Back to Table of ContentsA Quantitative Evaluation of an Instructional Design Support System: Assessing the Structural Knowledge and Resulting Curricula of Expert and Novice Instructional Designers
Theresa L. McNelly, Winfred Arthur, Jr.
Department of Psychology, Texas A&M University
Winston Bennett, Jr., & Dennis J. Gettman
Human Resources Directorate, Armstrong Laboratory Brooks AFB
The Guide to Understanding Instructional Design Expertise (GUIDE) is an automated instructional design (ID) tool developed at Armstrong Laboratory's Technical Training Research Division. Based on Gagn
é’s nine events of instruction, GUIDE was developed to aid content domain experts in the planning, development, and implementation of quality instruction. A systematic, quantitative evaluation is being conducted to determine whether GUIDE can be used to acquire the skills that would otherwise be obtained by means of long-term, on-the-job, instructional design experience. Reaction, learning, and behavioral measures will be collected. Learning will be assessed by administering traditional paper-and-pencil knowledge tests and by investigating the structural knowledge of the participants. A comparison of the knowledge structure of novice instructional designers and expert instructional designers will be conducted before and after the implementation of GUIDE. Several knowledge representation techniques (namely, multidimensional scaling and Pathfinder) will be used to assess the underlying mental models of novice and expert instructional designers to determine (1) the similarity of novice and expert mental models before the implementation of GUIDE, and (2) whether the implementation of GUIDE results in an increase in similarity between the expert and novice mental models of instructional design. Behavior will be assessed by obtaining courseware samples from the participants, which will be rated as to the extent that the courseware incorporates the nine events of instruction proposed by Gagné.Background
Subject matter experts (SMEs) are often used to design curricula and develop technical training courses for their area of specialized knowledge. The typical SME is therefore knowledgeable about a particular content domain, but not necessarily versed in formal instructional design practices. The Guide to Understanding Instructional Design Expertise (GUIDE)' is an instructional design support system developed at Armstrong Laboratory, Brooks Air Force Base, in San Antonio, Texas. It is a CD-ROM based multi media software program designed to aid novice and expert instructional designers in the development of quality instruction. GUIDE is based on the nine events of instruction proposed by Gagn
é (1985) to be essential in the instructional design process. These nine events are: gaining attention, informing the learner of the objectives, stimulating recall of prerequisite knowledge, presenting the stimulus material, providing learning guidance, eliciting the performance, providing feedback about performance correctness, assessing the performance, and enhancing retention and transfer. GUIDE presents step-by-step guidance in applying Gagné’s nine events of instructional design.GUIDE operates in two modes: lesson and guidance. In the lesson mode, the user can peruse an assortment of interactive courseware examples. This lesson library or casebase demonstrates instruction for a variety of training objectives and presentation techniques. In the guidance mode, an explanation of each of the nine events of instruction is presented, as well as demonstrations of how to incorporate each of the nine events effectively to create meaningful interactive instruction. The two modes work in conjunction with each other, allowing the user to jump from one mode to the other (Gettman & Whitehead, 1995). This provides instructional designers with complete control over the organization of the sequence of learning throughout the program. Further, GUIDE is equipped with an on-line note taking feature that allows the user to record thoughts and ideas while interacting with the software. It is important to note, however, that GUIDE does not actually author the instruction. The intention of GUIDE is to aid in the development of the instruction content through the introduction of applicable instructional design practices.
Several formative evaluations of GUIDE have been conducted (Gagn
é , 1992; Tennyson & Gettman, 1995) and the initial results have been overwhelmingly positive (Gettman, 1995). An initial evaluation of GUIDE's general approach was conducted (Gagné , 1992). Findings indicated that (1) students experienced no difficulty in understanding or using the instruction of the GUIDE lesson, (2) the subsequent lessons they designed followed closely the model task provided as an example, and (3) 83% of the students judged the lesson they designed to be useful and effective.In a more extensive evaluation, Tennyson and Gettman (1995) used instructional technology graduate students enrolled in an instructional design course over several semesters to assess attitudes toward and perceptions of GUIDE, as well as assess student learning. Several findings resulting from this evaluation are noteworthy. First, participants rated the quality of GUIDE fairly high. The quality of the texts and graphics received excellent ratings. Further, participants reacted positively to GUIDE's ability to give the user control of the sequence of learning within the program. Second, participant attitudes about GUIDE's ability to aid in designing instruction shifted from indifference, at the beginning of the course, to positive, by the end of the course. Finally, results indicate that the use of GUIDE was somewhat more effective than a traditional textbook version in presenting Gagn
é's nine events of instruction. Using the grade received on end-of-course projects as a criterion, participants who opted to learn the nine events using GUIDE technology received higher grades on the project than students who chose to learn the nine events from a textbook.These two initial evaluations were highly favorable of GUIDE. Other evaluations have not been as positive, however. Asiu (1995) conducted focus groups with novice Air Force instruction developers and instructional technology graduate students using GUIDE. Responses from the group of Air Force developers indicated that GUIDE did not always provide a coherent link between the lesson objectives represented by the example lessons and content areas of interest to the Air Force users. For example, Air Force developers using GUIDE had difficulty understanding how the program could aid them in designing and organizing instruction when the case examples/samples did not closely resemble their area of expertise. Results from the second focus group activity, the instructional technology graduate students, reflected similar findings. Additionally, the second focus group identified some general notions about GUIDE not previously recognized. For example, those who had more computer experience, especially with Windows-based programs, had higher perceptions of the use and utility of GUIDE. To date, the evaluations of GUIDE have been qualitative in nature and the results have been somewhat mixed. The present study will evaluate GUIDE in a more systematic, quantitative manner.
This study proposes to assess GUIDE through the evaluation of reaction, learning, and behavioral criterion measures. First, reactions of instructional designers will be assessed after being exposed to GUIDE. Previous evaluations of GUIDE (Gagn
é, 1992; Tennyson and Gettman, 1995) have indicated that reactions have been overwhelmingly positive. Therefore, in alignment with past research, it is hypothesized participants will generally have favorable attitudes towards GUIDE. Specifically, it is hypothesized that they will report that GUIDE is easy to understand and use when developing instruction.Second, learning of instructional design in general and Gagn
é's nine events in particular will be assessed by administering traditional paper-and-pencil knowledge tests before and after the implementation of GUIDE. It is hypothesized that there will be a significant increase in test scores between the pre-training administration and the post-training administration.In addition to traditional measures of learning, the structural knowledge of the participants will be assessed before and after the implementation of GUIDE. A number of researchers have become interested in measuring how knowledge is organized in memory by studying knowledge structures or mental models (Dorsey & Foster, 1996). This interest reflects the recognition that the organization of knowledge stored in memory is perhaps of equal or greater importance to the amount of knowledge stored in memory (Johnson-Laird, 1983; Kraiger, Ford, & Salas, 1993; Rouse & Morris, 1986). Mental models have been defined as a "rich and elaborative structure, reflecting the user's understanding of what the system contains, how it works, and why it works that way. It can be conceived as knowledge about the system sufficient to permit the user to try out actions mentally before choosing one to execute." (Carroll & Olson, 1988, p.51). By collecting information on both the amount of learning (traditional paper-and-pencil test) and the organization of learning (assessment of mental models), the convergence between these two metrics of learning, and the relationship between the organization of knowledge and training outcomes can be determined. It is hypothesized that there will be an increase in similarity between the novice and expert instructional designers after the implementation of GUIDE. Further, it is hypothesized that there will be high variability across the mental models derived from the novice instructional designers at time one, and that this variability across the novice mental models will decrease at time two.
Finally, behavior measures will be collected before and after the implementation of GUIDE. Behavior, in this case, will be displayed in the extent to which Gagn
é's nine events of instruction are incorporated in the curricula developed after interacting with GUIDE. It is hypothesized that there will be an increase in the degree to which Gagné’s nine events are incorporated in the instruction developed before and after interacting with GUIDE.Method
Participants
The participants for this study will be obtained from the Texas Engineering Extension Service (TEEX). TEEX is a member of the Texas A&M University System and provides vocational and technical training, continuing education, technical assistance and technology transfer services to public and private agencies, businesses and industries. Approximately 170 instructors are employed by TEEX's 13 different divisions (e.g., Fire Protection Training Division, Management and Leadership Development Division, etc.). Approximately 30 instructors will be used in the current study. The instructors will be classified as either novice or expert instructional designers according to several criteria (e.g., years in instructional design, exposure to formal instructional design classes/workshops, etc.).
Measures
Reactions - Trainee's affective response to the training will be assessed by collecting traditional end-of-course reaction evaluations. Items included in the reaction measure will assess the degree to which trainee's feel that GUIDE's instructions are clear, comprehensible and workable, the extent to which they feel that GUIDE assists them in developing instruction, the extent to which they intend to incorporate the principles outlined by GUIDE in subsequent instruction they develop, and the extent to which they will recommend GUIDE to other peers and colleagues involved in developing instruction.
Learning - Learning will be assessed by administering a traditional paper-and-pencil knowledge measure before and after the implementation of GUIDE. The items on the measures will cover the content of GUIDE to assess the extent that the knowledge of the nine events has been acquired by the trainee.
Learning will also be assessed by investigating the participants' mental models of instructional design. The knowledge domain for this study is instructional design in general and Gagn
é's nine events of instruction, in particular. When devising a list of concepts particular to a specific content domain, it is advisable to limit the number of items in the set to simplify data collection and interpretation of results (Cooke & McDonald, 1987). By limiting the number of concepts to be analyzed, the researcher is assured that only relevant concepts are being investigated. A comprehensive list of concepts relevant to instructional design and Gagné’s nine events will be obtained. Subject matter experts (instructional designers, in this case) will then rate each of the concepts in terms of importance in the design of instruction. The concepts with the highest overall importance ratings will be used in this study.Proximity values between all possible pairs of concepts will be obtained through a paired comparison technique. In the paired comparison technique, participants will rate each pair of concepts on a 1 to 9 scale as to the degree of similarity between the two concepts in that pair. Once all pairs have been rated, distance estimates are computed as the inverse of the similarity ratings. The matrix of distance estimates can then be analyzed to determine the underlying knowledge structure for each participant.
It is advisable to utilize multiple scaling techniques in the knowledge representation stage (Cooke & McDonald, 1987). This study will incorporate multidimensional scaling (MDS) and Pathfinder techniques to analyze the participant's structural knowledge. Although each of these techniques uses the proximity value matrix to analyze the knowledge structures, each technique emphasizes different representations of the data (Cooke & McDonald, 1991; Gonzalvo, Cafias, & Bajo, 1994). Pathfinder captures information about local relationships (i.e., pairs of items that are highly related). MDS captures information about global relations among the set of concepts as a whole (i.e., dimensions). Furthermore, MDS and Pathfinder differ in terms of the type of representation generated (e.g., hierarchical in MDS, network in Pathfinder).
Behavior - In order to determine whether the knowledge and skills learned in training has been transferred to the job, behavioral measures will be collected. The knowledge and skill imparted in GUIDE is best assessed by evaluating courseware designed by the participant before and after interacting with GUIDE. A courseware sample will be obtained from each participant before and after interacting with GUIDE. Each courseware sample will be rated by a panel of research psychologists/scientists as to the extent that the courseware incorporates Gagn
é’s nine events of instruction.Procedures
An initial courseware sample for each participant will be obtained prior to being exposed to GUIDE. Participants' prior knowledge of instructional design will be determined by administering the paper-and-pencil knowledge test and collecting proximity values between the concepts to assess the novice mental models of instructional design. Proximity values for the list of concepts will be collected from expert instructional designers in order to determine the expert mental model of instructional design with which to compare the mental models of novice instructional designers. After the novice instructional designers have had adequate time (approximately three months) to interact with GUIDE, reactions to GUIDE, as well as a second administration of the two learning measures (paper-and-pencil knowledge test and proximity values between concepts) will be collected. The participants will then have an additional 30 days in which to submit their second courseware sample for evaluation.
References
Asiu, B. (1995). GUIDE Focus Group Activity. Unpublished report. Brooks AFB, TX: Armstrong Laboratory, Technical Training Research Division, Human Resources Directorate.
Carroll, J. M. & Olson, J. R. (1988). Mental models in human-computer interaction. In M. Helander (Ed.) Handbook of Human-Computer Interaction. Amsterdam: Elsevier (North-Holland).
Cooke, N.M., & McDonald, J. E. (1987). The application of psychological scaling techniques to knowledge elicitation for knowledge-based systems. International Journal of Man-Machine Studies, 26, 533-500.
Dorsey, D. W. & Foster, L. L. (1996). An application of knowledge structure assessment in the computer knowledge domain: Relations with experience and implications for training. Symposium paper presented in R. Kumoski's (chair) Knowledge Structures and Training:Measurement and Validity Issues. 11th Annual Conference of the Society for Industrial and Organizational Psychology, San Diego, California.
Gagne', R. M. (1985). The conditions of learning (4th ed.). New York, NY: Holt, Rinehart, & Winston.
Gagne', R. M. (1992). Tryout of an organizing strategy for lesson design:Maintenance procedure with checklist. Brooks AFB, TX: Armstrong Laboratory, Human Resources Directorate (AL-TP- 1992-0016).
Gettman, D. J. (1995). The Guided approach to instructional design advising: Theory history and research. Unpublished manuscript. Brooks AFB, TX: Technical Training Research Division, Human Resources Directorate.
Gonzalvo, P., Ca
ñ as, J. J., & Bajo, M. (1994). Structural representations in knowledge acquisition. Journal of Educational Psychology, 85, 601-616.Johnson-Laird, P. (1983). Mental models. Cambridge, MA: Harvard University Press.
Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78,311-328.
Rouse, W. B., & Morris, N.M. (1986). On looking into the black box: Prospects and limits in the search for mental models. Psychological Bulletin, 100, 349-363.
Tennyson, R. D. & Gettman, D. 3. (in press). Guided Approach to Instructional Design Advising (GAIDA) evaluation: Student learning, quality, attitudes, and perceptions. Brooks AFB, TX: Armstrong Laboratory, Technical Training Research Division, Human Resources Directorate.