Steven E. Lammlein
Caroline C. Cochran
Personnel Decisions Research Institutes, Inc.

It is hardly possible to overemphasize the individual and societal importance of effective training. However, the need for effective training is on a collision course with an increasing emphasis on cost controls in public- and private-sector organizations. Training is important, to be sure, but it is also coming under greater scrutiny for its efficiency and effectiveness.

The sine qua non of training is retention. Designing training to ensure retention in a cost-effective manner requires, of course, an understanding of how training design decisions and other aspects of training situations influence retention. Many studies have been conducted over the years of the influences on retention, to the point where the most significant recent work has been able to adopt a cumulative or meta-analytical focus (cf. Arthur, 1995; Bennett, 1995; Farr, 1986).

Yet there is a wide gulf between research and practice in this area. It is still too difficult to take a conclusion from training research and (a) translate it into a practical training decision, (b) predict with reasonable precision the effects of the decision on retention, and (c) determine whether the decision is an optimal one. In part this is due to the fact that some training practitioners lack the technical sophistication needed to profit from much of the research literature. It is also due to factors such as inadequate ecological validity of some of the training research and a dearth of research on the effects of multiple retention-relevant factors taken together.

The study described in this paper was intended as a start at solving these problems. In this study, we developed a prototype training design tool; one that allows training research to be easily utilized by practitioners to develop more effective and efficient training. This tool is a computerized model, called the LTM, that allows a user to input various aspects of the training situation and predict the retention that would result. It also allows the user to investigate various scenarios for improving retention.

In the process of developing the LTM prototype we conducted a literature review on the factors influencing retention, operationalized some of the retention-relevant constructs in an Inventory of Task and Instructional Characteristics (ITIC), and programmed the prototype LTM. Each is discussed in turn, followed by a brief description of the follow-up research now underway.

The first part of our study was a literature review devoted to the influences on retention. Our initial view of retention focused on the idea that to demonstrate retention of a skill or piece of information we must first learn the skill or information (acquisition), then we must be able to access and recover this skill or information from memory after some amount of time has passed (retrieval), and finally we must want or have to perform the skill or demonstrate knowledge of the information (performance/motivation). We then operated under the assumption that these three processes; acquisition, retrieval, and performance/motivation, are represented by certain key concepts in the literature and that these concepts are in turn predicted by other relevant variables. For example, degree of original learning represents acquisition and an individual’s cognitive ability predicts degree of original learning or acquisition (Farr, 1986). With these assumptions in mind we conducted several comprehensive literature searches of relevant databases and reviewed the literature.

Based on the literature review, we formulated a conceptual process model of retention, shown in Figure 1. In the model, all of the trainee characteristics, including ability, prior knowledge, and motivation to learn are hypothesized to affect degree of original learning through depth of processing, elaboration, and cognitive effort. Additionally, the situational variables of task characteristics and instructional strategies are thought to influence degree of original learning. The situational characteristic of the length of the retention interval is affected by the degree of original learning and in turn affects the accessibility of information in long-term memory. Context and cue similarity and testing methods are also hypothesized to directly affect retrieval processes. Finally, reinforcement contingencies are proposed to affect acquisition and performance/motivation.

The three major processes of acquisition, retrieval, and performance are arranged linearly, as they must occur in sequence in order to result in retention. Final retention performance represents the dependent or criterion variable. This is the variable that is often used to evaluate the effectiveness and efficiency of training.

We desired to operationalize a subset of the variables in the conceptual model in a prototype ITIC. We chose this subset to sample various parts of the model. The variables chosen for measurement in this prototype were degree of original learning, prior knowledge, task organization/complexity, elaboration, context/cue similarity, ability, and motivation.

We wrote multiple-choice items to operationalize each of the selected variables. Our goal was effective measurement, and to this end we first strove to represent in construct-valid fashion each key facet of the selected retention-relevant variables. For most, this required more than one item. Response alternatives were written to be as generally applicable as possible, so that the ITIC would be relevant to a wide variety of training situations.

Two sets of prototype ITIC items were developed for all of the variables except ability and motivation; one set for procedural/motor tasks and one for declarative/verbal information. Though the same variables are measured in each set of items, the two types of training content are different enough in nature to necessitate specialized wording for each. Since ability and motivation refer to characteristics of the trainees and not the training content, only one set of items was needed for each of these variables.

"Ability" was conceptualized in the ITIC items as having three facets; cognitive ability, perceptual ability, and psychomotor ability. For each of these, there is a pair of ITIC items. The first asks for the level of ability possessed by trainees, and the second asks for the importance of the ability for successful performance in training.

Following is an example ITIC item that assesses prior knowledge for procedural/motor tasks:

1. To a very little extent
2. To a little extent
3. To some extent
4. To a great extent
5. To a very great extent

Fifty-one such items were developed for the original prototype of the ITIC. The prototype ITIC was successful in that retention-relevant variables were operationalized in a closed-end response format useful for describing a variety of training situations. We are encouraged by this; however, we did find some of the variables in our model somewhat difficult to adequately represent in the ITIC. This is because some of the variables are rather complex and thus were difficult to depict in unidimensional, multiple-choice items. As such, we currently plan to investigate the summary/anchored rating scale as an alternate assessment format for the ITIC.

Programming the LTM involves developing software to (a) introduce the user to the program and provide instructions on how to use it, (b) administer and score the ITIC assessment of the training situation, (c) generate and display the predicted retention, and (d) allow the user to investigate the effects on retention of various design scenarios involving the retention-relevant variables. This software was programmed using Microsoft Visual Basic for the Windows 95 environment.

Algorithm. In the program, the ITIC is administered according to a branching procedure. The first ITIC question asks whether the task or material to be trained is procedural/motor vs. declarative/verbal; the answer to this question then branches program execution through the appropriate set of questions for the chosen task type. Then, the common set of items for ability and motivation are presented.

Each variable is assessed in the ITIC by one or more items. For all the variables except ability, an average score is computed based on the ITIC responses, with higher scores corresponding to more favorable retention conditions. For ability, the importance of each of the three facets (cognitive, perceptual, and psychomotor) for training performance is multiplied by its corresponding level among trainees, the products are summed across the three facets, and the resulting sum is scaled so that it has a metric similar to that of the other variables.

The scores for the ITIC variables are then combined in a weighted, linear equation. The weighting scheme used in the prototype is an ad hoc one based loosely on combining the findings of Arthur (1995) and Farr (1986). This places slightly higher weight on degree of original learning and context/cue similarity than on the other variables. It bears mention that the program offers the user the option of easily changing the weights assigned to the variables.

The LTM prototype depicts predicted retention as the percentage of the maximum possible given the ITIC responses. In other words, the predicted value is as follows:

Weighted Composite of ITIC Responses/Maximum Possible ITIC Composite Score*100

We decided to use this as our outcome measure in the prototype because the current literature makes it difficult to generate precise, time-based predictions of retention. This is a very simplified prediction model, used only to demonstrate the feasibility of the LTM concept. We believe that the actual influences on retention are more complex, and an important part of our future work will be to map these more complex relationships and generate time-based predictions of retention.

Figure 2 shows the program flow for the prototype LTM.

User Interface. The LTM user interface presents the ITIC items one-by-one. Users select an answer by clicking on it. When the ITIC items are completed, the user is given the option of reviewing the item responses or viewing the results. The results screen consists in part of a graphic that depicts the predicted retention level in terms of a thermometer analogue, from 0% to 100% of the maximum possible retention.

The scores for each of the seven retention-relevant variables are also provided, and the user is given an opportunity to modify any or all of them. This is the "explore mode" portion of the results screen. If the user does decide to modify any of the variables, a screen is presented which explains the variable and how it influences retention. The user is then given the opportunity to modify the score based on how he/she might be able to change the training situation. Then, a predicted retention value is generated based on these modifications and compared to the original one. The user can make as many modifications as desired, and is given the option at any time to delete the already-made modifications and start over. This capability allows the user to explore in an iterative fashion how the training situation might be changed to increase retention or to reach a certain level of retention in the most cost-effective fashion. As noted earlier, the user also has the option of changing the weights of the variables used to predict retention.

Our evaluation of the LTM prototype in this preliminary study was informal. Those who used it gave us excellent feedback on its straightforwardness and usefulness. We were greatly encouraged by the success of the LTM concept; that it is indeed possible to develop a product that allows inputting aspects of the training situation, generating a prediction of retention, displaying the prediction, and allowing users to explore how the retention might be increased.

This was an exploratory study to identify retention-relevant variables, operationalize them, and develop prototype software to test the concept of assessing training situations and generating predicted levels of retention. The algorithm used to predict retention was an ad hoc one drawn loosely from the literature for illustration purposes only.

A follow-up study is currently underway to develop and validate an empirical algorithm for predicting retention. In this study, a revised ITIC that includes all the known retention-relevant variables will be used to assess a variety of carefully-sampled US Air Force training programs. Persons trained in these programs will complete a number of performance measures to assess their retention of the trained content. Performance will be assessed at the end of training and at three-month intervals up to one year after training. This will provide the data needed to generate and validate the empirical algorithm.

This follow-up study is scheduled for completion in early 1999.

Arthur, W. (1995). Skill retention and decay: A meta-analysis. The Armstrong Laboratory Human Resources Directorate. Brooks AFB, TX.

Bennett, W. R. (1995). A meta-analytic review of factors that influence the effectiveness of training in organizations. Unpublished doctoral dissertation, Texas A & M University.

Farr, M. J. (1986). The long-term retention of knowledge and skills: A cognitive and instructional perspective. IDA Memorandum Report M-205. Alexandria, VA: Institute for Defense Analyses.

Lammlein, S. E., & Cochran, C. C. (1996). Revised report: Predicting skill and knowledge retention: Development of the LTM prototype (Institute Report No. 281). Minneapolis: Personnel Decisions Research Institutes, Inc.