Analysis of Aptitude-Treatment Interactions from a Personnel Classification Perspective

Robyn Maldegen
Texas A&M University

MaryAnn Statman
Monica Gribben
HumRRO

Robert Yadrick
Armstrong Laboratory
Human Resource Directorate

Overview of the Project

This paper describes the conceptual background for a three year aptitude-treatment interaction (ATI) study being conducted by the authors at the Human Resources Research Organization (HumRRO) and the Human Resource Directorate of the Armstrong Laboratory, and presents a brief review of the general status of ATI research. Before presenting the observations on and critique of ATI research, the project entitled, Using the Differential Classification Research Paradigm to Study Aptitude-Treatment Interactions in Training will be discussed.

We are presently at the end of the first year of a three-year project designed to examine the relationship between learner characteristics (e.g., cognitive aptitudes, affective factors, and conative factors) and the training environment (e.g., instructional methods and course content) using the differential prediction paradigm adopted from personnel classification. The importance of ATI research to instructional design and training evaluation is captured in the following statement by Snow and Lohman (1984):

Educational treatment comparisons, including the program evaluations, must at least incorporate tests of plausible ATI hypotheses in order to interpret their intended main effect conclusions properly. Any treatment environment can serve some learners well and others poorly. Research on treatment design should thus always use what is known about individual differences to determine for whom any particular instructional method is appropriate and for whom it is not appropriate (pp. 358-359).

The idea for applying the classification paradigm to ATI research comes from Cronbach and Snow (1977), who observed that the personnel classification problem is directly analogous to ATI phenomenon in training. The personnel classification problem is based on the assumption that an interaction between worker characteristics (e.g., aptitudes, interests, motivation), and job characteristics (e.g., technical content) exists. The operational classification process involves the assessment of a group of applicants simultaneously for several different jobs or occupations. Each individual is placed in (i.e., differentially assigned or classified to) the job for which he or she has the highest estimated performance. Differential classification capitalizes on the match between each individualís aptitude and interest profile and the unique requirements of the job, thus enhancing individual performance and overall organizational productivity (Statman, 1993).

By applying the classification paradigm to ATI research, advances in the fields of training design and evaluation, especially for Adaptive Tutors, which attempt to capitalize on ATIs through student interaction with the tutor, may be made more possible. Use of the classification paradigm will make it possible to simulate the benefits (if any) of capitalizing on ATIs through student-course matching and provide an index of training effectiveness. Neither of these capabilities are possible with the traditional ATI design. Further, the classification paradigm provides a psychometric foundation for expanding our knowledge and understanding of individual characteristics and training variables that interact to enhance or degrade student performance in a particular instructional setting.

In this study the classification paradigm is applied to examine whether learner characteristics and training environments interact to produce intra-individual differences in final course grades across Air Force technical courses. Using student-course assignment simulations, the average training performance in a sample of courses after optimal placement of students in the course that best matches their aptitude profiles will be measured. These results will be compared to random (and possible actual) assignments. It is hypothesized that training performance averages across 20 to 30 courses will be better with optimal placement than with random or actual assignments.

The project will be conducted in three Phases. The major tasks of Phase I are the design of the research method, the selection of individual difference variables to describe learner characteristics, and the development of a training characteristics inventory. Approximately 30 Air Force courses, which will vary across occupations, will be sampled. The major consideration in selecting the course sample is to obtain differentiation in the instructional methods and course content.

Phase II will entail data collection. Phase III will consist of two main tasks: the development of course prediction models and the classification cross-validation study. The classification procedure will be based on the Zeidner and Johnson (1994) method, which has received a good deal of attention within the past several years. The approach uses a Monte Carlo procedure to supplement sample data. Training effectiveness will be measured by conducting computer-based course placement simulations.

The study is being conducted over a 36 month period. The outcome of the project will be a series of reports describing the research design, variables, methodology, and results. The reports will also discuss the implications for the advancement of psychological theory in the study of ATIs, instructional theory development, and the design and evaluation of adaptive and conventional training programs.

Current Research Findings

As part of the preparation for this study, the literatures in ATI, instructional design, and training evaluation were reviewed. The goal was to explore trends in the results and to identify possible course characteristics that may account for ATIs. Currently, these findings are being used to select the sample of Air Force technical courses and to develop a questionnaire on the training environment. The following is a brief synopsis of the ATI research and some general conclusions derived from the review.

The review showed that most of the studies examined cognitive aptitudes. More specifically, 48 data points from the studies could be attributed to the investigation of cognitive factors, 16 to affective factors, and 5 to conative factors. The majority of the studies examined 6 types of cognitive aptitudes, including verbal/crystallized aptitudes (14), quantitative aptitudes (7), general knowledge (7), spatial aptitudes (6), field dependence independence (5), and fluid/analytical (5). Conative factors were examined primarily using locus of control (4). The remaining 6 of the data points were divided equally among motivation, achievement via independence, and achievement via conformity. Finally, 11 data points were associated with affective factors. These factors included anxiety (6), attitudes/preferences (3) and self-efficacy (2).

The review also indicated that a variety of different treatments were used. The most frequently manipulated treatment variables were structure (26%) and elaboration (23%). The typical manipulation for structure involved providing one group with a more self-directed course of study and providing the control group with lecture in a more controlled atmosphere. Elaboration was typically manipulated by providing the experimental group with analogies or some other means of clarifying the material while the control group typically received only text. Other experimental manipulations included either providing one group with deductive training while the other received inductive training, providing one group with cognitive modeling and comparing that group to a control group, or comparing a group that participated primarily in small groups to a group that participated in a large group.

Table 1 presents study outcomes for the cognitive factors examined by treatment. Study outcomes had to have at least two data points to be considered. The results indicated that most studies did not find an aptitude treatment interaction by manipulating structure (67%), the elaboration of the text (65%), or mapping (88%). On the other hand, the majority of data points (75%) in studies in which deductive and inductive training were manipulated indicated that subjects low in cognitive ability performed better when given deductive training and subjects high in cognitive ability performed better when given inductive training. Also, it should be noted that no apparent trend could be detected by manipulating either cognitive modeling or the group size. Finally, although the data indicated that manipulating the elaboration of the text was not likely to result in an ATI, some trends should be noted. For example, subjects high in verbal ability tended to benefit from verbal elaboration while those low in verbal ability tended to benefit more from visual elaboration or analogies as did subjects high in spatial ability.

Table 2, presents the frequency of study outcomes for conative and affective factors by treatment. In general, there were too few data points to detect any patterns within a given treatment manipulation for conative factors. One data point indicated that subjects with an internal locus of control performed better with low structure. The other data point indicated that there was no ATI for locus of control and deductive vs. inductive training. Finally, there were also too few data points to detect any patterns between treatment and affective factors. Although, one finding is particularly interesting and may not be clear from the table. The outcome of manipulating the group size was that students who had a preference for large groups actually performed better in small groups while students who had a preference for small groups performed better in large groups.

Critique of ATI Research

In general, we join with most other researchers before us in concluding that there are no consistent findings of ATIs in the training and education literatures. There are some possible methodological explanations for this. First, the most obvious, and possibly the most damaging, is the small sample sizes in ATI studies. For example, in the 37 studies reviewed in this paper, the overall N ranged from 35 (e.g., DeKeyser, 1993) to 297 (e.g., Canino & Cicchelli, 1988) and the sample size within treatment ranged from 16 (e.g., DeKeyser, 1993) to 94 (e.g., Canino & Cicchelli, 1988). And related to sample size, the review revealed a surprising diversity in the study populations used to examine ATIs. More specifically, study populations ranged from lower and upper elementary students to high school and college students to managers, administrators, and Air Force trainees.

Second, many of the studies reviewed had compounded problems because not only did they have small sample sizes but they also used a large number of variables and analyzed every main effect and interaction possible. For example, one study examined 11 different cognitive abilities in a sample of 147 (Mills, Dale, Cole, & Jenkins, 1995). Another study conducted 240 F tests and found that 15 of the aptitude-treatment interactions were significant. They, not surprisingly, had to conclude that the significant interactions were not meaningful, and therefore, did not contribute to knowledge about differential educational programming (Ysseldyke, 1977). Cohen (1990) pleasantly reminds researchers that too many variables and indiscriminant analyses increase the probability of making a Type I error, thereby, reducing the likelihood of finding significant results.

Third, there was very little consistency across studies in aptitude constructs and the measures used to examine these constructs. In fact, it may be more logical to refer to aptitudes as person characteristics due to the variety of possible variables. For example, Shute, Lajoie, and Gluck (in press) discuss three categories of person characteristics that may interact with the treatment: cognitive, conative, and affective. Cognitive factors refer to the mental processes that make knowledge and skill acquisition possible. Conative factors refer to the stable traits unique to a person, such as motivation. And affective factors refer to the less stable personality traits and mood, such as neuroticism and fatigue. An examination of the ATI studies reviewed indicated that the range of cognitive factors investigated was the most diverse of the three types of person characteristics. Cognitive ability was defined as verbal/crystallized ability, quantitative ability, general knowledge, spatial ability, field dependence independence, fluid/analytical ability, motor skills, working memory, natural science, and operation learning. In addition, the measures used to assess these aptitudes ranged from standardized tests such as the California Achievement Test and the Group Embedded Figures Test to measures developed by the researcher to other types of measures such as GPA.

In addition to the discrepancies found among the person characteristics, a lack of consistency for the type of treatment was also observed. For example, although the majority of data points from studies were associated with either the structure of the course or the elaboration of the text, there were also 13 other types of treatments examined in the studies. Nine of these treatments only contributed one data point. The diversity of the treatments used coupled with the relative lack of data points for many of the treatments made it difficult to draw conclusions about possible interactions with person characteristics.

Finally, the literature review revealed a surprising diversity of outcome measures. For example, preestablished measures included achievement tests, and reading and math tests. Measures developed by the researcher took the form of multiple choice, essay, or open ended questions and generally tested the subjectís learning of the course material. Other types of measures included accuracy, number of errors, number of appropriate links, and number of concept words.

Conclusions

There is a great deal of room for improvement in ATI research. We choose to view this state of affairs as our cup being half full. Our ATI/classification study is designed to address several of the important areas, i.e., the need for: 1) large sample sizes within treatments, 2) well-established measures of person characteristics (we are using the ASVAB subtests), and 3) a research paradigm that provides a foundation for theory development and the measurement of training effectiveness.

References

Canino, C. & Cicchelli, T. (1988). Cognitive styles, computerized treatments on mathematics achievement and reaction to treatments. Journal of Educational Computing Research, 4(3), 253-265.

Cohen, J. (1990). Things I have learned so far. American Psychologist, 45(12), 1304-1312.

Cronbach, L.J., & Snow, R.E. (1977). Aptitudes and instructional methods: A handbook for research on interaction. New York, NY: Irvington.

DeKeyser, R.M. (1993). The effect of error correction on L2 grammar knowledge and oral proficiency. The Modern Language Journal, 77, 501-514.

Mills, P.E., Dale, P.S., Cole, K.N., & Jenkins, J.R. (1995). Follow-up of children from academic and cognitive preschool curricula at age 9. Exceptional Children, 61(4), 378-393.

Shute, V.J., Lajoie, S.P., & Gluck, K.A., (in press). Individualized and group approaches to training. To appear in S. Tobias & D. Fletcher (Eds.), Handbook on Training.

Statman, M.A. (1993). Improving the effectiveness of employment testing through classification: Alternative methods of developing test composites for optimal job assignment and vocational counseling. Dissertation Abstracts International, 54, (08B, University Microfilms No. AAG9403360).

Ysseldyke, J.E. (1977). Aptitude-treatment interaction research with first grade children. Contemporary Educational Psychology, 2, 1-9.

Zeidner, J., & Johnson, C.D. (1994). Is personnel classification a concept whose time has passed? In M.G. Rumsey, C.B. Walter, & J.H. Harris (Eds.), Personnel Selection and Classification (pp. 103-125). Hillsdale, NJ: Lawrence Erlbaum Associates.


Table 1
Frequency of Study Results for Cognitive Factors and Treatment

Treatment

Significant / Not Significant Results

High Cognitive Ability

Low Cognitive Ability

structure

sig.
not sig.

2
4

1 - highs benefited from low structure

1 - lows benefited from low structure

elaboration

sig.
not sig.

8
15

6 - highs benefited from high elaboration

2 - lows benefited from high elaboration

deductive/
inductive

sig. 3
not sig.

3
1

3 - highs benefited from inductive

3 - lows benefited from deductive training

cognitive modeling

sig.
not sig.

2
2

1 - highs benefited from modeling

1 - lows benefited from modeling

mapping

sig.
not sig.

1
7

0 - highs benefited from mapping

1 - lows benefited from mapping

small/large group

sig.
not sig.

3
1

2 - highs benefited from small group

1 - lows benefited from small group

Note. The numbers represent the number of data points associated with each outcome.

Table 2
Frequency of Study Results for Conative / Affective Factors and Treatment

Treatment

Significant / Not Significant Results

High Conative / Affective Ability

Low Conative / Affective Ability

structure

sig.
not sig.

1
0

1 - high conative benefited from low structure

0 - low conative benefited from low structure

structure

sig.
not sig.

1
2

1 - high affective benefited from low structure

0 - low affective benefited from low structure

deductive/
inductive

sig.
not sig.

0
1

0 - high conative benefited from deductive

0 - low conative benefited from deductive training

cognitive modeling

sig.
not sig.

1
0

0 - high affective benefited from modeling

1 - low affective benefited from modeling

small/large group

sig.
not sig.

1
0

0 - high affective benefited from small group

1 - low affective benefited from small group

Note. Conative refers only to locus of control in this table. The numbers represent the number of data points associated with each outcome.

Back to Table of Contents