Stephen A. Truhon
Winston-Salem State University
and
James R. Van Scotter
Air Force Institute of Technology

Paper Presented at
International Military Testing Association
San Antonio, TX
November 12-14, 1996

The current study was concerned with three questions: 1) has a reported decline in mechanical abilities among applicants affected the quality of recruits in Mechanical Air Force specialties (AFSs)?; 2) given the changes in the backgrounds of applicants, has the Armed Services Vocational Aptitude Battery (ASVAB) retained its value as a predictor of training outcomes for Mechanical specialties?; and 3) what other factors could improve prediction of mechanical performance? The records of 48,009 first-term recruits who enlisted in the service between January, 1990 and September, 1995 and were assigned to a Mechanical AFS were examined. Using the Mechanical (MECH) composite from the ASVAB, there does not appear to be a decline in the quality of airman selected for assignments in Mechanical AFSS. The correlation between MECH and final school grades in technical training is r -.32, (when corrected for unreliability and range restriction r -.60), comparable to earlier studies. MECH correlates well with the number of high school shop classes (uncorrected r = .31, corrected r = .48), and somewhat less well with the number of physical and applied science classes (uncorrected r = .20, corrected r = .31). These and other factors are discussed in improving the prediction of outcomes for Mechanical AFSs.

INTRODUCTION

The Air Force, like the other American military organizations, uses the Armed Services Vocational Aptitude Battery (ASVAB) to select applicants for military service and to assign accepted applicants to specific jobs (Air Force Specialties [AFSs]). The ASVAB is made up of 10 subtests, which measure aptitudes in verbal, mathematical, clerical-speed, and technical areas. The Air Force uses four composites from the ASVAB to assign recruits to jobs in specialties: Mechanical, Administrative, General, and Electronic; and a fifth composite, the Armed Forces Qualification Test (AFQT), to select applicants for entry into the Air Force, prior to job assignment.

A number of changes occurring in the military, in general, and in Mechanical specialties, in particular, have created concern among recruiters. With the end of the Cold War, the amount of money available for military budgets has declined (Grier, 1995). The resulting drawdown has led to a smaller military force. A smaller force could work to the advantage of the military because it allows the military to be more selective in the applicants it chooses. However, this drawdown has led some in the civilian population to believe that the military does not need new recruits and that it is not a stable career option. Surveys indicate the percentage of 16- to 21-year-olds interested in enlisting has been dropping from 33% to 25% among males (Chapman, 1996).
______________________________
¹ This study was completed when the first author was a summer research associate with the Air Force Office of Summer Research. The views expressed are those of the authors and not necessarily those of the Department of the Air Force, the Department of Defense, or the Government of the United States.

Meanwhile, other changes have more primarily affected Mechanical specialties. While the number of commissioned officers in the Air Force (often pilots) has remained steady in recent years, the number of enlisted recruits (from which those in Mechanical AFSs are often selected) has been declining (Chapman, 1996). At the same time, increased demands are being made of mechanics. In 1992 as part of its Year of Training initiative, the Air Force began the Mission Ready Technician (MRT) program (Rankin, 1995). As part of MRT, airmen receive classroom and controlled technical training and then are sent to their assignments to receive operational training (Kuhn, 1995). Thus graduates of technical training in Mechanical AFSS are more productive members of their units much sooner than in the past. In addition, there has been a decline in mechanical skills among applicants to the Air Force (Skinner, 1996, unpublished data). Similar trends have been observed by the other branches of the military (Defense Manpower Data Center, 1996, unpublished data) and by civilian industries (e.g., Orenstein, November 20, 1995).

Another concern has been the validity of the ASVAB in predicting performance in Mechanical AFSs. Studies from those recruits tested in the 1980s suggested that the ASVAB had acceptable validity. Wilbourn, Valentine, & Ree (1984) reported a median uncorrected correlation between Mechanical score and final school grade of .41. Similarly Ree & Earles (1992) reported a weighted mean uncorrected validity of .43 (.75, when corrected for restriction in range). Carey (1994) reported that, among Marine Corps automotive and helicopter mechanics, the ASVAB and time in service accounted for a weighted mean uncorrected validity of .52 (.68, when corrected for restriction in range). In general, the ASVAB predicts performance in Mechanical specialties well

The current study was concerned with three questions: 1) has the decline in mechanical abilities among applicants affected the quality of Air Force recruits in Mechanical specialties?; 2) given the changes in the backgrounds of applicants, has the ASVAB retained its value as a predictor of training outcomes for Mechanical specialties?; and 3) what other factors could improve prediction of mechanical performance?

Subjects

The subjects were 48,009 first-term recruits who enlisted in the service between January, 1990 and September, 1995 and were assigned to a Mechanical AFS. These recruits were primarily male, white, single, and had completed high school.

Measures

A database on the recruits was created from their PACE, MEPS, and technical school training records. In all, 155 variables were gathered. Of primary interest were the recruits' Mechanical score on the ASVAB (MECH) and their final school grade (FSG) in technical school. Ree and Earles (1992) reported that the MECH has an internal consistency of .90. For purposes of this study, the FSG was considered to have a reliability of .80 (Pearlman, Schmidt, & Hunter, 1980).

Criterion Groups

Sixty-two Mechanical AFSS were considered for examination. Interviews of training course managers identified changes in course content, emphasis, length, location and performance approach for each of the AFSs. These changes were used to form 113 samples.

Changes in Mechanical Ability Over Time

TABLE 1

Comparison of Mean Mechanical Scores from the ASVAB
by Year of Entry²

The Relationship between MECH and AFQT

The relationship between the MECH and AFQT was also examined. The correlation between MECH and AFQT was moderate and significant (r = .34, df = 46,792, p < .001). Performing separate correlations by year of entry revealed that the relationship was fairly consistent across the years (1990: r = .31; 1991: r = .35; 1992: r = .32; 1993: r = .37; 1994: r = .36; and 1995: r = .39).

Validity of Mechanical Score in Predicting Final School Grade

The weighted average correlation between MECH scores and FSG across the 113 samples was .31 (N = 40,654). Most of the validity correlations are between .20 and .40, although they range from .06 to .89. (Most of the extreme values occur when the sample size is small). It was also notable that most of the validities are quite similar in magnitude for samples derived from the same AFS Code.

These separate analyses can be viewed as having been derived from separate samples from the same population and thus can be combined into a meta-analysis (Hunter & Schmidt, 1990). A meta-analysis was conducted with corrections for artifacts due to unreliability and range restriction, using Metaquik (Stauffer, 1996). Estimates of reliability for the predictor and criterion were described earlier (i.e., .90 for the MECH, .80 for FSG). The estimate of range restriction relative to the ASVAB norm group was determined by dividing the standard deviation for the MECH from each of the samples by 26.28³. When corrected for unreliability and range restriction, the correlation between MECH and FSG is estimated at .60.

Metaquik also tests whether the samples are homogeneous, i.e., whether the sample statistics could have been derived from the same population parameter. A lack of homogeneity suggests the presence of one or more moderator variables, i.e., variables that cause differences in the correlation between the two variables being examined in the meta-analysis.
_______________________________
² Year of entry analyses for the MECH score are based on accessions for the entire calendar year (January-December) in 1990-1994, but for the (January-September) period in 1995. The same applies for analyses by year of entry described later in this paper.

Explaining the Relationship between MECH and FSG

The usual method for testing possible moderator variables is to categorize the samples into subsets based on a potential moderator and then perform a meta-analysis for each subset (Hunter & Schmidt, 1990, pp. 292-293). With the limited amount of time available to complete this report, that approach was forgone. This is, however, an area for further exploration and possible moderator variables will be described in the Discussion section.

Instead of the typical meta-analysis it was decided to examine relationships among other variables, MECH, and FSG. This approach is akin to using meta-analysis as a means to model-building (Borman, White, Pulakos, & Oppler, 1991; Viswesvaran & Ones, 1995).

Two variables seemed to be promising: the number of high school shop classes, and the number of high school physical and applied sciences classes. The high school shop classes variable (HANDSON, so called because these classes provide students with hands-on mechanical experience) was calculated by counting the number of such classes (i.e., electronics, radio repair, auto repair, hydraulics, industrial arts, and mechanics) that the recruit had completed. The physical and applied sciences class variable (PHYSCI) was calculated by counting the number of such classes (i.e., physics, chemistry, general science, blueprint reading, and shop math) that the recruit had completed.

The correlations among MECH, FSG, HANDSON, and PHYSCI were then calculated for the 113 samples. Meta-analyses were conducted to determine the weighted mean correlations. The meta-analyses were also corrected for unreliability. Reliabilities for HANDSON and PHYSCI were determined by calculating the internal consistencies of the scales for each of the 113 samples. The reliability of HANDSON was generally higher than that of PHYSCI (average r's = .67 and .32 respectively). The restrictions in range were unknown. The results are presented in Table 2.

TABLE 2

Correlations among MECH, FSG, HANDSON, and PHYSCI⁴

HANDSON correlates strongly with MECH (uncorrected r = .37, corrected r = .48), but does not correlate well with FSG (uncorrected r = .07, corrected r = .10). PHYSCI has a somewhat lower correlation with MECH (uncorrected r = .20, corrected r = .31), but its correlation with FSG is somewhat higher (uncorrected r = .16, corrected r = .17). Finally, HANSON and PHYSCI are moderately correlated (uncorrected r = .32, corrected r = .71).
_______________________________
⁴ The correlations above the diagonal are corrected for unreliability and, in the relationship between MECH and FSG, for range restriction. The correlations below the diagonal are the weighted average of correlations from the samples.

DISCUSSION

The current study was concerned with changes that have been occurring in mechanical abilities and in Mechanical AFSs in the 1990s. The results suggest that some of the worries about Mechanical AFSS are unfounded.

While there have been reports of a decline in mechanical abilities among applicants (Skinner, 1996, unpublished data), this decline does not seem to have affected the quality of airmen selected for assignment in Mechanical AFSs. While there are changes in the mean MECH scores for those in Mechanical AFSs during the 1990s, the overall pattern is not one of decline. While the decline in the number of applicants and in the mechanical ability of those applicants have created difficulties for recruiters (Chapman, 1996), they still seem to be attracting a high quality of recruits for Mechanical AFSs.

Hypotheses about the causes of the decline in mechanical ability among accessions have generally focused on the changing nature of accessions (i.e., gender and racial composition). Yet the gender composition of accessions in Mechanical AFSs does not appear to have changed appreciably in the 1990s and the racial composition has only changed slightly. Follow-on analyses revealed that, from 1990 to 1995, the percentage of females ranged from 4.2% to 6.0%, of blacks from 6.4% to 8.7%, and of other races from 2.7% to 7.3%.

Another hypothesis is that the Air Force, in seeking recruits with higher AFQT scores, has made it more difficult to obtain recruits with good mechanical skills. However, other analyses from this study found that MECH and AFQT were positively and moderately correlated.

The current study also replicated ASVAB validity studies done in the 1980s (Ree & Earles, 1992; Wilbourn et al., 1984) and demonstrated the validity of the ASVAB in predicting performance by recruits in Mechanical AFSs in the 1990s. All of the validity coefficients are positive, meaning that recruits with higher MECH scores were expected to demonstrate higher levels of academic performance on entry-level material taught during technical training.

Time constraints prevented this study from examining possible moderator variables between MECH and FSG. Clearly this is an area for future research. Potential moderators include: 1) level of aptitude requirement (i.e., how high a MECH score is required for entry into a particular AFS); 2) the type of aptitude required (i.e., whether the AFS requires a specific score on the MECH alone, on both the MECH and the Electronics composites, or on either the MECH or Electronics composites); 3) type of career field (either determined by the first two digits of the AFS Code or the clusters described by Alley & Ree, in preparation); and 4) an indicator of the nature of the training (e.g., the time or location of the training).

Viswesvaran and Ones (1995) present a strong case for the use of meta-analyses in the process of model building. They suggest procedures for using the estimated true correlations obtained through meta-analysis as input for structural equations modeling. Future analyses could follow this approach in helping to explain the relationship between MECH score and FSG.

The number of high school shop and applied science classes appear to be salient factors. The variables derived from them (HANDSON and PHYSCI) correlate well with each other. HANDSON correlates fairly strongly with MECH but not FSG. PHYSCI follows the same pattern but with lower correlations. These results suggest a basic model in which HANDSON and PHYSCI have a direct effect on MECH and an indirect effect on FSG, such as seen in Figure 1.

This basic model is open to improvements. Improvements could be made to the measurement of HANDSON and PHYSCI by obtaining estimates of range restriction. In addition a variable counting the number of mathematics classes taken in high school (MATH) could be added. Gender differences have been noted in the number of females in mechanical AFSs, in their scores on several of these measures, as well as in the curriculum they pursue in high school. Such a model should also include gender. Instead of using the MECH score itself, the subtests (MC, GS, and AS) that make up MECH and the other subtests of the ASVAB (Arithmetic Reasoning [AR], Word Knowledge [WK], Paragraph Comprehension [PC], Numerical Operations [NO], Coding Speed [CS], Math Knowledge [MK], and Electronic Information [El]) could be used in a refined model. An index of training performance could be added for those AFSs for which MRT is applicable. Indicators of job performance in various job types within individual Mechanical AFSs after completion of entry-level training (e.g., six months, a year, or as long as four to eight years after training) could be used to illustrate how level of achievement in training, as measured by FSG, is related to later performance on technical tasks on the job.

These suggestions and refinements are shown in the more comprehensive model in Figure 2. This model depicts relatively simple and direct relationships among educational, aptitude, training, job assignment, and job performance. Follow-up research exploring the accuracy of alternate models is needed. As part of this research, the potential for using occupational surveys to explore measures such the number, type, and difficulty of tasks performed for capturing the productivity of airmen is recommended.

REFERENCES

Alley, W.E., & Ree, M.J. (in preparation). Improved training performance through optimum personnel classification. (AL/HR-TP-XX-XX) Brooks AFB, TX: Human Resource Directorate, Armstrong Laboratory.

Borman, W.C., White, L.A., Pulakos, E.D., & Oppler, S.H. (1991). Models of supervisory job performance ratings. Journal of Applied Psychology, 76, 863-872.

Carey, N.B. (1994). Computer predictors of mechanical job performance: Marine Corps findings. Military Psychology, 6,1-30.

Chapman, S. (1996). Uncertainty on the personnel front. Air Force Magazine, 79 (3), 40-43.

Grier, P. (1995). Snapshots of a force on the move. Air Force Magazine, 78 (6), 58-62.

Hunter, J.E., & Schmidt, F.L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage Publications.

Kuhn, T. (1995). Crew chiefs: Mission ready! Airman, 39 (3), 30-35.

Neter, J., Wasserman, W., & Whitmore, G.A. (1988). Applied statistics (3rd edition). Boston: Allyn and Bacon.

Orenstein, B.W. (November 20, 1995). Machinist shortage plagues manufacturers. Eastern Pennsylvania Business Journal.

Pearlman, K., Schmidt, F.L., & Hunter, J.E. (1980). Validity generalization results for tests to predict job proficiency and training success in clerical occupations. Journal of Applied Psychology, 65, 373-406.

Rankin, K.S. (1995). The year of tech training. Airman, 39 (10), 38-41.

Ree, M.J., & Earles, J.A. (1992). Subtest and composite validity of ASVAB Forms 11, 12. and 13 for technical training courses (AFHRL-TR-1991-0107). Brooks AFB, TX: Human Resources Directorate, Manpower and Personnel Research Division.

Stauf fer, J.M. (1996). Metaquik 16: Psychometric meta-analysis program for Windows [Computer Program]. Terre Haute, IN: Department of Management and Finance, School of Business, Indiana State University.

Viswesvaran, C. & Ones, D.Z. (1995). Theory testing: Combining psychometric meta-analysis and structural equations modeling. Personnel Psychology, 48, 865-885.

Wilbourn, J.M., Valentine, Jr., L.D., & Ree, M.J. (1984).Relationships of the Armed Services Vocational Aptitude Battery (ASVAB) Forms 8, 9, 10 to Air Force technical school final grades (AFHR-TP-84-8). Brooks AFB, TX: Manpower and Personnel Division.