Tasks and Competencies for U

Tasks and Competencies for U.S. Army First Sergeants

Jimmy L. Mitchell
The Institute for Job and Occupational Analysis, San Antonio, TX, USA

Darrell A. Worstine and William T. Badey
U.S. Army Research Institute for the Behavioral Sciences, Alexandria, VA, USA

Johnny J. Weissmuller
Metrica, Inc., San Antonio, TX, USA

ABSTRACT

A study of U. S. Army First Sergeants was undertaken by the Army Research Institute to validate training
requirements for the First Sergeant Course conducted by the U.S. Army Sergeants Major Academy (USASMA),
Ft. Bliss, Texas. In addition to collecting data on pertinent tasks, this survey also included a list of competencies
potentially relevant to such duty, as well as training emphasis ratings by supervisors of First Sergeants at all
levels of command. Rating of the competencies was voluntary both for the First Sergeants themselves, and
for their supervisors; voluntary participation ranged from 64 - 75 percent for various raters. The competencies
list was adapted from Fleishman's standard skills list (also used in the O*NET) which had been reviewed by a
group of Army Subject-Matter Experts to identify items relevant for Army jobs. Analysis of the data indicated
a high degree of correlation between average ratings (group means) of commanders and other supervisors of
First Sergeants. In addition, the highest rated competencies were generally the same as First Sergeants had
indicated as significant for their job. Thus there appears to be substantial convergent validity for these
constructs. Correlations among rating groups were very high (.96-.98) as were correlations between group
mean descriptions for disk-based and Internet survey respondents.

The United States Army Sergeants Major Academy (USASMA), Fort Bliss, TX., requested the Army Research Institute (ARI) conduct an occupational study, to collect data to evaluate the First Sergeant course. A preliminary Army study revealed that mandated manpower and skill reductions had created situations whereby soldiers were being placed into jobs previously held by soldiers of higher rank. Consequently, these soldiers were becoming skilled in corresponding tasks much sooner than soldiers of equal rank had in the past. Personnel at USASMA developed the following mission statement to describe the intended purpose of this study.
"To validate First Sergeant (1SG) and Detachment Sergeant (DET SGT) duty position
performance requirements and to facilitate the development of a life-cycle training model
to support effective and efficient training of future 1SGs and DET SGTs."

USASMA coordinated a request for this study with ARI. ARI, in turn, contracted with the Institute for Job and Occupational Analysis (IJOA) to conduct the study. The study was initiated in April 1999 and was completed in June 2000.

The initial 1SG and DET SGT task list was developed using a combination of task statements provided by course managers at USASMA and task statements defined during a prior study (i.e., Enlisted Common Soldier Tasks, February 1999). It was supplemented with a list of personal skills and abilities relevant to the performance of job tasks in an attempt to gain more information about the requirements of Army 1SG and Det SGT jobs. The initial competency list was derived from lists of knowledge, skills and abilities contained in the Occupational Information Network (O*NET) Content Model, September 1995 (Peterson, et al, 1999). Many of the competency items are ultimately derived from the work of Fleishman and his associates (Fleishman & Quiantance, 1984).

Field validation of 1SG and DET SGT tasks and competencies was accomplished through interactions with senior NCOs designated as subject matter experts (SMEs). At the time of the interviews, the SMEs were performing duties as 1SGs and DET SGTs. When presented with the competency list, the NCOs were asked to add any competencies deemed as required but not listed. Interviews were conducted at Fort Bliss, Texas. Task and competency lists were then forwarded to the Army Training and Doctrine Command (TRADOC), ARI and USASMA for review and comment.

Two types of computer-administered surveys were developed: Job Incumbent and Training Emphasis. ARI mandated the use of the Raosoft^TM application for disk survey development. First, a Job Inventory survey was developed. The primary purpose for this survey was to collect, from job incumbents, ratings of the significance of tasks and competencies to their jobs. Job incumbents were requested to base their ratings on the importance of each task as related to the objectives of their jobs. The design for this survey required completion by soldiers having appropriate duty positions. Further, the design required responses to all questions related to tasks, while responses to questions related to competencies were strictly on a voluntary basis. Participants were also provided opportunities to make open-ended comments relevant to the effectiveness of training in the 1SG Course and to list tasks they performed but that were not addressed in the surveys. A total of three thousand, one hundred and fifty (3,150) survey diskettes were distributed to field organizations.

Next, a Training Emphasis survey was developed. The primary purpose for this survey was to collect information from supervisors of the job incumbents relevant to how much emphasis each task should receive in the 1SG course and which competencies were required for job incumbents to perform effectively in their jobs. Supervisors were requested to base their ratings on four factors. These four factors were as follows: 1) the importance of each task to mission accomplishment, 2) the consequence of each task not being performed to standard, 3) the chance of each task being performed by course graduates, and 4) the consequence of delays in performing the task within six months following graduation.

The sample for these surveys required participants to be in specified command or supervisory positions, but otherwise replicated the sampling of the job incumbent surveys in terms of organizational levels. In addition to disk surveys, a test of an emerging technology used for administering surveys via the Internet was employed in part to expedite Reserve and Guard participation. This technology, titled "GenSurv," was developed for the U.S. Government under an earlier research and development project. A goal was set to have at least two hundred (200) GenSurv surveys completed via the Internet. ARI stipulated that administration of the 1SG surveys via the Internet was to be in addition to those surveys administered via diskettes. Soldiers targeted to participate in surveys administered via diskettes were assigned to organizations with war-fighting missions, while soldiers targeted to participate in surveys administered via the Internet were primarily assigned to organizations with support missions.
Distribution of surveys on diskettes was made to thirty (30) Active Duty, thirty (30) Army National Guard and thirty (30) Army Reserve component Brigades. Surveys were provided in sufficient numbers to accommodate administration to a predetermined sampling of 1SGs and DET SGTs and their supervisors within each Brigade, plus three subordinate Battalions. Specific organizations to receive surveys were determined by USASMA. One thousand, four hundred (1,400) 1SG Job Incumbent surveys were forwarded to specified organizations. One thousand, seven hundred and fifty (1,750) 1SG Training Emphasis surveys were forwarded to the same organizations.

Survey Responses
The percentage of returned surveys was less than the initially anticipated fifty percent (50%). Only a total of 730 incumbent disk surveys were returned to IJOA, for a twenty-three percent (23%) return rate. A breakdown of these surveys by component is provided in Table 1. Only 328 of the returned 1SG Job Incumbent surveys had been appropriately completed and were usable for analysis.

Table 1. Returned 1SG Survey Diskettes by Component and Type

COMPONENT /YPE OF SURVEY	NUMBER OF SURVEYS MAILED	NUMBER OF SURVEYS RETURNED	PERCENT OF SURVEYS RETURNED
Active (1SG Job Incumbent)	480	103	22%
Active (1SG Training Emphasis)	600	112	19%
Guard (1SG Job Incumbent)	480	169	37%
Guard (1SG Training Emphasis)	600	205	36%
Reserves (1SG Job Incumbent)	480	67	14%
Reserves (1SG Training Emphasis)	600	74	12%

While a relatively small number of soldiers were expected to participate in the GenSurv surveys (200), via the Internet, the number was actually greater than those participating in the disksurveys. Nine hundred (900) soldiers logged into the 1SG Job Incumbent survey. Four hundred and ninety five (495) soldiers actually completed the 1SG Job Incumbent survey to the point that data were usable for analysis. This represents a fifty five percent (55%) completion rate. The number of surveys completed by component and type are portrayed in Table 2, Completed Internet Surveys.

Table 2. Completed 1SG Internet Surveys by Component and Type

COMPONENT / TYPE OF SURVEY	NUMBER OF SURVEY LOGINS	NUMBER OF SURVEYS COMPLETED	PERCENT OF SURVEYS COMPLETED
Active (1SG Job Incumbent)	642	398	62%
Active (1SG Training Emphasis)	259	152	59%
Guard (1SG Job Incumbent)	124	92	74%
Guard (1SG Training Emphasis)	44	29	66%
Reserves (1SG Job Incumbent)	9	5	56%
Reserves (1SG Training Emphasis)	5	3	60%

The automated occupational data analysis tool, atCODAP, was used to support analysis of survey responses. Significant results are those related to the experience of job incumbents (hours worked per week, deployments, etc.), their opportunities to attend the 1SG Course and how they perceive the effectiveness of the course. The vast majority of respondents who reported having completed the course indicated the 1SG Course well prepared them to perform their jobs.
A significant area of interest expressed by 1SG Course managers prior to conducting the survey was whether soldiers in operational units had sufficient access to computers (with specified capabilities and peripherals) and the Internet to enable expanded implementation of modern training delivery technologies. Analysis of background items revealed availability varies significantly by component and that multiple delivery media may continue to be required for some time, pending increases in computer and Internet availability.

Training Emphasis Ratings
Of particular interest is how supervisors of 1SG/DET SGT job incumbents (Commanders and Command Sergeants Major) perceive the emphasis each duty and task should receive in the 1SG Course. The final sample of TE raters included 542 supervisors (358 completed disk TE surveys, 184 provided their ratings via the internet). Overall, about 64% of TE raters were commanders and 36% were CSM. The vector of group mean TE ratings for tasks for Commanders and CSM correlated .96 and the group mean ratings of competencies for these two types of raters correlated .97, indicating that their priorities are almost identical.
Disk Versus Internet Data Collection
This study also provided an ideal opportunity to compare data collected through disk-based surveys with such data collected via the Internet. The Internet has been utilized fairly successfully for data collection (Stanton, 1998). Recent research on organizational surveys with paper-and-pencil and via the Internet yielded "nearly identical coefficient alphas and item mean scores" but greater item response variability for the Internet (Neilsen & Halfhill, 2000). These authors recommended further research.
In this study, the key question is how similar or different the composite job description of the First Sergeant position is produced using each survey tool; thus the level of analysis is the group job description. The results of this comparison of First Sergeant job description derived from disk-based surveys and internet surveys are shown below. The correlations were computed separately for the task list and the competencies list since they involved different rating scales. In addition, data are separated into the three major types of jobs (company versus headquarters units, and detachments) to control for variance due to type of unit. The internet cases were restricted to personnel in "warfighting" units so as to be comparable with the disk sample.

Table 3. Disk-Based vs. Internet Survey Comparison

These data strongly suggest that group descriptions derived from data collected via the internet and disks are essentially the same. The few units of the Army spearheading the adoption of Digital Systems tasks, however, were all sent disk surveys rather than participating in the internet survey; hence, one of the nine duties was not comparable. The only figure below .90 is for Detachment Sergeants, and even this correlation is much higher than normally considered acceptable for job analysis data (.65-.75 - See McCormick, 1979) or what might occur by chance (i.e., Monte Carlo studies - see Harvey & Wilson, 1998).
When correlations are computed for each section of the task inventory, the differences (lack of agreement) between the Detachment Sergeant groups can be identified as involving Digital Systems, where very few Det. Sgts are involved with the new digital technologies. For most duty areas, agreement between the groups are comparable to those for Company and Headquarters First Sergeants.

Conclusions

The correlations among group mean job vectors in this analysis suggests that there is almost no difference in group descriptions, either for task lists or competencies. The clear inference which can be made for such results is that the two quite different data collection methodologies are yielding the same information. Thus, it is very realistic to merge such data into a single database for analysis, and to use the resulting information with confidence in making training decisions.
This study demonstrates that merging occupational data from disks and Internet surveys is both realistic and practical. Based on this experience, additional ARI studies are proceeding employing this multi-method data collection approach (Sergeants Major, Command Sergeants Major, etc.). The multi-method approach helps improve participation of various groups who are geographically or organizationally remote, thus presumably improving critical training decision making and other human resource utilization and management issues.

References

Fleishman, E.A., & Quiantance, M.K., (1984). Taxonomies of Human Performance: The Description of Human Tasks. New York, Academic Press.
Harvey, R.J., & Wilson, M.A. (1998, April). Monte Carlo baselines for interrater agreement when rating KSA requirements: How much is enough. Presentation in the symposium, R. J. Harvey (Chair), Measurement issues in job analysis: Good news and bad news. Annual conference of the Society for Industrial and Organizational Psychology, Dallas, Texas.
McCormick, E.J. (1979). Job analysis: Methods and applications. New York, New York; AMACOM, a division of the American Management Association.
Nielsen, T.M., & Halfhill, T. (2000, April). Organizational survey data collected via the Internet: Is it really the same? Poster presentation at the Annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA.
Peterson, N.G., Mumford, M.D., Borman, W.C., Jeanneret, P.R., & Fleishman, E.A. (1999). An Occupational Information System for the 21^st Century: The Development of the O*NET, Washington, D.C., American Psychological Association.
Stanton, J.M. (1998). An empirical assessment of data collection using the Internet. Personnel Psychology 51:709-729.

posted December 18, 2000

Next Paper