Several innovations in software, instructions, and procedures have resulted from the need to deal with long, multilevel task lists, under the assumption that people tire quickly and their responses degrade in quality and validity. One approach to such problems is to let individuals prioritize inventory items (duties, modules) to ensure that all "important" parts of their job are dealt with early in the survey administration process. Initial trials of this new
procedure in two major studies suggests that this is a very workable safeguard against the "fatigue effect" and that it probably produces improved quality and validity. Additional operational experiments with several alternative procedures are reported which may further enhance data quality.  

The substantive type of problem is associated with uncertainties from switching elicitation and recording methodologies. In other words, "Does the bubble-marking process systematically affect the quality of the data?" In these substantive issues one is obliged to conduct a careful scientific analysis to ensure continued (or improved) validity. The transition impact on the 1970s survey respondent was minimal and, after minor logistical adjustments, comparable data from old to new systems were easy to demonstrate. The optically scanned occupational survey booklet became the standard for the high-volume Job Incumbent (JI) data, with keypunched booklets being reserved for low-volume supplementary ratings from Subject Matter Experts (SMEs) such as Task Learning Difficulty (TD) and Training Emphasis (TE).

The transition reviewed today is a much more complicated situation. The present goal is more than moving occupational survey technology from optically scanned survey booklets of the 1970s to computer-diskette surveys of the mid-1990s (Mitchell, J. L, Weissmuller, J. J., Bennett, W. R., Agee, R. C, & Albert, W. C., 1995). Today’s complex environment is mandating a suite of survey technologies using ever-evolving multi-method approaches including combinations of paper-and-pencil (manual data entry), optical scan sheet, computer-diskette, email, local area network, and Internet surveys (Stanton, 1998). The goal is to bring all available tools and techniques to bear on the three pillars of occupational survey validity: maximum population coverage, effective elicitation of information, and faithful linkage of data into the master database.

In designing new systems, one must be mindful of the special requirements of the job at hand. The focus here is on servicing occupational analysis (OA) surveys that, as a group, have very special needs. Occupational analysis surveys target individual job incumbents within a job family because every position is potentially unique. Standard personnel tracking systems tend to track individuals by "job title" and "paygrade" in a job family. In many cases, job titles are not highly reliable indicators of actual job content. Moreover, as jobs change with new tasks or new technologies over time, the job titles are not always adjusted to reflect changed content. When surveys are constructed for only a single purpose, say common training for an entire job family, or training evaluation for a specific course, the standard personnel tracking system may be sufficient to support rational, stratified sampling (Tsai, 1993; Mitchell, Tucker, Fast, Bennett & Albert, 1997). Such focused, single-purpose surveys often do not reveal changing job content, nor are they designed to do so. Further, as training budgets shrink and we need to target training so as to optimize preparation for specific jobs (subgroups, etc.), then generalized training surveys may not be sufficient.

One goal of responsible occupational analysis is to undercover reliable, identifiable, subgroups with stable commonalties that can be used for recruitment, selection, training, training evaluation, promotion, restructuring, multi-level organizational modeling, up sizing, and out placement. When occupational data are used for applications across the entire personnel spectrum, discussions about appropriate sampling levels become less than productive. The basis of scientific sampling is knowledge of the population’s distribution across underlying dimensions – and the purpose of occupational analysis is to help establish the identity of those dimensions. For this same reason, the survey methods must be capable of surveying personnel in all locations, not just central-site office workers who tend to have computers and hence, paper and pencil forms may never go out of style.

OA Surveys are Long & Complex

OA surveys tend to be very large, typically 1,000 to 2,000 responses. OA surveys are growing in size and are servicing increasingly diverse job families. The computer’s ability to support complex branching within a survey is a new opportunity that will save countless hours at the computer screen for future survey respondents (Phalen, Tucker, & Weissmuller, 1993). Paper and pencil surveys can mimic this ability by embedding written routing instructions like programmed instruction texts of old. Whereas routing instructions in paper and pencil forms can be misapplied or ignored by survey respondents, in the computerized forms compliance is automated and immediate. Perhaps more importantly, reproduction costs will limit the enthusiasm of the agency conducting the survey and the physical size of a "programmed instruction" survey in the person’s in-basket will crush the enthusiasm of the survey recipients.

Unfortunately, the most portable survey format (the manually entered paper and pencil format) is the most costly to data enter and delays the analysis phase of the project. The optical scan sheet approach is relatively fast in creating a data base from the returned booklets, but the booklets themselves are expensive and involve a greater lead-time for set up. Local area networks and email systems tend to present a host of site-specific barriers to automated survey distribution and data flow-back. Although telephone survey systems may be practical for "quick response" attitude surveys, the typical operational Occupational Analysis (OA) survey contains well in excess of 1,000 items and thereby precludes use of the telephone in this area.

When some survey methods have new opportunities that other methods can’t duplicate, thought needs to be given to merging multi-method data into a master database. In a recent example where new opportunities are exploited, a computer administration system was used to provide raters with real-time feedback on actual time-spent ratings. A paper-and-pencil "parallel" form would not be able to limit or interactively guide a rater’s responses and without this "rater training" gross errors in time ratings are the expected norm (Phalen, 1995). The dilemma then is, should the coverage pillar be sacrificed and ignore job incumbents who, because of their assignment location, could only use paper and pencil forms, or should some analytical accommodation be developed to recognize and integrate all rating paradigms into the master database?

 
Self-Prioritized Inventory

A disk-based survey of 20,000 cases was conducted in April. It was deemed important to both maximize response validity (by combating fatigue effects) and minimize job incumbent time spent taking the 900 item survey. A two-level branching design was used. The master survey was pre-coded for items appropriate to each of five paygrade groups. Once a person indicated his or her paygrade, the appropriate duty areas were displayed and rated on an eight point "importance to job" scale. After the duty ratings, the survey software presented tasks under each selected duty, sequencing from highest rated duty to lowest rated duty area. This effort was successful in reducing the number of items each individual had to rate, and in controlling the total amount of time required to complete the survey.

In a recent study of Air Force Security Force patrolmen, incumbents were presented each major duty area in ordered sequence based on their own ratings of the degree to which the duties were "part of the job" (defined as time and effort required). This ensured that the main focus of the work was rated first (Holt, Mitchell, & Zuniga, 1998).

Early work on exploiting new scales within the automated survey environment also validated the technique of having job incumbent rate the importance of duty areas then present tasks under that self-prioritized order (Dittmar, M, Hand, D., Tucker, D., & Weissmuller, J. 1995). In that study it was shown that there is a statistically significant correlation between the "importance" ratings and the actual number of tasks selected within a duty. A second phase of the present study is collecting Training Emphasis ratings from raters in both automated and paper and pencil versions. As raters are expected to rate all tasks, no branching is used in either the automated or paper and pencil forms and parallel results are expected.
 
Lessons Learned

In a special purpose, 90-item job satisfaction survey, 200 job incumbents were surveyed in each of two job families (Weissmuller, Grimes, Siem & Kenny, 1997). Every person surveyed was sent a diskette with the computer-based survey and a printed copy of the survey. Incumbents were given the option of completing and returning either form. In both cases, 51% of the returns were on diskette. The other 49% had to be data entered. The data entry was accomplished by using a survey diskette and "transcribing" the answers per the prompts on the screen. The lessons learned from this include the fact that the people who used the paper and pencil forms ignored limits and instructions on the form. The automated survey format used for "data entry" had to be modified to record "invalid" responses coming in from the paper versions.

An observation was made that automated surveys had longer "write-in" comments than paper and pencil surveys. No demographic differences were found between the groups selecting "disk" or "paper". Various reasons have been offered to explain the difference with the most compelling being that the incumbents viewed the survey disk as more "secure" for personal comments. If this is the case, then it is the true that incumbents view the media as qualitatively different – and this speaks to the relative validity of data collected.

Albert, W.G., Phalen, W.J., Selander, D.M. Yadrick, R.M., Weissmuller, J.J., Dittmar, M.J., & Tucker, D.L. (1993, June). Research and development of computerized occupational survey administration software and scaling procedures. Proceedings of the Eighth International Occupational Analyst Workshop, pp 232-236. San Antonio, TX: USAF Occupational Measurement Squadron.

Christal, R.E. (1974). Collecting, analyzing, and reporting information describing jobs and occupations. (AFHRL-TR-74-19, AD-774 575). Lackland AFB, TX: Occupational Research Division, Air Force Human Resources Laboratory.

Dittmar, M., Hand, D., Tucker, D., & Weissmuller, J.J. (1995, February). Development of occupational measurement technology (OMT): Automated surveys, tailored task lists, and system documentation. Draft report submitted to Armstrong Laboratory, Human Resources Directorate (AL/HRMS) under Contract F33615-91-D-0010. San Antonio, TX: Metrica, Inc.

Holt, C. N., Mitchell, J. L. & Zuniga, J. (1998, October). Career Field Modeling: Estimating Time Utilization in Law Enforcement Patrol Jobs. Proceedings of the Military Testing Association (in press). Pensacola, Fl: Naval Education and Training Professional Development & Technology Center.

Hudspeth, D.R., Fayfich, P.R., & Price, J.S. (1990, November). Automating the administration of USAF occupational surveys. Proceedings of the 32nd Annual Conference of the Military Testing Association (pp. 70-75). Orange Beach, AL: Naval Education and Training Program Management Support Activity.

Mitchell, J.L., Tucker, D., Fast, J., Bennett, W., Jr., & Albert, W.G. (1997, October). Research and development of new occupational analysis and training evaluation technologies. Presentation in the symposium, J.S. Tartell and H.W. Ruck, co-chairs, Advanced Technology Research and Applications in Occupational & Training Analysis & Organizational Assessment, at the 38th annual conference of the International Military Testing Association (IMTA), Sydney, Australia (Proceedings in press).

Mitchell, J.L., Weissmuller, J.J., Bennett, W., Jr., Agee, R.C., & Albert, W.G. (1995). Final results of a field study of the

feasibility of computer-assisted occupational surveys: Stability of task and job information. Proceedings of the 37th annual conference of the International Military Testing Association (IMTA), pp. 231-236. Toronto, Ontario, Canada: Canadian Forces Applied Research Unit.

Mitchell, J.L., Weissmuller, J.J., Tucker, D.L., Waldroop, P., & Bennett, W. Jr. (1996, November). Development and application of a computer-assisted survey authoring tool for training needs assessment. Proceedings of the 38th Annual Conference of the International Military Testing Association. San Antonio, TX: Air Force Personnel Center, Armstrong Laboratory/Human Resources Directorate, and the Air Force Occupational Measurement Squadron.

Phalen, W.J. (1995). A critical evaluation of various procedures for estimating time spent. Proceedings of the 37th Annual Conference of the International Military Testing Association, pp. 418 - 423. Toronto, Ontario, Canada: Canadian Forces Applied Research Unit.

Phalen, W.J., Mitchell, J.L. (1993, June). Innovations in occupational measurement technology for the U.S. Military. In the symposium, Military Occupational Analysis: Issues and Advances in Research and Application (H.W. Ruck, chair). Proceedings of the Eighth International Occupational Analysts Workshop, pp 12-16. San Antonio, TX; USAF Occupational Measurement Squadron.

Phalen, W.J., Tucker, D.L, & Weissmuller, J.J. (1993, June). A multi-purpose procedure for computer-adaptive administration of task lists. Proceedings of the Eighth International Occupational Analysts Workshop. San Antonio, TX; USAF Occupational Measurement Squadron.

Sage, J.E. (1993, June). Automated self-report questionnaires: Highlights of a CODAP pilot study. In the symposium, Military Occupational Analysis: Issues and Advances in Research and Application (H.W. Ruck, chair). Proceedings of the Eighth International Occupational Analysts Workshop (pp 85-113). San Antonio, TX; USAF Occupational Measurement Squadron.

Stanton, J.M., (1998). An empirical assessment of data collection using the internet. Personnel Psychology, 51:709-725.

Tsai, C. (1993). Seeking out engineering training needs: A computer-based survey can provide answers. Technology Today, June 1993, Southwest Research Institute.

Weissmuller, J. J., Grimes, G. R., Siem, F. M., & Kenny, J. E. (1997, June). Automated Surveys or Paper and Pencil: A Real Choices. Proceedings of the Tenth International Occupational Analysts Workshop (pp 217-223). San Antonio, TX; USAF Occupational Measurement Squadron.

Weissmuller, J. J. , Kauffman, B. (October 1976). Data Processing Problems in Collecting Job Survey Information. Proceedings of the 18^th Annual Conference of the Military Testing Association, pp 783-796. Gulf Shores, Alabama, Naval Education and Training Program Development Center.