The 737th Training Group at Lackland AFB, TX, is implementing portable computer terminals into Basic Military Training (BMT) to automate many of the paper and pencil processes performed by their instructors. For example, each terminal provides access through a network server to the instructor’s lesson plans, relevant Air Force regulations, standard forms that the instructor needs to complete on a regular basis, e-mail, and word processing, spreadsheet, presentation graphics, and data-base management software. The effort involves optimizing the human-terminal interface and efficiency of the software, designing instructor training, and identifying the costs and benefits associated with use of the new technology. Subject matter experts developed tailored task lists describing the activities performed by the instructors. Computer-based surveys will be used to evaluate savings or costs in hours and minutes associated with automating tasks performed by BMT instructors and the instructors’ attitudes and opinions concerning the software, hardware, and changes in their work processes.

The Armstrong Laboratory’s Human Resource Directorate (AL/HR) has been conducting research on the best methods to use in automating occupational-type surveys. One area of particular interest has been in developing valid and reliable estimates of the amount of time that individuals spend performing the tasks associated with their jobs. As a result, the Air Force is in the process of automating the administration of occupational surveys. The paper-and-pencil surveying process is very time consuming and labor intensive and results from experimental studies show that automated tools and procedures can radically improve the process. In particular, a feasibility study of the use of computer-based methods to gather occupational survey data in the operational Air Force conducted in the Aircraft Fuel Systems and Information Management specialties by Institute for Job and Occupational Analysis (IJOA), USAF Occupational Measurement Squadron (USAFOMS), and AL/HR researchers demonstrated that automating data collection has the potential for significant cost and time savings while still delivering comparable data quality (Mitchell, Weissmuller, Bennett, Agee & Albert, 1995).

Automation has also facilitated exploration of new methods of gathering estimates of actual time in hours and minutes spent performing each task by each respondent. One of the scales that has exhibited promising results in gathering this information is the Actual Time Spent (ATS) Scale. In using the ATS Scale, each respondent is asked how frequently he/she performs each task comprising his/her job during some time period such as a week, month, or year and how long it takes on average for him/her to perform the task once, excluding delays, interruptions, or breaks. The product of these two components of time is the estimate of the total time spent by the respondent on that task. Then, the tasks are fed back to the rater in high-to-low order on total time spent for evaluation and revision of estimates and automatic recalculation and reordering of tasks (until the respondent is satisfied with the results). Therefore, the actual time spent procedure decomposes total time into its two basic components: "time to perform a task once" and "frequency of performance." "Frequency" has been shown to be a highly reliable counter mechanism, a perceptive ability that is consistently more accurate than that which governs our perception of time (Hasher & Zacks, 1984). In addition, "time to perform a task once" has the advantage of representing an average or median value rather than a total. For example, if you were to look at a long column of numbers, you could more quickly approximate a reasonably accurate mean or median value than a sum for that set of numbers.

A laboratory test of automated occupational survey procedures for 572 subjects across 67 Air Force specialties conducted at the Armstrong Laboratory’s Experimental Testing Facility at Lackland AFB has provided evidence for the viability of the ATS Scale (Albert, et al., 1994). For example, the time1/time 2 reliability with surveys being administered approximately two weeks apart was .65 for the Actual Time Spent Scale versus .55 for the Nine-Point Relative Time Spent Scale, the actual time spent estimates were judged by respondents to be more accurate than the relative time spent estimates (p<.001), 70% of the tasks that had been identified as tasks performed at least once a week in the actual time spent procedure were also identified as having been performed by the respondent within the last five working days, the relationship of weekly versus not weekly and recently performed versus not recently performed was significant (p<.001), and tasks performed within the last five working days were associated with significantly higher frequencies than tasks not identified as being performed within the last five working days (p<.001). Further evidence of the validity of actual time spent estimates is provided by job performance measurement data where subject matter expert estimates of time to perform correlated .79 with clock time (Harville, 1996).

A number of field applications of the automated survey technology have been accomplished so far. These include: (a) an impact analysis of the reengineering of the Midwest Region of the Defense Commissary Agency (DeCA); due to this successful application, it is currently being used by personnel in the European Region of DeCA to reengineer their organization; (b) needs assessment and requirements prioritization of job content for advanced training technology development for Reserve Medical Technicians; (c) determining training efficiency and effectiveness for Radiology Technicians; and (d) a Security Police study which showed that the software is useful for gathering task-based time spent data at the squadron level and subject matter experts can accurately tailor a task list for the specific requirements of an organization although development of the list is very time consuming. A key finding across all studies is that supervisors and subordinates have very positive reactions to the software as a data collection tool. Exit surveys show that instructions are clear, automated surveys are preferred to paper-and-pencil surveys, time to complete is reasonable and typically shorter than paper and pencil, and others in the career field can be expected to successfully complete the survey.

Based on their extensive experience in automating both occupational-type surveys and general purpose surveys, AL/HR personnel have been requested to evaluate savings or costs associated with the introduction of portable computer terminals into the 737th Training Group (TRG) at Lackland AFB and to determine instructors’ attitudes and opinions concerning the software, hardware, and changes in their work processes. The 737th TRG is implementing the terminals into Basic Military Training (BMT) to automate many of the paper and pencil processes performed by their instructors. For example, each terminal provides access through a network server to the instructor’s lesson plans, relevant Air Force regulations, standard forms that the instructor needs to complete on a regular basis, e-mail, and word processing, spreadsheet, presentation graphics, and data-base management software. To make the transition to automation smoother, AL/HR is also providing expertise in optimizing the human-terminal interface and efficiency of the associated software and in designing training for new users of the terminals. In addition to expanding AL/HR’s experience with computer-based surveys, this effort will provide a good test-bed for the automated ATS scale.

Focus group discussions were held with instructors in the initial squadron selected for implementation of the new technology to identify work processes that will benefit from the new technology and to identify difficulties and areas of concern that will need to be addressed prior to full implementation. Although attitudes concerning implementation of the new technology were mixed, instructors with significant computer experience and experience in use of the technology were positive about its potential to lessen their workload. In particular, they liked the e-mail and record maintenance features of the software. It became clear that an effort should be initiated to minimize requirements to use the accompanying keyboard by making as much use as possible of the terminal’s "touch screen" capability.

An important part of this research effort will focus on determining how much time is saved by automating various tasks performed by BMT instructors. The ATS scale will be used in this evaluation. With the help of subject matter experts (SMEs) assigned to the 737th TRG, task lists were created containing all job-related tasks performed by users of the new technology, which include Military Training Instructors (MTIs), trainers of MTIs, MTIs in training, and Military Drill and Ceremony (MDC) NCOs. The starting point for task list development was a task list developed by USAFOMS, which the SMEs tailored to the specific requirements of this study. Instructions given to the SMEs specified that the tasks should be worded such that they are time ratable using the ATS scale (that is, accurate frequency and duration estimates can be provided), the task lists should include all activities performed by the respondents, and the tasks should be mutually exclusive. The task list was reviewed by senior personnel assigned to the 737th TRG for accuracy and completeness.

All job related tasks are being included in the survey so that the total workload requirements of personnel assigned to the 737th TRG can be determined. In addition, time estimates associated with the tasks that can be performed using the new technology will be used to estimate the workload savings associated with introduction of the new technology. To accomplish this, surveys will be administered to personnel prior to implementation of the new technology and after they have had an opportunity to become familiar with its capabilities. Thereafter, periodic survey administrations are planned to track the impact of automation longitudinally. The idea is to have continual feedback from users of the new technology on how its capabilities can be improved, thereby optimizing its implementation.

Two other surveys were developed that will be administered in conjunction with the occupational survey. The first of these will provide background information on respondents such as their rank, job assignment, experience and attitude towards using computers, and experience and opinions concerning the implementation of the new technology. The responses should provide information on whether or not there is a strong relationship between experience with and attitudes towards using computers and attitudes towards the new technology, areas where additional training would be helpful, identifying other areas where the new technology is potentially beneficial, and where future efforts should be directed on making the new technology more user friendly. The second survey requests opinions concerning use of the automated survey and the ATS scale. These responses will provide information on the respondents’ perceptions of the accuracy of their resulting job descriptions and potential improvements to the survey software. Many respondents will also be interviewed on-site to probe particular areas of interest identified from analyses of the survey responses.

A briefing was prepared that will familiarize respondents with the purpose and importance of this research effort and will be delivered to them immediately prior to survey administration. After a pilot test administration of the automated survey to approximately 15 personnel from all job categories, it will be administered to as close to 100% of the personnel assigned to those jobs as possible. As much of the survey administration as possible will be proctored by 737th TRG personnel; however, AL/HR will provide personnel to proctor some survey administrations if necessary. AL/HR personnel will perform all data analyses and will provide a briefing on the baseline results to 737th senior leadership early in FY98 and periodically throughout the effort as additional data is gathered.

AL/HR is also supporting an effort to make the software associated with the new technology more efficient. In particular, efforts are underway to sort the logic of the software so that it can be better maintained, to implement a naming convention that will aid in dataflow analysis, and to rearrange the layouts of forms to make them easier for training instructors to use. AL/HR personnel are also providing expertise in improving the training for new users of the technology.

AL/HR and IJOA personnel are collaborating with the USAFOMS to field an operational automated survey capability. Testing and final refinements to the technology for USAFOMS applications are underway. Near term efforts will focus on implementing a survey administration and data gathering capability over the Internet. This will broaden the applicability of the new tool, reduce surveying costs, and will permit survey data to be automatically uploaded into data analysis software at USAFOMS. We are currently developing a cooperative research agreement to commercialize the survey technology. Future plans include developing adaptive surveys, expanding occupational measurement methods, and applying recent advances in the use of intelligent search agents to identify cross-career field related task and performance information. These efforts will help to further decrease survey administration time and increase data precision, thereby improving the timeliness and the quality of the data used in many research efforts conducted by AL/HR and throughout the Air Force. The accuracy and timeliness of these estimates is becoming more important with the continued downsizing of the Air Force and the broadening range of responsibilities associated with various jobs. In particular, these estimates are useful for many Air Force applications such as costing training outcomes, learning curve development, costing force composition and changes, job design and process reengineering, comparing work activities to civilian analogues, and linking individual and team performance to mission requirements and effectiveness.

Albert, W.G., Phalen, W.J., Selander, D.M., Dittmar, M.J., Tucker, D.L., & Weissmuller, J.J. (1994, October). Large-scale laboratory test of occupational survey software and scaling procedures. Paper presented at the 36th Annual Meeting of the International Military Testing Association, Rotterdam, The Netherlands.
 

Harville, D.L. (1996, November). Analyzing work sample task performance using three data sets. Paper presented at the 38th Annual Meeting of the International Military Testing Association, San Antonio, TX.
 

Hasher, L. & Zacks, R.T. (1984). Automatic processing of fundamental information: The case of frequency of occurrence. American Psychologist, 39 (12), 1372-1388.
 

Mitchell, J.L., Weissmuller, J.J., Bennett, W.R., Agee, R.C., & Albert, W.G.(1995, October). Final results of a field study of the feasibility of computer-assisted occupational surveys: Stability of task and job information. Paper presented at the 37th Annual Meeting of the International Military Testing Association, Toronto, Canada.
 
 
 

 Next Page 

Back to IJOA Home Page