The Air Force Basic Military Training (BMT) Group at Lackland AFB, TX, has been examining the impact of technology on the daily work activities of Training Instructors (TIs) and their supervisors. The group has implemented portablecomputer terminals (CruisePads) into BMT to automate many of the paper andpencil processes performed by their TIs. Data collection, using a BMT-developed task list and an actual time spent (ATS) Scale, was performed prior to, andshortly after, the new technology was implemented to determine if there would be timesaving using the CruisePad technology. Participants’ commentswere also collected to identify problems and potential solutions. In addition,BMT,in collaboration with the Air Force Research Laboratory (AFRL) has developed a prototype electronic classroom where the impact of automation on studentfeedback assessments and on end-of-course testing is beingassessed. This paper will highlight some of this work, concentrating on the impact study of the CruisePad technology.  

One of the scales that has exhibited promising results in gathering this information is the Actual Time Spent (ATS) Scale (Phalen, 1995). In using the ATS Scale, each respondent is asked how frequently he/she performs each task comprising his/her job during some time period such as a week, month, or year and how long it takes on average for him/her to perform the task once, excluding delays, interruptions, or breaks. The product of these two components of time is the estimate of the total time spent by the respondent on that task. Then, the tasks are fed back to the rater in high-to-low order on total time spent for evaluation and revision of estimates and automatic recalculation and reordering of tasks (until the respondent is satisfied with the results). The ATS procedure decomposes total time into its two basic components: "time to perform a task once" and "frequency of performance." "Frequency" has been shown to be a highly reliable counter mechanism, a perceptive ability that is consistently more accurate than that which governs our perception of time (Hasher & Zacks, 1984). In addition, "time to perform a task once" has the advantage of representing an average or median value rather than a total. For example, if you were to look at a long column of numbers, you could more quickly approximate a reasonably accurate mean or median value than a sum for that set of numbers. The ATS scale has considerable potential for assessing the impact of new technologies and other operational studies.
 
 

737^th personnel wrote their own CruisePad software (in Microsoft Access), provided initial CruisePad training and brought a few of the CruisePads on line. It was soon apparent that help was needed, however, and AFRL agreed to assist with identifying and solving problems associated with this effort. In addition to providing critical programming assistance, AFRL offered expertise in optimizing the human-terminal interface and efficiency of the associated software, assisted in designing training for new CruisePad users and offered guidance in solving hardware-related problems. In addition, AFRL completed an evaluation of the impact of the implementation of the CruisePad technology on tasks normally performed by TIs. This exercise expanded AFRL’s experience with computer-based surveys and provided a good test-bed for the automated ATS scale.

The primary objective of the Operational Study was to determine if the implementation of the CruisePad technology has had an impact on the amount of time TIs spend on their normal work tasks. It was hypothesized that there would be an overall reduction in time spent on tasks; it was also anticipated that there would be a large amount of variance due to the turmoil created with the implementation of the CruisePad technology.

Methodology

With several experienced TIs serving as Subject Matter Experts (SMEs) and using the Air Force Occupational Squadron-developed TI task list as a starting point, the Institute for Job and Occupational Analysis (IJOA), under contract to the AFRL, developed a 126-item task list which was incorporated into a computer-based survey using the Air Force Survey Authoring System (AFSAS; see Mitchell, Weissmuller, Tucker, Waldroop, & Bennett, 1996). An ATS Scale (Albert, Phalen, Selander, Dittmar, Tucker, & Weissmuller, 1994; Phalen, 1995) was used to collect time estimates by incumbents. In addition to the task list and background sections, several open-ended questions were included with the expectation that comments from CruisePad users would assist in identifying and improving the CruisePad system.

Identical surveys were administer on two occasions, once just prior to the full implementation of the CruisePad in March, 1998, and three and a half months later, in early July, 1998. During the intervening period, there were several significant changes to the software (notably the inclusion of signature recognition software) and hardware (notably the FM antenna system). Also during this period, several TIs utilized both the existing paper and pencil methodology and the new CruisePad system for a portion of that four-month period. Additionally, several technical problems identified during the first survey were in the process of being addressed at the time of the second survey; only a portion of those problems had been resolved by then. Both survey administrations were proctored by AFRL, IJOA and BMT staff and were completed in several group sessions held in a computer laboratory in the 737^th BMT Squadron Headquarters. The first sample consisted of 125 TIs while the second sample consisted of 112 TIs. Two cases were eliminated from the first administration and one case was removed from the second study due to response patterns that suggested that the respondents misunderstood the requirements of the survey. In addition, 12 data points were removed from the first survey and 9 data points were removed from the second survey after being identified by BMT SME personnel as providing obviously out-of-range responses.

A similar decrease in the number of hours incumbents reported working per week (in response to a background item asking for an estimate of "hours worked per week") was observed (see Table 2).

The ATS data document average timesaving of just over three hours per week per TI from the first survey to the second, or about 157 hours per year per TI. Table 2 reveals that when estimating the hours worked per week (a more global estimate), the average TI said he/she was working about 4 hours less per week. Both these estimates are in the same direction and in about the same magnitude, and show a trend towards less time being spent on tasks with the use of the CruisePad. Of the 108 tasks performed by the majority of TIs, 56 showed a decrease in actual time spent, while 52 reflected an increase. The estimated savings may be an underestimate since a portion of the respondents was actually using both paper and pencil and CruisePad methodologies. Table 3 shows a direct comparison of the top 12 tasks from each administration.

Considerable information was also provided from the comment section of the survey. Table 4 summarizes these results.

Results

A comparison of actual time data and the more global estimate (survey item asking for "hours per week worked") showed both estimates to be similar (see Table 5).

The experimental study confirms that ATS estimations were consistent with a more global estimate of time spent working on tasks, indicating that further study of ATS is warranted. As evidenced by the high standard deviations for both surveys, there may be some problem(s) with how the ATS data collection was operationalized. Future development of ATS and its related software should address these overestimation issues.

In addition to the current operational study, BMT, in collaboration with the Air Force Research Laboratory (AFRL) has developed a prototype electronic classroom where the impact of automation on student feedback assessments and on end-of-course testing is being assessed. Customer surveys are provided to students at the 2^nd and 5^th weeks of training to track satisfaction and to identify potential conflict areas. Typically done as paper and pencil surveys, BMT is now using distributed data collection and reporting for these surveys. For the automated testing, AFRL and BMT have developed a preliminary test item bank and automated version of the current paper and pencil end-of-course test. Researchers are presently applying Item Response Theory (IRT) methodology to develop and adaptive testing item bank for future implementation. AFRL has designed and implemented a comprehensive evaluation study of these innovations and their impact on BMT work processes, products and overall efficiency.

Albert, W.G., Phalen, W.J., Selander, D.M., Dittmar, M.J., Tucker, D.L., & Weissmuller, J.J. (1994). Large-scale laboratory test of occupational survey software and scaling procedures. Proceedings of the 36th Annual Conference of the International Military Testing Association. Rotterdam, The Netherlands: European Members of the IMTA.

Hasher, L. & Zacks, R.T. (1984). Automatic processing of fundamental information: The case of frequency of occurrence. American Psychologist, 39 (12), 1372-1388.

Mitchell, J.L., Weissmuller, J.J., Bennett, W.R., Agee, R.C., & Albert, W.G. (1995, October). Final results of a field study of the feasibility of computer-assisted occupational surveys: Stability of task and job information. Paper presented at the 37th Annual Meeting of the International Military Testing Association, Toronto, Canada.

Mitchell, J.L., Weissmuller, J.J., Tucker, D.L., Waldroop, P., & Bennett, W., Jr. (1996, November). Development and application of a computer-assisted survey authoring tool for training needs assessment. In the symposium, H. W. Ruck, Chair, Recent Research and Applications in Training Needs Assessment and Evaluation. Proceedings of the 38th Annual Conference of the International Military Testing Association, pp.486-491. San Antonio, TX: Air Force Personnel Center, Armstrong Laboratory Human Resources Directorate, & the Air Force Occupational Measurement Squadron.

Phalen, W.J. (1995). A critical evaluation of various procedures for estimating time spent. Proceedings of the 37th Annual Conference of the International Military Testing Association, pp. 418 - 423. Toronto, Ontario, Canada: Canadian Forces Applied Research Unit.
 

 

Next Paper