The initial study of Internet surveying was conducted for the Air Force Research Laboratory under the GenSurv development project. In this initial survey, the Air Force Behavioral Scientist project was used as a vehicle with which to study the effect of presentation order on survey responses. Respondents linking up with the GenSurv software via the Internet were randomly assigned to one of two treatment groups. Treatment One subjects hadtasks presented in traditional inventory order while Treatment Two subjects saw tasks in order of the rated importance of duties ("to what degree are the tasks in this duty a part of your job?"). The survey also included logical branching around items which did not apply (based on some earlier reported response, such as whether the individual was an officer, enlisted, or civilian employee). Where such branching occurs, a response can be forced in order to proceed to only relevant questions. Such logic branching minimizes the amount of time required to complete the survey, and focuses respondents on those items of most interest. Other capabilities tested and refined in this project included scrolling lists (base assigned, academic major, etc.), background images, header graphics, and free ranging write-in comment sections. In pilot testing, the background image proved to be distracting to respondents, and was removed prior to operational data collection.
 

This project was successful in terms of fairly rapid data collection (one month), immediate availability of data for analysis, and random assignment to experimental treatment. Respondents proved quite capable of utilizing the enhanced write-in comment capability in a constructive way. Survey participants also recommended some structural changes in future surveys, including a "look up" capability to display the tasks involved in each duty they were asked to rate. In terms of the research objective, no significant differences in time to complete the survey were found between groups, although there was a trend in the expected direction. Job descriptions of the two groups were essentially the same. The study concluded that this "rated duty" presentation capability might be extremely useful with longer, more complex task lists. Exit survey indicated that respondents preferred Internet surveys and felt that the time needed to complete the survey was reasonable.

A second Internet project completed this summer was the U. S. Army Petroleum Officer training analysis survey. In this project, a normal Army Research Institute (ARI) occupational analysis survey was developed to collect information for the Quartermaster Corps and the Petroleum Officer course.
 

Notification of officers in this project was done by Course Personnel at the school, and entry into the survey was made via a page on the school’s website. Entry was password protected and each respondent selected a unique User ID to permit reentry if they had to stop for any reason, or were timed out. Some difficulty was encountered in notifying individuals to participate, so that the initial one month data collected had to be extended at the request of the school to collect a sample sufficient for their needs. The Internet survey itself, however, was successful and demonstrated a number of very useful features. Exit surveys again found that respondents found the time required to complete the survey was reasonable, and very few reported any difficulty in getting access to the Internet in order to complete the survey.

One operational Air Force occupational survey was also posted to the Internet early this fall, at the request of the Air Force Occupational Measurement Squadron (AFOMS). Data collection for this project is continuing, and data analysis has not yet begun. Some problems were encountered at some bases (particularly in Alaska) with local base restrictions on Internet access (only .mil addresses could be opened). It was solved by AFOMS coordinating a temporary change of policy at the base for the duration of this survey. This type of restriction of access would be eliminated when GenSurv is fully operational on the AFOMS computer (rather than a contractor’s system). Additional AFOMS Internet Surveys are anticipated this fall.

A third major project this year is the U. S. Army Band survey which became operational last Thursday. The basic band course is taught by the U.S. Navy in a joint school, but the Army Band staff want data on field utilization of Band personnel, performance level requirements, extra duty assignments, and deployment data as well as basic instrument qualifications. This makes for a more complex survey, but one which has the potential of multiple analysis possibilities and application in critical training and policy decision making.
 

In this project, additional layers of logic branching were used, so that respondents select first the type of band to which they are assigned and then see only the units and bases where those types of bands are used. This eliminates the need for long scrolling lists of units and facilitates quicker movement through the survey. Since this is expected to be a substantial (n = 4,000) survey (Active Duty, Army National Guard, Army Reserve; possibly also members of other services), any reduction in the time required to survey will result in substantial time savings.

As with the Petroleum Officer survey, access to the Band survey is password protected and is entered via the Army Band website.

Since there is the potential for members of other services to participate, several pages of instructions have been posted on the Band website. The response of bandmembers has been exceptional since the survey was posted to the net last Thursday – this promises to be an interesting and challenging data collection and analysis project.

Additional Army Internet studies are anticipated. In the impending First Sergeant survey, conducted for the U.S. Army Sergeants Academy, Ft. Bliss, Texas, most of the surveys will be disk-based using RaoSoft delivery. An Internet form of the survey is also being completed at will be used to collect a good sample (n = 400) of cases, so that the equivalence of survey responses of the two forms of survey can be evaluated. The results of this analysis should be available for reporting by this time next year.

Several Internet surveys have been initiated this year, with a substantial degree of success. Some problems have been encountered in Internet access and we have learned a great deal about the web and web access in each of these studies. While the potential for time and cost savings are great, there are also potential problems which must be solved before this will be an efficient process. Website hosting is more involved and complex than anticipated, yet a number of efficiencies appear feasible, such as hosting surveys on military computers, creation of special .mil domains, co-location of surveying software with Internet Service Providers (ISP), etc. The potential for interactive data quality control are also being explored. We will be continuing to evaluate such possibilities and to refine Internet Survey authoring capabilities in the coming year.

Albert, W.G., Phalen, W.J., Selander, D.M., Dittmar, M.J., Tucker, D.L., & Weissmuller, J.J. (1994). Large-scale laboratory test of occupational survey software and scaling procedures. Proceedings of the 36th Annual Conference of the International Military Testing Association. Rotterdam, The Netherlands: European Members of the IMTA.

Mitchell, J.L., Weissmuller, J.J., Bennett, W.R., Agee, R.C., & Albert, W.G. (1995, October). Final results of a field study of the feasibility of computer-assisted occupational surveys: Stability of task and job information. Paper presented at the 37th Annual Meeting of the International Military Testing Association, Toronto, Canada.

Mitchell, J.L., Tucker, D.L., Waldroop, P., Bennett, W., Jr., & Tartell, J.S. (1997, June) a. Operational Implementation of the Air Force Survey Authoring System. In the symposium, Archie M. Smith, II, Chair, Innovation and Advanced Technology Research and Applications in Military Occupational Analysis. Proceedings of the Tenth International Occupational Analysts Workshop. San Antonio, TX: Air Force Occupational Measurement Squadron.

Mitchell, J.L., Tucker, D.L., Waldroop, P., Bennett, W., Jr., & Tartell, J.S. (1997, October) b. Operational Use of the Air Force Survey Authoring System. In the symposium, Advanced Technology Research and Applications in Occupational & Training Analysis & Organizational Assessment. Sydney, Australia: Australian Department of Defence.

Mitchell, J.L., Weissmuller, J.J., & Bennett, Winston, Jr.. (1999, May).  The impact of presentation order and sequence on survey time: OA R&D via the internet. In the symposium, Smith, Archie, Maj., Chair, New applications of quantitative occupational analysis data to support critical decision making and strategic planning.  11th International Occupational Analysts Workshop, San Antonio, Texas.

Mitchell, J.L., Gosc, B., Weissmuller, J.J., Tartell, D., Worstine, D., & Badey, B. (1999, May).  Common enlisted soldier tasks: A "muddy boots" database for revising multiple Army skill-level courses. In the symposium, Smith, Archie, Maj., Chair, New applications of quantitative occupational analysis data to support critical decision making and strategic planning.  11th International Occupational Analysts Workshop, San Antonio, Texas.

Holt, C.N., Mitchell, J.L., & Zuniga, J. (1998, October). Career field modeling: Estimating time utilization in law enforcement patrol jobs. Presentation in the symposium, J. L. Mitchell and J. S. Tartell, co-chairs, Evaluating innovations in training assessment and occupational modeling technologies, at the 40^th annual conference of the International Military Testing Association (IMTA), Pensacola, Florida

Back to IJOA main page

posted November 30, 1999