Various technologies exist for automating survey administration, but this paper is only concerned with the one in use at the Air Force Occupational Measurement Squadron (AFOMS). We have a long history with paper-and-pencil survey instruments and the salient point to make about them is: they work. We’ve kept our eyes on emerging technologies that might improve survey administration, but we’ve waited for one that offers improvement while adapting to the way we do business, rather than one which demands that we change our processes for the overriding goal of adapting to the technology. You’ve heard the saying "When the only tool you have is a hammer, every problem looks like a nail." Well, we’ve got a whole bunch of nails (surveys) which need driving, so we’ve been looking for a hammer. The Survey Authoring System is a hammer to drive our nails, not a Torxä screwdriver which makes us acquire a new set of screws before we begin.

The Survey Authoring System is a "black box" in the middle of our occupational analysis process. We already had the technology, program, and process for developing a survey’s content, both the task inventory and appropriate background questions for each specialty: the Task Inventory Analysis and Requirements Application (TIARA). We already had the technology, program, and process for analyzing collected data: the Comprehensive Occupational Data Analysis Program (CODAP). The black box in between TIARA and CODAP had to be plug-compatible on both ends if we were to be able to improve the overall process without having to tinker with the TIARA black box or the CODAP black box. The intermediate black box turns out to be the Air Force Survey Authoring System (AFSAS, commonly referred to as AUTHOR in its AFOMS incarnation), which has been extensively described elsewhere (Albert, et al., 1993, 1995; Mitchell, et al., 1996). The purpose of this paper is to recount our real-world experiences in getting AUTHOR to be the servant, not the master, now that disk survey administration has matriculated from the laboratory (literally) into the operational world. Recall that the theme of this paper is "Nothing Is Ever Easy."

The first challenge was getting AUTHOR to fit between TIARA at the start and CODAP at the finish. The determination of whether a survey would be disk-administered, paper-and-pencil, or a combination of both methods could not always be made before the project was underway, or even before TIARA processing was complete, so there could be no consideration of tailoring TIARA output for the different media. Similarly, the behemoth that is CODAP wants its input one way and one way only. With the input and output parameters for AUTHOR thus clearly defined and set, one might expect that a straightforward, regular set of options for AUTHOR could be identified and one might be correct. One might, but then there would be less to write about. AUTHOR was developed for and is used for more than the single application described in this paper, so AUTHOR grew an "Import TIARA Database" option to start the disk survey authoring process and an "Output CODAP File" option for data collected by the disk survey.

AUTHOR bears all the responsibility for data conversion, input and output. Initially, AUTHOR expected its input from TIARA to be regular and predictable. Thanks to AUTHOR, we learned more than we thought we needed to know about the TIARA-CODAP interface. A danger with "black boxes" is that over the years folks tend to forget what’s actually going on in those black boxes and what gyrations are involved in the interface. For instance, TIARA doesn’t care about the presentation order of some responses because it has an internal table to keep the sequence consistently ordered. If that same key is available to CODAP, and it is, then CODAP doesn’t care about the order either. However, if you stick AUTHOR in the middle with the assumption that there’s some logical order to those variables which is important to preserve, and that the same logical order persists between surveys, then you might end up with a pocket of 150 airmen stationed at an obscure radar site in Alaska who spend their days working on air conditioners. 150 airmen who, by the way, might be identified as non-active duty reserves on weekend training. So, AUTHOR had to be taught to read from the same playbook as TIARA and CODAP, even if some of those plays involved three 90-degree left turns to go right.

No matter how slavishly one might try to pattern disk administration of a survey to be the same experience as booklet administration, they’re just not the same. Given that disk and booklet are different, the difference should be turned to our advantage. Automation should not simply be adopted as an evolutionary progression, it should be exploited. Disk administration affords the opportunity for more control to influence the respondent’s behavior than is possible with a survey booklet. Is this change in the survey administration process introducing data influences which didn’t exist before? Is it easier, or more tempting, to cruise through a series of computer screens entering "5" for every response than it is to fill in the "5" bubble in a stack down the center of a booklet page? Is it easier, or more tempting, to look at a group of tasks and bypass them with an unconscious "That doesn’t sound like my job"? Is it easier, or more tempting, to return the survey untouched, despite admonishments of mandatory participation? As you probably realize, the simple answers are maybe, maybe, no, and yes.

Some perceived improvements, variables which couldn’t be controlled in a booklet survey, revolve around controlling the respondent’s behavior. Up-front data edits can limit answers to valid choices. If only a 6, 7, 8, or 9 is a reasonable choice, the disk survey will not accept a 1, 2, 3, 4, or 5. Branching based on previous answers can keep a self-identified sergeant from later answering an item about his officer specialty code - that item will simply never be presented. If a respondent chooses "Other" as one of the choices from an equipment list, the disk survey will immediately present an opportunity to type in that other equipment. With a survey booklet, the respondent has to take the initiative to write in "Other" responses at the end of the booklet (and we have to hope that we can decipher the writing). When appropriate, the disk survey can force the respondent to step through each line one at a time and even control how fast one can move from each line to the next. In a booklet, one could simply turn the page with nary a glance at any entry on the page. The answer to the more important question, What difference does any of this make?, won’t be addressed in this paper. Nothing is ever easy.

Despite extensive testing (Albert & Bennett, 1997), every time we put an operational survey together - and we’ve built 11 of them now - something new would crop up. These learning experiences led to exchanges like the following:

Fortunately, the fun did not stop here. Once a survey had been created, tested, and approved, it still had to be duplicated and distributed, and the data had to be collected after the disks were returned from the field. None of these steps were new; all applied to survey booklets as well as survey disks. These were the areas in which automation offered the greatest time savings and time is, after all, money. Instead of 30-60 days to allow a contractor to print survey booklets, one person could duplicate thousands of survey disks in 1 week. Instead of a staff to cut apart survey booklets and scan the responses one page at a time, one person could collect the response data directly from the survey disks and CODAP couldn’t tell the difference between disk input and booklet input. Automation provided time savings for everyone involved except that "one person."

Although many processing steps (packaging, mailing, tracking) are similar with both disk and booklet surveys, disk processing provides several new opportunities. The occasional survey booklet comes back with an unidentifiable food stain, but there’s normally no concern about checking the booklet for toxicity. Survey disks, on the other hand, must absolutely be checked for the presence of a virus. Disk surveys also reallocate some workloads, summarized in the following table:

Recall that the title of this paper is "Automation Exploitation." Conversion of booklet survey administration to disk survey administration is the automation portion. Judge for yourself where the exploitation comes in.

Actual field administration of the disk surveys has, so far, been far less interesting (as in the curse "May you live in interesting times") than anticipated. Most of the inquiries we’ve received ("My disk doesn’t work!") can, upon analysis, be resolved with three (four, to the purist) simple troubleshooting instructions:

1) The numeral 0 (zero) and the letter O (oh) are not interchangeable. Use the zero.
1a) No Air Force Specialty Code has a letter O, not even yours.

2) Your skill level code is a 3, 5, 7, 9, or 0; never an X, not even if you’re not sure.

3) Use the Quick Exit key to stop the survey, then restart it: it will return to the question where you left off, but all will be well.

That third step is the silver bullet. It works, but we don’t know how or why. ("Trust us." "Oh. Okay.")

One area in which disk administration clearly leads booklet administration is the volume of guidance required with the survey. Unfortunately, this is a sport in which a low score is better. For the first disk survey, we attempted to substitute disks for booklets with no more adaptation than changing the word "booklet" to the word "disk" everywhere in the narrative guidance. Well, maybe a little more adaptation than that. "Use only a No. 2 pencil to mark your answers in the booklet" didn’t translate so well to "Use only a No. 2 pencil to mark your answers in the disk." Folks have a lifetime of experience with paper-and-pencil surveys and tests, but automated administrations are not yet ubiquitous. First, we had to expand the instructions on the survey administration letter which went to survey control monitors (SCMs). It was always assumed that folks knew how to set a booklet down on a hard, flat surface, turn the pages, and color in circles with that trusty No. 2 pencil. Two out of three’s not bad: coloring in the circles was a challenge. But now we had to explain what kind of computer could be used, how to actually convince the disk to let you take the survey, and how to tell when it was safe to take the disk back out of the computer. ("Don’t remove your survey booklet from the desktop before it stops spinning" was never a required instruction.) And it was far easier for an SCM to leaf through a booklet to ensure it had been completed than for the SCM to weigh a survey disk to determine whether enough data had been added. Or however they check those things. Automation leads to more administrative overhead and an increase in written guidance. Nothing is ever easy.

As is often the case, the best has been saved for last. In these, the kinder, gentler, more sensitive ‘90’s, our environmentally conscious organization is now undertaking recycling of survey disks for their next reuse. The only obstacle to overcome is removal of the disk labels from the prior survey administration. Aside from the tacky look of label applied atop label, the machinery which duplicates the disks is also, if not kinder and gentler, a sensitive ‘90’s kind of machine and does not like thick disks. So, the original labels have to go. Did we mention there are over 10,000 disks in the first group of surveys?

If you’ve ever peeled industrial-strength labels from floppy disks, you know what a challenge this is. Once again, the obvious solution was TSgt O’Neil. We never asked how, but somehow it was learned that sitting on a disk from 5-12 minutes (depending on individual physical characteristics - but that’s a topic for another day) significantly improved peelability. Eventually, others were convinced (Remember that dialogue: "Huh?" "Trust me." "Oh. Okay.") of the efficacy of staging the disks in a warming environment before peeling and production subsequently improved. Picture a mixed (military and civilian, enlisted and officer) group sitting around a room, pulling disks out of their ¼ well, out of their "staging areas" and peeling labels. Picture the enchantment on the faces of passersby when they notice this activity. ("You’re doing what?" "Trust us." "Oh. Okay.")

Serendipitously, we replicated the Tom Sawyer experiment. As we explained to each enchanted passerby that we really were seriously engaged in productive work and there was a certain level of fun involved (a low level, to be sure, but who knew how exciting the rest of their lives might be?), the look of disbelief would dissolve and they’d be hooked. "Let me try one of those." Another volunteer! It was learned that this was a particularly effective technique at organizational functions, such as luncheons, where there was a large captive audience and little to do other than await the arrival of food. Automation or exploitation? You be the judge.

As scientists, you recognize that most of this narrative is anecdotal experiences. We’re still testing the waters with operational disk administrations. Of the 11 surveys selected for this new method, we have only finished data collection for one and data analysis for that one is currently underway. Rest assured that you will hear more about this adventure, with actual tables, statistical comparisons, and other scientific trappings, at a future symposium. AFOMS has convened a group of practical experts, from hands-on media preparers and shippers through survey developers and occupational analysts, to evaluate the overall experience and determine hard, defensible answers to these three questions: Does disk administration gather data which is at least as reliable as paper-and-pencil survey booklets? Are there compelling reasons to use disk administration in place of survey booklets? How can the survey administration process, regardless of media, become more effective?

An unscientific, but practical question that we all face every day, is "How much money will automation save for us?" We can calculate how much less expensive it is to produce and use a survey disk, relative to a survey booklet, but the answer to this question alone is irrelevant outside the framework of the preceding three questions.

There is one abiding tenet, true for every evolving system: Nothing is ever easy.

Albert, W.G & Bennett, W. Jr. (1997, June). Automated occupational survey research and applications. Proceedings of the Tenth International Occupational Analyst Workshop. San Antonio, TX.

Albert, W.G., Phalen, W.J., Selander, D.M., Dittmar, M.J., Tucker, D.L., Hand, D.K., & Weissmuller, J.J. (1995, May). Large-scale laboratory test of occupational survey software and scaling procedures. Proceedings of the Ninth International Occupational Analyst Workshop. San Antonio, TX.

Albert, W.G., Phalen, W.J., Selander, D.M., Yadrick, R.M., Rouse, I.F., Weissmuller, J.J., Dittmar, M.J., & Tucker, D.L. (1993, November). Development and test of computer-administered survey software. Proceedings of the 35th Annual Conference of the International Military Testing Association. Williamsburg, VA.

Mitchell, J.L., Weissmuller, J.J., Tucker, D.L., & Waldroop, P. (1996, November). Development and application of a computer-assisted survey authoring tool for training needs assessment. Proceedings of the 38th Annual Conference of the International Military Testing Association. San Antonio, TX.
 
 
 

Next Page 

Back to IJOA Home Page