Data Collection Through Analysis in the

U. S. Army's Enlisted Common Soldier Task (ECST) Project
 

Johnny J. Weissmuller, Metrica, Inc., San Antonio, TX

Bob Gosc, Institute for Job & Occupational Analysis, San Antonio, TX


 

Background

The Institute for Job and Occupational Analysis (IJOA) was founded in 1993 as a not-for-profit organization. IJOA is dedicated to improving and sharing methodologies and technologies used in occupational analysis and other applications related to the world of work. Although IJOA undertakes work which many may view as "operational", the overriding goal of this scientific and educational organization is to research, develop, validate, implement, and share improved methods (not project data) with government, academic, and private sector agencies.

Introduction

In 1998, the Institute for Job and Occupational Analysis (IJOA), in conjunction with Metrica, Inc, developed, administered, and analyzed a worldwide survey of 20,000 soldiers in the U.S. Army. This Army-wide survey of common training needs spanned across all Military Occupational Specialties (MOSs) and across all Skill Levels. Previous reports have detailed the results and conclusions from this study for the internal use by the U.S. Army. The purpose of this paper is to focus on the challenges encountered, new technologies developed and lessons learned in support of the Enlisted Common Soldier Task (ECST) Project.

Stages in the Occupational Analysis (OA) Process

The normal stages in the occupational analysis (OA) process include:

Each of these will be reviewed in order to address the challenges, technologies or lessons learned in that stage of the OA process for the ECST project.

Inventory Development (Content)

The Use of Word 97 & Hyper-Text-Markup-Language (HTML)

Although several inventory development tools have been investigated over the past several years by IJOA, the one used in this project was Microsoft Word 97. Many task list development systems may be "OK" if the task list is nearing completion and is fairly stable, our concern is, however, on the development of a task list during its early, volatile stage. At this point of the OA process, flexibility is the key requirement.

Using Microsoft Word in a table format, we can transmit draft task lists all over the world, and get rapid feedback. Earlier versions of Word may be used during this phase, although the use of versions with the "Track Revisions/Highlight" options is highly desirable. This on-line review process permits a rapid focus in on a draft task list for formal review/scrub-down by a live Subject Matter Expert (SME) panel.

The IJOA/Metrica team developed specifications for new conversion software to take this volatile information in a form optimized for inventory development (see below) and mechanically convert it to "ready-to-survey (on disk)" format in minutes. The Word "97" version is required at the end of this process because the "Save as HTML" option is required for this conversion software (HTM2Flat & HTM2Data). This first use of the HTML protocol for data and text transfer spawned a flurry of OA tool advancements.

Word 97 Task List Table Requirements

Our method has been to create a Word Document with a TABLE to hold the task list. Only one column is REQUIRED in the table to hold the task statement. Note that Duty Titles are recorded in the same table column as the Task Statements, immediately preceding the tasks "under" that duty. The Duty Titles must be displayed in a font color other than BLACK. We use RED for Duty Title text. The same method, of course, can be used to record Knowledge, Skill, Ability, and Other Characteristics (KSAO) lists or Tool/Equipment Lists, etc.

Word 97 Task List Table Options

We tend to use two or three columns to hold such things as notes about the source of the task, manual/regulation references, and target skill levels for each task. Some users may wish to use columns for nouns or verbs to allow resorting of tasks based on those grammatical dimensions. The use of an actual "task number" column is discouraged because it causes more work than it is worth -- although an automatic sequence number column is good to facilitate communication between reviewers.
 

ECST Project Challenges

Use of "Author" and a changing "OASurv"

Author is a user-friendly system developed by IJOA to capture the desired content for occupational surveys. Author then formats detailed survey script files. These script files define survey content, branching, and protocols for survey administration when fed into the survey engine called OASurv.. The Author system was used to create a first draft of "job incumbent" survey script. Because of new requirements in this ECST project, OASurv underwent several upgrades (discussed below). The final script had to be repeatedly hand-edited and tweaked to meet new requirements and to take advantage of the new options being added to OASurv.

The Eight Training Emphasis (TE) Surveys

The ECST Project surveyed not only job incumbents, it also surveyed Training Emphasis (TE) from supervisors of each of the eight (8) major sub-samples in this study (i.e., 4 of 5 skill levels, two grade levels and two "additional duty" designators). The same Master Task List used for the job incumbents was also used in the 8 TE surveys. Each TE survey required only a few background questions along with the ECST Master Task List.

Production Efficiencies in Multiple Surveys

Production of the TE surveys was expedited (as was analysis) by two main factors. First, the project "task list", (in this case, 847 items) is treated as a single unit. In any survey within a project, one simple "cut and paste" of the original job incumbent task list filename can clone all task statements. Second, OASurv scripts are standard flat-ASCII text files. Once a "Master TE" survey script was made, a "mail merge" system was set up to copy the Master TE (with appropriate changes) to each of the 8 TE variation surveys.

Survey Instrument Preparation (Implementation)

Survey Software Used in ECST

    OASurv - The Survey Engine

The Enlisted Common Soldier Task Project used a diskette-based survey to collect data. As expected, inventory content had to be automated in the format required by the survey software. The survey software on the disk was "OASurv.exe" which was developed by Metrica based upon the "atSURVEY" design developed by Sensible Systems, Inc. (SSI) in 1988. Both OASurv and atSURVEY were designed to feed data directly into the atCODAP analysis package. Hence, the atCODAP tools that import the task inventory and other lists, also prepare the analysis package to receive the data when they are available.

    Virus Checking System

Floppy disks containing an OA survey can be used by multiple respondents. Disks with over 20 cases have been observed in some instances. For this reason, the survey disks also included VIRUS checking software. The VIRUS checking software was written by Metrica exclusively for use in occupational (OA) surveys. In OA surveys the right set of conditions exist to trap even previously unknown Boot Sector and Trojan Horse viruses -- and the latest craze of macro viruses may be ignored given a properly controlled environment. This feature prevented the spread of viruses in the field and alerted survey respondents if a virus was detected on their computer. Worldwide, the Army computers were shown to be amazingly virus-free as compared to comparable agencies.

    Computer Inventory Program

Some agencies want to collect data about the kinds of computers used to take OA surveys. This may be done to "inventory" what kinds and how many computers are being used (versus shared) in taking the OA surveys. This information may be used by agencies to determine the computers available for possible future training programs, either on CD-ROM or over the Internet.

    Software to Use "Extra" Columns in Task List Tables

With the introduction of the Word 97 document as a formal input to the automated survey, it became clear that it would be nice to be able to use the "extra" columns from the Word Table. For example, the column recording the "Target Skill Level" for tasks became important because the Army wanted to restrict the tasks shown to soldiers to minimize "wasted time." Hence, the extraction software was modified to pull out the task (or other) list AND carefully extract associated columns to be used either in screening items (like the skill levels) or to be displayed later as normal text tacked onto the task statement (like references to training manuals).

OASurv Upgrade to Implement "One Up, One Down"

Although the "Target Skill Level" column had been extracted from the Word Table, the OASurv software had no capacity to use that information. In addition, to look for task migration (i.e., tasks being performed either higher or lower than in the past), the Army wanted a "One Up, One Down" policy. Hence, if the task Target Skill Level was "3", they wanted that task shown to soldiers at Skill Levels 2, 3, and 4. A few small programs were written to translate the original specification (e.g., "teach at SL 3") into something OASurv could handle. To complicate matters, two more "additional duty" job requirements had to be added on top of the standard 5 skill level jobs. For each task these requirements were coded into a "Yes/No" (1/0) string of 7 values (e.g., "01110 00") that were tied in the survey script to three key questions: "What is your Skill Level?", "Are you a First Sergeant?", and "Are you in a Battle Staff Position?" OASurv itself had to be upgraded to use that information by inspecting the respondent's self-reported answers to these questions and determine, for each task, if this respondent should be shown the current task.

OASurv Upgrade to Use Prioritized Duties

In keeping with the Army's desire to minimize time requirements on soldiers, OASurv was also modified to use the Prioritized Duty Approach. The Prioritized Duty Approach has job incumbents rate the Duty Titles first, then, based on those duty ratings, present tasks in descending order of importance of the duties. Note that the "importance" of the duty may be defined by the needs of the current study. Typical scales include "Part of Job", "Time Spent", "Importance", etc. In the ECST project, "Part of Job" was used for rating duties. Also, the survey author may choose to force the job incumbent to see ALL DUTIES (regardless of rated level) or may choose to not show the incumbent tasks under those duties rated low (or 0). In the ECST project, we did not show job incumbents tasks under duties rated "Not Part of My Job."
 

Field Administration

Once a master disk was created, it was Fed Exed to a commercial duplicator who copied, labeled, and returned disks within 72 hours. There were 20,000 disks shipped to 60 locations worldwide. Where possible, receiving locations were verified prior to shipment. CONUS packages were sent via FedEx to obtain expeditious delivery and prompt confirmation of delivery. APO packages were sent via Express Mail where available and Priority Mail for others. Returns were tracked on a revised version of the ship-out spreadsheet.
 

Data Aggregation

Copying Respondent Data to Hard Disk Directory

At first, as the ECST disks came in, they were uploaded on several PCs, using the PC's built-in disk drive. About one-third of the way through the upload phase, an automatic disk reader was purchased and that became the primary (but not sole) source for loading disks.

Files were copied from the floppy disk (data file, log file, computer inventory file), merged, assigned a unique Quality Control Number (QCN), split between "data (QCNnnnnn.QCN)" and "comments (QCNnnnnn.WIT)" and then stored on the hard drive of the upload computer. Each upload computer had a unique range for QCN numbers, assigned in blocks of 5000 - (0-4999), (5000-9999), etc. In this way, many machines could be used in the upload process and yet, all files could (eventually) be consolidated onto a single machine with no conflicts.

ECST Return Rates

Approximately 7,000 of the 20,000 were returned with good data. As this was a Total Army survey, it should be noted that the return rate for the Active Component was nearly 70% while the Guard and Reserve return rates were about 20% and 15%,

respectively. The Guard and Reserve returns were low, as expected, due to a need to re-package and distribute disks to remote sites (in some cases, a five-state area). In addition, the Guard and Reserve only meet one weekend a month. The survey response window was left open for 135 days instead of the 45 originally planned.

ECST Aggregation Dates

Actual aggregation of the data files occurred four times throughout the life of this project. This step is accomplished periodically in order to determine the number and source of "good cases" so that the non-responding units can be identified and contacted. The final aggregation was in November 1998 and was the source of all products generated for both the final report and for use by the Critical Task Pre-Screen Panel (Dec 98) and the official Army Critical Task Selection Panel in Jan 1999.

Data Processing

Standard occupational analysis processing is accomplished by a version of the CODAP system – i.e., the Comprehensive Occupational Data Analysis Programs (CODAP) system. The CODAP approach was pioneered and publicly documented by the U.S. Air Force over the period of 1964 to 1995.

The ECST project performing data processing using the commercial atCODAP System by Sensible Systems, Inc. atCODAP is an OA statistical analysis package producing historical CODAP analysis products, but from a newly designed computational platform, created specifically for occupational analysis performed by small units with few support personnel. "atCODAP" means "anchored to CODAP, but not limited by it." While atCODAP incorporates all the open and publicly documented algorithms of CODAP, it goes well beyond the CODAP system to facilitate easy of use and training and provides many project management features not available in the clones of old mainframe versions of CODAP.
 

Analysis, Interpretation, Report Writing, and End-User Assistance

In keeping with design changes at the beginning of this project (i.e., using Word 97 to capture HTML protocols), several new capabilities were added to increase the user-friendly interface of atCODAP products. For example, the processing for all "comment" files was standardized. The text of key "write-in" comments was categorized, formatted in HTML and merged with user-selected data extracted from the atCODAP data base for the corresponding case and also formatted into HTML. The resulting file could be loaded directly into Word 97 and sorted as desired on any data column.

Only a few atCODAP programs output HTML format reports for display on the Internet (JOBHTM). Most atCODAP programs assume the user will transfer data vectors from standard products such as Job Descriptions (JOBPRT) and Task Factor (GRPREL) Reports using the OADecks program for insertion into Microsoft Access (or Excel), allowing End-Users to manipulate the data as needed.

Next paper 

Back to list of papers

Back to IJOA main page

posted November 30, 1999