Historically, job and occupational studies gather and examine data obtained from one level or component of an organization at a time. Rarely is the entire hierarchical structure of the organization surveyed simultaneously. This is especially true in the military environment where one career field at a time is normally studied using task lists developed especially for that career field. In fact, the Air Force Survey Authoring Software (AUTHOR) was designed primarily to develop and administer these task list-based surveys. Because of this, AUTHOR has rarely been utilized to develop or conduct organizational climate studies or opinion polls.

Likewise, data that are obtained from respondents at a given supervisory level in organizational climate studies are normally not "rolled up" to provide supervisors up the chain of command with data obtained from all respondents under his supervisory control. More typically, data might be reported by individuals, supervisory units, as an overall mean or across selected variables.

The SA-ALC Supervisor Feedback Survey has been administered twice a year at Kelly AFB since 1994 under the direction of the Kelly Air Force Base Civilian Transition Office (DPCEL), 76 SPTG. This anonymous survey was designed to provide insight on how well supervisors are perceived to be performing by their direct subordinates in six critical areas: communication, decision making, human development, management support, safety, and transition issues. The same 26-item survey had been administered in house since 1994 using a paper and pencil methodology, except for the second of two 1997 administrations which utilized an in-house computerized version using relational database software. The Center contracted with the Institute for Job and Occupational Analysis (IJOA) to conduct the 1998 study since they had encountered numerous problems developing, administering and analyzing their computer-based version. Analysis in previous surveys was limited to summary counts of responses, unweighted averages and counts by category.

In the current study, nearly 5,000 job incumbents at the Center were asked to evaluate their immediate supervisor’s performance in the same six broad areas listed above. Over 1,000 supervisors were evaluated in a base-wide study to evaluate individual and organizational supervisory performance throughout the organization. Personnel from all levels within the hierarchy were surveyed and each supervisor with two, or more, subordinates, were provided with feedback in the form of average scores and standard deviations generated from data provided by their direct subordinates. The data from each supervisory level was also combined and "rolled up" so that supervisors at the various organizational levels (division, organization and overall) were provided the data obtained not only from their immediate subordinates, but also from all the individuals within their division or organization. The survey results were intended to shape supervisory behavior and leadership characteristics, to assess supervisors’ effectiveness and to identify training needs for individual supervisors.

This study, then, provided an excellent opportunity to test the ability of AUTHOR to generate, administer and collect data from a survey not normally developed with the software.

Methodology

The current Supervisor Feedback Survey was conducted in August and September 1998. IJOA developed, reproduced and distributed the survey disks, uploaded and analyzed the survey data, and generated individual and summary reports for this iteration of the survey.

IJOA utilized the Air Force Survey Authoring Software (AFSAS), developed jointly by IJOA, Metrica, Inc., and the Air Force Research Laboratory (AFRL), as the survey platform. The computer-based survey was duplicated on high-density floppy disks that were then distributed to the survey population though Points of Contact (POC) designated by the Civilian Transition Office.

One of the attractive features of the AFSAS software is that multiple respondents are able to use the same disk to record their responses, allowing IJOA to minimize the cost of administering the survey. The floppy disk format permitted respondents to respond to the survey at a time and location that was convenient.

Transition Office personnel returned the completed disks to IJOA for data uploading and processing. Data were uploaded using AUTHOR’s Collect function while data were analyzed by, and reports were generated by, the Statistical Package for the Social Sciences (SPSS) software program. SPSS was used to permit the generation of means and standard deviations for each organizational unit and to facilitate the summarization of data to multiple levels within the organization.

The Survey

The survey consisted of 30 multiple-choice items and four open-ended questions. The multiple choice items were answered using a six-point "agreement" scale, with "1" being "Strongly Disagree", and "6" being "Strongly Agree." The four open-ended items asked respondents to comment on things the rater would like their supervisor to "Stop Doing, Start Doing, Continue Doing" and solicited "General Comments" as well. The multiple-choice survey items, open-ended questions and rating scale were identical to those used in previous versions of the survey except that five additional multiple-choice items were added to obtain information regarding transition issues.

Survey respondents anonymously rated only their immediate supervisor. Each supervisor was assigned a four-digit code that matched individual respondents to their supervisor. Likewise, supervisors rated their immediate supervisors up the entire chain of command.

It should be noted that there were very few problems associated with the use of computer disks for this survey. IJOA staff received only a handful of "trouble" calls from respondents, most of which involved a misunderstanding on the part of the respondent (e.g. lack of understanding of the meaning of "drive a"). However, a potential problem surfaced early in the survey administration when IJOA staff discovered that the survey would not run on new, high-speed (266 Megahertz, or greater) computers that used a particular combination of logic chips. This problem was easily resolved by requesting that respondents at the two locations where the new computers were in use take the survey on older, slower machines. A more permanent solution has been developed utilizing a "patch" developed to correct a problem in the commercial software compiler used in developing the software.

Completed survey disks were electronically uploaded into a common database using the Collect function in AUTHOR. The full database was then imported into SPSS, Version 8.0, which was used to compute a grand mean for each of the six critical areas and individual means for each supervisory unit, each division and each organization. It was planned that management personnel from Kelly AFB would determine which supervisors would be offered remedial supervisory training but that IJOAA would identify individuals scoring two standard deviations, or more, below the mean.

Survey Results

In all, 4,840 individuals rated their corresponding 1099 immediate supervisors.

IJOA generated 1099 individual reports, 38 division reports and 3 organizational reports for a total of 1140 statistical reports in about six weeks. This was no small feat, since each supervisory report had to be generated by SPSS one at a time, as did each of the division and organizational reports. Rolling up the data presented its own set of challenges since SPSS, Version 8, was not primarily designed to manipulate databases, but rather perform statistical analysis on a relatively fixed database. Frequently, the database was imported into Microsoft Excel to more easily group supervisory units and otherwise manipulate the database. Once the desired format was obtained in Excel, that portion of the database was imported back into SPSS where the analysis for that level, or unit, within the organizational structure was performed.

In general, the overall results were very positive. The overall means (on the six-point agreement scale described above) by critical area are shown below:
 

Management Support               4.96

Communication                          5.07

Human Development                4.98

Decision Making                       5.02

Safety                                         5.44
 

By division, only one stood out as being problematic. However, only eight individuals in that division completed surveys which, depending on the total number of employees in that area, may, or may not, be a sufficiently large sample to accurately rate the division’s supervisors. Their means (on the six-point agreement scale described above) by critical area are shown below:
 

Management Support                  2.00

Communication                             2.15

Human Development                   1.98

Decision Making                         2.08

Safety                                           2.79

Transition                                    1.86

From an IJOA generated report that included the overall scores for every supervisor who was evaluated, the researcher identified approximately 40 supervisors (who were rated by at least two respondents) who scored at, or below, two standard deviations below the overall mean.

In addition, there were over 1,000 pages of narrative comments in response to the four open-ended questions. While it was beyond the scope of IJOA’s contract to provide a complete semantic analysis, the comments were carefully analyzed for threatening or abusive statements. In order to perform this analysis, the comments from each survey were compiled into on contiguous Microsoft Word document. Then, using the "find" function, the researcher entered one key word at a time from a list of specifically threatening words, such as hate, anger, hurt, kill, murder, gun, knife, bomb, plot, repercussion, abuse, harass, a list of profane words and a list of race-specific derogatory words. When a threatening word was found, the researcher reviewed the comment containing the word to determine the context. At least one individual was identified as making threats against his/her supervisor and was reported to the base commander.

Discussion

Overall, this appeared to be a realistic survey of supervisors at Kelly AFB, at least in terms of the large number of participants and the generally meaningful participation across many organizational units. The picture that emerged was generally very favorable with most individual employees being pleased with their supervisors’ performance.

The large number of write-in comments, while providing a very rich and highly desirable data, presented a serious analysis problem. While AUTHOR was easily able to handle the volume (over 1,000 pages of text was generated from the four open-ended items) the researcher had no expedient way to read and group comments into meaningful categories. While it was possible to do so, time and personnel constraints prevented this activity. A reasonable procedure to identify threatening comments was developed using the "word search" capacity within Microsoft Word. Comments containing apparent threats were identified using this somewhat crude semantic analysis technique and an individual was reported to the appropriate commander. However, it would have been helpful if a more rigorous of semantic analysis were available to categorize comments and to identify threatening or hostile comments.

A related issue is that survey respondents were assured that their comments would be anonymous. That an individual respondent was identified and perhaps ultimately faced negative consequences due to his or her comments raises an important ethical question regarding respondent confidentiality. However, it is not the intent of this paper to take a stand one way or the other, other than to recognize that there is potential for abuse whenever candid comments are provided to individuals being rated by subordinates.

The successful completion of this organizational climate study was a milestone for the AUTHOR software. It clearly demonstrated its ability to develop and administer surveys other than job or occupational surveys. While SPSS was utilized to perform the data analysis, atCODAP could have been utilized since the Collect function in AUTHOR is able to format the database in a flat ASCII format, a tab-delimited format or in the CODAP format. A lesson learned from this study is that the data analysis requirements should have been clearly identified prior to survey administration and CODAP used to perform the data analysis and generate automated reports upon the completion of data upload. This would have added additional time on the front end of the project, but would have saved an even greater amount of time during the report generation process.

Next report 

Back to IJOA main page

posted November 30, 1999