Adaptive, Internet Survey Techniques for Determining
Critical Pilot Training Tasks and Requirements
 
 

Capt. Gregory Sidor, Air Force Research Laboratory, Mesa, AZ

Maureen Bergondy, Naval Air Warfare Center, Orlando, FL

Dr. Winfred Arthur, Jr., Dept of Psychology, Texas A&M University, College Station, TX

Dr. Jimmy Mitchell, IJOA, San Antonio, TX

Dr. Winston Bennett, Air Force Research Laboratory, Mesa, AZ
 
 

Background

Aircraft and system acquisitions are expensive investments, both in terms of initial acquisition and lifecycle costs. A primary focus of most new system acquisition programs is "affordability." This entails reducing the costs of development, production, and ownership through definition, demonstration, and transition of innovative concepts and advanced technologies that will significantly reduce system costs while meeting the operational requirements of the customers (i.e., the services). One strategy being used to reduce costs is to maximize commonality among air vehicle variants and systems, while providing unique features that maximize capabilities for diverse missions. Another strategy for aircraft acquisition includes eliminating the need for a two-seat aircraft version for training. This would provide cost savings in terms of design, production, maintenance, and manpower. However, in order to ensure that there would be no reduction in training effectiveness or safety, a methodology needed to be developed to examine the purpose of current two-seat training and to explore training and aircraft design options that might help mitigate the need for this expensive training variant. The purpose of this paper is to describe a methodology for collecting field data to support the acquisition and design process.
 
 

Objectives

The research team was tasked to conduct a study analyzing elements of new aircraft pilot training that would potentially be impacted by the decision not to build a two-seat training variant. The goal of this study was to assist acquisition decision makers in the Engineering, Manufacturing, and Development phase by identifying the critical pilot training issues of safety and effectiveness for tasks currently trained in two-seat aircraft variants. This study’s findings and recommendations will help the program offices ensure that the critical issues are adequately addressed and resolved by the weapon system contractors through aircraft design or training system innovation.

This study gathered key information about appropriate training strategies from instructor pilots (IPs) of existing strike aircraft and will employ these findings in generating recommendations to ensure that new aircraft and training system designs will mitigate any risks. Several training strategies and aircraft configurations are plausible and combinations of strategies may provide the best options for the various aviation communities. To enhance the validity of decisions made based on information derived from this study, this project used subject matter expert (SME) input from current strike fighter aircraft IPs from various services and countries participating in new aircraft acquisition programs. There are two primary objectives of this survey:

Objective 1 - Determine why certain tasks for current aircraft are trained in 2-seat models.

Objective 2 - Determine what training methods could be used to safely and effectively train tasks currently trained in a 2-seat aircraft if there are no 2-seat aircraft available.
 

Methods Workshop

The project team for this study consisted of two occupational survey analysts, an industrial/organizational psychology professor from Texas A&M, and two research psychologists who specialize in U.S. Air Force and Navy aviation training. The project team conducted a two-day workshop with a diverse group of IP subject matter experts

to construct the appropriate survey items and response options to meet the defined objectives. The IPs represented legacy strike fighter aircraft communities. Specifically, the instructor pilots consisted of:

a UK Royal Navy F/A2 former IP) who works at a new aircraft program office as the lead for UK operational requirements;

two USAF F-16 pilots who were also exchange pilots with the RAF flying GR7s, one of whom was an IP for F-16 initial qualification training and the other was an IP for undergraduate pilot training;

one USN F/A-18 IP and

one USMC AV-8B IP who accomplished the task and syllabus matching off-line from the workshop.

The project team began the workshop by giving an overview of a new aircraft acquisition program and its various roles/missions, performance characteristics, and planned system capabilities. The project team then, defined for the SMEs the objectives of the study and suggested types of survey items that might be appropriate for answering those objectives. The IPs reviewed each of their respective services' legacy aircraft training syllabi, course training standards, and master training task lists to identify the tasks and skills currently trained in two-seat versions of those legacy aircraft. The IPs then generated a single list, using common terminology, of all the major tasks that are trained in the various services’ two-seat aircraft.

Having a common list will enable an analysis of the data across the different services for tasks that are common to several or all of the services. The master list contained 44 tasks, of which, the USAF F-16 used 33, the USN and USMC F/A-18 used 26, the USMC AV-8B used 28, and the UK RN and RAF used 43. In addition, each IP identified the appropriate legacy syllabus event in which those tasks are trained, and what type of instructor pilot is responsible for teaching that task.

To meet the first objective of the study, determining why certain tasks are currently trained in two-seat models, the project team and SME's constructed a set of survey items that asked respondents to assess the importance of each of the four roles for an IP who is in the back-seat while training each of the identified two-seat tasks. The four major functions of the IP in a back seat were determined to be (a) Ensure Safety of Flight,

(b) Demonstrate Execution of Task, (c) Provide Immediate Feedback/Instruction, and (d) Assess Performance. Since an IP often performs more than one of these functions concurrently from the back seat, we asked the survey respondents to rate the importance of each function for each specified task on a scale of 0 to 4 (0 being "Not at all important" and 4 being "Absolutely Important").

Since new aircraft might only have single-seat versions of the aircraft, even for training, this study collected data from survey participants on potential aircraft capabilities or training system technologies that might be required to train these types of tasks with minimal levels of risk and high levels of training effectiveness. With the combined knowledge of the project team and SMEs, a list was generated with supporting definitions of aircraft capabilities that might enable, or obviate the need for, an IP to perform the four functions identified above in the absence of a two-seat training aircraft. These aircraft capabilities included such items as a chase aircraft, ground-collision avoidance system, and advanced handling characteristics (See Table 1).

In addition to the list of aircraft capabilities, the project team and SMEs developed a list of simulation technologies with supporting definitions, for both existing and emergingconcepts, that could be used to facilitate or enhance training, or for evaluation of competency before a student must perform those tasks in the aircraft (see Table 1).
 
 

Table 1. List of aircraft and simulator capability response options used on the survey

 
Survey Construction and Layout

Because of the requirement to survey populations at disparate sites, as well the uniqueness of the survey based on the different tasks from the master list that are actually relevant to the different services, the project team used an adaptive, internet-based survey construction and delivery tool called GenSurv (Bennett, 1999; Mitchell & Weissmuller 1999). This survey tool allowed the survey to branch based on the service, aircraft type, and instructor pilot qualifications.

The survey began by explaining the purpose of the study and providing some background information on the new aircraft, to include its expected missions and capabilities. The survey then had a section to gather demographic information from each respondent which was branched and tailored to each service. The next section was constructed to collect data to address the first objective, assessing the importance of the IP functions for each task currently trained in two-seat training aircraft. This section began by defining the functions and then asked respondents to assess the importance for each of the tasks relevant to their legacy aircraft type and their instructor qualifications.

The next section addressed the second objective of determining aircraft and simulation capabilities that would be required to safely and effectively train those same tasks with only a single-seat aircraft available. It began by having the respondent read through the response options and their detailed definitions. Participants were then asked to identify any and all of the aircraft and simulation capabilities they thought would be required for each relevant task (see Table 1).

The final section included a general comment section for respondents to provide any additional inputs on these training issues they felt might be useful and gave an option for the respondents to provide personal contact information.

In addition to being able to create an adaptive survey based on the branching described above, there are other features of GenSurv that were extremely useful for constructing the survey and using the instrument to collect the data. GenSurv allowed the survey developers to require an appropriate response before a participant could proceed to the next item. Another key feature used throughout the survey, was allowing for or requiring, open-ended responses or comments which helped to collect a richer set of information for the study. Because the survey is Internet-based, the instrument developers inserted links to a new aircraft program office web site for respondents seeking more information. There were also pictures from simulation visual databases to illustrate some of the advanced simulation capabilities that new aircraft training systems could employ.

Alpha and Beta Tests

After the survey was developed with GenSurv and hosted on a server, an alpha test of the survey was conducted with the project team and SMEs who helped develop the survey. Based on feedback from project team members and SMEs, changes were made to clarify the objectives and to enhance the presentation and usability of the survey. A limited beta test was also conducted with IPs who were not part of the survey development to ensure usability and clarity to pilots of the targeted population, and to ensure complete and accurate coverage of two-seat tasks for each legacy aircraft community. The feedback from the beta test showed that the survey was ready for implementation across the targeted population.

Field Data collection

The project team’s goal for the survey was to generate a sample of at least thirty current or recent instructor pilots from each legacy aircraft community, resulting in a final sample of 120 participants. Data collection began on September 7 and the web site was closed on October 6, due to program schedule constraints. At the conclusion of the data collection, 48 completed survey responses had been received from the following aircraft communities: 8 from United Kingdom GR7 and F/A2 IPs, 10 from AV-8B IPs, 18 from F-16 IPs, and 12 from F/A-18 IPs.
 
 

Conclusion

Analysis of the data has just begun. Currently, the background data and raw database are being evaluated for completeness. Results are expected to highlight areas that may require risk mitigation, either through innovative aircraft design or implementation of innovative training technologies and strategies. Results may indicate a need to extend the data collection effort to increase the sample size or highlight areas that require a more detailed study and analysis to fully understand the critical safety and training effectiveness issues. The data need to be summarized in such a way as to represent the professional opinions of this sample of pilots as to how training should be conducted with new systems. This will assist new aircraft acquisition decision makers in guiding the development of the aircraft and its pilot training systems to mitigate risks and ensure effective training.
 

Next Paper