Air Force (AF) Major Commands (MAJCOMs) often have difficulty in understanding how much improved war-fighting capability can be expected from investments in Manpower, Personnel, and Training (MPT) or Human Performance Research and Development (R&D). This is because MPT R&D findings usually have not related directly to war-fighting Measures of Effectiveness (MOEs). To convince MAJCOMs of the utility of their research, MPT R&D personnel must demonstrate how combat capability will be increased by these investments. To clarify these relationships, CSERIAC was tasked to identify MOEs and Measures of Performance (MOPs) which could serve as criteria for human system R&D, and to hierarchically organize these MOEs for linkage to a set of improved training effectiveness measures. Overall, the goal is to enable commanders to predict how, for example, specified screening and training technology can contribute to unit effectiveness. In Phase I of the project, CSERIAC searched the literature, brainstormed with readiness research experts, and examined human-system interface criteria for judging mission effectiveness for aeronautical systems. We found that the literature contains no comprehensive listing of human-related aircraft MOEs; therefore we compiled a listing from the literature. In Phase II using the MOE hierarchy, CSERIAC will interview AF decision-makers with wartime missions and responsibilities to capture their criteria for judging unit readiness and effectiveness for comparison with the compiled MOE hierarchy. Measures prioritized as most important will then be linked to MPT effectiveness metrics. With these linkages, the value of advanced training and screening technology can be more clearly demonstrated. This paper will provide Phase I results and highlight plans for the remaining project.

Declining R&D Budget Demands Research Targeted to Combat Capability. MPT R&D traditionally validates its products and procedures using outcome criteria such as course grades, "time-to-master," cost-benefits analyses, or more recently, job performance indices. With major R&D program budgets declining, MPT researchers must demonstrate their contribution to overall mission effectiveness to gain support from MAJCOM officials, and higher priority in the budgeting cycle. This project is based on the assumption that research funding prioritizations may be greatly influenced by demonstrating war-fighting capabilities and readiness impacts. Therefore, incorporating relevant readiness and war-fighting criteria into research program descriptions and impact statements is critical to future research.

MPT Scientists Not Using Combat Capability Criterion Link. Unfortunately, MPT scientists have not empirically shown this link, and top decision-makers may use this perceived lack of evidence as an indicator of the relative merit of MPT R&D during today's "downsizing" budget climate. One reason why research scientists do not consistently use empirical links to readiness and war-fighting capability is that they do not know what these higher level war-fighting criteria are, or where and how to obtain these criterion measures. They have been accustomed to using course and job criteria. Especially in these times, MPT effectiveness measures must be linked with war-fighting capability measures. But few MPT scientists are aware of what these top decision-makers and potential users of their technology will accept as indicators of merit, nor are they are fully aware of which decision-maker they should target for demonstrations of MPT R&D technology with what kind of wartime links. To extend the connection from these typical MPT measures to criteria that MAJCOM decision-makers value, scientists need a taxonomy to assist them in identifying relevant and organizationally valued criteria. The basic problem is that MPT scientists do not know the identity and interests of decision-makers who might be educated on the true return on human systems R&D investments (Gould, 1995).

Definition of MOEs & MOPs. One way to target human performance improvement research is to focus on the Measures of Effectiveness (MOEs) and Measures of Performance (MOPs) used to evaluate the performance of weapon systems, organizational units, teams or crews, and individual airmen. MOEs are a measure of how well an operational task or task element is accomplished through using a system (ordinarily a single operator using one aircraft or weapons system to perform one task) (AFOTEC/XRC, 1995; Lane, 1986; USAF/TEP, 1994). MOPs are qualitative or quantitative measures of system capabilities or characteristics (USAF/TEP, 1994). They indicate the degree to which that capability or characteristic performs or meets the requirement under specific conditions (USAF/TEP, 1994). MOPs are components, or subsets, of MOEs; i.e., the "degree-to-which" a system performs is one of a number of possible measures of "how well" a system's task is accomplished. Therefore, MOPs can be accumulated to assess an MOE that is not directly measurable (AFOTEC/XRC, 1995). In these definitions, however, there appears to be much overlap and interchangeability between MOEs and MOPs (Lane, 1986), with some MOPs being labeled as MOEs; therefore, we refer to these terms with a slash between them (i.e., MOE/MOP). Additionally, since the AF Operational Test and Evaluation Center (AFOTEC) focuses on total system performance, and since the "total system" as defined by Department of Defense (DoD) Directive 5000.1 (DoD, 1996, p.5) includes "the people who operate and maintain the system" and their "training and training devices," the broader definition of "total system" MOEs/MOPs would include the effectiveness of units, team or crew, and individual performance associated with the system.

Other Essential Definitions. Other important definitions include the following: Mission Effectiveness is defined as the probability that a system (also read this to include the unit, team or crew, or individual) is available to initiate its mission and will complete its mission when initiated (USAF/LGMM, 1994). War-fighting (Combat) Capability is determined by the number of consecutive events (e.g., sorties, miles, orbits, hours) that a weapon system can successfully execute its mission under specified conditions (USAFILGMM, 1994). Readiness has been defined as the ability of forces, units, weapon systems, or other equipment to deliver the output for which they were designed, including the ability to deploy and employ without unacceptable delay (USAF/XOOO, 1993).

Based on the need to identify potential sponsors of MPT R&D and criterion measures that these decision-makers will accept as essential, the Armstrong Laboratory Technical Training Research Division (AL/HRT) tasked CSERIAC to accomplish the following objective: Develop a hierarchical taxonomy of war-fighting capability metrics linking the priorities and preferences of AF operational command decision-makers for specific MOEs or MOPs. The overall project is divided into two phases, with a CSERIAC Review and Analysis documenting the results of the first phase. The specific tasks and approach are listed below.

Phase I tasks include the following: (a) survey the literature and network with experts on aircraft-related MOEs/MOPs; (b) develop a candidate listing of AF operational flying command officials responsible for wartime missions and identify MOEs/MOPs; (c) based on the literature search and networking findings, develop a consolidated taxonomy of MOEs/MOPs for aircraft systems, aircrew, and maintenance, and display these MOEs/MOPs together with their meaning, computational formula, and source; and (d) document the findings in a CSERIAC Review and Analysis report.

Phase II tasks include the following: (a) construct a structured interview based on the findings in Phase I designed to capture the preferred MOEs/MOPs and their priority from MAJCOM decision-makers; (b) develop a taxonomy of these MOEs/MOPs showing hierarchical relationships of the criteria of war-fighting capability through decomposing those criteria associated with human systems implications; and (c) link decision-makers and their preferences.

After a review of the approach taken in this study, this paper reviews Phase I findings. Leading sources of MOEs/MOPs were found to be military standards and AF instructions, mission area assessments and plans, acquisition documents, test and evaluation (T&E) reports, the readiness reporting systems, operational readiness inspections, command status briefings, exercises, wargames and simulations, competitions, and related surrogate measures. Based on these findings, recommendations are made for Phase II of this study.

CSERIAC conducted extensive literature searches of government and commercial databases, such as the Defense Technical Information Center (DTIC), to collect and organize background information on MOEs and MOPs for aeronautical systems, training, and MAJCOMs. The main source of MOEs in the scientific literature was T&E reports; however, most of these reports were either "classified" or "limited distribution" documents, and were not used for the present search. Because there was no comprehensive listing of aircraft-related MOEs/MOPs, CSERIAC began constructing one. Since the literature failed to contain summary articles on MOEs/MOPs, it was necessary to begin extensive "networking" with MAJCOM officials to identify the MOEs/MOPs which were not listed in DTIC. CSERIAC requested MAJCOM officials to provide information on these "undocumented" measures which are being used to determine the health of their aircraft systems and evaluate unit or team performance and force readiness. The resulting MOE/MOP information was organized into a MOE listing, organized by functional and mission area.

Numerous and Many Variants. While there is a limited set of AF missions, mission tasks and subtasks, AF MOEs and MOPs are as numerous as there are organizations attempting to fulfill those missions.

Not Well Documented in Literature. A major finding of this literature search is that because of the tremendous variety and number of MOEs/MOPs, a comprehensive listing does not exist in the literature, and is unlikely to be constructed and maintained. Further, MOEs/MOPs are not well documented in the literature, and must often be inferred or derived.

Sources of MOEs/MOPs. The following sources yielded significant MOEs/MOPs:

MIL-STD 1776A. The most extensive source of aircraft-related MOEs/MOPs was Military Standard (MIL-STD) 1776A, Aircrew Station and Passenger Accommodations (DoD, 1994). It contains a table of aeronautical MOEs/MOPs together with air tasks and subtasks, although it is not so comprehensive as to include all missions, tasks, subtasks, and formulas to compute the MOEs/MOPs. We identified no other current MIL-STD that contained MOEs/MOPs to this extent.

AFI 10-602. The most comprehensive listing of aircraft maintenance-related MOEs/MOPs was found in Air Force Instruction (AFI) 10-602, Determining Logistics Support and Readiness Requirements, Attachment 3 (USAF/LGMM, 1994). This listing provides formulas to compute MOEs/MOPs like "maintenance hours per flying hours," "mean time between failure," and "mean time to repair," which are the measures used to judge success at the AF wing organizational maintenance units. These measures are also used for evaluation purposes up the chain of command and for the evaluation of proposed acquisitions and modifications.

MAAs & MAPs. A third source of high-level MOEs/MOPs is the Defense Modernization Planning process (USAF/XOXP, 1996). In this structured process Mission Area Assessments (MAAs) are being developed for each mission area. The products of these assessments are documented in Mission Area Plans (MAPs) which include both MOEs/MOPs and deficiencies (Air Mobility Command [AMC]/Plans [XP], 1995). Also by implication, one can develop additional MOEs/MOPs from these deficiencies.

Acquisition Documents. Related to the MAAs and MAPs are the Mission Need Statements (MNSs) and Operational Requirements Documents (ORDs) which contain MOEs/MOPs for new or modified acquisitions. In the ORD a table called the Requirements Correlation Matrix (RCM) contains detailed MOEs/MOPs that are requested by the using MAJCOM for the new or modified system to achieve.

Test and Evaluation. Later in acquisition, the Test and Evaluation Master Plan (TEMP) is developed, detailing how the system will be tested. Either the AF Flight Test Center at Edwards AFB or AFOTEC will develop an extensive hierarchy of Critical Operational Issues (COIs), MOEs, and MOPs to assess operational effectiveness and suitability of the system (AFOTEC/XRC, 1995). T&E reports were helpful in identifying some MOEs/MOPs; however, classified reports were not examined to date. Remaining limited or classified T&E reports could be examined more closely (time permitting) for hierarchical relationships and MOEs/MOPs associated with each weapon system. Researchers may want to examine TEMPs which are negotiated with MAJCOM input, and

later in acquisition, T&E reports, to determine what the using command will accept as proof that a system is meeting the requirements documented in the RCM.

Readiness Reporting Systems. AF Policy Directive (AFPD) 10-2, Readiness (USAF/XOOO, 1993), requires that commanders assess the readiness of their resources. The prime readiness reporting system is the Status of Resources and Training System (SORTS). SORTS is the mechanism for assessing the readiness or "C-Rating" of each operational unit and reporting it up the chain of command. SORTS concentrates on four areas of determination: (1) personnel, (2) equipment on hand, (3) training, and (4) equipment condition. The AF version of SORTS is called the AF Integrated Readiness Measurement System (AFIRMS) (USAF/XOOOR, 1995; SofTech, 1985). Other forms of readiness data and reporting systems have been developed or tested to supplement SORTS because it was considered by the Government Accounting Office (1996) to be inadequate. These other systems include the following: the Joint Readiness Management System (Gillis, 1996), Joint Mission Essential Task Lists (JMETL) (Wagner, 1996), Joint Automated Readiness System (JARS) (Neal, 1996), and the Readiness Baseline Indicators project (Medlock, 1996). Each of these R&D systems offers a different emphasis, including the ability to assess the readiness to perform specific missions and the tasks associated with those missions. This readiness and reporting system emphasizes the importance of relating MOEs/MOPs to the mission.

Operational Readiness Inspections (ORIs). The famous "We're here to help." ORI visits by the Inspector General (IG) result in a report often used to justify additional funding or to shift resources to "fix" problems. AFI 90-201, Inspecior General Activities (SAF/IGI, 1996) describes the subjects that will be covered during ORIs. The Air Combat Command (ACC), Inspector General (ACC/IGIX 1996), has published supplements to this instruction that detail specific objective grading criteria, while other MAJCOM supplements contain more general and subjective criteria (AFSOC/IG, 1995; AMC/IGPS, 1996).

Command Status Briefings. These briefings provide MAJCOM commanders and their key staff highlights of the readiness of their units by weapon system. These briefings are especially important since they contain metrics which measure the "health" of the force that the commanders and key staff members have requested (AMC/LGQP, 1996). These briefings tell us their priorities. In some cases, the metrics are clearly defined in a MAJCOM pamphlet (AMC/LGQA, 1995), while other status briefings may tend to be more variable, be computed by staff analysts on an ad hoc basis, and lack full documentation.

Military Exercises. Both the AF and the Joint Chiefs of Staff conduct exercises on a continuing basis to ensure the force is ready to perform a variety of missions and contingencies. Exercises use real aircraft and simulated combat situations to test readiness and train war-fighting ability. Some last a few hours or days, while others are extended over a longer period. Blue (AF News Service [AFNS], 1996a) and Red Flag (AF Flight Weapons Center [AFFWC]/Public Affairs [PA], 1992) are examples of highly visible exercises; however, exercises are numerous and most are classified. Metrics are developed to aid in scoring the exercises; however, overall assessments are often subjective.

Wargames and Simulations. Wargames and simulations involve replicating warfare without actual combat, often involving computer simulations. Distributed simulation technology is now integrated into many exercises, competitions, and training programs. While the basic data on number of kills per mission and straightforward MOEs are readily available, detailed metrics are difficult to find in the literature about these distributed simulations. Wargames and model-type simulations include BATMAN & ROBIN (Federico, et al., 1991), AF Commander, Conflict: Korea, TAC Brawler (Survivability/Vulnerability Information Analysis Center [SURVIAC], 1994), and the Logistics Composite Model (LCOM) (Clark, 1989; Boyle, 1990). The latter two have also been used during systems acquisition to optimize MPT requirements with other weapon system requirements and design factors on aircraft systems like the F-22 and Joint Strike Fighter.

Military Competitions. Every year AF teams compete for top honors in their mission areas. Competitions appear to be one of the more interesting sources of MOEs/MOPs. For example, ACC's William Tell (ACC/DOOO, 1996) combines competitions for pilot and weapons director accuracy, munitions loading speed and safety, and aircraft maintenance proficiency. Other examples of competitions include the ACC-sponsored combined force competitions, Gunsmoke (AFFWC/PA, 1992) and combined mid- and long-range bombing competition, Long Shot (AFNS, 1996b), and AMC's transportation tanker and airlift Rodeo (AMC/PA, 1996).

Surrogate Measures. Surrogate measures are often developed for evaluation of exercises, wargames, simulations, competitions, operational tests, ORIs. Like MOPs, surrogate measures often do not measure the full MOE. Despite the fact that surrogate measures cover only some aspects of the MOE and only are able to approximate the MOE, these measures are often considered acceptable indicators of readiness conditions by commanders and high-level decision-makers. Since MAJCOM decision-makers are often willing to accept these surrogate measures as strong indicators of war-time outcomes, and since the surrogate measures may be more easily measured than MOEs, these measures may be more readily linked with MPT research and effectiveness measures than MOEs. Since these surrogate measures have already been established as credible, they may convey credibility to the MPT technology being tested.

Few Formal Comprehensive MOE/MOP Listings Available. Lack of comprehensive aircraft-related MOEs/MOPs listing and a dearth of unclassified scientific articles on the topic forced the labor-intensive use of networking to identify AF aeronautical MOEs/MOPs. Considerable information was available, but not from widely published sources. Two exceptions to this general rule were (1) MIL-STD-1776A, which contained an extensive, though incomplete, listing of operator/system-related MOEs/MOPs, and (2) AFI 10-602, Attachment 3, which contained a comprehensive listing of maintenance MOEs/MOPs. Through extensive networking, a number of other good sources of AF MOEs/MOPs were identified. Many of these sources are not formally published, are in the form of informal guides to analysts, or are published as MAJCOM supplements (not widely available).

MOE/MOP Taxonomy - A Work In-Progress. The CSERIAC Review and Analysis (Best, Gentner, Cunningham, & Schopper, in preparation) documents major MOEs/MOPs that were available in a taxonomy, and while this is not fully comprehensive, we believe that it contains an extensive sample of MOEs/MOPs, together with typical sources of MOEs/MOPs and a process for deriving them. As this taxonomy is refined, it should assist scientists in identifying MOEs/MOPs that might benefit from their research.

MAPs and Acquisition Documents Offer Research Targets. Since the MAJCOMs are defining deficiencies in MAPs and criteria-for-success in acquisition documents, it would behoove scientists to examine these deficiencies for possible research projects to improve human performance in the deficiency areas. If MPT researchers want to target their research toward a new weapon system or major modification, they should consult the RCM of major systems to determine how system performance could be improved through improved human performance.

Remaining Work. Since few displays of MOE/MOP hierarchies are found in the literature, it appears that the most difficult part of this study will be to develop an appropriate hierarchy for the taxonomy. It is possible that some of the classified and limited documents, such as T&E and Cost and Effectiveness Analyses (COEAs), which have not been reviewed to this point, may contain hierarchical listings of MOEs which could be helpful in developing ones tailored to depict common aircraft-related MOE/MOP structures. Based on the findings of this phase, a structured interview survey instrument will be constructed, and high-level decision-makers will be polled about their MOE/MOP priorities. The results will be compared with the written documentation on MOEs/MOPs, and adjustments made to the MOE/MOP taxonomy. MOFIMOP priority variances will be noted.

Specific recommendations for the next phase include the following: (1) review limited distribution documents (and classified, if necessary) to develop hierarchical listings of AF MOEs; (2) expand the MOE/MOP taxonomy to include late-arriving documents requested from AF sources; (3) review deficiencies documented in MAPs to identify current issues that could be linked with human performance metrics; (4) develop a structured interview and conduct a survey of high-level AF decision-makers, focusing on those in AF offices charged with maintaining war-fighting capability and keeping MAJCOM commanders informed on the readiness of their forces; and (5) focus on a limited number of mission areas where MOEs/MOPs are documented and where personnel can afford time to assist in specifying MOEs and MOPs when they are not available from the literature.

Air Combat Command (ACC/IGIX). (1996). AFI 90-201/ACC SUP 1, Inspector General Activities, 1 Jan 96. Langley AFB VA: Author.

Air Force Fighter Weapons Center (AFFWC/PA). (1992). Red Flag. Nellis AFB, NV: Directorate of Public Affairs: Author. [Online]. Available: ftp://ftp.primnet.com/users/c/cdaleb/Briefings/RedFlag.txt.

Air Force Operational Test and Evaluation Center (AFOTEC/XRC). (1995). AFOTECH 99-101, Test Concept Handbook, Jan 95. Kirtland AFB, NM: Author.

Air Force Special Operations Command (AFSO/IG). (1995). AFI 90-201/AFSOC SUP 1, Inspector General, 1 Dec 95. Hurlburt Field, FL: Author.

Air Mobility Command (AMC/LGQA). (1995). AMC Pamphlet (AMCPAM) 21-102. Unit Health of the Force Reports and Maintenance Analysis Guide, 18 Dec 95. Scott AFB, IL: Author.

AMC (AMC/XP). (1995). 1996 Air Mobility Master Plan (AMMP), 2 Oct 95. Scott AFB, IL: Author.

AMC (AMC/IGPS). (1996). AMC Instruction (AMCI) 90-201, The Inspection System, 15 Sep 96. Scott AFB, IL: Author.

Best, P.S., Gentner, F.C., Cunningham, P.H., & Schopper, A.W. (In Preparation). Evaluating war-fighting capabilities of aircraft systems, aircrew, and maintenance using measures of effectiveness and performance. Wright-Patterson AFB, OH: CSERIAC.

Boyle, E. (1990). LCOM Explained. Logistics and Human Factors Division, Wright-Patterson AFB, OH. Report No. AFHRL-TP-90-58. (DTIC No. AD-A224 497).

Clark, G. (1989). LCOM: An analysis tool for MPT. IMPACTS Bulletin, Vol 1(2). Washington, DC: HQ USAF.

Department of Defense (DoD). (1994). Military Standard. Aircrew Station and Passenger Accommodations (MIL-STD 1776A). Washington, DC: Author.

DoD. (1996). DoD Directive 5000.1, Defense Acquisition, 15 Mar 96. Washington, DC: Author.

Federico, P.A., Ullrich, R.R., Van de Wetering, B.L., Tomlinson, C.I., & Long, D.J. (1991). Human-computer interfaces for tactical decision making, analysis, and assessment using artificially intelligent platforms: Volume 1, software design and data base descriptions for BATMAN & ROBIN (NPRDC TN 91-20). San Diego. CA: Navy Personnel Research and Development Center. (DTIC No. AD-A241 509)

General Accounting Office/National Security and International Affairs Division (GAO/NSIAD)-96-105. (1996). Military Readiness: A Clear Policy is Needed to Guide Management of Frequently Deployed Units, 8 Apr 96. Washington, DC: Author.

GAO/NSIAD-96-170. (1996). Bottom-Up Review: Analysis of DOD War Game to Test Key Assumptions. 21 Jun 96. Washington, DC: Author.

Gillis, P.F. (1996). Joint Readiness Automated Management System (JRAMS). (Abstract and presentation material). Proceedings of the Military Operations Research Society (MORS) Symposium, Leveraging Technology for the Military Analyst. 18-20 Jun 96. Ft. Leavenworth KS: US Army Combined Arms Center.

Gould, B. (1995). Criteria for evaluating human systems technology effects on war-fighting capabilities. Brooks AFB, TX: Armstrong Laboratory/Human Resources Directorate (AL/HRT).

GUNSMOKE History. (1995). [Online]. Available: ftp://ftp.primenet.com/users/c/cdaleb/Briefings/Gunsmoke.txt

Lane, N.E. (1986). Issues in performance measurement for military aviation with application to air combat maneuvering (NTSC TR-86-008). Orlando, FL: Essex Corporation.

Medlock, K.V.T. (1996). The readiness baseline. (Abstract and presentation material). Proceedings of the Military Operations Research Society (MORS) Symposium, Leveraging Technology for the Military Analyst. 18-20 Jun 96. Ft. Leavenworth, KS: US Army Combined Arms Center.

Neal, P. (1996). Joint automated readiness system (JARS). (Abstract and presentation material). Proceedings of the Military Operations Research Society (MORS) Symposium, Leveraging Technology for the Military Analyst. 18-20 (Jun 96. Ft. Leavenworth, KS: US Army Combined Arms Center.

Secretary of the Air Force (SAF/IGI). (1996). AFI 90-201, Inspector General Activities, 1 May 96. Washington, DC: Author.

SofTech, Inc. (1985). Air Force Integrated Readiness Measurement Systems (AFIRMS): Analysis of Military Airlift Command (MAC) Capability Assessment Metrics. Alexandria, VA: Author. (DTIC No. AD-A170 523)

Survivability and Vulnerability Information Analysis Center (SURVIAC). (1994, Nov). Model spotlight: TAC BRAWLER. SURVIAC Bulletin. Wright-Patterson AFB, OH: Author.

United States Air Force (USAF/XOOO). (1993). AFPD 10-2, Readiness, 14 Apr 93. Washington, DC: Author.

USAF (USAF/LGMM) (1994). Air Force Instruction (AFI)10-602, Determining Logistics Support And Readiness Requirements, 20 Jun 94. Washington, DC: Author.

USAF (USAF/TEP). (1994). AFI 99-103, Test and evaluation process, 25 Jul 94. Washington, DC: Author.

USAF (USAF/XOOOR). (1995). AFI 10-201, Status of Resources and Training System, 1 Oct 95. Washington, DC: Author.

Wagner, M. (1996). Joint Mission Essential Task Lists (JMETL): A foundation for mission-based readiness assessment. (Abstract and presentation material). In Proceedings of the Military Operations Research Society (MORS) Symposium. Leveraging Technology for the Military Analyst. 18-20 Jun 96. Ft. Leavenworth, KS: US Army Combined Arms Center.