Back to Table of ContentsExpert System Planning: German Federal Armed Forces Psychological Service1
W. A. Sands
M. F. Wiskoff
INTRODUCTION
The number of young people eligible for conscription into the German Federal Armed Forces is decreasing, making the successful planning and execution of manpower management plans and policies increasingly important. In conjunction with the decreasing number of potential conscripts, the necessity for savings requires the downsizing of the Psychological Service of the Federal Armed Forces. Thus, the various tasks performed in processing personnel for entrance into the armed forces, including screening, assessment, and placement, must be performed by fewer people. Accomplishing this workload with fewer personnel resources can be achieved, in part, by leveraging the effectiveness of the remaining personnel. Time spent on routine tasks must be reduced to allow the remaining psychologists to focus on the situations and decisions that require their individual attention and expertise.
All military offices staffed by the Psychological Service of the Bundeswehr using psychological diagnostics will be equipped with computer-assisted testing (CAT) facilities. These CAT facilities will use computers to perform many of the tasks that, historically, have been accomplished by professional personnel. This computer-based support is a key to leveraging the productivity of the reduced psychological staff. As part of this effort most effectively to leverage the expertise of the remaining psychologists, a CAT-assisted expert system wilt be designed, developed, evaluated, deployed, and implemented. This system will provide a knowledge base containing expertise of the psychologists and a set of rules for making decisions based upon their expert knowledge. The plan is to incorporate the new expert system into the "Computer-Assisted and Adaptive Testing (CAT) of the Psychological Service," an approved system. 2
The development of the proposed expert system is being accomplished in two phases: (1) a market survey of the existing products and conceptual models, and (2) development of the actual system, specifically designed for the Psychological Services working environment. The work described in this paper addresses the first phase.
The study requirement entailed a review of the existing products and research literature on expert systems, as applied to personnel management. Early in the study, it was determined that focusing solely on expert systems would exclude a vast amount of information that was relevant to the objectives of the study. Many computer-based systems and models provide information that personnel decision-makers find very useful. These systems and models, while not strictly speaking "expert systems," are computer-based Decision Support Systems (DSS). Therefore, it was decided to expand the scope of the study and address the broader subject of DSSs for personnel management. This would include, but not be limited to, expert systems.
_______________________________________________________________________
1
Other approaches which appear to offer promise include neural networks and genetic algorithms. In addition, many of the models in use by the various U.S. Armed Services address personnel flows, projections, resource allocations, and person-job matching algorithms. All of these approaches fall under the DSS umbrella. The products and the literature reviewed for this study, as well as the interviews conducted, reflect this broader scope.
APPROACH
The first step in reviewing products was to identify candidate products. This was accomplished by reviewing the research literature and advertisements in professional journals describing commercially available products that seemed promising. A facsimile (fax) letter was sent to each of the companies identified, explaining the purpose of the study and requesting product information. Regular mail was used for those companies having no listed fax numbers.
Interviews were scheduled and conducted with sixteen individuals having a variety of types and levels of responsibilities for personnel management decision support systems. The responsibility levels of the interviewees ranged from manpower analysts to policy-makers.
A computer-based literature search was conducted in two very large reference databases: (I) the Educational Resources Information Center (ERIC), and (2) the on-line version of Psychological Abstracts (PsycINFo).
RESULTS AND DISCUSSION
Products
Product information was requested from forty-five companies. Information was received from twenty-nine companies. The information obtained from four of these companies indicated that the products were not relevant to the objectives of the study. Product reviews were conducted on the information received from the remaining twenty-five companies. The quantity and quality of the information provided by the companies covered a wide range, from a few pages of information to extensive packages containing product brochures, feature descriptions, product reviews, examples of product applications, information on the company, training opportunities, product-ordering information, and demonstration diskettes. A total of 284 printed items was received, containing over 800 pages of information. In addition, five of the companies provided diskettes containing product information and/or product demonstrations.
Products were reviewed for eight criteria specified in the project requirements: (1) open/relational database structure, (2) psychological diagnostic rules, (3) psychological diagnostic classification, (4) robust for incomplete datasets, (5) case-based reasoning (CBR) techniques, (6) complex statistical links, (7) complex filters or rule algorithms, and (8) recommendation for use. Companies were rated "Yes" on an individual criterion if the materials they submitted indicated that their product would support that criterion, or if support for the criterion could be readily inferred from the information received. Companies were rated "No" if their materials explicitly indicated a lack support for that criterion, or if no information supporting a '"Yes" evaluation could be readily inferred from the materials received. Eight companies were recommended for consideration as primary tools, six companies were recommended as secondary tools, and the remaining eleven companies were not recommended for further consideration.
Interviews
A total of sixteen people were interviewed, including manpower managers with policy-making and operational responsibilities, and researchers with experience in the design and development of decision support systems. The information presented below summarizes various comments and suggestions obtained during the interviews. Many of the points were raised independently by multiple people. The comments reflect the different perspectives contributed by policy-makers, operational analysts, and researchers.
Decision Support Systems offer many important advantages. They enable analysts to address complex questions that arise from within their own organizational hierarchy or from outside (e.g., Congressional staffers) in a timely manner. The information is quantitative, accurate, consistent, verifiable, and defensible.
The models leverage the limited resources (personnel and time) which can be dedicated to addressing a problem. Answers that once took days to produce can be generated in hours or even minutes. Alternatively, more scenarios can be addressed in a fixed period of time.
Various problems associated with some DSS models were also identified. For example, the development time for a model sometimes becomes so long that the problem which initiated the development effort has disappeared by the time the model is available. in other cases, the model turns out to be prohibitively expensive.
Some models are so complex and cumbersome to use that they "die under their own weight." This can happen when researchers become focused on using the "latest and greatest" methodological approaches and lose sight of the original rationale for the model. While this scientific focus may lead to many journal article publications and, thereby, enhance the professional stature of the researcher, it does not contribute to solving the original problem. If a model is either too complex to be understood or too cumbersome to use (or both), it will not be fully utilized.
Personnel turnover presents other potential problems.. For example, transfer of knowledge about a model to incoming personnel may not be adequate (and may be nonexistent). If the model documentation is poor, the likelihood of full model utilization diminishes. Another potential problem is that new personnel may adopt a "not invented here" attitude about a model if they played no role in its development. The high turnover in many units makes these problems particularly important considerations in military organizations.
Finally, if a model is delivered to the user without adequate testing, serious problems can result. If the model developer shortcuts the test and evaluation phase, some problems may not be discovered and solved prior to operational use. When a model suddenly produces results that are obviously erroneous to an experienced decision-maker, the credibility of the model can be seriously damaged. The problem might be very easy for the developer to fix, but the model’s (and perhaps the modeler’s) reputation can remain tarnished. In most cases, extensive and thorough testing can prevent this type of problem.
A number of useful recommendations were offered by the persons interviewed. Identification of the key players in a model development effort is essential. The model developer should work very closely with the customer(s) before, during, and after the model is built. Many times, this "handholding" is the key to sustained model use. The customer must define the model objectives as precisely as possible. Key operational constraints should be identified.
An extreme concern for global optimization in a DSS can be counterproductive. The forced translation of quantities that are naturally expressed in different metrics into a common metric (to provide an index of optimization) can reduce a model’s usefulness.
An alternative approach is to supply the decision-maker with the tradeoff information on the components and let the person integrate the information and make a decision. This suggestion was described as "keeping the human in the decision loop."
Models should be built and tested incrementally. Evaluation of model results should emphasize producing information that leads to the correct decision, not statistical accuracy. Users should be prepared for, and comfortable with, approximate answers. It is possible for a model answer to be only moderately accurate from a statistical standpoint, yet be very valuable if it influences the policy-maker to make the best decision, especially in large-scale or high-cost applications. Models developed for different purposes should be used to check on each other whenever possible. Finally, a concern for "user friendliness" in the model interface should be paramount to facilitate user acceptance and support.
In summary, the vast majority of persons interviewed strongly supported the value of DSSs. They stated that these systems provide critical information to the decision-makers in a timely fashion. In large organizations (e.g., the military), even small improvements can yield considerable benefits. The people interviewed believed that the funds invested in DSSs yield a high return on investment.
Literature Search
The initial step in the search strategy was to determine the keywords that would identify those references relevant to the study objectives ("hits"). In some cases, the search for a single keyword yielded an excessive number of references. When this situation occurred, an additional keyword was paired with the initial keyword to narrow and refine the search. For example, the keyword "evaluation" produced over l 00,000 hits in the ERIC database, because such a broad search locates references involving "evaluation" that are totally unrelated to the purpose of this study. Narrowing the search by pairing "artificial intelligence" with "evaluation" produced a total of 139 hits. The number of hits for a keyword was different for each database. Therefore, the paired keyword searches required were somewhat different.
After determining the number of hits in a database for a particular keyword, the abstract of each targeted reference was reviewed. Relevant references were included in the computerized reference database produced as part of this study. Having the reference information in a computerized database enables the user to sort the references with flexibility, according to different needs.
RECOMMENDATIONS
No existing commercial product or DSS developed by a military organization was discovered that could be used directly, without further development, to meet the needs of the Psychological Service for an expert system for use with potential conscripts for the German Federal Armed Forces. It is necessary to build such a system, either by developing entirely new software (not recommended) or by using a commercially-available, off-the-shelf, expert system shell and custom-tailoring a system for the operational environment.
Eight of the twenty-five commercial products reviewed could serve as a vehicle for developing a custom DSS. Information from these eight companies should be evaluated in greater depth, using the results of the ongoing needs assessment study and a more detailed and comprehensive list of technical and operational criteria. Input for developing the new evaluation criteria should be provided by both policy and operational personnel. Examples of technical criteria might include ease of programming the system, computer platforms supported, computer hardware requirements (e.g., random access memory size, hard disk storage capacity), flexibility to accommodate future changes, etc. Examples of operational criteria might include ease of operation and costs (purchase base price, license fees, etc).
The most promising DSS models identified in the technical literature should be reviewed further. It is possible that some of the operational problems identified in the field (but not the entire system) could be addressed using these models. Additional information that would be useful to obtain on the most promising existent DSS models might include availability, documentation, programming language, etc.
It would be beneficial for the Psychological Service of the Bundeswehr to convene a meeting of experts in the application of expert systems and DSS models to military personnel management problems. This meeting could provide valuable insights for planning, building, and implementing the required system for use by the Psychological Service.