Assembling just the right people with just the right knowledge and skills is increasingly critical for the successful conduct of missions. Doing so is most important, and most difficult, when missions respond to novel challenges, call for heterogeneous knowledge widely dispersed in the organization, involve systems or procedures for which there are few experts—for example for new or short production run devices—and when there is little time to prepare. These are all situations to be expected in future war-fighting scenarios. The problem would be to simultaneously find those servicemen (or civilians) who can most rapidly be trained to operate new systems, and to design special training that will optimally build on each person’s prior training and experience rather than spending time teaching things already known. Current solution methods for such problems require large investments of expert labor and are often either unacceptably slow or insufficiently effective. While a well developed engineering art, in the best case course design takes many months of specialized analysis and trial.

Thus, more effective methods are desired for characterizing, locating, and training servicemen who can optimally perform the set of duties required by any new mission. This calls for information technologies that can (a) represent knowledge and skills, (b) identify persons with all or parts of the knowledge and task experience required by a mission—wherever and in whatever occupation they are currently, (c) determine precisely what, if any, retraining each person needs in order to perform which new duties, (d) reduce the effort required to create new training programs, and (e) minimize the time required for training and retraining. The objective of this research was to develop and test the practical capability of a new knowledge representation technology, Latent Semantic Analysis (LSA).

Latent Semantic Analysis (LSA) is a powerful new machine-learning method for automatically extracting and representing knowledge in massive databases of relevant electronic text (Deerwester, Dumais, Furnas, Landauer, & Harshman, 1990). It was developed through ten years of basic and applied research supported by Bell Communications Research (now SAIC), DARPA, ONR, ARI, NASA, AFRL and others. LSA has been extensively validated in both controlled experiments and field tests (Landauer & Dumais, 1997; Landauer, Foltz, and Laham, 1998; Landauer, 1998). The present innovative data mining application of LSA exploits the explicit and implicit knowledge that already exists in extensive computer files of systems documentation, training and test materials, task analyses, and service records. The current research used a small subset of existing Air Force data relevant to the tasks performed in three Air Force occupations. The results showed that the technology could accurately estimate the similarity of each task or occupation to every other task or occupation, determine the degree of match of each airman to every task or occupation, determine which airmen could most easily take the place of others, and indicated that LSA has the potential to identify in detail and match the knowledge components required by new systems with those contained in segments of existing training materials, and with the experience of individual airmen.

Job placement or occupation assignment. Applications to Air Force job assignment were what was most directly illustrated by the research. The simplest case is direct replacement of one airman with another. For this, a query takes the form of the to-be-replaced airman’s identification number, and the k most similar airmen are listed known to the system (potentially all those in the Air Force plus others where relevant) are returned and listed in terms of their overall task-experience pattern—the closeness of their points in the joint semantic space representing tasks, occupations and airmen. Their complete service records can then be displayed and compared. If it is desired to add a new member to a work group, the descriptions of those tasks that are most in need of additional help can be entered as the query and the system will list in order those airmen whose total experience is most like the new job requirements. Note that in performing this match, LSA goes beyond simply counting the number of tasks in common between the wanted list and the service record, instead factoring in previous experience (and, later, training) in occupations and tasks that are similar but not identical to those in need of performance. Thus, it would be quite possible, in the absence of any airman who had done any of the prescribed tasks, to nevertheless find one or more candidates who had done similar work, the estimate of similarity having been automatically induced by LSA from the entire corpus of data without human intervention.

The technique could be used to add people to perform new jobs, by adding to the query a free-form description of the tasks involved. Because LSA captures semantic and conceptual similarity of verbal expressions, it will correctly match ad hoc task descriptions with official task definitions and job descriptions. The system can also form a representation of the overall mix of tasks required by a group by combining representations of the knowledge possessed by all its present members. In case of downsizing, the system would make it possible to find a set of airmen to transfer out of the group that would either leave it most like its previous composition, or desirably modified, again without relying on a crude counting operation or intuition.

The opportunity and manner of application for selecting airmen for missions, for example expeditionary war-fighting missions with unique challenges, is relatively straightforward. Given a careful verbal description of the mission, including all the tasks to be performed, the equipment, weapons, devices, procedures, numbers of airmen needed in each role, and perhaps even factors such as locale, terrain and likely weather and other challenges, the LSA matching technique would rank airmen for suitability to each task on the basis of the totality of their previous task and occupational experience, along with, if available, relevant (as determined by LSA) test scores and performance ratings.

Curriculum overlap analyses. The Air Force (like other military and civilian organizations) offers hundreds of specialty training courses, many of which overlap substantially in content, many of which may contain content no longer relevant to tasks currently in demand, and some which are missing content made desirable by changes in technology, missions or staffing. In many cases it would be desirable to combine, condense, or modify courses. Teaching unnecessary numbers of courses or redundant components in multiple courses is expensive in instructional staff and facilities, and even more expensive in wasted student time and resources. Teaching material that is sub-optimally matched to work requirements, either by being superfluous, redundant, or by failing to equip airmen with the best skill sets for all the tasks it would be desirable for them to be able to perform, is probably even more expensive in the long run.

To rationalize the content and organization of content for multiple training programs, a method is needed by which the overlap in course content can be easily assessed. Presently such analyses are performed, if at all, by highly labor intensive efforts by subject matter experts and training specialists. We have already demonstrated that LSA can do this kind of analysis automatically to a quite useful degree. Our studies were based on analysis only of AKT items, but appeared to give a great deal of useful information about course overlap.

LSA can also measure the overlap between course content and the full range of tasks performed in many different occupations. Information from such analyses will suggest where the training needed for different occupations overlaps and might be combined, where training is lacking, point to components that may not actually be needed at all, and, in some instances, suggest ways in which occupations might be restructured to increase training efficiency. LSA methods will not solve these problems completely, but we believe it can offer highly useful information for planners that is currently unobtainable or prohibitively expensive.

Instant rapid training materials. In brief, the way in which we envision that LSA would be employed in helping to create "instant" rapid training programs might be as follows. The component knowledge needed and tasks to be performed for a new device, system, or procedure would be carefully described by designers and relevant subject matter experts. LSA would determine the degree of match of each component to a wide range of tasks performed in the Air Force and to every paragraph in every possibly relevant training or operations manual. Tasks and paragraphs would be ranked by estimated relevance to the new system, and the LSA similarity of each paragraph to each task determined. In the quickest and dirtiest version, a custom retraining document for each candidate could initially be compiled from paragraphs highly relevant to the new system that are not highly similar to tasks the candidate has previously performed. In the case of urgent need for a small number of trainees, a subject matter or training expert could then edit each version. In case of need for large numbers and more available time, the collection of paragraphs could be crafted into a simple computer-based training program with branching to permit trainees to skip parts they already know.

Description of Latent Semantic Analysis.

Latent Semantic Analysis (LSA) is an advanced intelligent search engine technology that goes beyond matching on keywords and their Boolean combinations. LSA uses a powerful matrix computation, singular value decomposition (SVD), that exploits recent computer power to automatically estimate the degree of contextual similarity among words and text passages in a domain by a training analysis on large bodies of representative text. After training, LSA represents natural-language passages, either old or new, as vectors in a high dimensional semantic space, and measures similarity of the meaning by the closeness of vectors. LSA has been shown effective for applications such as those proposed here in several ways, e.g. by choosing instructional text optimally matched for learning by individual students with varying prior knowledge (Wolfe, Schreiner, Rehder, Laham, Foltz, Kintsch, & Landauer, 1998), by correctly grouping AF AKT items by occupation, and by improving automatic text retrieval in the face of terminological variation that causes frequent failures in other search methods (Landauer, Laham, & Foltz, 1998).

LSA concentrates on learning to represent the similarity of the meanings of words. It does this in such a way that a representation of the meaning of combinations of words, e.g. paragraphs, can be computed as a simple additive function of the word representations. In brief, LSA induces a representation of the meaning of words and of passages of words, such as sentences or paragraphs, by analyzing the way in which all the unique word types, usually 20-100 thousand of them, are used in a large collection of text passages, usually 2-100 thousand of them. In a sense, it represents the meaning of a word as a kind of average of the overall meanings of all the passages it appears in, and the meaning of a passage as a kind of average of the meanings of all the words it contains. It accomplishes this "conjoint" fitting, or mutual constraint satisfaction, with a matrix-algebraic technique called singular value decomposition (SVD). SVD has become applicable to such large databases only in recent years as computers have grown in speed and capacity and algorithms have grown in efficiency.

In this research LSA was "trained" on many thousands of Air Force task descriptions and individual airman task-analysis reports, supplemented with a large corpus of text of the kind that a typical airman would have read since third grade. After LSA processes such a dataset it produces a representation of each word and each passage as a point in a high-dimensional "semantic space." It can also generate a point for any new passage by a matrix algebraic combination of its words.

Here is the important result. An estimate of the similarity of meaning of any two words, or of any two passages, or of any word to any passage, can be easily calculated as the angle between them in this semantic space. Even more importantly, a large amount of prior research has shown that these measures of semantic similarity correspond closely to how similar the two words or passages are in meaning as interpreted by people. The measure of similarity is a continuous number, reflecting the fluidity and fuzziness of language. The measure of similarity between passages reflects much more than the number of literal words they have in common (although if all the words are identical, the semantic distance is zero). Indeed, in many cases it will show two passages with no words in common to be similar in meaning, if they are, and two passages with many overlapping words to be quite different, if they are.

Moreover, in many comparisons, direct overlap measures yield useless results, while LSA measures simulate human judgment and performance accurately (Landauer, Laham, Rehder & Schreiner, 1997). Single words have meaning. Whole paragraphs have overall meanings. LSA captures and represents the information inherent in the relation between the two, and what it captures empirically turns out to be sufficient for determining the degree to which two utterance mean the same thing to a close enough approximation for many important purposes. LSA reflects the overall semantic or conceptual content of a passage, what it is about, and ignores many details of what it says or how it says it that might be important in some cases. However, for representing how much two tasks or airman service records, or the like, have in common, the similarity of overall semantic content, what kinds of things were done, is what counts, and, as in many other applications of LSA has proven to be more than adequate.

Description of LSA representations

Air Force occupational analysts have successfully employed a "Task Inventory" approach to categorize common tasks within occupational specialties. The "Airman Classification Structure Chart" details the 190 Specialty Occupations currently active. An extensive amount of data developed by AFOMS exists describing occupations, at least in part, as lists of their relevant task duties. In Occupational Surveys, personnel are directed to estimate their time on various tasks. These Occupational Surveys served as the source of data for the research.

The data from three air force specialty codes, Aerospace Mechanics (2A5X1), Physical Therapists (4J0X2), and Medical Services (4N0X1) were used. A semantic space was developed which included 20,000 documents, or object representations. The objects were classified as occupations (full set of tasks for each AFSC; N=3), duty lists (tasks grouped into functional units; N=44), tasks (individual task units; N=2121) and airmen (task & case record data; N=9215). Additional randomly selected documents (N=8617) not related to the AF data, were included to provide additional examples of words used in context so that LSA could create a more robust space with a more extensive vocabulary. The total number of "objects" in the system is 20,000—a small dataset compared to LSA’s capabilities, but enough to do a fair job in estimating statistical regularities.

By adding less than 9000 general reading documents, the semantic representations of more than 50,000 additional words were included in the space. This additional representational power allows for user synonyms to match task analysis keywords. For example, the space would consider "tire" to be a synonym for "wheel", or "physician" a synonym for "doctor" even if "tire" and "physician" were never used in the task analysis data.

Results from Modeling Studies

Several experiments were conducted over the course of this research to validate the quality of the representations within the developed semantic space.

Course content overlap. We conducted a study on the similarities of content between a set of USAF specialties based on the content of their course exam items. We were provided with text of the question and correct alternative answer for each of 100 items for each of 95 AKTs, the content having been encrypted before being sent by replacing each unique word wherever it appeared with a previously assigned string of random digits. From this we constructed a matrix of ~9,500 AKT items by ~20,000 word-types and performed LSA with 300 dimensions to obtain a vector representing each test item.

To test how well LSA had captured the semantic similarities of the domain, we computed the similarity, measured by cosine, of each item with that of 100 randomly chosen items from the same test and with 100 randomly chosen items from other tests. Item pairs from the same test had higher average cosines for every test (p<.0001), significantly so in 82 of 95 cases (p<.05). A subsequent hierarchical clustering of vectors for whole tests (each the vector average of its item vectors), done blind, produced groupings later judged to be intuitively meaningful and useful by knowledgeable AF training researchers.

Comparison of Airman similarity. In the first analysis of the semantic space developed from the AFOMS data, the system’s judgments of similarity for airmen within an AFSC were compared to the judgments of similarity between AFSCs. The average similarity rating were calculated between 3000 randomly sampled pairs of airmen either within an occupation, or between occupations. Each airman was compared with a random sample of others both within their own AFSC and between the alternative AFSC. The judgments of similarity of airmen within an AFSC are significantly greater (p<.0001) than the judgments of similarity between airmen (see Table 1).
 

Clustering of Duty List Items. Another experiment looked at how the representations for the three Occupations and the 44 Duty Lists clustered in the semantic space. The analysis shows the cluster distribution of shared and unique duty lists among the occupations. The within AFSC duty lists clustered tightly around their respective occupation, while those duties that are shared across occupations, such as training, management and supervisory tasks, clustered together and distinct from the occupations regardless of originating specialty and differences in specific task representations.

HeadHunter—A WWW-based Intelligent Search Agent

The HeadHunter software represents an initial demonstration of a usable World Wide Web based Intelligent Search Agent based on the LSA technology.. Currently the system only has knowledge of three AFSCs, however, even with its limited knowledge base, it demonstrates the necessary capabilities to match mission and job requirement statements with military personnel and training data. By measuring semantic similarity of training materials and tests, it facilitates combining occupations based on core competencies and similar work activities. It also helps to identify individuals qualified for work activities for which no current occupation exists.

An organization that has acquired a new or revised system can develop detailed descriptions of the activities required to operate or to maintain the system, based on system requirements documents, operations manuals or provided by subject matter experts. Given such descriptions, and assuming an increase in its knowledge, HeadHunter could automatically identify current jobs on fielded systems that are similar in component work activities and in their requirements for training. It could also identify similar paragraphs in existing course materials and rank them by probable relevance to work with the new system.

New occupations could be structured around these activities and new sets of training materials assembled, at least in major part, from subsets of existing material. In addition, individuals who work in jobs that use subsets of the competencies and experience required can be identified. This may permit the immediate employment of appropriate personnel or their more rapid and effective re-training for work in support of new systems. In this way, HeadHunter could help the military to exploit Internet resources to achieve information superiority.

In the occupational domain, this effort may ultimately produce a cost-effective capability to systematically mine occupational personnel and training databases to develop new job and training structures to support a variety of requirements. This capability will help employers identify critical characteristics and competencies associated with work activities and then to identify individuals who have the requisite experience and competencies to perform the identified work activities.
 

Acknowledgements. This research was performed as part of a Phase I SBIR award to Knowledge Analysis Technologies, LLC by the Air Force Research Laboratory and is currently in Phase II (Contract no. F41624-99-C-5003) The HeadHunter system is located on the internet at http://LSA.Colorado.edu/HeadHunter. Please contact Darrell Laham at dlaham@knowledge-technologies.com or Dr. Winston Bennett, Jr. at winston.bennett@williams.af.mil for a required password for using this system.

Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing By Latent Semantic Analysis. Journal of the American Society For Information Science, 41(6), 391-407.

Landauer, T. K. (1998). Learning and representing verbal meaning: The Latent Semantic Analysis theory. Current Directions in Psychological Science, 7, 161-164.

Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The Latent Semantic Analysis theory of the acquisition, induction, and representation of knowledge. Psychological Review, 104, 211-240.

Landauer, T. K., Foltz, P. W., & Laham, D. (1998). Introduction to Latent Semantic Analysis. Discourse Processes, 25, 259-284.

Landauer, T. K., Laham, D., Rehder, B. & Schreiner, M.E. (1997) How well can passage meaning be derived without using word order: A comparison of Latent Semantic Analysis and humans. In M. G. Shafto & P. Langley (Eds.), Proceedings of the 19th annual meeting of the Cognitive Science Society (pp. 412-417). Mawhwah, NJ: Erlbaum.

Landauer, T. K., Laham, D., & Foltz, P. W., (1998). Learning human-like knowledge by Singular Value Decomposition: A progress report. In M. I. Jordan, M. J. Kearns & S. A. Solla (Eds.). Advances in Neural Information Processing Systems 10, (pp. 45-51). Cambridge: MIT Press.

Wolfe, M. B., Schreiner, M. E., Rehder, B., Laham, D., Foltz, P. W., Kintsch, W., & Landauer, T. K. (1998). Learning from text: Matching readers and text by Latent Semantic Analysis. Discourse Processes, 25, 309-336.
 

Next paper