Frederick M. Siem
Air Force Armstrong Laboratory Human Resources Directorate

This session consists of four presentations and demonstrations relating to research sponsored through the Small Business Innovative Research (SBIR) program. The first speaker will discuss a pilot selection test designed to measure Crew Resource Management (CRM) skills. This test, the Situational Test of Aircrew Response Styles (STARS), is administered using a flexible software shell that allows for on-line scoring and the collection of examinee characteristics such as response latencies. Data from a test validation study will be presented, along with a demonstration of the software.

The next speaker will describe and demonstrate a video-based version of the STARS instrument. In the video, viewers see a CRM problem situation acted out and are asked to identify most and least effective behavioral responses. Use of such a video-based assessment method for both screening and training personnel will be discussed.

The third speaker will discuss how principles of computer adaptive testing can be used to develop computerized adaptive rating scales (CARS). The research objective is to develop more precise estimates of ratee performance than is possible with typical rating scales, with particular applications to non-technical or contextual performance. A sample CARS will be demonstrated.

The final speaker will discuss a software product, LTM, that allows for users to assess the practical effects of various training design decisions on outcomes such as retention of knowledge and skill. The speaker will discuss the conceptual basis for the LTM tool, the results of a feasibility study, and current efforts to validate empirically the LTM algorithm for predicting retention.