For this effort, we propose to explore methodologies that facilitate scenario-based testing. Several factors combine to make such an approach feasible. The first is recognition of the importance of training and assessment in job contexts and the theoretical support for such approaches. Second, event-based measurement approaches can be applied to the development of meaningful performance measures with excellent psychometric properties, minimizing some of the problems traditionally seen in measuring performance in job or simulated job settings. Third, computer-based formats may serve to standardize the implementation of hands-on tests, which already work pretty well, and automate the implementation and scoring process. In Phase I, we plan to develop a functional specification for such a system through the research and development of a prototype system for one enlisted Air Force specialty. A key issue addresed for proof of concept in Phase I is the reliability and validity of domain sampling techniques. In Phase II, the methods will be applied to additional Air Force Specialties. The result will be a software development tool for a scenario-based testing system.
Keywords: SCENARIO-BASED TESTING COMPUTER-BASED TESTING COGNITIVE TASK ANALYSIS HANDS-ON TESTING HUMAN PERFORM
