This proposal outlines a technique for developing a performance methodology for evaluating artificial intelligence systems. The methodology proposed is based on research done by the futures group under the sponsorship of the national science foundation and the department of defense. The methodology, termed "state of the art" (SOA) technology measurement analysis has been successfully used to analyze and forecast the performance of advanced weapon systems and other complex technologies. The methodology uses informational gathered from expert system users to develop performance system parameters. The information gathered is structured interviews (Delphi techniques) and is analyzed using a linear combination of performance parameters and a weighting factor describing the parameter's importance for a specific task. The analysis is carried out by developing a history of the advancement in technology of the subject. Advancements in technology are identified and explanations for them are sought. Future impediments to progress in the subject are identified. This methodology has been successfully used to analyze the performance of subjects as diverse as infrared sensor technology and computer software. Although the methodology is open and flexible, a quantitative assessment of the technology from the point of view of the user is produced.
