SBIR-STTR Award

Lumina Decision Systems, Inc. will develop software which obtains expert judgment on uncertain quantities in the form of probability. Bayesian probability techniques are increasingly used for a wide variety of decision-making and risk-analysis. Currently, specially trained analysts are required to obtain expert probability judgments to avoid cognitive biases. The time now required of the analyst and expert is burdensome, often rendering the approach impractical. A reliable and easily used automated tool for the elicitation of uncertainty could substantially reduce these costs, and render the approach far more widely usable, as a natural adjunct to decision and risk management software tools. The software will allow assessors to use qualitative phrases, such as unlikely or very frequent, and map them into numerical probabilities. They will develop and evaluate methods to calibrate each user's language to obtain reliable and stable mappings. A second focus will be to develop geometric, animated density assessor for the direct assessment of continuous probability densities. This method will use a flexible family of distributions, allowing rapid graphical specification of unimodal, bimodal, bounded, and unbounded distributions. They will compare the time, effort, and reliability of continuous and discrete probabilities obtained using standard methods with the results from our prototype assessment tool. They will develop calibration methods for mappings between verbal phrases and numerical probabilities, and evaluate their reliability and stability.

Quantitative models, virtually all of which are subject to uncertainties and most of which must rely on expert opinion for estimating key uncertain input quantities, are increasingly widely used to support the making of key decisions in public and private sectors. This Small Business Innovation Research Phase II project from Lumina Decision Systems, Inc. describes research seeking to develop and evaluate an innovative decision support system software tool to assist experts in expressing their uncertain knowledge in the form of probability distributions. There is a growing recognition on the part of builders and users of quantitative models of the need for a such a tool, but heretofore usage has been constrained by the lack of availability of a tool that is both easy to use and designed to overcome the difficulties people have expressing uncertainty in meaningful and coherent ways. In Phase I, the practicality of two innovative methods was developed and experimentally established: (1) animated density functions and (2) verbal probability phrases, such as probable or very seldom. In Phase II, a prototype tool for probability assessment will be refined, providing these two and six other assessment methods with associated calibration and debiasing techniques...These methods will be evaluated experimentally to measure ease of learning and use, effort required, confidence in results, and reliability and calibration of the resulting distributions. These results will be used to guide users in choosing the most cost-effective methods to suit their needs and skills. Methods to assess multivariate distributions and multimedia courseware on probability assessment will also be integrated into the package. The expression of expert judgment in the form of subjective probability distributions is becoming accepted for risk analysis in areas where uncertainty is critical, including finance, oil and gas exploration, risks to the environment, medical diagnosis, and bidding decisions . This project will contribute effective means of eliciting beliefs regarding the likelihood of outcomes, and thus it will serve to enhance existing capability of quantifying expected costs and benefits in a variety of contexts