This Small Business Innovation Research Phase I project will carry out research and development on the AIDE assistant for data analysts. The goal of this Phase I project is determine which research issues must be addressed before AIDE can become a commercially viable product. Currently, analysts in government, corporations and universities rely on software that has changed little in decades. Exploratory data analysis, by which we mean finding hidden structure in large datasets, is painfully slow, because conventional statistical packages require users to specify every operation. Moreover, conventional packages lack the intelligence to interpret the results of operations. In contrast, our AIDE package can interpret intermediate results, so when, for example, it discovers clusters in data, it autonomously examines each for internal structure and then combines these local analyses into a summary of the entire dataset. AIDE is designed to conduct thousands of statistical operations under the strategic guidance of analysts. The analyst steers the exploration of a dataset instead of specifying every operation; AIDE is an assistant instead of a fancy calculator. Estimates are that three million statistics packages are installed in the U.S. If even a fraction of those users switched to AIDE, the commercial impact would be very significant. Potentially AIDE could revolutionize exploratory data analysis and save countless hours and dollars.
