SBIR-STTR Award

One of the major problems facing data management today is that collection capabilities far outstrip analysis capabilities. This is particularly true of scientific and engineering data sets. Analysis methods require access that cross file/data granule boundaries. We propose to address this problem by integrating unified data models of scientific data as embodied by the widely used data formats CDF, NetCDF, and HDF through the use of a commercially available data management system, METEOR. We propose to customize METEOR to handle the access requirements as imposed by the CDF, NetCDF, and HDF scientific data models. In particular this will involve clustering techniques to optimize access to large multidimensional datasets through temporal, spatial, or special sampling ordering rules. In Phase I we propose to demonstrate a prototype system utilizing one of the above data formats integrated with the METEOR data management system. This will involve a set of extraction and composition filters to populate the METEOR databases with metadata from the scientific datasets against which standard spatial and temporal, or attribute based queries can be formulated. The datasets retrieved from the data management system using the composition filters, can be merged into a single dataset for analysis. Phase II would involve specific ordering rules, and building a robust system for integration into a commercial product.

Keywords:
Hierarchical Storage Management Metadata Cdf Common Data Format Hdf Hierarchical Data Format

A number of papers and prototypes have demonstrated the utility of coupling metadata with database functionality, heirarchical storage management systems and visualization systems to facilitate scientific computing; few of these prototypes have been applied to flight-test and analysis systems. Recently, it is becoming recognized that for large distributed, computerized information management systems to evolve, a standardized form of handling data about data, or metadata is necessary. Ideally, this metadata should functionally work not only at the cosmic level of distributed mass storage systems, but also at the quantum level to meet the mixed I/O architectures of high performance massively parallel computing systems, and a flight test engineer's desktop system. This Phase II proposal addresses the requirements of US Air force SBIT Topic AF95-243 by detailing a robust design and implementation of the prototype metadata management system demonstrated in Phase I. Phase I introduced the concept of an object-oriented meta/server working with hierarchically formatted data (HDF), arbitrating and passing HDF data between browse, database, archive and analysis applications in a world-wide-web environment.

Keywords:
Hierarchical Storage Management Metadata Cdf Common Data Format Hdf Hierarchical Data Format