SBIR-STTR Award

Nuclear forensic analysis is critical to preventing nuclear terrorism. This analysis involves combining information from multiple microanalytical imaging technologies and other spectroscopic methods in a serial fashion to provide all required information; specifically the size, morphology, composition, and molecular and elemental makeup of the samples. Image registration is a crucial step in all image analysis tasks in which the final information is gained from the combination of various data sources. Image registration can overcome issues such as image rotation, scale, and skew effect that are common when combining this data. However, this methodology has not been widely used to correlate features between optical microscopy platforms and analytical image platforms due to several challenges, including the large differences in the Region of Interest (ROI), resolution, and morphology. ChemImage Sensor Systems (CISS) proposes the development of the Rapid Multi-Modal Microscopy Feature Correlation Toolkit (RM-MFCT). RM-MFCT is a set of algorithms that provides rapid automated feature correlation between optical microscope images and images obtained by SEM, SEM/EDS, SIMS, and other analytical imaging methods. RM-MFCT combines the advantages of deep learning and computer vision, therefore, it can better address the key challenges during the feature correlation process, include large scale and morphology differences, blurred boundary, lack of salient features and skew effects. RM-MFCT provides the most viable path to a fully automated, cost-effective, small footprint design that provides rapid, robust and highly accurate feature correlation results. CISS will develop the RM-MFCT utilizing both the advanced deep learning technique and classical computer vision approach. The deep learning technique will be applied initially to quickly correct large scale and appearance differences between optical and analytical microscopic data. Then, we will fine-tune the registration results using the traditional computer vision methods.RM-MFCT would have applicability beyond the nuclear forensics industry. The automation of image registration has significant commercialization potential in federal and civilian markets, in any field that utilizes sensor fusion technique, particularly within the medical/surgical fields and security industries.

One of the most difficult challenges faced to combine microscope information is the relocation of particulate matter or region of interest from one instrument platform to the other. This is often due to differences in morphological appearances of the particles and magnification changes between the scanning electron microscopy and the optical-based instrument. In addition, skew effect caused by optical aberration, sample movement and system drift also pose challenges for the feature correspondence work. Develop and design a novel rapid multi-modal microscopy feature correlation tool combining traditional computer vision techniques with Deep Learning, which is anticipated to provide quick and accurate registration performance while addressing region of interest relocation, correction of aberration effect, cost requirements, and the ability to meet multiple Concept-of-Operations. The rapid multi-modal microscopy feature correlation tool algorithm pipeline has been tested on 1,350 image pairs collected using three modalities, visible and near-infrared camera, Short-wave infrared camera and visible light camera. The overall correction success rate is 90.4% and the overall correlation error is within one pixel (0.7 pixel in average). In addition, the proposed algorithm pipeline is able to address skew effect due to optical aberration. Based on the success of this Phase I work, Phase II will consist of refining the rapid multi-modal microscopy feature correlation tool notional design, developing more robust correlation algorithm, fully transitioning the algorithm pipeline to a software application, and conducting the testing work on various of microscope data. Developing and testing this software application is the first step towards finalizing a set of requirements that can guide the commercialization for microscope software market. The resulting product of the Micro-Core project will deliver value immediately to the forensics community, then more broadly to machine vision market, the image processing market, and the image recognition market. Micro-Core will automate the process of image registration that is a necessity in any process that takes images from more than one camera or sensor. Micro-core will improve the speed, the accuracy, and reduce the need for human intervention in image- registration processes.