SBIR-STTR Award

Materials sample characterization by electron microscopy with multiple modes of detection provides more robust and information-rich data, but its adoption remains limited because of the dramatic increase in off- the-microscope data analysis work. Essentially, the necessary, but labor-intensive post-acquisition data analysis for multi-modal microscopy has kept it a low throughput technique that is completed off- microscope. The two primary contributing factors to the post-acquisition labor are (i) poor integration of post-acquisition image analysis systems and (ii) low interoperability between proprietary control software for each detector connected to a microscope. An integrated, platform-agnostic system with automated, concurrent image analysis and quantitation for multiple detectors will be developed. Such a system will enable real-time multi-modal microscopy image analysis, aggregation, and registration at the point-of- acquisition with output displayed as a set of layers over the microscope control software. Real-time electron microscopy image analysis for a single detector has already been demonstrated using a web-based software stack, recent advances in artificial intelligence, and a graphical processing unit-equipped edge computing device. The proposed technology will scale the established single detector image analysis platform to run concurrently for multiple detectors to enable real-time multi-modal microscopy. Success will be demonstrated by deployment of a fully integrated real-time multi-modal microscopy image analysis system within a Nuclear Sciences User Facility microscopy center. It is anticipated this system will transform the laborious, manual multi-modal microscopy image analysis workflow into being identical to the scalable, automated real-time single detector-like workflow and user experience. This transformation will enable wider adoption of multi-modal microscopy for materials discovery and qualification based on characterization tasks with additional growth into other forms of microscopy, such as optical and X-ray for biology and medical applications.

Quantitative digital microscopy involves identifying features within captured images and counting, categorizing, and measuring each feature to extract higher order metrics to make informed decisions in a variety of fields, such as materials, biology, metrology, medicine, and additive or advanced manufacturing. The quantitative digital microscopy workflow has been a low throughput affair with a high degree of skilled human labor that is difficult to scale in the current state to meet the demands of next generation microscopy and materials characterization systems. This is exacerbated by difficult to use legacy and often “quirky” data analysis software that cannot process multiple data streams in parallel or automatically adapt to variations in workflow and analysis requirements. The feasibility of an automated quantitative microscopy image analysis platform that leverages the latest techniques in artificial intelligence and machine learning and overcomes the issues of scale and adaptability has been demonstrated. The proposed effort then seeks to refine this technology for full-scale commercialization. The technology, which uses a collection of independent graphical processing unit-equipped edge computing devices running state-of-the-art artificial intelligence/machine learning models for simultaneous automated image analysis, was demonstrated at a national user facility with a multi-detector electron microscope for materials characterization. This technology will be further advanced to deploy on most microscopy instruments in a user facility while still maintaining an easy-to-use software interface and seamless integration with modern microscopy workflows. The proposed effort will then extend past the single experiment and use case demonstrated in the initial feasibility study by leveraging the zero-configuration and platform agnostic nature of the technology. Specifically, the technology’s edge computing device and user interface stacks will be advanced through concentrated efforts on optimizing network communication, analysis pipelines, and visualizations for near-infinite device growth and extensibility to enable widespread adoption in industry, energy, and medical sectors using optical, charged particle, and scanning probe microscopes. The resulting outcome will be a scalable, future-proof product that ultimately converts a low throughput, manual data analysis process into a high throughput, automated platform that accelerates the innovation lifecycle of new technologies that rely on quantitative microscopy with multiple data streams, such as products and processes under development in materials science, biology, medical imaging, geology, energy storage, and energy consumption.