SBIR-STTR Award

Electron microscopy image analysis workflows are currently a non-scalable, biased process that requires extensive time and expertise. Attempts to generate a scalable and non-biased automated image analysis workflow using artificial intelligence and machine learning technologies at the point-of-acquisition have been thwarted by (i) limited, or impossible, access to external cloud computing resources and environments, (ii) lack of a consistent, streamlined end-user experience for distribution and deployment within these network-constrained environments, and (iii) poor interactivity of real-time image analysis results in closed electron microscope system software. An open software-hardware platform that overcomes these deficiencies will be developed and include capabilities for displaying real-time image analysis results and machine learning output as a live dashboard and overlay in microscope control software. The proposed technology, which uses a web-based interface hosted on a graphical processing unit-equipped edge computing device, has already been demonstrated to overcome the currently identified issues through internal proof-of-concept efforts. This proof-of-concept will be driven to commercialization by incorporating a container-based machine learning model deployment framework and optimizing the communication pipeline between the microscope and edge computing device. Success will be demonstrated and evaluated by creating the first ever augmented reality electron microscope running a community driven automated feature detection algorithm. It is anticipated the interactive, real-time platform, which augments the feature set of any digitally controlled microscope, will result in an 80% reduction of person hours associated with labor intensive, time consuming image analysis tasks, while providing reproducible results unbiased by human-based imaging detection, classification, and quantification tasks. The proposed flexible platform ensures applicability to a range of microscopy workflows including material science, biology, medical imaging, geology, and hyperspectral imaging with significant impact in high-use equipment, such as those residing in government-supported interdisciplinary research centers and vision-based industrial quality control centers.

Microscopy workflows can include a detailed image analysis and quantification effort that has proven to be time consuming, cost ineffective, and potentially biased. Recent developments in machine learning and artificial intelligence methods have signaled a revolution in these efforts leading to accelerated and consistent quantification of digitally generated microscopy images. Given this, these innovative methods typically require cloud computing resources and at least some knowledge of software development and computing programming. This is further exacerbated by the seemingly daily release of new machine learning methods. An open software-hardware platform that overcomes these issues and embraces the rapidly evolving field has surpassed the feasibility stage and will be developed for full-scale commercialization. The technology, which uses a web-based interface hosted on a graphical processing unit-equipped edge computing device, was successfully demonstrated to host and run a state-of-the-art machine learning model to enable the first augmented reality in-situ transmission electron microscopy experiments for irradiated materials development and qualification. This technology will be further advanced to host a range of additional machine learning techniques while providing a flexible platform for integrating future techniques and methods for any automated, real-time image analysis and quantification effort. It has already been demonstrated that the interactive, real-time platform at the heart of the technology can augment the features of advanced electron microscopes resulting in reduced person hours (and in turn costs) within microscopy workflows. The proposed efforts will then extend past the niche application demonstrated in the initial feasibility study by leveraging the agnostic nature of the communication protocols used for interfacing with microscopes. Specifically, the platform’s edge computing device and user interface software stacks will be advanced through concentrated efforts on expanding the hosting, deployment, and dissemination of current and future machine learning methods to enable widespread adoption in industry, energy, and medical sectors using optical, charged particle, and scanning probe microscopes. The resulting outcome will be a flexible platform that reduces labor costs by up to 80%, decouples expertise in routine microscopy tasks, and accelerates the innovation lifecycle of new technologies that rely on quantitative microscopy, such as technologies and products under development in material science, biology, medical imaging, geology, energy storage, and energy consumption.