Supervised, Machine Learning based oncologic AI algorithms degrade over time and don't generalize well. The FDA requires the conduct of standalone testing of radiologic, AI products to characterize their performance. Sourcing representative data with sufficient variability is time consuming, expensive, and often under-represent the variety of image quality experienced in clinical reality. FDA Reviewers also cannot compare new products to predicate devices. We propose a process for defining Reference Datasets and software allowing developers of imaging-based oncology AI products to test using the datasets. A Reference Dataset (R.D.) is an imaging dataset whose oncologic condition is confirmed by pathologic and/or radiographic confirmation. Objective 1 will be defining the rules and process to define R.D.s. Objective 2 will result in search and curation methodology to improve the extraction of oncologic data from our study library. Objective 3 will create an end-end workflow for the testing an imaging AI model against a RD curated. Finally, we will develop and submit a qualification plan for a Medical Device Development Tool to FDA in Objective 4. Machine Learning/AI can improve the detection and characterization of cancers from medical imaging but there are no common, ground-truth reference data to test and compare new AI algorithms.
Public Health Relevance Statement: Terms:
