In general, ML models solve discrete, fine-grained problems such as the classification of documents into one of a fixed number of classes. A useful rule of thumb to fit a task to ML is: could a human do this in under a second? When designing a classifier, for example, decisions need to be made such as: (1) Which classes should be included? (2) What metrics should be utilized to train and evaluate the classifier; should we emphasize some classes at the expense of others? (3) How do the classes relate to one another? (4) Might a given document fall legitimately into more than one of these classes? (5) How obtainable is training data that aligns to these classes? (6) How will low-confidence predictions or potential latent, novel classes be handled? (7) And most importantly: does this classifier correspond directly to a specific problem, or does a more complex system need to be composed utilizing this classifier as one coordinated component among many? JaxonÂ’s commercial platform utilizes (manually-created) problem specifications to great success for framing the creation of training datasets and their derived ML models. Despite impressive advances in automation with respect to training individual models, this process of specifying a cooperative set of models to address a mission objective or specific problem is still a painstakingly manual one. Data scientists and data analysts spend copious effort and time designing useful systems and experiments - and must frequently revise them as the mission and/or data contexts change. Furthermore, the prototype code is often wasted as it is not reusable. Jaxon will develop an automated system AutoSpec that will analyze a collection of datasets (optionally with guidance around domains of interest) and propose candidate ML solution specifications. These specifications consist of detailed descriptions of ML models, as well as a graph linking the cooperative application of multiple models in order to compose a system that addresses a high-level problem utilizing these fine-grained models as building blocks. This concept is at TRL 3 and is not something that exists yet in production. We believe that automating the design process is a key to unlocking one of the remaining large bottlenecks in modern ML. Prototyping using language models (GPT-3, in our experiments) to derive candidate classes from data samples shows promise as one avenue for defining and enhancing these specifications.
