SBIR-STTR Award

Hundreds of thousands of children (669, 799) were confirmed victims of maltreatment in the United States in 2017; in that same year, of the 442, 733 children in foster care, 34% had been in more than one placement and 11% were in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these children because some placement types, and multiple placements, are associated with poor outcomes. With the Family First Prevention Act, states will be required to pay the average $88, 000 per year to keep a child in residential care if that high level of care is not authorized. But which children require --and, more importantly, would benefit-- from a placement in residential care? Decision-making support tools currently used by states to recommend specific level of care (LOC) placements for children do not maximize the rich data and innovative methodological approaches that are being explored in other fields like medicine. In addition, structured decision making (SDM) has been used to guide decisions about risk in child welfare settings but, in comparison to predictive modelling, SDM is limited by the use of a smaller group of factors to make recommendations. Outcome Referrals, Inc. has employed sophisticated machine learning techniques over the past 10 years to risk-adjust behavioral health outcome data for clients using baseline characteristics. Initial models predicted more than 30% of the outcome variance (i.e., it was possible to predict 30% of the variance in how depressed a client would be at follow-up). The next model improved that prediction to more than 50%, and our latest model has increased this to an average of 71%. With the assistance of Phase I NIH SBIR funding, we plan to improve the success rates of children in the child welfare system with an innovative, scientifically-derived product called "Placement Success Predictor." To guide level-of-care decision-making, this product will use site-customized, machine learning algorithms to predict the likelihood of an adolescent having a good outcome in a particular placement type in a specific community. We have preliminary evidence supporting the feasibility of developing these models based on work supported by the Duke Endowment Foundation. During this six-month Phase I project, we propose to 1) validate these preliminary machine learning models by applying them to new client data from our partner behavioral health organization, 2) explore options for sharing results of these models to facilitate their use in practice (e.g., aggregate predictions across different domains in a weighted way), 3) assess key stakeholder satisfaction with a new prototype, and 4) develop and test customized models for multiple placement types with a state-wide child welfare and juvenile justice dataset.

Public Health Relevance Statement:
NARRATIVE Hundreds of thousands of children were confirmed victims of maltreatment in the United States in 2017; in that same year, of the 442, 733 children in foster care, approximately one out of 10 were placed in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these vulnerable children because some placements, and multiple placements, are associated with poor outcomes. The likelihood of success recommendations provided by the proposed "Placement Success Predictor" tool will help placement staff and administrators identify the best placement setting for each child using machine-learning statistics to predict the child's chances of success in each potential treatment setting. Terms: Child; 0-11 years old; Child Youth; Children (0-21); youngster; Child Care; Puericulture; Child Welfare; adolescent welfare; child well being; child wellbeing; Client; Communities; Decision Making; Endowment; Family; Federal Government; National Government; Foundations; Future; Grant; Group Homes; Health Personnel; Health Care Providers; Healthcare Providers; Healthcare worker; health care personnel; health care worker; health provider; health workforce; healthcare personnel; medical personnel; treatment provider; Justice; Kidney; Kidney Urinary System; renal; Medicine; Mental Health Services; Mental Hygiene Services; Methodology; United States National Institutes of Health; NIH; National Institutes of Health; Team Nursing; Patients; Personal Satisfaction; well-being; wellbeing; Probability; Receiver Operating Characteristics; Receiver Operator Characteristics; Recommendation; Residential Treatment; residential care; Resources; Research Resources; Risk; statistics; Testing; United States; Work; County; Treatment outcome; Health Care Costs; Health Costs; Healthcare Costs; Administrator; Healthcare; health care; Managed Care; Schedule; Data Set; Dataset; Caring; Custom; Friction; base; improved; Procedures; Brain imaging; brain visualization; Psychiatric therapeutic procedure; psychiatric care; psychiatric therapy; psychiatric treatment; Site; Area; Clinical; Phase; Medical; Adolescent; Adolescent Youth; juvenile; juvenile human; Individual; Databases; Data Bases; data base; satisfaction; Funding; depressed; sadness; Depressed mood; machine learned; Machine Learning; foster care; Source; Techniques; System; Country; Operative Procedures; Surgical; Surgical Interventions; Surgical Procedure; surgery; Operative Surgical Procedures; Surgeon; experience; success; Structure; prevention service; treatment program; Self-Report; Patient Self-Report; model-based simulation; models and simulation; Prevention; Reporting; gatekeeper; Gatekeeping; Admission; Admission activity; Modeling; behavioral health; Intervention Strategies; interventional strategy; Intervention; health organization; Provider; Institution; Data; Small Business Innovation Research Grant; SBIR; Small Business Innovation Research; Characteristics; follow-up; Active Follow-up; active followup; follow up; followed up; followup; Development; developmental; cost; virtual; predictive modeling; computer based prediction; prediction model; maltreatment; mistreatment; severe mental illness; chronic mental illness; persistent mental illness; serious mental disorder; serious mental illness; severe mental disorder; Outcome; Population; innovation; innovate; innovative; usability; prototype; support tools; improved outcome; predictive tools; care costs; care coordination; coordinating care; machine learning algorithm; machine learned algorithm; Home

Hundreds of thousands of children (669,799) were confirmed victims of maltreatment in the United States in 2017. In that same year, 34% of the 442,733 children in foster care had been in more than one placement and 11% were in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these children because some placement types, and multiple placements, are associated with poor outcomes. But which children require --and, more importantly, would benefit-- from a placement in residential care? Decision-making support tools currently used by states to recommend specific level-of-care (LOC) placements for children do not maximize the rich data and innovative methodological approaches that are being explored in other fields like medicine. In addition, structured decision making (SDM) has been used to guide decisions about risk in child welfare settings but, in comparison to predictive modelling, SDM is limited by the use of a smaller group of factors to make recommendations. Outcome Referrals, Inc. has employed sophisticated machine learning techniques over the past 10 years to risk-adjust behavioral health outcome data for clients using their current characteristics. We have evidence of the predictive validity of this approach for generating risk-adjusted scores via the PCORI-funded RCT in which clients were matched with clinicians who had previous clients who did well in treatment. With the assistance of NIH SBIR funding, we plan to improve the success rates of children in the child welfare system with an innovative, scientifically- derived tool called "Placement Success Predictor" to guide level-of-care decision-making. This product will use machine learning algorithms to predict individualized outcomes for children and adolescents in a particular placement type. During the Phase I project, we developed and validated machine learning models to predict the probability of each youth's success within placement types and conducted usability testing of a prototype to assess potential barriers to implementation. The likelihood of success recommendations provided by the machine learning models developed for the "Placement Success Predictor" clinical decision-making support tool were associated with improved youth well-being (i.e., those children that were placed in the Placement Success Predictor-recommended placement had four times higher success rates). Potential future users found the prototype of this tool easy to use. The machine learning models appear to distinguish clients who will do well in various placement types and suggest that only 10% of clients would be successful in higher-cost level of care. With the Family First Prevention Act, states are now required to pay the average $88,000 per year to keep a child in residential care if that high level of care is not authorized. We submit this tool could help improve overall outcomes and reduce healthcare costs when and if high-cost residential care options are judiciously provided to children who are most likely to benefit from them. In Phase II we propose to develop this flexible program to generate placement recommendations per client based on automatically updated custom, site-specific machine learning prediction models then assess its efficacy and effectiveness.

Public Health Relevance Statement:
NARRATIVE Hundreds of thousands of children were confirmed victims of maltreatment in the United States in 2017. In that same year, of the 442,733 children in foster care, approximately one out of 10 were placed in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these vulnerable children because some placements, and multiple placements, are associated with poor outcomes. The likelihood of success recommendations provided by the proposed "Placement Success Predictor" tool will help placement staff and administrators identify the best placement setting for each child using machine-learning statistics to predict the child's chances of success in each potential treatment setting. Terms: <0-11 years old; Child Youth; Children (0-21); kids; youngster; Child; Puericulture; Child Care; adolescent welfare; child well being; child wellbeing; Child Welfare; Client; Decision Making; Family; Future; Grant; Group Homes; Justice; Kidney Urinary System; renal; Kidney; Medicine; Mental Health Services; Mental Hygiene Services; mental health care; mental healthcare; Methodology; United States National Institutes of Health; NIH; National Institutes of Health; Team Nursing; Patients; Personal Satisfaction; well-being; wellbeing; Probability; Recommendation; Residential Treatment; residential care; Resources; Research Resources; Risk; statistics; Testing; United States; Treatment outcome; Health Costs; Healthcare Costs; Health Care Costs; Administrator; health care; Healthcare; Managed Care; Youth 10-21; Youth; Data Set; Caring; customs; Custom; improved; brain visualization; Brain imaging; psychiatric care; psychiatric therapy; psychiatric treatment; Psychiatric therapeutic procedure; Site; Clinical; Phase; Adolescent Youth; juvenile; juvenile human; Adolescent; randomized, clinical trials; Individual; Data Bases; data base; Databases; Funding; tool; machine based learning; Machine Learning; programs; foster care; Source; Techniques; System; Operative Surgical Procedures; Operative Procedures; Surgical; Surgical Interventions; Surgical Procedure; surgery; Surgeon; success; Risk Adjustment; Structure; treatment program; Patient Self-Report; Self-Report; models and simulation; model-based simulation; Prevention; Reporting; Gatekeeping; gatekeeper; Modeling; behavioral health; health organization; Institution; Effectiveness; Data; Small Business Innovation Research Grant; SBIR; Small Business Innovation Research; Update; Characteristics; Development; developmental; cost; predictive modeling; computer based prediction; efficacy evaluation; determine efficacy; efficacy analysis; efficacy assessment; efficacy determination; efficacy examination; evaluate efficacy; examine efficacy; maltreatment; mistreatment; severe mental illness; chronic mental illness; persistent mental illness; serious mental disorder; serious mental illness; severe mental disorder; Outcome; Population; Quasi-experiment; Quasi-experimental analysis; Quasi-experimental approach; Quasi-experimental design; Quasi-experimental methods; Quasi-experimental research; Quasi-experimental study; Quasi-experimental technique; innovation; innovate; innovative; usability; prototype; flexibility; flexible; clinical decision-making; clinical predictors; support tools; predictive tools; care costs; care coordination; coordinating care; personalized predictions; individualized predictions; machine learning algorithm; machine learned algorithm; machine learning based algorithm; data curation; curating data; implementation barriers; barriers to implementation; implementation challenges; Home; homes; machine learning prediction; machine learning based prediction model; machine learning based predictive model; machine learning prediction model; machine learning model; machine learning based model