SBIR-STTR Award

Current approaches to medical education produce new physicians with insufficient clinical competency to practice effectively with limited supervision, which has serious implications for patient outcomes during hospital off hours. Up to 70% of patients admitted to a hospital are admitted on nights or weekends, when staffing is low and residents may be in charge of patient care. Off-hours care is linked to an increase in medical errors, surgical complications, and readmission rates. Negative patient outcomes during off hours are 2 to 8 times more likely to occur under the care of residents in their first postgraduate year. We propose to improve undergraduate medical education to minimize the time to clinical competency for first year residents through targeted diagnostic reasoning skill development that (1) integrates basic science and clinical instruction; (2) provides deliberate practice with structured, case-based learning opportunities; and (3) enables anytime/anywhere learning that fits with the demanding schedules of most medical students. Southern Illinois University School of Medicine (SIUSOM) is a recognized leader in using performance-based clinical competency exams to enhance reasoning skill acquisition among medical students. These exams feature clinical scenarios with standardized patients followed by diagnostic justification essays which require students to explicitly describe the thought process used to reach a final diagnosis. These essays are the most reliable method of assessing diagnostic strategies but are not in use in the majority of medical schools, though interest in improving diagnostic reasoning instruction and assessment during undergraduate medical education is widespread. Barriers to the widespread adoption of this approach are 1) the time-consuming need to hand score each essay; and 2) the difficulty in accurately and consistently identifying the causes of strategy failures. This project will develop an application to provide automated scoring of diagnostic justification essays, identification of the underlying causes of failure whenstudents perform poorly, and feedback with instructional strategies for remediation specific to each deficit. We propose these specific aims: 1) Develop a scoring algorithm that automatically evaluates students' written justifications of diagnostic strategy. 2) Build a taxonomy of strategy failures from students' diagnostic justifications. 3) Work closely with SIUSOM and other medical schools contemplating the addition of diagnostic reasoning assessment to their curriculum. Phase II will focus on additional specific aims: 4) Develop an application based on the Phase I proof of concept with refined algorithms for scoring, categorizing reasoning failures and targeted remediation. 5) Evaluate the learning application for usability, acceptance via focus groups, and effectiveness via a limited field trial with students at SIUSOM. The proposed product represents a significant shift in undergraduate medical training and through Phase III dissemination will address a critical gap between education and practice in academic medicine.

Thesaurus Terms:
Address;Adoption;Algorithms;Base;Basic Science;Caring;Case-Based;Categories;Charge;Clinical;Competence;Consult;Development;Diagnosis;Diagnostic;Differential Diagnosis;Educational Aspects;Educational Curriculum;Educational Process Of Instructing;Educational Technology;Effectiveness;Ensure;Environment;Essays;Evaluation;Evidence Base;Faculty;Failure (Biologic Function);Feedback;Focus Groups;Foundations;Future;Goals;Hand;Hospitals;Hour;Human;Illinois;Improved;Incubators;Innovation;Instruction;Interest;Knowledge;Lead;Leadership;Learning;Left;Link;Machine Learning;Maps;Medical;Medical Education;Medical Errors;Medical Malpractice;Medical Schools;Medical Students;Medicine;Methods;Nature;Online Systems;Outcome;Patient Care;Patient Safety;Patients;Performance;Phase;Physicians;Plant Roots;Preparation;Prevalence;Process;Public Health Relevance;Remediation;Research And Development;Residencies;Role;Schedule;Semantics;Skill Acquisition;Skills;Structure;Students;Success;Supervision;Surgical Complication;System;Taxonomy;Technology;Technology Development;Testing;Theories;Thinking, Function;Time;Training;Undergraduate Medical Education;Universities;Usability;Validity And Reliability;Work;Writing;

Today's clinical learning environments do not provide the level of deliberate practice, direct supervision, and rigorous assessment and feedback needed to develop diagnostic reasoning expertise. Clinical performance assessment emphasizes learner evaluation over learner development, lacks rigor and utility for developmental purposes, and clinical teachers have expressed particular difficulty with diagnosing reasoning deficits for remediation purposes. Further, medical students' diagnostic reasoning does not improve over the course of clinical training and senior medical students have highly variable diagnostic performance that is often rated below expectations according to theory-based and validated scoring criteria. Independent practice does not necessarily enhance the context for clinical reasoning; the majority of physicians' medical errors are thought to be diagnostic in nature. We propose to improve undergraduate medical education to minimize the time to clinical competency for first year residents through targeted diagnostic reasoning skill development that (1) integrates basic science and clinical instruction; (2) provides deliberate practice with structured, case-based learning opportunities; and (3) enables anytime/anywhere learning that fits with the demanding schedules of most medical students. Southern Illinois University School of Medicine (SIUSOM) is a recognized leader in using performance-based clinical competency exams to enhance reasoning skill acquisition among medical students. These exams feature clinical scenarios with standardized patients followed by diagnostic justification essays which require students to explicitly describe the thought process used to reach a final diagnosis. These essays are the most reliable method of assessing diagnostic strategies but are not in use in the majority of medical schools, though interest in improving diagnostic reasoning instruction and assessment during undergraduate medical education is widespread. Barriers to the widespread adoption of this approach are 1) the time-consuming need to hand score each essay; and 2) the difficulty in accurately and consistently identifying the causes of strategy failures. This project will develop an application to provide automated scoring of diagnostic justification essays, identification of the underlying causes of failure when students perform poorly, and feedback with instructional strategies for remediation specific to each deficit. We propose these specific aims: 1) Improve reliability of human scoring of DXJ essays. 2) Extend the automated scoring algorithms. 3) Automated reasoning failure categorization and remediation. 4) Complete the software development required for delivering the commercial product. 5) Evaluate predictive validity of automatically scored DXJ essays. The proposed product represents a significant shift in undergraduate medical training and through Phase III dissemination will address a critical gap between education and practice in academic medicine.

Public Health Relevance Statement:
Project Narrative Today's clinical learning environments do not provide the level of deliberate practice, direct supervision, and rigorous assessment and feedback needed to develop diagnostic reasoning expertise. Better preparation during undergraduate medical education can shorten the time to competency of first year residents, improving patient outcomes. We propose to develop and test a technology-enabled, deliberate-practice approach to training diagnostic strategy that includes automated scoring of diagnostic justification essays, identification of specific diagnostic strategy failures and targeted remediation. The proposed product represents a significant shift in undergraduate medical training and through Phase III dissemination will address a critical gap between education and practice in academic medicine.

Project Terms:
Address; Adopted; Adoption; Algorithms; base; Basic Science; Caring; Case Based Learning; Case Study; Charge; Classification; Clinical; Clinical Competence; Community Health Education; Competence; Consult; Custom; Data; Databases; Development; Diagnosis; Diagnostic; Education; educational atmosphere; Educational process of instructing; Educational Technology; Ensure; Environment; Equation; essays; Evaluation; evidence base; expectation; Faculty; Failure; Feedback; Future; Goals; Hand; Hospitals; Human; Illinois; improved; Incubators; innovation; Instruction; interest; Leadership; Learning; Letters; Machine Learning; Measures; Medical; Medical Education; Medical Errors; medical schools; Medical Students; Medicine; Methods; Modeling; Nature; Patient Care; Patient-Focused Outcomes; Patients; Pattern Recognition; Performance; Phase; Physicians; Preparation; Process; prototype; Recommendation; remediation; research and development; Role; Sales; Schedule; Semantics; skill acquisition; skills; software development; Standardization; Structure; Students; Suggestion; Supervision; System; Taxonomy; teacher; Teaching Method; Technology; Testing; Text; theories; Time; tool; Training; undergraduate medical education; undergraduate student; Universities; Validity and Reliability; Variant; virtual