SBIR-STTR Award

The important task of training raters and establishing and maintaining reliability for the duration of a psychiatric clinical trial is a major problem that is often ignored. This problem is particularly vexing for trials involving many raters at multiple sites. Furthermore, factors effecting rater expectancy (bias) with standard psychiatric rating instruments has never been fully explored. This proposal will develop an interactive video system that can train research raters to use rating instruments with shared rating conventions and assess interrater reliability across sites and over time. Programs will be written that will enable this video system to be used as (I) an interactive "tutorial" system that can provide immediate feedback to trainees, and (2) a "reliability testing" system, that uses an integrated database to record ratings and compute interrater reliability within an identified group of raters or with preset ratings ("a gold standard"). Phase I will create a prototype for assessing interrater reliability for the Hamilton Depression Rating Scale (HDRS). Using scripts from actual patient interviews, a male and a female actor will each portray three separate HDRS interviews reflecting depressive symptoms of various severity. The functionality of the prototype system will be assessed. In preparation for Phase II, feasibility will be established by 1) having experienced raters assess videos of the HDRS interviews of both actors and actual patients to establish the validity of the actors' portrayals. 2) assessing interrater reliability in a group of experienced raters and a group of novice raters using the system, and 3) assessing the positive and negative attributes of the system with both experienced and novice raters. Phase II will develop a complete library of 30 variations of the I-IDRS using 6 different actors of diverse age, gender and ethnicity and implement the tutorial system. The Phase II study will assess variability and rater bias as an interaction of demographic and professional characteristics of the raters and "patients." Future product development will utilize the software and statistical methodology with other rating instruments, including those used to assess schizophrenia, anxiety disorders, and dementia. PROPOSED COMMERCIAL APPLICATION: The primary market for this product consists of any large organization involved in the conduct of psychiatric clinical trials, including academic medical centers, pharmaceutical companies, and contract research organizations. Another potential market is the managed care companies that use a growing number of rating scales to assess outcomes and quality of psychiatric and behavioral health services across multiple clinical sites.

Thesaurus Terms:
Internet, clinical trial, computer assisted instruction, computer program /software, computer system design /evaluation, diagnosis quality /standard, mental disorder diagnosis, psychometrics bias, interactive multimedia, interview, psychiatry human subject

The important task of training raters and establishing and maintaining reliability for the duration of a psychiatric clinical trial is a major problem that is often ignored. This problem is particularly vexing for trials involving many raters at multiple sites. Furthermore, factors effecting rater expectancy (bias) with standard psychiatric rating instruments has never been fully explored. This Phase II project is intended to develop an interactive video system for training research raters to use rating instruments with shared rating conventions and assess interrater reliability across sites and over time. Programs are to be written that will enable this video system to be used as (1) an interactive "tutorial" system that can be used to train raters on the use of the rating instrument, and (2) a "reliability testing" system, that uses an integrated database to record ratings and compute interrater reliability within an identified group of raters or with preset ratings, i.e., a "gold standard". The plan involves developing a complete library of 30 variations of the Hamilton Depression Rating Scale (HDRS) using four different actors of both genders and diverse ethnicity. The Phase II study will assess variability and rater bias as an interaction of demographic and professional characteristics of the raters and "patients" with 200 novice raters. Phase I created a prototype for assessing interrater reliability for the HDRS. Using scripts from actual patient interviews, a male and a female actor each portrayed three separate HDRS interviews reflecting depressive symptoms of various severity. In preparation for Phase II, the functionality of the system was found to be excellent. Feasibility was established by 1) having experienced raters assess videos of the HDRS interviews of both actors and actual patients and establish the validity of the actors' portrayals, 2) calculating interrater reliability scores for a group of experienced raters (0.97) and a group of novice raters (0.96) using the system prototype, and 3) convening focus groups to assess the positive and negative attributes of the system with both experienced and novice raters. Future product development will utilize the software and statistical methodology with other rating instruments, including those used to assess schizophrenia, anxiety disorders, and dementia.

Thesaurus Terms:
Internet, clinical trial, computer assisted instruction, computer program /software, computer system design /evaluation, depression, diagnosis quality /standard, mental disorder diagnosis, psychometrics bias, interactive multimedia, interview, psychiatry, video recording system clinical research, human subject