Evaluation of inpatient medication guidance from an artificial intelligence chatbot

PURPOSE

To analyze the clinical completeness, correctness, usefulness, and safety of chatbot and medication database responses to everyday inpatient medication-use questions.

METHODS

We evaluated the responses from an artificial intelligence chatbot, a medication database, and clinical pharmacists to 200 real-world medication-use questions. Answer quality was rated by a blinded group of pharmacists, providers, and nurses. Chatbot and medication database responses were deemed "acceptable" if the mean reviewer rating was within 3 points of the mean rating for pharmacists' answers. We used descriptive statistics for reviewer ratings and Kendall's coefficient to evaluate interrater agreement.

RESULTS

The medication database generated responses to 194 (97%) questions, with 88% considered acceptable for clinical correctness, 76% considered acceptable for completeness, 83% considered acceptable for safety, and 81% considered acceptable for usefulness compared to pharmacists' answers. The chatbot responded to only 160 (80%) questions, with 85% considered acceptable for clinical correctness, 65% considered acceptable for completeness, 71% considered acceptable for safety, and 68% considered acceptable for usefulness.

CONCLUSION

Traditional search methods using a drug database provide more clinically correct, complete, safe, and useful answers than a chatbot. When the chatbot generated a response, the clinical correctness was similar to that of a drug database; however, it was not rated as favorably for clinical completeness, safety, or usefulness. Our results highlight the need for ongoing training and continued improvements to artificial intelligence chatbots for them to be incorporated reliably into the clinical workflow. With continued improvement in chatbot functionality, chatbots could be a useful pharmacist adjunct, providing healthcare providers with quick and reliable answers to medication-use questions.

Adapting Cognitive Task Analysis Methods for Use in a Large Sample Simulation Study of High-Risk Healthcare Events

Cognitive task analysis (CTA) methods are traditionally used to conduct small-sample, in-depth studies. In this case study, CTA methods were adapted for a large multi-site study in which 102 anesthesiologists worked through four different high-fidelity simulated high-consequence incidents. Cognitive interviews were used to elicit decision processes following each simulated incident. In this paper, we highlight three practical challenges that arose: (1) standardizing the interview techniques for use across a large, distributed team of diverse backgrounds; (2) developing effective training; and (3) developing a strategy to analyze the resulting large amount of qualitative data. We reflect on how we addressed these challenges by increasing standardization, developing focused training, overcoming social norms that hindered interview effectiveness, and conducting a staged analysis. We share findings from a preliminary analysis that provides early validation of the strategy employed. Analysis of a subset of 64 interview transcripts using a decompositional analysis approach suggests that interviewers successfully elicited descriptions of decision processes that varied due to the different challenges presented by the four simulated incidents. A holistic analysis of the same 64 transcripts revealed individual differences in how anesthesiologists interpreted and managed the same case.

Decision-making in anesthesiology: will artificial intelligence make intraoperative care safer?

PURPOSE OF REVIEW

This article explores the impact of recent applications of artificial intelligence on clinical anesthesiologists' decision-making.

RECENT FINDINGS

Naturalistic decision-making, a rich research field that aims to understand how cognitive work is accomplished in complex environments, provides insight into anesthesiologists' decision processes. Due to the complexity of clinical work and limits of human decision-making (e.g. fatigue, distraction, and cognitive biases), attention on the role of artificial intelligence to support anesthesiologists' decision-making has grown. Artificial intelligence, a computer's ability to perform human-like cognitive functions, is increasingly used in anesthesiology. Examples include aiding in the prediction of intraoperative hypotension and postoperative complications, as well as enhancing structure localization for regional and neuraxial anesthesia through artificial intelligence integration with ultrasound.

SUMMARY

To fully realize the benefits of artificial intelligence in anesthesiology, several important considerations must be addressed, including its usability and workflow integration, appropriate level of trust placed on artificial intelligence, its impact on decision-making, the potential de-skilling of practitioners, and issues of accountability. Further research is needed to enhance anesthesiologists' clinical decision-making in collaboration with artificial intelligence.

The decision aid is the easy part: workflow challenges of shared decision making in cancer care

Delivering high-quality, patient-centered cancer care remains a challenge. Both the National Academy of Medicine and the American Society of Clinical Oncology recommend shared decision making to improve patient-centered care, but widespread adoption of shared decision making into clinical care has been limited. Shared decision making is a process in which a patient and the patient's health-care professional weigh the risks and benefits of different options and come to a joint decision on the best course of action for that patient on the basis of their values, preferences, and goals for care. Patients who engage in shared decision making report higher quality of care, whereas patients who are less involved in these decisions have statistically significantly higher decisional regret and are less satisfied. Decision aids can improve shared decision making-for example, by eliciting patient values and preferences that can then be shared with clinicians and by providing patients with information that may influence their decisions. However, integrating decision aids into the workflows of routine care is challenging. In this commentary, we explore 3 workflow-related barriers to shared decision making: the who, when, and how of decision aid implementation in clinical practice. We introduce readers to human factors engineering and demonstrate its potential value to decision aid design through a case study of breast cancer surgical treatment decision making. By better employing the methods and principles of human factors engineering, we can improve decision aid integration, shared decision making, and ultimately patient-centered cancer outcomes.

Medication Safety Amid Technological Change: Usability Evaluation to Inform Inpatient Nurses' Electronic Health Record System Transition

BACKGROUND

Electronic health record (EHR) system transitions are challenging for healthcare organizations. High-volume, safety-critical tasks like barcode medication administration (BCMA) should be evaluated, yet standards for ensuring safety during transition have not been established.

OBJECTIVE

Identify risks in common and problem-prone medication tasks to inform safe transition between BCMA systems and establish benchmarks for future system changes.

DESIGN

Staff nurses completed simulation-based usability testing in the legacy system (R1) and new system pre- (R2) and post-go-live (R3). Tasks included (1) Hold/Administer, (2) IV Fluids, (3) PRN Pain, (4) Insulin, (5) Downtime/PRN, and (6) Messaging. Audiovisual recordings of task performance were systematically analyzed for time, navigation, and errors. The System Usability Scale measured perceived usability and satisfaction. Post-simulation interviews captured nurses' qualitative comments and perceptions of the systems.

PARTICIPANTS

Fifteen staff nurses completed 2-3-h simulation sessions. Eleven completed both R1 and R2, and seven completed all three rounds. Clinical experience ranged from novice (< 1 year) to experienced (> 10 years). Practice settings included adult and pediatric patient populations in ICU, stepdown, and acute care departments.

MAIN MEASURES

Task completion rates/times, safety and non-safety-related use errors (interaction difficulties), and user satisfaction.

KEY RESULTS

Overall success rates remained relatively stable in all tasks except two: IV Fluids task success increased substantially (R1: 17%, R2: 54%, R3: 100%) and Downtime/PRN task success decreased (R1: 92%, R2: 64%, R3: 22%). Among the seven nurses who completed all rounds, overall safety-related errors decreased 53% from R1 to R3 and 50% from R2 to R3, and average task times for successfully completed tasks decreased 22% from R1 to R3 and 38% from R2 to R3.

CONCLUSIONS

Usability testing is a reasonable approach to compare different BCMA tasks to anticipate transition problems and establish benchmarks with which to monitor and evaluate system changes going forward.

Antibiotic clinical decision support for pneumonia in the ED: A randomized trial

BACKGROUND

Electronic health record-based clinical decision support (CDS) is a promising antibiotic stewardship strategy. Few studies have evaluated the effectiveness of antibiotic CDS in the pediatric emergency department (ED).

OBJECTIVE

To compare the effectiveness of antibiotic CDS vs. usual care for promoting guideline-concordant antibiotic prescribing for pneumonia in the pediatric ED.

DESIGN

Pragmatic randomized clinical trial.

SETTING AND PARTICIPANTS

Encounters for children (6 months-18 years) with pneumonia presenting to two tertiary care children s hospital EDs in the United States.

INTERVENTION

CDS or usual care was randomly assigned during 4-week periods within each site. The CDS intervention provided antibiotic recommendations tailored to each encounter and in accordance with national guidelines.

MAIN OUTCOME AND MEASURES

The primary outcome was exclusive guideline-concordant antibiotic prescribing within the first 24 h of care. Safety outcomes included time to first antibiotic order, encounter length of stay, delayed intensive care, and 3- and 7-day revisits.

RESULTS

1027 encounters were included, encompassing 478 randomized to usual care and 549 to CDS. Exclusive guideline-concordant prescribing did not differ at 24 h (CDS, 51.7% vs. usual care, 53.3%; odds ratio [OR] 0.94 [95% confidence interval [CI]: 0.73, 1.20]). In pre-specified stratified analyses, CDS was associated with guideline-concordant prescribing among encounters discharged from the ED (74.9% vs. 66.0%; OR 1.53 [95% CI: 1.01, 2.33]), but not among hospitalized encounters. Mean time to first antibiotic was shorter in the CDS group (3.0 vs 3.4 h; p = .024). There were no differences in safety outcomes.

CONCLUSIONS

Effectiveness of ED-based antibiotic CDS was greatest among those discharged from the ED. Longitudinal interventions designed to target both ED and inpatient clinicians and to address common implementation challenges may enhance the effectiveness of CDS as a stewardship tool.

Decision-Making During High-Risk Events: A Systematic Literature Review

Effective decision-making in crisis events is challenging due to time pressure, uncertainty, and dynamic decisional environments. We conducted a systematic literature review in PubMed and PsycINFO, identifying 32 empiric research papers that examine how trained professionals make naturalistic decisions under pressure. We used structured qualitative analysis methods to extract key themes. The studies explored different aspects of decision-making across multiple domains. The majority (19) focused on healthcare; military, fire and rescue, oil installation, and aviation domains were also represented. We found appreciable variability in research focus, methodology, and decision-making descriptions. We identified five main themes: (1) decision-making strategy, (2) time pressure, (3) stress, (4) uncertainty, and (5) errors. Recognition-primed decision-making (RPD) strategies were reported in all studies that analyzed this aspect. Analytical strategies were also prominent, appearing more frequently in contexts with less time pressure and explicit training to generate multiple explanations. Practitioner experience, time pressure, stress, and uncertainty were major influencing factors. Professionals must adapt to the time available, types of uncertainty, and individual skills when making decisions in high-risk situations. Improved understanding of these decisional factors can inform evidence-based enhancements to training, technology, and process design.

Understanding Patient and Clinician Reported Nonroutine Events in Ambulatory Surgery

OBJECTIVE

Nonroutine events (NREs, i.e., deviations from optimal care) can identify care process deficiencies and safety risks. Nonroutine events reported by clinicians have been shown to identify systems failures, but this methodology fails to capture the patient perspective. The objective of this prospective observational study is to understand the incidence and nature of patient- and clinician-reported NREs in ambulatory surgery.

METHODS

We interviewed patients about NREs that occurred during their perioperative care using a structured interview tool before discharge and in a 7-day follow-up call. Concurrently, we interviewed the clinicians caring for these patients immediately postoperatively to collect NREs. We trained 2 experienced clinicians and 2 patients to assess and code each reported NRE for type, theme, severity, and likelihood of reoccurrence (i.e., likelihood that the same event would occur for another patient).

RESULTS

One hundred one of 145 ambulatory surgery cases (70%) contained at least one NRE. Overall, 214 NREs were reported-88 by patients and 126 by clinicians. Cases containing clinician-reported NREs were associated with increased patient body mass index ( P = 0.023) and lower postcase patient ratings of being treated with respect ( P = 0.032). Cases containing patient-reported NREs were associated with longer case duration ( P = 0.040), higher postcase clinician frustration ratings ( P < 0.001), higher ratings of patient stress ( P = 0.019), and lower patient ratings of their quality of life ( P = 0.010), of the quality of clinician teamwork ( P = 0.010), being treated with respect ( P = 0.003), and being listened to carefully ( P = 0.012). Trained patient raters evaluated NRE severity significantly higher than did clinician raters ( P < 0.001), while clinicians rated recurrence likelihood significantly higher than patients for both clinician ( P = 0.032) and patient-reported NREs ( P = 0.001).

CONCLUSIONS

Both patients and clinicians readily report events during clinical care that they believe deviate from optimal care expectations. These 2 primary stakeholders in safe, high-quality surgical care have different experiences and perspectives regarding NREs. The combination of patient- and clinician-reported NREs seems to be a promising patient-centered method of identifying healthcare system deficiencies and opportunities for improvement.

A pilot study to determine the incidence, type, and severity of non-routine events in neonates undergoing gastrostomy tube placement

BACKGROUND

Non-routine events (NRE) are defined as any suboptimal occurrences in a process being measured in the opinion of the reporter and comes from the field of human factors engineering. These typically occur well up-stream of an adverse event and NRE measurement has not been applied to the complex context of neonatal surgery. We sought to apply this novel safety event measurement methodology to neonates in the NICU undergoing gastrostomy tube placement.

METHODS

A prospective pilot study was conducted between November 2016 and August 2020 in the Level IV NICU and the pediatric operating rooms of an urban academic children's hospital to determine the incidence, severity, impact, and contributory factors of clinician-reported non-routine events (NREs, i.e., deviations from optimal care) and 30-day NSQIP occurrences in neonates receiving a G-tube.

RESULTS

Clinicians reported at least one NRE in 32 of 36 (89%) G-tube cases, averaging 3.0 (Standard deviation: 2.5) NRE reports per case. NSQIP-P review identified 7 cases (19%) with NSQIP-P occurrences and each of these cases had multiple reported NREs. One case in which NREs were not reported was without NSQIP-P occurrences. The odds ratio of having a NSQIP-P occurrence with the presence of an NRE was 0.695 (95% CI 0.06-17.04).

CONCLUSION

Despite being considered a "simple" operation, >80% of neonatal G-tube placement operations had at least one reported NRE by an operative team member. In this pilot study, NRE occurrence was not significantly associated with the subsequent reporting of an NSQIP-P occurrence. Understanding contributory factors of NREs that occur in neonatal surgery may promote surgical safety efforts and should be evaluated in larger and more diverse populations.

LEVEL OF EVIDENCE

IV.

Preventing clinical deterioration in cancer outpatients: Human centered design of a predictive model and response system

e13567

Background: Patients (pts) with cancer in the outpatient setting are at a high-risk for adverse events, such as unplanned hospitalizations and ER visits. A recent study found that up to 30% of hospital admissions were preventable. Identifying pts at risk of avoidable clinical deterioration remains a challenge, as clinicians may not be aware of pts’ experiences at home. The growing use of health IT presents an opportunity to identify and respond to clinical deterioration in patients before an adverse event occurs. In this study, we describe a human-centered design (HCD) process used to develop a clinical deterioration risk prediction system to improve the detection of and response to deterioration in cancer outpatients. Methods: Predictive model: We enrolled eligible cancer pts and collected data from each one including: FitBit, geolocation, EHR, and weekly patient-reported outcome measures (PROMs). Pts and their family caregivers could also report non-routine events (NREs), defined as any deviation from expected optimal care. We also captured unplanned treatment events (UTEs), a clinically meaningful change in the pt’s treatment course or care pathway. We developed a predictive model that generates a pt’s 7-day risk of clinical deterioration. Response system: We are developing a risk communication system (RCS) to communicate predicted risk scores to clinical teams. Using a HCD process, we first conducted 36 observations across 100 patient encounters to understand the environment of use. Next, we conducted 18 clinician interviews to define user needs. We have conducted 7 multi-disciplinary design sessions to iteratively develop prototypes of the RCS. We are currently conducting formative usability testing to assess the prototype and gather clinician feedback. Results: Predictive model: We have enrolled 36 cancer outpatients (24 head & neck, 9 gastrointestinal, and 3 lung). Pts completed a total of 219 weekly PROM surveys, reported 107 NREs and experienced 18 UTEs (e.g., infection). So far, models using EHR and PROM data are the most sensitive and precise (AUC: 0.983; 0.999). More patient data are required to develop higher quality stable models. Response system: We identified key design elements to include in the RCS, such as the caregiver’s phone number and the pt’s weight over time. Preliminary findings demonstrate high usability of the prototype RCS. Oncologists identified opportunities for the system to better support team communication and coordination, and to improve the identification and response to clinical deterioration in cancer outpatients. Conclusions: We have developed and tested a clinical deterioration risk prediction system for cancer outpatients. Future studies will implement the response system and evaluate its impact on clinical care.

Defining the Epidemiology of Safety Risks in Neonatal Intensive Care Unit Patients Requiring Surgery

OBJECTIVE

The aim of the study was to determine the incidence, type, severity, preventability, and contributing factors of nonroutine events (NREs)-events perceived by care providers or skilled observers as a deviations from optimal care based on the clinical situation-in the perioperative (i.e., preoperative, operative, and postoperative) care of surgical neonates in the neonatal intensive care unit and operating room.

METHODS

A prospective observational study of noncardiac surgical neonates, who received preoperative and postoperative neonatal intensive care unit care, was conducted at an urban academic children's hospital between November 1, 2016, and March 31, 2018. One hundred twenty-nine surgical cases in 109 neonates were observed. The incidence and description of NREs were collected via structured researcher-administered survey tool of involved clinicians. Primary measurements included clinicians' ratings of NRE severity and contributory factors and trained research assistants' ratings of preventability.

RESULTS

One or more NREs were reported in 101 (78%) of 129 observed cases for 247 total NREs. Clinicians reported 2 (2) (median, interquartile range) NREs per NRE case with a maximum severity of 3 (1) (possible range = 1-5). Trained research assistants rated 47% of NREs as preventable and 11% as severe and preventable. The relative risks for National Surgical Quality Improvement Program - pediatric major morbidity and 30-day mortality were 1.17 (95% confidence interval = 0.92-1.48) and 1.04 (95% confidence interval = 1.00-1.08) in NRE cases versus non-NRE cases.

CONCLUSIONS

The incidence of NREs in neonatal perioperative care at an academic children's hospital was high and of variable severity with a myriad of contributory factors.

Using Fitbit data to predict clinical deterioration and unplanned treatment events in cancer outpatients

e13560

Background: A common cause of preventable harm is the failure to detect and appropriately respond to clinical deterioration. Timely intervention is needed, particularly in cancer patients, to mitigate the effects of adverse events, disease progression, and medical error. This problem requires effective clinical surveillance, early recognition, timely notification of the appropriate clinician, and effective intervention. Methods: Applying a user-centered systems engineering design approach, we designed and implemented a surveillance-and-response system to improve the detection and response to clinical deterioration in cancer outpatients. The surveillance system predicts 7-day risk of UTEs, defined as clinically meaningful changes in the patient’s treatment course or cancer care pathway (e.g., any unplanned/unexpected: clinic or ER visit, hospital admission, or major treatment change and/or delays, and/or death). Data inputs consist of: 1) patient activity and health data collected by a Fitbit monitor; 2) geolocation data to measure activity outside the home (i.e., locations preselected at study onset); 3) clinical data from the hospital’s electronic health record; and 4) patient-reported outcomes measures (i.e., PROMs; the NCCN Distress Thermometer, the Comprehensive OpeN-Ended Survey or CONES, Global Health Score, items from the Consumer Assessment of Healthcare Providers and Systems (CAHPS)). Herein, we measured the effectiveness of Fitbit data alone to UTEs in a pilot sample of patients. Dimension reduction of Fitbit variables was first carried out by using Pearson correlation analysis to eliminate redundant variables. As UTEs are rare events, they were oversampled using the Synthetic Minority Oversampling Technique (SMOTE) to balance the dataset. A random forest classification model was trained to predict 7-day UTE risk. Model accuracy was determined by calculating the mean of Stratified 5-Fold Cross-Validation with 10 repeats. Results: Fitbit data was collected over a 6-8-week period from 14 head and neck cancer patients receiving surgical resection, outpatient chemotherapy, and/or radiotherapy. We identified six UTEs in 5 patients. A random forest classification model was developed from 10 variables derived from 7 Fitbit measures. The following variables were averaged or summed daily: average heart rate (HR), resting HR, below 50% or zone 1 of maximum HR, zone 2 and zone 3 HR combined (i.e., 70-100% of max HR), total daily calories, steps, and sleep in minutes. We achieved a model accuracy of 94% (ROC AUC: 0.984, Precision-Recall AUC: 0.985). Conclusions: Activity and health data collected by a commercial activity monitor demonstrated effectiveness in predicting patient UTEs when an oversampling procedure was used to adjust for class imbalance (i.e., low UTE rate). Future studies are recommended to verify and validate this result in a larger patient sample.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Prevalence of burnout and its relationship to health status and social support in more than 1000 subspecialty anesthesiologists

BACKGROUND

Physician burnout may be at 'epidemic' proportions due to factors associated with modern healthcare practice and technology. Practice attributes vary appreciably among subspecialists. Understanding burnout incidence and its associated factors could illuminate potential causes and interventions. We evaluated burnFout, mental and physical health, and social support and coping skills in acute and chronic pain physicians and pediatric and cardiac anesthesiologists.

METHODS

We administered the Maslach Burnout Inventory Human Services Survey (MBI-HSS), a two-item self-identified burnout measure, the Veterans RAND 12-item Health Survey and the Social Support and Personal Coping Survey to subspecialty society members practicing acute and chronic pain management, pediatric anesthesiology and cardiac anesthesiology. Multivariable regression analysis compared the groups, and adjusted burnout prevalence was compared with an all-physician and an employed general population sample.

RESULTS

Among 1303 participants (response rates 21.6%-35.6% among the subspecialty groups), 43.4% met established burnout criteria (range 30.0%-62.3%). Chronic pain physicians had significantly worse scores (unadjusted) than the other three groups of subspecialty anesthesiologists, the all-physician comparator group and the general population comparator group. Mental health inversely correlated with emotional exhaustion and depersonalization in all groups. Self-identified burnout correlated with the full MBI-HSS (R=0.54; positive predictive value of 0.939 (0.917, 0.955)). Physicians' scores for personal accomplishment were higher than population norms.

CONCLUSIONS

This study provides data on burnout prevalence and associated demographic, health and social factors in subspecialist anesthesiologists. Chronic pain anesthesiologists had significantly greater burnout than the other groups. The self-identified burnout metric performed well and may be an attractive alternative to the full MBI-HSS.

Health information technology:Fallacies and Sober realities - Redux A homage to Bentzi Karsh and Robert Wears

Since the publication of "Health Information Technology: Fallacies and Sober Realities" in 2010, health information technology (HIT) has become nearly ubiquitous in US healthcare facilities. Yet, HIT has yet to achieve its putative benefits of higher quality, safer, and lower cost care. There has been variable but largely marginal progress at addressing the 12 HIT fallacies delineated in the original paper. Here, we revisit several of the original fallacies and add five new ones. These fallacies must be understood and addressed by all stakeholders for HIT to be a positive force in achieving the high value healthcare system the nation deserves. Foundational cognitive and human factors engineering research and development continue to be essential to HIT development, deployment, and use.

Improving the Effectiveness of Health Information Technology: The Case for Situational Analytics

Health information technology has contributed to improvements in quality and safety in clinical settings. However, the implementation of new technologies in health care has also been associated with the introduction of new sociotechnical hazards, produced through a range of complex interactions that vary with social, physical, temporal, and technological context. Other industries have been confronted with this problem and have developed advanced analytics to examine context-specific activities of workers and related outcomes. The skills and data exist in health care to develop similar insights through situational analytics, defined as the application of analytic methods to characterize human activity in situations and identify patterns in activity and outcomes that are influenced by contextual factors. This article describes the approach of situational analytics and potentially useful data sources, including trace data from electronic health record activity, reports from users, qualitative field data, and locational data. Key implementation requirements are discussed, including the need for collaboration among qualitative researchers and data scientists, organizational and federal level infrastructure requirements, and the need to implement a parallel research program in ethics to understand how the data are being used by organizations and policy makers.

CYP2D6 Genotype-guided Metoprolol Therapy in Cardiac Surgery Patients: Rationale and Design of the Pharmacogenetic-guided Metoprolol Management for Postoperative Atrial Fibrillation in Cardiac Surgery (PREEMPTIVE) Pilot Study

OBJECTIVES

The Preemptive Pharmacogenetic-guided Metoprolol Management for Atrial Fibrillation in Cardiac Surgery (PREEMPTIVE) pilot trial aims to use existing institutional resources to develop a process for integrating CYP2D6 pharmacogenetic test results into the patient electronic health record, to develop an evidence-based clinical decision support tool to facilitate CYP2D6 genotype-guided metoprolol administration in the cardiac surgery setting, and to determine the impact of implementing this CYP2D6 genotype-guided integrated approach on the incidence of postoperative atrial fibrillation (AF), provider, and cost outcomes.

DESIGN

One-arm Bayesian adaptive design clinical trial.

SETTING

Single center, university hospital.

PARTICIPANTS

The authors will screen (including CYP2D6 genotype) up to 600 (264 ± 144 expected under the adaptive design) cardiac surgery patients, and enroll up to 200 (88 ± 48 expected) poor, intermediate, and ultrarapid CYP2D6 metabolizers over a period of 2 years at a tertiary academic center.

INTERVENTIONS

All consented and enrolled patients will receive the intervention of CYP2D6 genotype-guided metoprolol management based on CYP2D6 phenotype classified as a poor, intermediate, extensive (normal), or ultrarapid metabolizer.

MEASUREMENTS AND MAIN RESULTS

The primary outcome will be the incidence of postoperative AF. Secondary outcomes relating to rates of CYP2D6 genotype-guided prescription changes, costs, lengths of stay, and implementation metrics also will be investigated.

CONCLUSIONS

The PREEMPTIVE pilot study is the first perioperative pilot trial to provide essential information for the design of a future, large-scale trial comparing CYP2D6 genotype-guided metoprolol management with a nontailored strategy in terms of managing AF. In addition, secondary outcomes regarding implementation, clinical benefit, safety, and cost-effectiveness in patients undergoing cardiac surgery will be examined.

Use of a Portable Functional Near-Infrared Spectroscopy (fNIRS) System to Examine Team Experience During Crisis Event Management in Clinical Simulations

Objective: The aim of this study was to investigate the utilization of a portable functional near-infrared spectroscopy (fNIRS) system, the fNIRS Pioneer^TM, to examine team experience in high-fidelity simulation-based crisis event management (CEM) training for anesthesiologists in operating rooms.

Background: Effective evaluation of team performance and experience in CEM simulations is essential for healthcare training and research. Neurophysiological measures with wearable devices can provide useful indicators of team experience to compliment traditional self-report, observer ratings, and behavioral performance measures. fNIRS measured brain blood oxygenation levels and neural synchrony can be used as indicators of workload and team engagement, which is vital for optimal team performance.

Methods: Thirty-three anesthesiologists, who were attending CEM training in two-person teams, participated in this study. The participants varied in their expertise level and the simulation scenarios varied in difficulty level. The oxygenated and de-oxygenated hemoglobin (HbO and HbR) levels in the participants’ prefrontal cortex were derived from data recorded by a portable one-channel fNIRS system worn by all participants throughout CEM training. Team neural synchrony was measured by HbO/HbR wavelet transformation coherence (WTC). Observer-rated workload and self-reported workload and mood were also collected.

Results: At the individual level, the pattern of HbR level corresponded to changes of workload for the individuals in different roles during different phases of a scenario; but this was not the case for HbO level. Thus, HbR level may be a better indicator for individual workload in the studied setting. However, HbR level was insensitive to differences in scenario difficulty and did not correlate with observer-rated or self-reported workload. At the team level, high levels of HbO and HbR WTC were observed during active teamwork. Furthermore, HbO WTC was sensitive to levels of scenario difficulty.

Conclusion: This study showed that it was feasible to use a portable fNIRS system to study workload and team engagement in high-fidelity clinical simulations. However, more work is needed to establish the sensitivity, reliability, and validity of fNIRS measures as indicators of team experience.

Human performance measures for the evaluation of process control human-system interfaces in high-fidelity simulations

We reviewed the available literature on measuring human performance to evaluate human-system interfaces (HSIs), focused on high-fidelity simulations of industrial process control systems, to identify best practices and future directions for research and operations. We searched the available literature and then conducted in-depth review, structured coding, and analysis of 49 articles, which described 42 studies. Human performance measures were classified across six dimensions: task performance, workload, situation awareness, teamwork/collaboration, plant performance, and other cognitive performance indicators. Many studies measured performance in more than one dimension, but few studies addressed more than three dimensions. Only a few measures demonstrated acceptable levels of reliability, validity, and sensitivity in the reviewed studies in this research domain. More research is required to assess the measurement qualities of the commonly used measures. The results can provide guidance to direct future research and practice for human performance measurement in process control HSI design and deployment.

Acoustic features of auditory medical alarms-An experimental study of alarm volume

Audible alarms are a ubiquitous feature of all high-paced, high-risk domains such as aviation and nuclear power where operators control complex systems. In such settings, a missed alarm can have disastrous consequences. It is conventional wisdom that for alarms to be heard, "louder is better," so that alarm levels in operational environments routinely exceed ambient noise levels. Through a robust experimental paradigm in an anechoic environment to study human response to audible alerting stimuli in a cognitively demanding setting, akin to high-tempo and high-risk domains, clinician participants responded to patient crises while concurrently completing an auditory speech intelligibility and visual vigilance distracting task as the level of alarms were varied as a signal-to-noise ratio above and below hospital background noise. There was little difference in performance on the primary task when the alarm sound was -11 dB below background noise as compared with +4 dB above background noise-a typical real-world situation. Concurrent presentation of the secondary auditory speech intelligibility task significantly degraded performance. Operator performance can be maintained with alarms that are softer than background noise. These findings have widespread implications for the design and implementation of alarms across all high-consequence settings.

Integrating Foundational Sciences in a Clinical Context in the Post-Clerkship Curriculum

PURPOSE

To design, implement, and launch courses that integrate foundational science learning and clinical application in a post-clerkship undergraduate medical school curriculum.

METHOD

In Academic Year (AY) 15-16, as part of a comprehensive curricular revision, Vanderbilt University School of Medicine (VUSM) formally implemented "Integrated Science Courses" (ISCs) that combined rigorous training in the foundational sciences with meaningful clinical experiences. These courses integrated foundational sciences that could be leveraged in the clinical environment, utilized a variety of instructional modalities, and included quantitative and qualitative (competency-based milestones) student assessments. Each ISC underwent a rigorous quality improvement process that required input on foundational science content, student experience, and student performance assessment.

RESULTS

Eleven ISCs were delivered to 173 students in AY15-16, with some students taking more than one ISC. Immediately after completing each course, 93% (n=222) of ISC enrollees completed a course evaluation. Students (91%; n=201) 'agreed' or 'strongly agreed' that foundational science learning informed and enriched the clinical experiences. Furthermore, 94% (n=209) of students thought that the clinical experiences informed and enriched the foundational science learning. Ninety-four percent of the students anticipated using the foundational science knowledge acquired in future clinical training and practice.

CONCLUSION

The teaching of foundational sciences in the clinical workplace in the post-clerkship medical curriculum is challenging and resource-intensive, yet feasible. Additional experience with the model will inform the mix of courses as well as the breadth and depth of foundational science instruction that is necessary to foster scientifically-based clinical reasoning skills in each student.

The influence of stress responses on surgical performance and outcomes: Literature review and the development of the surgical stress effects (SSE) framework

BACKGROUND

Surgical adverse events persist despite several decades of system-based quality improvement efforts, suggesting the need for alternative strategies. Qualitative studies suggest stress-induced negative intraoperative interpersonal dynamics might contribute to performance errors and undesirable patient outcomes. Understanding the impact of intraoperative stressors may be critical to reducing adverse events and improving outcomes.

DATA SOURCES

We searched MEDLINE, psycINFO, EMBASE, Business Source Premier, and CINAHL databases (1996-2016) to assess the relationship between negative (emotional and behavioral) responses to acute intraoperative stressors and provider performance or patient surgical outcomes.

RESULTS/CONCLUSIONS

Drawing on theory and evidence from reviewed studies, we present the Surgical Stress Effects (SSE) framework. This illustrates how emotional and behavioral responses to stressors can influence individual surgical provider (e.g. surgeon, nurse) performance, team performance, and patient outcomes. It also demonstrates how uncompensated intraoperative threats and errors can lead to adverse events, highlighting evidence gaps for future research efforts.

A simulation-based curriculum to introduce key teamwork principles to entering medical students

BACKGROUND

Failures of teamwork and interpersonal communication have been cited as a major patient safety issue. Although healthcare is increasingly being provided in interdisciplinary teams, medical school curricula have traditionally not explicitly included the specific knowledge, skills, attitudes, and behaviors required to function effectively as part of such teams.

METHODS

As part of a new "Foundations" core course for beginning medical students that provided a two-week introduction to the most important themes in modern healthcare, a multidisciplinary team, in collaboration with the Center for Experiential Learning and Assessment, was asked to create an experiential introduction to teamwork and interpersonal communication. We designed and implemented a novel, all-day course to teach second-week medical students basic teamwork and interpersonal principles and skills using immersive simulation methods. Students' anonymous comprehensive course evaluations were collected at the end of the day. Through four years of iterative refinement based on students' course evaluations, faculty reflection, and debriefing, the course changed and matured.

RESULTS

Four hundred twenty evaluations were collected. Course evaluations were positive with almost all questions having means and medians greater than 5 out of 7 across all 4 years. Sequential year comparisons were of greatest interest for examining the effects of year-to-year curricular improvements. Differences were not detected among any of the course evaluation questions between 2007 and 2008 except that more students in 2008 felt that the course further developed their "Decision Making Abilities" (OR 1.69, 95% CI 1.07-2.67). With extensive changes to the syllabus and debriefer selection/assignment, concomitant improvements were observed in these aspects between 2008 and 2009 (OR = 2.11, 95% CI: 1.28-3.50). Substantive improvements in specific exercises also yielded significant improvements in the evaluations of those exercises.

CONCLUSIONS

This curriculum could be valuable to other medical schools seeking to inculcate teamwork foundations in their medical school's preclinical curricula. Moreover, this curriculum can be used to facilitate teamwork principles important to inter-disciplinary, as well as uni-disciplinary, collaboration.

Human Factors Research in Patient Safety: A Candid Assessment

Five panelists, playing active research roles in the Agency for Healthcare Research and Quality's patient safety research initiative, present their views on challenges to human factors research for enhancing patient safety. Bogner advocates a systems structure for linking the findings of various research projects so that the missing pieces of the patient safety puzzle can serve as fruitful targets for subsequent research. Carayon adopts a macroergonomic framework for designing interventions to clinical work systems while Cook focuses on the complexity that underlies configurable clinical devices. With respect to anesthesia and critical care, Weinger cites the successful use of task analysis, workload assessment, and video analysis, yet notes challenges regarding concerns about patient privacy, disruption of patient care, and cultural barriers. Xiao cites impressive HF/E work on team coordination and performance shaping factors and sees the need for greater use of video and information technology to improve institutional learning and coordination of patient care.

Assessing Medication Safety Technology in the Intensive Care Unit

Understanding the strengths and weaknesses of a technology in the context of the distributed system in which it is working is critical to assessing and improving the performance of that system. Taking a systems approach requires knowledge about how all agents in a system work together to achieve the goals of that system. With these aims, the alerting mechanism of infusion pumps containing Dose-Error Reduction Software (DERS) was studied to determine its effectiveness in the Intensive Care Units (ICU's) of three hospitals. In 1,146 of the 9,557 pump alerts (12.0%), the alert caused the clinician to change the input. Of these, 1,030 were changed to within the hospital's recommended dosing limits. The alert was overridden for 8,400 (88.0%) of the alerts. The data show that this technology successfully informed clinicians over 1000 times that unintended doses had been inputted and stopped those doses from reaching the patient, thereby averting potential Medication Events. The data also suggest that, because nearly 90% of the alerts were overridden, a well-intended and valuable alert may be perceived by the clinicians as a false alarm and may be overlooked. Another key finding from this analysis was that clinicians may have used potentially unsafe workarounds to administer intravenous drug boluses (i.e., more rapid infusion of a defined dose or volume) and to keep the patient's line active between infusions. In a separate parallel study, clinician self-report of potentially harmful medication events was studied. During 559 hours of direct observation, clinicians detected 27 (IV and non-IV) medication events. All of the reported events were outside of the scope of what DERS technology was designed to detect. In addition, during the same time period the technology detected five potentially harmful IV medication events that the clinicians did not report. The results of these two studies indicate two possible classes of solutions that could reduce the impact and likelihood of medication administration errors. One class of solutions involves the procedures and policies of the hospital, ensuring that process and technology implementations are optimally tuned, taking human performance and the current practice of the clinicians into account. The other class of solutions involves using new strategies and technologies to ensure that each system agent has access to other agents' perspectives, and the broader system's perspective. Studies such as these can provide insight into the use of safety technology during critical care processes and provide direction for future research, including more effective design of alerting mechanisms of ICU devices.

Developing a Technique to Measure Anesthesiologists' Real-Time Workload

Workload is a construct used to describe the extent to which an operator has engaged the cognitive and physical resources required for task performance. As task difficulty increases, operators allocate more resources to maintain acceptable performance. The purpose of the present study is to develop a technique to measure workload continuously, with high-resolution in real-time, and in a way that accounts for the contribution of multiple individual task components of the job of administering anesthesia. Workload associated with 51 clinical tasks performed by anesthesiologists was assessed using a written survey instrument, developed for this study. Participants were 241 anesthesia providers. Interval scale values were computed for each of the tasks using direct estimatation procedures. These values were then used to produce workload density maps for actual anesthesia cases by weighting the tasks performed during the case by their workload scale values. Moment-by-moment data on the specific tasks performed by anesthesiologists were obtained in real-time by trained observers during actual anesthetic cases. This type of analysis can be used to evaluate the costs associated with technical anesthesia procedures, increase our understanding of the anesthesiologist's job, and guide rational optimization of procedures, equipment, scheduling, and training.

Panel: The Role of Human Factors in Healthcare – 2020

What role will human factors professionals play in healthcare 2020? Health systems throughout the world face a number of common pressures, related to demography, epidemiology, science and technology developments, and medical demand. In particular, while developments in technology do not just provide health care with new possibilities for human factors engineering, medical interventions and therapies. They also produce changes the in our understanding of sickness and health, the possibilities and needs for managing the systems, for innovation, for standardization, and the political and economic relationships. The health care providers not only have to cope with these technological developments but assure their successful implementation and acceptance. The uncertainties and expectations linked to these innovations face major issues within the research and health care system, such as policies for managing scarcity of resources and changes in the relative frequency of diseases because of factors like ageing, and mobile global population and the like. A panel of both healthcare and human factors experts will discuss the role that human factors will play in healthcare in 2020.

Impact of electronic health record technology on the work and workflow of physicians in the intensive care unit

OBJECTIVE

To assess the impact of EHR technology on the work and workflow of ICU physicians and compare time spent by ICU resident and attending physicians on various tasks before and after EHR implementation.

DESIGN

EHR technology with electronic order management (CPOE, medication administration and pharmacy system) and physician documentation was implemented in October 2007.

MEASUREMENT

We collected a total of 289 h of observation pre- and post-EHR implementation. We directly observed the work of residents in three ICUs (adult medical/surgical ICU, pediatric ICU and neonatal ICU) and attending physicians in one ICU (adult medical/surgical ICU).

RESULTS

EHR implementation had an impact on the time distribution of tasks as well as the temporal patterns of tasks. After EHR implementation, both residents and attending physicians spent more of their time on clinical review and documentation (40% and 55% increases, respectively). EHR implementation also affected the frequency of switching between tasks, which increased for residents (from 117 to 154 tasks per hour) but decreased for attendings (from 138 to 106 tasks per hour), and the temporal flow of tasks, in particular around what tasks occurred before and after clinical review and documentation. No changes in the time spent in conversational tasks or the physical care of the patient were observed.

CONCLUSIONS

The use of EHR technology has a major impact on ICU physician work (e.g., increased time spent on clinical review and documentation) and workflow (e.g., clinical review and documentation becoming the focal point of many other tasks). Further studies should evaluate the impact of changes in physician work on the quality of care provided.

Survey of Infection Control Programs in a Large National Healthcare System

In light of consumers' and regulators' increasing focus on infection prevention, infection control practices and resources were surveyed at 134 hospitals owned by the Hospital Corporation of America. Infection control practices and resources varied substantially among hospitals, and many facilities reported difficulty acquiring the data they needed to report infection rates.

Barriers to Effective Preoperative Handover Communication in the Neonatal Intensive Care Unit

We sought to better understand the preoperative neonatal intensive care unit to operating room (NICU-to-OR) handover process and to elicit barriers to effective handovers. We first conducted observations of NICU-to-OR handovers to ascertain current handover practices, including the participants involved, handover content, as well as barriers and facilitators of effective handovers. We then developed a survey tool to assess the generalizability of our findings to other NICUs across the country. The resulting pilot data highlight key areas for future research and potential interventions to improve the quality of NICU-to-OR care transitions.

Operating room clinicians' ratings of workload: a vignette simulation study

OBJECTIVES

Increased clinician workload is associated with medical errors and patient harm. The Quality and Workload Assessment Tool (QWAT) measures anticipated (pre-case) and perceived (post-case) clinical workload during actual surgical procedures using ratings of individual and team case difficulty from every operating room (OR) team member. The purpose of this study was to examine the QWAT ratings of OR clinicians who were not present in the OR but who read vignettes compiled from actual case documentation to assess interrater reliability and agreement with ratings made by clinicians involved in the actual cases.

METHODS

Thirty-six OR clinicians (13 anesthesia providers, 11 surgeons, and 12 nurses) used the QWAT to rate 6 cases varying from easy to moderately difficult based on actual ratings made by clinicians involved with the cases. Cases were presented and rated in random order. Before rating anticipated individual and team difficulty, the raters read prepared clinical vignettes containing case synopses and much of the same written case information that was available to the actual clinicians before the onset of each case. Then, before rating perceived individual and team difficulty, they read part 2 of the vignette consisting of detailed role-specific intraoperative data regarding the anesthetic and surgical course, unusual events, and other relevant contextual factors.

RESULTS

Surgeons had higher interrater reliability on the QWAT than did OR nurses or anesthesia providers. For the anticipated individual and team workload ratings, there were no statistically significant differences between the actual ratings and the ratings obtained from the vignettes. There were differences for the 3 provider types in perceived individual workload for the median difficulty cases and in the perceived team workload for the median and more difficult cases.

CONCLUSIONS

The case difficulty items on the QWAT seem to be sufficiently reliable and valid to be used in other studies of anticipated and perceived clinical workload of surgeons. Perhaps because of the limitations of the clinical documentation shown to anesthesia providers and OR nurses in the current vignette study, more evidence needs to be gathered to demonstrate the criterion-related validity of the QWAT difficulty items for assessing the workload of nonsurgeon OR clinicians.

Presenting multiple drug alerts in an ambulatory electronic prescribing system: a usability study of novel prototypes

OBJECTIVE

This study explores alternative approaches to the display of drug alerts, and examines whether and how human-factors based interface design can be used to improve the prescriber's perception about drug alert presentation, signal detection from noisy alert data, and their comprehension of clinical decision support during electronic prescribing.

METHODS

We reviewed issues with presenting multiple drug alerts in electronic prescribing systems. User-centered design, consisting of iterative usability and prototype testing was applied. After an iterative design phase, we proposed several novel drug alert presentation interfaces; expert evaluation and formal usability testing were applied to access physician prescribers' perceptions of the tools. We mapped drug alert attributes to different interface constructs. We examined four different interfaces for presenting multiple drug alerts.

RESULTS

A TreeDashboard View was better perceived than a text-based ScrollText View with respect to the ability to detect critical information, the ability to accomplish tasks, and the perceptional efficacy of finding information.

CONCLUSION

A robust model for studying multiple drug-alert presentations was developed. Several drug alert presentation interfaces were proposed. The TreeDashboard View was better perceived than the text-based ScrollText View in delivering multiple drug alerts during a simulation of electronic prescribing.

Self-reported subjective workload of on-call interns

BACKGROUND

Workload has traditionally been measured by using surrogates, such as number of patients admitted or census, but these may not fully represent the complex concept of workload.

OBJECTIVE

We measured self-reported subjective workload of interns and explored the relationship between subjective workload and possible predictors of it.

METHODS

Trained research assistants observed internal medicine interns on call on a general medicine service. Approximately once an hour, the research assistants recorded the self-reported subjective workload of the interns by using Borg's Self-Perceived Exertion Scale, a 6 to 20 scale, and also recorded their own perceptions of the intern's workload. Research assistants continuously recorded the tasks performed by the interns. Interns were surveyed before and after the observation to obtain demographic and census data.

RESULTS

Our sample included 25 interns, with a mean age of 28.6 years (SD, 2.4 years). Mean self-reported subjective workload was 12.0 (SD, 2.4). Mean self-reported subjective workload was significantly correlated with intern age (r  =  0.49, P < .05), but not with team or intern census, number of admissions, or number of patients cross-covered. Self-reported subjective workload in the period after sign-out was significantly higher than in the period before and during sign-out (P < .001).

CONCLUSIONS

Self-reported subjective workload was not associated with traditional measures of workload. However, receiving sign-out and assuming the care of cross-coverage patients may be related to higher subjective workload in interns. Given the patient safety implications of workload, it is important that the medical education community have tools to evaluate workload and identify contributors to it.