Measuring compliance with preventive care guidelines: standardized patients, clinical vignettes, and the medical record

Randomized controlled trial demonstrates novel tools to assess patient outcomes of Indigenous cultural safety training

BACKGROUND

Health care routinely fails Indigenous peoples and anti-Indigenous racism is common in clinical encounters. Clinical training programs aimed to enhance Indigenous cultural safety (ICS) rely on learner reported impact assessment even though clinician self-assessment is poorly correlated with observational or patient outcome reporting. We aimed to compare the clinical impacts of intensive and brief ICS training to control, and to assess the feasibility of ICS training evaluation tools, including unannounced Indigenous standardized patient (UISP) visits.

METHOD

Using a prospective parallel group three-arm randomized controlled trial design and masked standardized patients, we compared the clinical impacts of the intensive interactive, professionally facilitated, 8- to10-h Sanyas ICS training; a brief 1-h anti-bias training adapted to address anti-Indigenous bias; and control continuing medical education time-attention matched to the intensive training. Participants included 58 non-Indigenous staff physicians, resident physicians and nurse practitioners from family practice clinics, and one emergency department across four teaching hospitals in Toronto, Canada. Main outcome measures were the quality of care provided during UISP visits including adjusted odds that clinician would be recommended by the UISP to a friend or family member; mean item scores on patient experience of care measure; and clinical practice guideline adherence for NSAID renewal and pain assessment.

RESULTS

Clinicians in the intensive or brief ICS groups had higher adjusted odds of being highly recommended to friends and family by standardized patients (OR 6.88, 95% CI 1.17 to 40.45 and OR 7.78, 95% CI 1.05 to 58.03, respectively). Adjusted mean item patient experience scores were 46% (95% CI 12% to 80%) and 40% (95% CI 2% to 78%) higher for clinicians enrolled in the intensive and brief training programs, respectively, compared to control. Small sample size precluded detection of training impacts on clinical practice guideline adherence; 100% of UISP visits were undetected by participating clinicians.

CONCLUSIONS

Patient-oriented evaluation design and tools including UISPs were demonstrated as feasible and effective. Results show potential impact of cultural safety training on patient recommendation of clinician and improved patient experience. A larger trial to further ascertain impact on clinical practice is needed.

TRIAL REGISTRATION

Clinicaltrials.org NCT05890144. Retrospectively registered on June 5, 2023.

A Randomized Controlled Study on Clinical Adherence to Evidence-Based Guidelines in the Management of Simulated Patients With Barrett's Esophagus and the Clinical Utility of a Tissue Systems Pathology Test: Results From Q-TAB

INTRODUCTION

Barrett's esophagus (BE) is a precursor to esophageal adenocarcinoma. Physicians infrequently adhere to guidelines for managing BE, leading to either reduced detection of dysplasia or inappropriate re-evaluation.

METHODS

We conducted a three-arm randomized controlled trial with 2 intervention arms to determine the impact of a tissue systems pathology (TSP-9) test on the adherence to evidence-based guidelines for simulated patients with BE. Intervention 1 received TSP-9 results, and intervention 2 had the option to order TSP-9 results. We collected data from 259 practicing gastroenterologists and gastrointestinal surgeons who evaluated and made management decisions for 3 types of simulated patients with BE: nondysplastic BE, indefinite for dysplasia, and low-grade dysplasia.

RESULTS

Intervention 1 was significantly more likely to correctly assess risk of progression to high-grade dysplasia/esophageal adenocarcinoma and offer treatment in accordance with US society guidelines compared with the control group (+6.9%, 95% confidence interval +1.4% to +12.3%). There was no significant difference in ordering guideline-recommended endoscopic eradication therapy. However, for cases requiring annual endoscopic surveillance, we found significant improvement in adherence for intervention 1, with a difference-in-difference of +18.5% ( P = 0.019). Intervention 2 ordered the TSP-9 test in 21.9% of their cases. Those who ordered the test performed similarly to intervention 1; those who did not, performed similarly to the control group.

DISCUSSION

The TSP-9 test optimized adherence to clinical guidelines for surveillance and treatment of both patients with BE at high and low risk of disease progression. Use of the TSP-9 test can enable physicians to make risk-aligned management decisions, leading to improved patient health outcomes.

Comparison of medical documentation between pharmacist-led anticoagulation clinics and physician-led anticoagulation clinics: A retrospective study

Background and objectives

High-quality documentation is critical in medical settings for providing safe patient care. This study was done with the objective of assessing the standard of medical records in anticoagulation clinics and investigating the distinctions between notes written by pharmacists and physicians.

Methods

A retrospective cross-sectional analysis of data from electronic health records (EHRs) was performed on patients who received anticoagulation and were observed at anticoagulation clinics from October to December 2020. Patients were monitored in two anticoagulation clinics, one administered by pharmacists and the other by physicians. The quality of the documentation was assessed using a score, and the note was assigned one of five categories according to its score: very good, good, average, poor, and very poor. The data was analyzed using Stata/SE 13.1. P value<0.05 was considered significant in all analytical tests.

Results

A total of 331 patients were included. While 160 patients (48.3%) were followed by the physician-led clinic, 171 (51.6%) were by the pharmacist-led clinic. The average age of the patients was 54 ± 15. 60.73% of them were female, and 90.3% of them were Saudi nationals. Warfarin was the most widely used anticoagulant (70%), followed by rivaroxaban (15.7%). Compared to physicians, pharmacists demonstrated very strong documentation (54% vs. 18%). The examination of the variables considered in the study revealed that physicians had significantly less drug-drug interaction documentation (17 vs. 71 times) or drug-food interaction documentation (23 vs. 71 times) than pharmacists. In terms of follow-up frequency, pharmacists were found to adhere to the clinic protocol (150 times) more frequently than physicians (104 times). However, there was no significant difference in therapeutic plan documentation between the two groups. (p = 0.416).

Conclusion

Pharmacists were more comprehensive in their documentation than physicians in anticoagulation clinics. Unified clinic documentation can ensure consistent documentation within EHRs across all disciplines.

Clinical Variation in the Treatment Practices for Patients With Type 2 Diabetes: A Cross-Sectional Patient Simulation Study Among Primary Care Physicians and Cardiologists

Background Cardiovascular disease risk stratification is necessary and critically important in patients with type 2 diabetes. Despite its known benefits to guide treatment and prevention, we hypothesized that providers do not routinely incorporate this into their diagnostic and treatment decisions. Methods and Results The QuiCER DM (QURE CVD Evaluation of Risk in Diabetes Mellitus) study enrolled 161 primary care physicians and 80 cardiologists. Between March 2022 and June 2022, we measured the care variation in risk determination among these providers caring for simulated patients with type 2 diabetes. We found a wide variation in the overall assessment of cardiovascular disease in patients with type 2 diabetes. Participants performed half of the necessary care items with quality-of-care scores, ranging between 13% and 84%, averaging 49.4±12.6%. Participants did not assess cardiovascular risk in 18.3% of cases and incorrectly stratified risk in 42.8% of cases. Only 38.9% of participants arrived at the correct cardiovascular risk stratification. Those who correctly identified a cardiovascular risk score were significantly more likely to order nonpharmacologic treatments, advising on their patients' nutrition (38.8% versus 29.9%, P=0.013) and the correct glycated hemoglobin target (37.7% versus 15.6%, P<0.001). Pharmacologic treatments, however, did not vary between those who correctly specified risk and those who did not. Conclusions Physician participants struggled to determine the correct cardiovascular disease risk and specify the appropriate pharmacologic interventions in simulated patients with type 2 diabetes. Additionally, there was a wide variation in the quality of care regardless of risk level, indicating opportunities to improve risk stratification.

Randomized prospective trial to detect and distinguish between medication nonadherence, drug-drug interactions, and disease progression in chronic cardiometabolic disease

BACKGROUND

Disentangling nonadherence (NA), drug-drug interactions (DDIs), and disease progression from each other is an important clinical challenge for providers caring for patients with cardiometabolic diseases. NAs and DDIs are both ubiquitous and often overlooked. We studied a novel chronic disease management (CDM) test to detect medication adherence and the presence and severity of DDIs.

MATERIALS AND METHODS

We conducted a prospective, randomized controlled trial of 236 primary care physicians using computer-based, simulated patients, measuring clinical care with and without access to the CDM test. The primary outcomes were whether use of the CDM test increased the accuracy of diagnoses and ordering better treatments and how effective the intervention materials were in getting participants to order the CDM test.

RESULTS

Physicians given the CDM test results showed a + 13.2% improvement in their diagnosis and treatment quality-of-care scores (p < 0.001) in the NA patient cases and a + 13.6% improvement in the DDI cases (p < 0.001). The difference-in-difference calculations between the intervention and control groups were + 10.4% for NA and + 10.8% for DDI (p < 0.01 for both). After controlling for physician and practice co-factors, intervention, compared to control, was 50.4x more likely to recognize medication NA and 3.3x more likely to correctly treat it. Intervention was 26.9x more likely to identify the DDI and 15.7x more likely to stop/switch the interacting medication compared to control. We found no significant improvements for the disease progression patient cases.

CONCLUSION

Distinguishing between nonadherence, drug-drug interactions, and disease progression is greatly improved using a reliable test, like the CDM test; improved diagnostic accuracy and treatment has the potential to improve patient quality of life, medication safety, clinical outcomes, and efficiency of health delivery.

TRIAL REGISTRATION

clinicaltrials.gov (NCT05192590).

Gaps in the Care of Pulmonary Hypertension: A Cross-Sectional Patient Simulation Study Among Practicing Cardiologists and Pulmonologists

Background Diagnosis of pulmonary hypertension (PH) is often delayed or missed, leading to disease progression and missed treatment opportunities. In this study, we measured variation in care provided by board-certified cardiologists and pulmonologists in simulated patients with potentially undiagnosed PH. Methods and Results In a cross-sectional study (https://www.clinicaltrials.gov, NCT04693793), 219 US practicing cardiologists and pulmonologists cared for simulated patients presenting with symptoms of chronic dyspnea and associated signs of potential PH. We scored the clinical quality-of-care decisions made in a clinical encounter against predetermined evidence-based criteria. Overall, quality-of-care scores ranged from 18% to 74%, averaging 43.2%±11.5%. PH, when present, was correctly suspected 49.1% of the time. Conversely, physicians incorrectly identified PH in 53.7% of non-PH cases. Physicians ordered 2-dimensional echocardiography in just 64.3% of cases overall. Physicians who ordered 2-dimensional echocardiography in the PH cases were significantly more likely to get the presumptive diagnosis (61.9% versus 30.7%; P<0.001). Ordering other diagnostic work-up items showed similar results for ventilation/perfusion scan (81.5% versus 51.4%; P=0.005) and high-resolution computed tomography (60.4% versus 43.2%; P=0.001). Physicians who correctly identified PH were significantly more likely to order confirmatory right heart catheterization or refer to PH center (67.3% versus 15.8%; P<0.001). Conclusions A wide range of care in the clinical practice among simulated patients presenting with possible PH was found, specifically in the evaluation and plan for definitive diagnosis of patients with PH. The delay or misdiagnosis of PH is likely attributed to a low clinical suspicion, nonspecific symptoms, and underuse of key diagnostic tests. Registration URL: https://www.clinicaltrials.gov; Unique identifier: NCT04693793.

Clinical variation in surveillance and management of Barrett's esophagus: A cross-sectional study of gastroenterologists and gastrointestinal surgeons

Appropriate surveillance and treatment of Barrett's esophagus (BE) is vital to prevent disease progression and decrease esophageal adenocarcinoma (EAC)-related mortality. We sought to determine the variation in BE care and identify improvement opportunities. 275 physicians (113 general gastroenterologists, 128 interventional gastroenterologists, 34 gastrointestinal surgeons) cared for 3 simulated patients, one each from 3 BE clinical scenarios: non-dysplastic BE (NDBE), BE indefinite for dysplasia (IND), and BE with low grade dysplasia (LGD), and care scores were measured against societal guidelines. Overall quality-of-care scores ranged from 17% to 85% with mean of 47.9% ± 11.8% for NDBE, 50.8% ± 11.7% for IND, and 52.7% ± 12.2% for LGD. Participants appropriately determined risk of progression 20.3% of the time: 14.4% for NDBE cases, 19.9% for LGD cases, and 26.8% for IND cases (P = .001). Treatment and follow-up care scores averaged 12.9% ± 17.5% overall. For the LGD cases, guideline-recommended twice-daily PPI treatment was ordered only 24.7% of the time. Guideline-based follow-up endoscopic surveillance was done in only 27.7% of NDBE cases and 32.7% of IND cases. For the LGD cases, 45.4% ordered endoscopic eradication therapy while 25.1% chose annual endoscopic surveillance. Finally, participants provided counseling on lifestyle modifications in just 20% of cases. Overall care of patients diagnosed with BE varied widely and showed room for improvement. Specific opportunities for improvement were adherence to guideline recommended surveillance intervals, patient counseling, and treatment selection for LGD. Physicians would potentially benefit from additional BE education, endoscopic advances, and better methods for risk stratification.

Preventive Health Screening in Veterans Undergoing Bariatric Surgery

INTRODUCTION

Individuals with obesity are vulnerable to low rates of preventive health screening. Veterans with obesity seeking bariatric surgery are also hypothesized to have gaps in preventive health screening. Evaluation in a multidisciplinary bariatric surgery clinic is a point of interaction with the healthcare system that could facilitate improvements in screening.

METHODS

This is a retrospective cohort study of 381 consecutive patients undergoing bariatric surgery at a Veterans Affairs Hospital from January 2010 to October 2021. Age- and sex-appropriate health screening rates were determined at initial referral to a multidisciplinary bariatric surgery clinic and at the time of surgery. Rates of guideline concordance at both time points were compared using McNemar's test. Univariate and multivariate analyses were performed to identify the risk factors for nonconcordance.

RESULTS

Concordance with all recommended screening was low at initial referral and significantly improved by time of surgery (39.1%‒63.8%; p<0.001). Screening rates significantly improved for HIV (p<0.001), cervical cancer (p=0.03), and colon cancer (p<0.001). Increases in BMI (p=0.005) and the number of indicated screening tests (p=0.029) were associated with reduced odds of concordance at initial referral. Smoking history (p=0.012) and increasing distance to the nearest Veterans Affairs Medical Center (p=0.039) were associated with reduced odds of change from nonconcordance at initial referral to concordance at the time of surgery.

CONCLUSIONS

Rates of preventive health screening in Veterans with obesity are low. A multidisciplinary bariatric surgery clinic is an opportunity to improve preventive health screening in Veterans referred for bariatric surgery.

Firearm injuries in children: a missed opportunity for firearm safety education

BACKGROUND

Surgeons frequently care for children who have sustained gunshot wounds (GSWs). However, firearm safety education is not a focus in general surgery training. We hypothesised that firearm safety discussions do not routinely take place when children present to a trauma centre with a GSW.

METHOD

A retrospective review of patients <18 years presenting with GSWs to a level 1 paediatric trauma centre from 2009 to 2019 was performed. The primary outcome was discussion of firearm safety with the patient or family. The secondary outcome was notification of child protective services (CPS).

RESULTS

A total of 226 patients with GSWs were identified, 22% were unintentional and 63% were assault. Firearm safety discussions took place in 10 cases (4.4%). Firearm safety discussions were more likely to occur after unintentional injuries compared with other mechanisms (16.0% vs 1.3%, p<0.001). CPS was contacted in 29 cases (13%). CPS notification was more likely for unintentional injuries compared with other mechanisms (40% vs 3.9%, p<0.001) and for younger patients (7 years vs 15 years, p<0.001).

CONCLUSION

At a paediatric trauma centre, firearm safety discussions occurred in 4.4% of cases of children presenting with a GSW. There is a significant room for improvement in providing safety education interventions.

Impact of an unannounced standardized veteran program on access to community-based services for veterans experiencing homelessness

Background

The United States Department of Veterans Affairs established a program in which actors incognito portray veterans experiencing homelessness with pre-determined needs to identify barriers to access and services at community-based organizations.

Methods

From 2017 to 2019, actors who varied in gender, skin color and age portrayed one of three scripts at all VA Community-Based Resource and Referral Centers (CRRCs) serving veterans experiencing homelessness in 30 cities and completed an evaluative survey. They carried authentic VA identification and were registered in a VA patient database for each identity. CRRCs were provided with reports annually and asked to implement corrective plans. Data from the survey were analysed for change over time.

Results

Access to food, counselling, PTSD treatment, and hypertension/prediabetes care services increased significantly from 68–77% in year 2 to 83–97% in year 3 (each P < 0.05 adjusted for script present). A significant disparity in access for African American actors resolved following more uniform adherence to pre-existing policies.

Conclusions

The ‘unannounced standardized veteran’ (USV) can identify previously unrecognized barriers to needed services and care. Audit and feedback programs based on direct covert observation with systematic data collection and rapid feedback may be an effective strategy for improving services to highly vulnerable populations.

Variation in availability and use of surgical care for female urinary incontinence: a mixed-methods study

Background

Urinary incontinence affects between 25% and 45% of women. The availability and quality of services is variable and inequitable, but our understanding of the drivers is incomplete.

Objectives

The objectives of the study were to model patient, specialist clinician, primary and secondary care, and geographical factors associated with referral and surgery for urinary incontinence, and to explore women’s experiences of urinary incontinence and expectations of treatments.

Design

This was a mixed-methods study.

Setting

The setting was NHS England.

Participants

Data were collected from all women with a urinary incontinence diagnosis in primary care data, and all women undergoing mid-urethral mesh tape surgery for stress urinary incontinence were included. Interviews were also carried out with 28 women from four urogynaecology clinics who were deciding whether or not to have surgery, and surveys were completed by 245 members of the Royal College of Obstetricians and Gynaecologists with a specialist interest in urinary incontinence.

Data sources

The sources were patient-level data from Hospital Episode Statistics, the Clinical Practice Research Datalink and the Office for National Statistics mortality data linked to Hospital Episode Statistics. Interviews were conducted with women. An online vignette survey was conducted with members of the Royal College of Obstetricians and Gynaecologists.

Main outcome measures

The main outcome measures were the rates of referral from primary to secondary care and surgery after referral, the rates of stress urinary incontinence surgery by geographical area, the risk of mid-urethral mesh tape removal and reoperation after mid-urethral mesh tape insertion.

Results

Almost half (45.8%) of women with a new urinary incontinence diagnosis in primary care were referred to a urinary incontinence specialist: 59.5% of these referrals were within 30 days of diagnosis. In total, 14.2% of women referred to a specialist underwent a urinary incontinence procedure (94.5% of women underwent a stress urinary incontinence procedure and 5.5% underwent an urgency urinary incontinence procedure) during a follow-up period of up to 10 years. Not all women were equally likely to be referred or receive surgery. Both referral and surgery were less likely for older women, those who were obese and those from minority ethnic backgrounds. The stress urinary incontinence surgery rate was 40 procedures per 100,000 women per year, with substantial geographical variation. Among women undergoing mid-urethral mesh tape insertion for stress urinary incontinence, the 9-year mesh tape removal rate was 3.3%. Women’s decision-making about urinary incontinence surgery centred on perceptions of their urinary incontinence severity and the seriousness/risk of surgery. Women judged urinary incontinence severity in relation to their daily lives and other women’s experiences, rather than frequency or quantity of leakage, as is often recorded and used by clinicians. Five groups of UK gynaecologists could be distinguished who differed mainly in their average inclination to recommend surgery to hypothetical urinary incontinence patients. The gynaecologists’ recommendations were also influenced by urinary incontinence subtype and the patient’s history of previous surgery.

Limitations

The primary and secondary care data lacked information on the severity of urinary incontinence.

Conclusions

There was substantial variation in rates of referrals, surgery, and mesh tape removals, both geographically and between women of different ages and women from different ethnic backgrounds. The variation persisted after adjustment for factors that were likely to affect women’s preferences. Growing safety concerns over mid-urethral mesh tape surgery for stress urinary incontinence during the period from which the data are drawn are likely to have introduced more uncertainty to women’s and clinicians’ treatment decision-making.

Future work

Future work should capture outcomes relevant to women, including ongoing urinary incontinence and pain that is reported by women themselves, both before and after mesh and non-mesh procedures, as well as following conservative treatments. Future research should examine long-term patient-reported outcomes of treatment, including for women who do not seek further health care or surgery, and the extent to which urinary incontinence severity explains observed variation in referrals and surgery.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research; Vol. 9, No. 7. See the NIHR Journals Library website for further project information.

Impact of Unannounced Standardized Patient Audit and Feedback on Care, Documentation, and Costs: an Experiment and Claims Analysis

BACKGROUND

Meaningful variations in physician performance are not always discernible from the medical record.

OBJECTIVE

We used unannounced standardized patients to measure and provide feedback on care quality and fidelity of documentation, and examined downstream effects on reimbursement claims.

DESIGN

Static group pre-post comparison study conducted between 2017 and 2019.

SETTING

Fourteen New Jersey primary care practice groups (22 practices) enrolled in Horizon BCBS's value-based program received the intervention. For claims analyses, we identified 14 additional comparison practice groups matched on county, practice size, and claims activity.

PARTICIPANTS

Fifty-nine of 64 providers volunteered to participate.

INTERVENTION

Unannounced standardized patients (USPs) made 217 visits portraying patients with 1-2 focal conditions (diabetes, depression, back pain, smoking, or preventive cancer screening). After two baseline visits to a provider, we delivered feedback and conducted two follow-up visits.

MEASUREMENTS

USP-completed checklists of guideline-based provider care behaviors, visit audio recordings, and provider notes were used to measure behaviors performed and documentation errors pre- and post-feedback. We also compared changes in 3-month office-based claims by actual patients between the intervention and comparison practice groups before and after feedback.

RESULTS

Expected clinical behaviors increased from 46% to 56% (OR = 1.53, 95% CI 1.29-1.83, p < 0.0001), with significant improvements in smoking cessation, back pain, and depression screening. Providers were less likely to document unperformed tasks after (16%) than before feedback (18%; OR = 0.74, 95% CI 0.62 to 0.90, p = 0.002). Actual claim costs increased significantly less in the study than comparison group for diabetes and depression but significantly more for smoking cessation, cancer screening, and low back pain.

LIMITATIONS

Self-selection of participating practices and lack of access to prescription claims.

CONCLUSION

Direct observation of care identifies hidden deficits in practice and documentation, and with feedback can improve both, with concomitant effects on costs.

Evaluating the care provision of a community-based serious-illness care program via chart measures

BACKGROUND

Although quality-of-care domains for home-based primary and palliative programs have been proposed, they have had limited testing in practice. Our aim was to evaluate the care provision in a community-based serious-illness care program, a combined home-based primary and palliative care model.

METHODS

Retrospective chart review of patients in an academic community-based serious-illness care program in central North Carolina from August 2014 to March 2016 (n = 159). Chart review included demographics, health status, and operationalized measures of seven quality-of-care domains: medical assessment, care coordination, safety, quality of life, provider competency, goal attainment, and access.

RESULTS

Patients were mostly women (56%) with an average age of 70 years. Patients were multi-morbid (53% ≥3 comorbidities), functionally impaired (45% had impairment in ≥2 activities of daily living) and 32% had dementia. During the study period, 31% of patients died. Chart review found high rates assessment of functional status (97%), falls (98%), and medication safety (96%). Rates of pain assessment (70%), advance directive discussions (65%), influenza vaccination (59%), and depression assessment (54% of those with a diagnosis of depression) were lower. Cognitive barriers, spiritual needs, and behavioral issues were assessed infrequently (35, 22, 21%, respectively).

CONCLUSION

This study is one of the first to operationalize and examine quality-of-care measures for a community-based serious-illness care program, an emerging model for vulnerable adults. Our operationalization should not constitute validation of these measures and revealed areas for improvement; however, the community-based serious-illness care program performed well in several key quality-of-care domains. Future work is needed to validate these measures.

Randomized Trial on the Clinical Utility of a Novel Biomarker Panel to Identify Treatable Determinants of Chronic Pain

Millions suffer daily from chronic pain diagnosed anatomically and treated with opioids. Research shows that underlying nutritional, metabolic and oxidative stressors, which drive the development or worsening of chronic pain, are not diagnosed despite the fact that treatment of these primary pain pathways relieves pain and increases function. One of the main reasons for this gap in care is the lack of a simple diagnostic assay to help clinicians make these diagnoses. We examined the clinical utility of a urine-based pain biomarker panel. Primary care physicians were randomized into the test group and compared to controls. We measured their ability to make the diagnosis and treat a total of nine standardized patients, with common but challenging cases of chronic pain, over two rounds of data collection in a pre–post design using a fixed-effects model. Intervention doctors received educational materials on a novel pain biomarker panel after the baseline round and had access to biomarker test results. Provider responses were measured against evidence-based criteria. The two study arms at baseline provided similar, poor care for three different primary pain pathways: nutritional deficiencies (5.0% control versus 9.2% intervention treated, p = 0.208), metabolic abnormalities (1.0% control versus 0% for intervention treated, p = 0.314), and oxidative stress (1.2% control versus 0% intervention treated, p = 0.152). After the introduction of the Foundation Pain Index (FPI) biomarker test, physicians in the intervention group were 41.5% more likely to make the diagnosis of a micronutrient deficiency, 29.4% more likely to identify a treatable metabolic abnormality and 26.1% more likely to identify an oxidative stressor. These diagnostic and treatment improvements were seen across all three case types, ranging from a relative +54% (p = 0.004) for chronic neuropathic pain to +35% (p = 0.007) in chronic pain from other causes to +38% (p = 0.002) in chronic pain with associated mental health issues. Intervention doctors were also 75.1% more likely to provide a non-opioid treatment to patients on chronic opioids (O.R. 1.8, 95% C.I. 0.8–3.7), 62% less likely to order unnecessary imaging for their patients with low back pain (O.R. 0.38, 95% C.I. 0.15–0.97) and 66% less likely to order an unnecessary pain referral (O.R. 0.34, 95% C.I. 0.13–0.90). This experimental study showed significant clinical utility of a validated pain biomarker panel that determines nutritional deficiencies, metabolic abnormalities and oxidative stressors that drive underlying treatable causes of pain. When integrated into routine primary care practice, this testing approach could considerably improve diagnostic accuracy and provide more targeted, non-opioid treatments for patients suffering from chronic pain.

Finding the clinical utility of 1,5-anhydroglucitol among primary care practitioners

Background

HbA1c is widely used as the standard measure to track glycemic control in patients with diabetes and pre-diabetes but measures average levels of glycated hemoglobin over two to three months, with limited utility in the presence of recent and/or short-term fluctuations in glycemic control, which are correlated with worse patient outcomes.

Methods

We examined the clinical utility of 1-5-anhydroglucitol (1,5-AG) in six different, but common, case types of diabetes patients with short-term glycemic variability. We conducted a randomized controlled trial of simulated patients to examine the clinical practice patterns of primary care physicians before and after introducing 1,5-AG. The 145 participants were randomly assigned into standard care or standard care + 1,5-AG arms. Provider care was reviewed against explicit evidence-based care standards.

Results

At baseline, we saw no difference between the two study arms in clinical quality of care provided (p = 0.997). After introduction of 1,5-AG, standard care + 1,5-AG providers performed 3.2% better than controls (p = 0.025. In diagnosis and treatment, there was a slight, but nonsignificant trend toward better care (+1.1%, p = 0.507) for intervention providers. Upon disaggregation by case, almost all the improvement occurred in the medication-induced hyperglycemia patients (+8.1%, p = 0.047).

Conclusions

A nationally representative sample of primary care physicians demonstrated that of six different cases used in this study, 1,5-AG was found to be most effective increasing awareness of poor glucose control in medication-induced hyperglycemia. If 1,5-AG is used in this particular circumstance, the overall savings to the healthcare system is estimated to be $28 million.

Pregnancy and multiple sclerosis: Risk of unplanned pregnancy and drug exposure in utero

Background

Multiple sclerosis is a central nervous system demyelinating disease that affects women of reproductive potential. It is important to identify the frequency and risk factors of unplanned or disease-modifying therapy-exposed pregnancies to create interventions to reduce these.

Methods

This retrospective, single-center, observational chart review study aims to identify risk factors for unplanned pregnancy to identify a target population for family counseling.

Results

In total, 63 live births in 45 patients (20 unplanned and 43 planned) were analyzed. The percentage of unplanned pregnancy was 32%. The proportion of those receiving family planning counseling was lower in the patients with unplanned pregnancies (p < 0.001). The main risk factors for unplanned pregnancy were younger age (p = 0.004), disease-modifying therapy exposure (p < 0.001), and being unmarried (p < 0.001). Overall, 16 pregnancies had disease-modifying therapy exposure and in a subsequent study the risk for disease-modifying therapy exposure was unplanned status (p < 0.001). Birth outcomes were not different between groups. There were more enhancing lesions in the post-partum magnetic resonance imaging of women with planned pregnancy (p < 0.04).

Conclusion

Prevention of unplanned pregnancy could lead to less disease-modifying therapy exposed pregnancies. This study suggests a targeted intervention of family planning counseling in younger, unmarried multiple sclerosis patients could potentially lead to less unintended in utero disease-modifying therapy exposure.

[Regional anesthesia in daily clinical practice: an economic analysis based on case vignettes]

BACKGROUND

The economic effect is a crucial aspect of every medical procedure. This article analyzes the economic implications of various methods in anesthesia based on three case vignettes.

METHODS

The management of anesthesia of a forearm fracture with sufficient brachial plexus blockade, general anesthesia and insufficient brachial plexus blockade with subsequent general anesthesia was analyzed with respect to the relevant cost factors (personnel costs, durables, consumables, fixed assets costs, anesthesia-related overhead costs).

RESULTS

Sufficient regional anesthesia was the least expensive method for a forearm fracture with 324.26 €, followed by general anesthesia with 399.18 € (+23% compared with regional anesthesia). Insufficient regional anesthesia was most the expensive method, which necessitated an additional general anesthesia (482.55 €, +49% compared with sufficient regional anesthesia).

CONCLUSION

Even considering that this cost analysis was calculated based on data from only one medical institution (General Hospital of Vienna, Medical University of Vienna), regional anesthesia appeared to be cost efficient compared with other anesthesia procedures. Main cost drivers in this example were personnel costs.

Variability in triage practices for critically ill cancer patients: A randomized controlled trial

PURPOSE

Intensive care triage practices and end-user interpretation of triage guidelines have rarely been assessed. We evaluated agreement between providers on the prioritization of patients for ICU admission using different triage guidelines.

MATERIALS AND METHODS

A multi-centered randomized study on providers from 18 different countries was conducted using clinical vignettes of oncological patients. The level of agreement between providers was measured using two different guidelines, with one being cancer specific.

RESULTS

Amongst 257 providers, 52.5% randomly received the Society of Critical Care Prioritization Model, and 47.5% received a cancer specific flowchart as a guide. In the Prioritization Model arm the average entropy was 1.193, versus 1.153 in the flowchart arm (P = .095) indicating similarly poor agreement. The Fleiss' kappa coefficients were estimated to be 0.2136 for the SCCMPM arm and 0.2457 for the flowchart arm, also similarly implying poor agreement.

CONCLUSIONS

The low agreement amongst practitioners on the prioritization of cancer patient cases for ICU admission existed using both general triage guidelines and guidelines tailored only to cancer patients. The lack of consensus on intensive care unit triage practices in the oncological population exposes a potential barrier to appropriate resource allocation that needs to be addressed.

Exploring factors that might influence primary-care provider discussion of and recommendation for prostate and colon cancer screening

Background

Primary-care providers may contribute to the use of low-value cancer screening.

Objective

We sought to examine circumstances under which primary-care providers would discuss and recommend two types of cancer screening services across a spectrum of net benefit and other factors known to influence screening.

Patients and methods

This was a cross sectional survey of 126 primary-care providers in 24 primary-care clinics in the US. Participants completed surveys with two hypothetical screening scenarios for prostate or colorectal cancer (CRC). Patients in the scenarios varied by age and screening-request status. For each scenario, providers indicated whether they would discuss and recommend screening. Providers also reported on their screening attitudes and the influence of other factors known to affect screening (short patient visits, worry about lawsuits, clinical reminders/performance measures, and screening guidelines). We examined associations between providers’ attitudes and their screening recommendations for hypothetical 90-year-olds (the lowest-value screening).

Results

Providers reported they would discuss cancer screening more often than they would recommend it (P<0.001). More providers would discuss and recommend screening for CRC than prostate cancer (P<0.001), for younger than older patients (P<0.001), and when the patient requested it than when not (P<0.001). For a 90-year-old patient, every point increase in cancer-specific screening attitude increased the likelihood of a screening recommendation (30% for prostate cancer and 30% for CRC).

Discussion

While most providers’ reported practice patterns aligned with net benefit, some providers would discuss and recommend low-value cancer screening, particularly when faced with a patient request.

Conclusion

More work appears to be needed to help providers to discuss and recommend screening that aligns with value.

Does health insurance coverage or improved quality protect better against out-of-pocket payments? Experimental evidence from the Philippines

This paper explores whether health insurance coverage or improved quality at the hospital level protect better against out-of-pocket payments. Using data from a randomized policy experiment in the Philippines, we found that interventions to expand insurance coverage and improve provider quality both had an impact on out-of-pocket payments. The sample consists of 3121 child-patient patient observations across 30 hospitals either at baseline in 2003/04 or at the follow-up in 2007/08. Compared to controls, interventions that expanded insurance and provided performance-based provider payments to improve quality both resulted in a decline in out-of-pocket spending (21% decline, p-value = 0.061; and 24% decline, p-value = 0.017, respectively). With lower out-of-pocket payments for hospital care, monthly household spending on personal hygiene rose by 0.9 (p-value = 0.026) and 0.6 US$ (p-value = 0.098) under the expanded insurance and provider payment interventions, respectively, amounting to roughly a 40-60% increase relative to the controls. With the current surge for health insurance expansion in developing countries, our study suggests paying increased and possibly, equal attention to supply-side interventions will have similar impacts with operational simplicity and greater provider accountability.

Increased Rates of Documented Alcohol Counseling in Primary Care: More Counseling or Just More Documentation?

BACKGROUND

Clinical performance measures often require documentation of patient counseling by healthcare providers. Little is known about whether such measures encourage delivery of counseling or merely its documentation.

OBJECTIVE

To assess changes in provider documentation of alcohol counseling and patient report of receiving alcohol counseling in the Veterans Administration (VA) from 2009 to 2012.

DESIGN

Retrospective time-series analysis.

PARTICIPANTS

A total of 5413 men who screened positive for unhealthy alcohol use at an outpatient visit and responded to a confidential mailed survey regarding alcohol counseling from a VA provider in the prior year.

MAIN MEASURES

Rates of provider documentation of alcohol counseling in the electronic health record and patient report of such counseling on the survey were assessed over 4 fiscal years. Annual rates were calculated overall and with patients categorized into four mutually exclusive groups based on their own reports of alcohol counseling (yes/no) and whether alcohol counseling was documented by a provider (yes/no).

KEY RESULTS

Provider documentation of alcohol counseling increased 23.6% (95% CI: 17.0, 30.2), from 59.4% to 83.0%, while patient report of alcohol counseling showed no significant change (4.0%, 95% CI: -2.3, 10.3), increasing from 66.1% to 70.1%. An 18.7% (95% CI: 11.7, 25.7) increase in the proportion of patients who reported counseling that was documented by a provider largely reflected a 14.7% decline (95% CI: 8.5, 20.8) in the proportion of patients who reported alcohol counseling that was not documented by a provider. The proportion of patients who did not report counseling but whose providers documented it did not show a significant change (4.9%, 95%CI: 0.0, 9.9).

CONCLUSIONS

If patient report is accurate, increased rates of documented alcohol counseling in the VA from 2009 to 2012 predominantly reflected improved documentation of previously undocumented counseling rather than delivery of additional counseling or increased documentation of counseling that did not meaningfully occur.

Misdiagnosis of obstetrical cases and the clinical and cost consequences to patients: a cross-sectional study of urban providers in the Philippines

BACKGROUND

Misdiagnosis may be a significant and under-recognized quality of care problem. In birthing facilities located in anurban Philippine setting, we investigated the diagnostic accuracy for three obstetric conditions: cephalopelvic disproportion (CPD), post-partum hemorrhage (PPH), and pre-eclampsia.

DESIGN

Identical simulated cases were used to measure diagnostic accuracy for every provider (n=103). We linked misdiagnosis - identified by the simulated cases - to obstetrical complications of the patients at the participating facilities. Patient-level data on health outcomes and costs were obtained from medical records and follow-home in-person interviews.

RESULTS

The prevalence of misdiagnosis among obstetric providers was 29.8% overall, 25% for CPD, 33% for PPH, and 31% for pre-eclampsia. Linking provider decision-making to patients, we found those who misdiagnosed the simulated cases were more likely to have patients with a complication (OR 2.96; 95% CI 1.39-3.77) compared with those who did not misdiagnose. Complicated patients were significantly less likely to be referred to a hospital immediately, were more likely to be readmitted to a hospital after delivery, had significantly higher medical costs, and lost more income than non-complicated patients.

CONCLUSION

Diagnosis is arguably the most important task a clinician performs because it determines the subsequent course of evaluation and treatment, with the direct and indirect costs of diagnostic error, placing large financial burdens on the patient.

Impact of financial incentives on alcohol intervention delivery in primary care: a mixed-methods study

BACKGROUND

Local and national financial incentives were introduced in England between 2008 and 2015 to encourage screening and brief alcohol intervention delivery in primary care. We used routine Read Code data and interviews with General Practitioners (GPs) to assess their impact.

METHODS

A sequential explanatory mixed-methods study was conducted in 16 general practices representing 106,700 patients and 99 GPs across two areas in Northern England. Data were extracted on screening and brief alcohol intervention delivery for 2010-11 and rates were calculated by practice incentive status. Semi-structured interviews with 14 GPs explored which factors influence intervention delivery and recording in routine consultations.

RESULTS

Screening and brief alcohol intervention rates were higher in financially incentivised compared to non-incentivised practices. However absolute rates were low across all practices. Rates of short screening test administration ranged from 0.05% (95% CI: 0.03-0.08) in non-incentivised practices to 3.92% (95% CI: 3.70-4.14) in nationally incentivised practices. For the full AUDIT, rates were also highest in nationally incentivised practices (3.68%, 95% CI: 3.47-3.90) and lowest in non-incentivised practices (0.17%, 95% CI: 0.13-0.22). Delivery of alcohol interventions was highest in practices signed up to the national incentive scheme (9.23%, 95% CI: 8.91-9.57) and lowest in non-incentivised practices (4.73%, 95% CI: 4.50-4.96). GP Interviews highlighted a range of influences on alcohol intervention delivery and subsequent recording including: the hierarchy of different financial incentive schemes; mixed belief in the efficacy of alcohol interventions; the difficulty of codifying complex conditions; and GPs' beliefs about patient-centred practice.

CONCLUSIONS

Financial incentives have had some success in encouraging screening and brief alcohol interventions in England, but levels of recorded activity remain low. To improve performance, future policies must prioritise alcohol prevention work within the quality and outcomes framework, and address the values, attitudes and beliefs that shape how GPs' provide care.

Factors Influencing the Use of Second-Generation Antipsychotics in Children with Psychosis

BACKGROUND

In 2011, the U.S. Department of Health and Human Services sent a letter to state Medicaid directors explaining the need for oversight of psychiatric prescriptions for children with mental health disorders. The National Committee for Quality Alliance proposed 3 quality measures for rating managed care organizations (MCOs) that involve use of second-generation or atypical antipsychotics in children. In order to ensure appropriate use and to effectively manage the use of second-generation antipsychotics in children, MCOs need to better understand the factors that influence medication treatment decisions for children.

OBJECTIVES

To (a) determine how patient-level and physician-level factors influence decisions to prescribe second-generation antipsychotics to children (aged under 18 years) diagnosed with psychosis and (b) evaluate how the influence of these factors may differ between primary care providers and psychiatrists.

METHODS

This study employed a cross-sectional survey of 193 primary care providers and psychiatrists. A web-based patient simulation survey using a fractional factorial design was administered via a commercial vendor. Respondents were presented with simulated patient profiles described by various levels of factors considered to be essential to decision making. Respondents were asked to make treatment recommendations for each profile evaluated. In addition to treatment recommendations, demographics and beliefs about products were measured. Modified Poisson regression accounting for multilevel data was used to identify the factors that significantly affect treatment recommendations.

RESULTS

Psychiatrists were more likely to recommend second-generation antipsychotics than primary care practitioners (unadjusted RR = 1.36, 95% CI = 1.23-1.51). Social factors such as foster status or parental concern were not found to be significant predictors of prescribing second-generation antipsychotics. The percentage of a provider's patients using second-generation antipsychotics (RR = 1.002, 95% CI = 1.0002-1.003), patient age (aged 4 years: RR = 0.75, 95% CI = 0.68-0.84; aged 10 years: RR = 0.94, 95% CI = 0.91-0.99; reference group: aged 15 years), and patient disease severity (severe: RR = 1.11, 95% CI = 1.04-1.18; moderate: RR = 1.10, 95% CI = 1.05-1.17; reference group: mild) significantly predicted prescription behavior among primary care providers and psychiatrists. Primary care providers were about twice as likely to recommend antipsychotics if they believed the use of antipsychotics was a labeled indication (RR = 2.16, 95% CI = 1.56-2.98) or a medically accepted use (RR = 1.88, 95% CI = 1.33-2.67), when compared with physicians who believed there was no evidence available. This effect was not significant among psychiatrists. Primary care providers, but not psychiatrists, were also significantly influenced by patient white blood cell (WBC) count. Patients with healthy WBC counts were 1.11 times as likely (95% CI = 1.05-1.17) to receive antipsychotics from primary care providers compared with those with low WBC count. Patient body mass index (BMI) was not found to significantly influence prescribing behavior. Nearly 50% of patients did not receive recommendations for psychosocial care. Primary care providers recommended antipsychotic polypharmacy in 23% of the patient profiles, while psychiatrists did so in 42% of the profiles.

CONCLUSIONS

This study provides valuable insight into physician-prescribing practices for antipsychotics. The lack of significance of foster status and parental concern, after controlling for other factors, shows that physicians base their decisions on clinical factors more than social factors. Results for patient BMI and frequency of recommendations of polypharmacy are concerning. The general lack of awareness of evidence supporting use of antipsychotics is also highly concerning. The effects of patient BMI, beliefs about evidence supporting use, and prescribing practices with regard to psychosocial care and antipsychotic polypharmacy provide actionable results for managed care programs looking to improve their quality metrics. The results of this study further demonstrate the need for the immediate implementation of the various proposed quality metrics in this area and for new practice guidelines to raise the current standard of care.

DISCLOSURES

No outside funding supported this research. Bentley reports the receipt of grants from PQA and the NACDS Foundation. Patel is employed by Medical Marketing Economics. The authors report no other conflicting interests, potential or otherwise. Study concept and design were contributed by Ramachandran, Banahan, West-Strum, and Bentley. Ramachandran, Banahan, and Patel collected data; data interpretation was performed primarily by Ramachandran, Banahan, and Patel, along with Bentley and West-Strum. The manuscript was primarily written by Ramachandran, along with Banahan and Bentley, and revised by Banahan, Bentley, West-Strum, and Patel.

Using Clinical Vignettes to Assess Quality of Care for Acute Respiratory Infections

Overprescribing of antibiotics for acute respiratory infections (ARIs) is common. Our objective was to develop and validate a vignette-based method to estimate clinician ARI antibiotic prescribing. We surveyed physicians (n = 78) and retail clinic clinicians (n = 109) between January and September 2013. We surveyed clinicians using a set of ARI vignettes and linked the responses to electronic health record data for all ARI visits managed by these clinicians during 2012. We then created a new measure of antibiotic prescribing, the comprehensive ARI management rate. This was defined as not prescribing antibiotics for antibiotic-inappropriate diagnoses and prescribing guideline-concordant antibiotics for antibiotic-appropriate diagnoses (and also included appropriate use of streptococcal testing for the pharyngitis vignettes). We compared the vignette-based and chart-based comprehensive ARI management at the clinician level. We then identified the combination of vignettes that best predicted comprehensive ARI management rates, using a partitioning algorithm. Responses to 3 vignettes partitioned clinicians into 4 groups with chart-based comprehensive ARI management rates of 61% (n = 121), 50% (n = 47), 31% (n = 12), and 22% (n = 7). Responses to 3 clinical vignettes can identify clinicians with relatively poor quality ARI antibiotic prescribing. Vignettes may be a mechanism to target clinicians for quality improvement efforts.

Do Health Reforms to Improve Quality Have Long-Term Effects? Results of a Follow-Up on a Randomized Policy Experiment in the Philippines

We tracked doctors who had previously participated in a randomized policy experiment in the Philippines. The original experiment involved 30 district hospitals divided equally into one control site and two intervention sites that increased insurance payments (full insurance support for children under 5 years old) or made bonus payments to hospital staff. During the 3 years of the intervention, quality-as measured by clinical performance and value vignettes-improved and was sustained in both intervention sites compared with controls. Five years after the interventions were discontinued, we remeasured the quality of care of the doctors. We found that the intervention sites continued to have significantly higher quality compared with the control sites. The previously documented quality improvement in intervention sites appears to be sustained; moreover, it was subject to a very low (less than 1% per year) rate of decay in quality scores.

Methods of Observing Variations in Physicians' Decisions: The Opportunities of Clinical Vignettes

To support their efforts to promote high quality and efficient care, policymakers need to better understand the key factors associated with variations in physicians' decisions, and in particular, physician deviations from evidence-based care. Clinical vignette survey instruments hold potential for research in this area as an approach that both allows for practical, large-scale study and overcomes the data quality challenges posed by analysis of clinical data. These surveys present respondents with a narrative description of a hypothetical patient case and solicit responses to one or more questions regarding the care of the patient. In this review, we describe various methods for measuring variations in physicians' decisions and highlight a range of design features researchers should consider when developing a clinical vignette survey. We conclude by identifying areas for future research.

Racial, gender, and socioeconomic status bias in senior medical student clinical decision-making: a national survey

BACKGROUND

Research suggests stereotyping by clinicians as one contributor to racial and gender-based health disparities. It is necessary to understand the origins of such biases before interventions can be developed to eliminate them. As a first step toward this understanding, we tested for the presence of bias in senior medical students.

OBJECTIVE

The purpose of the study was to determine whether bias based on race, gender, or socioeconomic status influenced clinical decision-making among medical students.

DESIGN

We surveyed seniors at 84 medical schools, who were required to choose between two clinically equivalent management options for a set of cardiac patient vignettes. We examined variations in student recommendations based on patient race, gender, and socioeconomic status.

PARTICIPANTS

The study included senior medical students.

MAIN MEASURES

We investigated the percentage of students selecting cardiac procedural options for vignette patients, analyzed by patient race, gender, and socioeconomic status.

KEY RESULTS

Among 4,603 returned surveys, we found no evidence in the overall sample supporting racial or gender bias in student clinical decision-making. Students were slightly more likely to recommend cardiac procedural options for black (43.9 %) vs. white (42 %, p = .03) patients; there was no difference by patient gender. Patient socioeconomic status was the strongest predictor of student recommendations, with patients described as having the highest socioeconomic status most likely to receive procedural care recommendations (50.3 % vs. 43.2 % for those in the lowest socioeconomic status group, p < .001). Analysis by subgroup, however, showed significant regional geographic variation in the influence of patient race and gender on decision-making. Multilevel analysis showed that white female patients were least likely to receive procedural recommendations.

CONCLUSIONS

In the sample as a whole, we found no evidence of racial or gender bias in student clinical decision-making. However, we did find evidence of bias with regard to the influence of patient socioeconomic status, geographic variations, and the influence of interactions between patient race and gender on student recommendations.

Directly observed care: can unannounced standardized patients address a gap in performance measurement?

There are three potential sources of information for evaluating a clinician's performance: documentation, patient report, and directly observed care. Current measures draw on just two of these: data recorded in the medical record and surveys of patients. Neither captures an array of performance characteristics, including clinician attention to symptoms and signs while taking a history or conducting a physical exam, accurate recording in the medical record of information obtained during the encounter, evidence based communication strategies for preventive care counseling, and effective communication behavior. Unannounced Standardized Patients (USPs) have been widely deployed as a research strategy for systematically uncovering significant performance deficits in each of these areas, but have not been adopted for quality improvement. Likely obstacles include concerns about the ethics of sending health professionals sham patients, the technical challenges of the subterfuge, and concerns about the relatively small sample sizes and substantial costs involved. However, the high frequency of significant and remediable performance deficits unmasked by USPs, and the potential to adapt registration and record keeping systems to accommodate their visits, suggest that their selective and purposeful deployment could be a cost effective and powerful strategy for addressing a gap in performance measurement.

Necessity of introducing postencounter note describing history and physical examination at clinical performance examination in Korea

PURPOSE

Information gathering ability had been evaluated mainly via checklists in clinical performance examinations (CPX). But, it is not proved yet if students write the information correctly in postencounter note (PN), although they asked questions or performed physical examinations (PE) about the information when they interacted with standardized patients in CPX. This study addressed the necessity of introducing PN to evaluate the ability in CPX.

METHODS

After patient encounters, students were instructed to write the findings of history taking and physical examination that they considered as important information in approaching the patient's problems in PN. PNs were scored using answer keys selected from checklist items, which were considered to be recorded in PN by CPX experts.

RESULTS

PNs of six CPX cases from 54 students were analyzed. Correlation coefficients between the key-checklist scores and PN scores of six cases were moderate to high (0.52 to 0.79). However, students frequently neglected some cardinal features of chief complains, pertinent findings of past/social history and PE, and pertinent negative findings of associated symptoms in PNs, which were checked as 'done' in the keys of checklists.

CONCLUSION

It is necessary to introduce PN in CPX to evaluate the students' ability of synthesis and integration of patient information.

Patient and provider perspectives on quality and health system effectiveness in a transition economy: evidence from Ukraine

Facing a severe population health crisis due to noncommunicable diseases, Ukraine and other former Soviet republics and Eastern European countries have a pressing need for more effective health systems. Policies to enhance health system effectiveness should consider the perspectives of different stakeholder groups, including providers as well as patients. In addition, policies that directly target the quality of clinical care should be based on objective performance measures. In 2009 and 2010 we conducted a coordinated series of household and facility-level surveys to capture the perspectives of Ukrainian household members, outpatient clinic patients, and physicians regarding the country's health system overall, as well as the quality, access, and affordability of health care. We objectively measured the quality of care for heart failure and chronic obstructive pulmonary disease using CPV(®) vignettes. There was broad agreement among household respondents (79%) and physicians (95%) that Ukraine's health system should be reformed. CPV(®) results indicate that the quality of care for common noncommunicable diseases is poor in all regions of the country and in hospitals as well as polyclinics. However, perspectives about the quality of care differ, with household respondents seeing quality as a serious concern, clinic patients having more positive perceptions, and physicians not viewing quality as a reform priority. All stakeholder groups viewed affordability as a problem. These findings have several implications for policies to enhance health system effectiveness. The shared desire for health system reform among all stakeholder groups provides a basis for action in Ukraine. Improving quality, strengthening primary care, and enhancing affordability should be major goals of new health policies. Policies to improve quality directly, such as pay-for-performance, would be mutually reinforcing with purchasing reforms such as transparent payment mechanisms. Such policies would align the incentives of physicians with the desires of the population they serve.

Adherence of French GPs to chronic neuropathic pain clinical guidelines: results of a cross-sectional, randomized, "e" case-vignette survey

Background and aims

The French Pain Society published guidelines for neuropathic pain management in 2010. Our aim was to evaluate the compliance of GPs with these guidelines three years later.

Methods

We used “e” case vignette methodology for this non interventional study. A national panel of randomly selected GPs was included. We used eight “e” case-vignettes relating to chronic pain, differing in terms of the type of pain (neuropathic/non neuropathic), etiology (cancer, postoperative pain, low back pain with or without radicular pain, diabetes) and symptoms. GPs received two randomly selected consecutive “e” case vignettes (with/without neuropathic pain). We analyzed their ability to recognize neuropathic pain and to prescribe appropriate first-line treatment.

Results

From the 1265 GPs in the database, we recruited 443 (35.0%), 334 of whom logged onto the web site (26.4%) and 319 (25.2%) of whom completed the survey. Among these GPs, 170 (53.3%) were aware of the guidelines, 136 (42.6%) were able to follow them, and 110 (34.5%) used the DN4 diagnostic tool. Sensitivity for neuropathic pain recognition was 87.8% (CI: 84.2%; 91.4%). However, postoperative neuropathic pain was less well diagnosed (77.9%; CI: 69.6%; 86.2%) than diabetic pain (95.2%; CI: 90.0%; 100.0%), cancer pain (90.6%; CI: 83.5%; 97.8%) and typical radicular pain (90.7%; CI: 84.9%; 96.5%). When neuropathic pain was correctly recognized, the likelihood of appropriate first-line treatment prescription was 90.6% (CI: 87.4%; 93.8%). The treatments proposed were pregabaline (71.8%), gabapentine (43.9%), amiptriptylline (23.2%) and duloxetine (18.2%). However, ibuprofen (11%), acetaminophen-codeine (29.5%) and clonazepam (10%) were still prescribed.

Conclusions

The compliance of GPs with clinical practice guidelines appeared to be satisfactory, but differed between etiologies.

What are tests for? The implications of stuttering steps along the US patient pathway

This article explores the implications of how US family physicians make decisions about ordering diagnostic tests for their patients. Data is based on a study of 256 physicians interviewed after viewing a video vignette of a presenting patient. The qualitative analysis of 778 statements relating to trustworthiness of evidence for their decision making, the use of any kind of technology and diagnostic testing suggests a range of internal and external constraints on physician decision making. Test-ordering for family physicians in the United States is significantly influenced by both hidden cognitive processes related to the physician's calculation of patient resources and a health insurance system that requires certain types of evidence in order to permit further tests or particular interventions. The consequence of the need for physicians to meet multiple forms of proof that may not always relate to relevant treatment delays a diagnosis and treatment plan agreed not only by the physician and patient but also the insurance company. This results in a patient journey that is made up of stuttering steps to a confirmed diagnosis and treatment undermining patient-centred practice, compromising patient care, constraining physician autonomy and creating additional expense.

The impact of performance incentives on child health outcomes: results from a cluster randomized controlled trial in the Philippines

Improving clinical performance using measurement and payment incentives, including pay for performance (or P4P), has, so far, shown modest to no benefit on patient outcomes. Our objective was to assess the impact of a P4P programme on paediatric health outcomes in the Philippines. We used data from the Quality Improvement Demonstration Study. In this study, the P4P intervention, introduced in 2004, was randomly assigned to 10 community district hospitals, which were matched to 10 control sites. At all sites, physician quality was measured using Clinical Performance Vignettes (CPVs) among randomly selected physicians every 6 months over a 36-month period. In the hospitals randomized to the P4P intervention, physicians received bonus payments if they met qualifying scores on the CPV. We measured health outcomes 4-10 weeks after hospital discharge among children 5 years of age and under who had been hospitalized for diarrhoea and pneumonia (the two most common illnesses affecting this age cohort) and had been under the care of physicians participating in the study. Health outcomes data collection was done at baseline/pre-intervention and 2 years post-intervention on the following post-discharge outcomes: (1) age-adjusted wasting, (2) C-reactive protein in blood, (3) haemoglobin level and (4) parental assessment of child's health using general self-reported health (GSRH) measure. To evaluate changes in health outcomes in the control vs intervention sites over time (baseline vs post-intervention), we used a difference-in-difference logistic regression analysis, controlling for potential confounders. We found an improvement of 7 and 9 percentage points in GSRH and wasting over time (post-intervention vs baseline) in the intervention sites relative to the control sites (P ≤ 0.001). The results from this randomized social experiment indicate that the introduction of a performance-based incentive programme, which included measurement and feedback, led to improvements in two important child health outcomes.

Physician decision making for colorectal cancer screening in the elderly

BACKGROUND

Although individualized decision making is recommended to appropriately screen for colorectal cancer (CRC) in older adults, it is unclear whether physicians solicit input from older patients before making a recommendation for or against CRC screening.

OBJECTIVE

The purpose of this study was to examine whether physicians elect to engage older patients in individualized decision making for CRC screening.

DESIGN AND PARTICIPANTS

We surveyed a random sample of 650 US primary care physicians by mail. Physicians responded to questions about three clinical vignettes involving 80-year-old female patients in good, fair, and poor health. We examined whether physicians reported that they would initiate a discussion with the patients about CRC screening and whether they would make a recommendation about screening or seek patient input first.

RESULTS

A total of 276 eligible physicians responded (52 % corrected response rate). Whether physicians indicated they would initiate a discussion about CRC screening varied by vignette: 91 % of physicians indicated they would do so for the patient in good health and 66 % and 44 % for the patients in fair and poor health, respectively (p<0.0001). The proportion of physicians that would seek patient input for their screening recommendation also varied by vignette (45 % for good, 49 % for fair, and 26 % for poor).

CONCLUSION

We found that physicians often individualize their CRC screening recommendations for older women by electing to engage patients in discussions and seeking their input before making a CRC recommendation. Physicians were more likely to elect to engage the patients represented by the good and fair health vignette, where the potential benefits likely outweigh the potential harms, than the patient in poor health, where the potential harms likely outweigh the potential benefits.

Stimulating preventive procedures in primary care. Effect of PIUPOZ program on the delivery of preventive procedures

INTRODUCTION

Educational meetings are one of the most frequently used strategies to change doctors' professional behavior; however, their effectiveness as a single intervention is limited. This study evaluated the effect of a multifactorial intervention, based on interactive workshops, on the GPs' knowledge and the delivery rates of preventive procedures in primary care.

MATERIAL AND METHODS

The study population comprised 106 GPs working in the Wielkopolska region recruited to the PIUPOZ program (Improving Quality in Primary Care). The intervention in the program consisted of lectures, interactive workshops and an audit, before and three months after the training. Trained medical students directly observed GPs to register which of 12 studied preventive procedures were performed during the consultation in patients aged 40+.

RESULTS

A total of 1060 consultations were recorded, during which 4899 preventive procedures were delivered: 2115 before and 2784 after workshops. The mean number of preventive procedures per patient before and after workshops was 3.84 and 5.25 respectively (p < 0.0001). The most commonly performed preventive procedures were blood pressure, blood glucose and lipid profile measurement. Mean number of correct answers for 16 questions in the initial knowledge test was 8.7 and 12.7 in the final test (p < 0.0001).

CONCLUSIONS

The observed number of delivered preventive procedures was below the recommended range. Preventive procedures based on laboratory tests were performed more often than lifestyle counseling.

Medical evidence of human rights violations against non-Arabic-speaking civilians in Darfur: a cross-sectional study

BACKGROUND

Ongoing conflict in the Darfur region of Sudan has resulted in a severe humanitarian crisis. We sought to characterize the nature and geographic scope of allegations of human rights violations perpetrated against civilians in Darfur and to evaluate their consistency with medical examinations documented in patients' medical records.

METHODS AND FINDINGS

This was a retrospective review and analysis of medical records from all 325 patients seen for treatment from September 28, 2004, through December 31, 2006, at the Nyala-based Amel Centre for Treatment and Rehabilitation of Victims of Torture, the only dedicated local provider of free clinical and legal services to civilian victims of torture and other human rights violations in Darfur during this time period. Among 325 medical records identified and examined, 292 (89.8%) patients from 12 different non-Arabic-speaking tribes disclosed in the medical notes that they had been attacked by Government of Sudan (GoS) and/or Janjaweed forces. Attacks were reported in 23 different rural council areas throughout Darfur. Nearly all attacks (321 [98.8%]) were described as having occurred in the absence of active armed conflict between Janjaweed/GoS forces and rebel groups. The most common alleged abuses were beatings (161 [49.5%]), gunshot wounds (140 [43.1%]), destruction or theft of property (121 [37.2%]), involuntary detainment (97 [29.9%]), and being bound (64 [19.7%]). Approximately one-half (36 [49.3%]) of all women disclosed that they had been sexually assaulted, and one-half of sexual assaults were described as having occurred in close proximity to a camp for internally displaced persons. Among the 198 (60.9%) medical records that contained sufficient detail to enable the forensic medical reviewers to render an informed judgment, the signs and symptoms in all of the medical records were assessed to be consistent with, highly consistent with, or virtually diagnostic of the alleged abuses.

CONCLUSIONS

Allegations of widespread and sustained torture and other human rights violations by GoS and/or Janjaweed forces against non-Arabic-speaking civilians were corroborated by medical forensic review of medical records of patients seen at a local non-governmental provider of free clinical and legal services in Darfur. Limitations of this study were that patients seen in this clinic may not have been a representative sample of persons alleging abuse by Janjaweed/GoS forces, and that most delayed presenting for care. The quality of documentation was similar to that available in other conflict/post-conflict, resource-limited settings.

The role of unconscious bias in surgical safety and outcomes

Racial, ethnic, and gender disparities in health outcomes are a major challenge for the US health care system. Although the causes of these disparities are multifactorial, unconscious bias on the part of health care providers plays a role. Unconscious bias occurs when subconscious prejudicial beliefs about stereotypical individual attributes result in an automatic and unconscious reaction and/or behavior based on those beliefs. This article reviews the evidence in support of unconscious bias and resultant disparate health outcomes. Although unconscious bias cannot be entirely eliminated, acknowledging it, encouraging empathy, and understanding patients' sociocultural context promotes just, equitable, and compassionate care to all patients.

Do doctors contribute to the social patterning of disease? The case of race/ethnic disparities in diabetes mellitus

Data from the Boston Area Community Health Survey show that both undiagnosed signs and symptoms and diagnosed type 2 diabetes mellitus (T2DM) are patterned by socioeconomic status (SES). Such patterning is corroborated by National Health and Nutrition Examination Survey data for diagnosed T2DM. Complementary data from an experiment concerning clinical decision making show T2DM is patterned by race/ethnicity, following diagnosis by a physician. Undiagnosed signs and symptoms of T2DM in the community are patterned by SES (rather than race/ethnicity), but following diagnosis by primary care physicians they are patterned more by race/ethnicity (rather than by SES). Race/ethnicity and SES in the United States are almost totally confounded, such that measuring one is essentially also measuring the other. Physician patterning of T2DM by race/ethnicity, however, motivates the search for genetic and biophysiologic explanations and distracts attention from the more important contribution of SES circumstances to the prevalence of diabetes mellitus.

The relative contribution of patient, provider and organizational influences to the appropriate diagnosis and management of diabetes mellitus

OBJECTIVE

To estimate the relative contribution of patient attributes, provider characteristics and organizational features of the doctors' workplace to the diagnosis and management of diabetes.

RESEARCH DESIGN AND METHODS

In a factorial experimental design doctors (n = 192) viewed clinically authentic vignettes of 'patients' presenting with identical signs and symptoms. Doctor subjects were primary care doctors stratified according to gender and level of experience. During an in-person interview scheduled between real patients, doctors were asked how they would diagnosis and manage the vignette 'patients' in clinical practice.

RESULTS

This study considered the relative contribution of patient, doctor and organizational factors. Taken together patient attributes explained only 4.4% of the variability in diabetes diagnosis. Doctor factors explained only 2.0%. The vast majority of the explained variance in diabetes diagnosis was due to organizational factors (14.3%). Relative contributions combined (patient, provider, organizational factors) explained only 20% of the total variance.

CONCLUSION

Attempts to reduce health care variations usually focus on the education/activation of patients, or increased training of doctors. Our findings suggest that shifting quality improvement efforts to the area which contributes most to the creation and amplification of variations (organizational influences) may produce better results in terms of reduced variations in health care associated with diabetes.

Differences between internists and family practitioners in the diagnosis and management of the same patient with coronary heart disease

It has been suggested that internists and family practitioners have somewhat different "disease" perspectives, which may be generated by use of different explanatory models during medical training (pathophysiological vs. biopsychosocial, respectively). This article explores differences between internists and family practitioners in their suggested diagnoses, level of diagnostic certainty, test and prescription ordering, when encountering exactly the same "patient" with coronary heart disease (CHD). Internists were more certain of a CHD diagnosis than family practitioners and were more likely to act on this diagnosis. Family practitioners were more likely to diagnose (and were more certain of) a mental health condition. While many physicians simultaneously entertain several alternate diagnoses, diagnostic certainty has shown to have an important influence on subsequent clinical actions, such as stress testing and prescription of beta blockers. These results may inform future educational strategies designed to reduce diagnostic uncertainty in the face of life-threatening conditions, such as CHD.

Influences of organizational features of healthcare settings on clinical decision making: qualitative results from a cross-national factorial experiment

A proliferating literature documents cross-national variation in medical practice and seeks to explain observed differences in terms of the presence of certain kinds of healthcare systems, economic, and cultural differences between countries. Less is known about how providers themselves understand these influences and perceive them as relevant to their clinical work. Using qualitative data from a cross-national factorial experiment in the United States and United Kingdom, we analyze 244 primary care physicians' explanations of how organizational features of their respective healthcare settings influence the treatment decisions they made for a vignette patient, including affordability of care; within-system quality deficits; and constraints due to patient behavior. While many differences are attributed to financial constraints deriving from two very differently structured healthcare systems, in other ways they are reflections of cultural and historical expectations regarding medical care, or interactions between the two. Implications, including possible challenges to the implementation of universal care in the USA, are discussed.

Does Receiving Clinical Preventive Services Vary across Different Types of Primary Healthcare Organizations? Evidence from a Population-Based Survey

OBJECTIVE

To measure the association between primary healthcare (PHC) organizational types and patient coverage for clinical preventive services (CPS).

METHOD

Study conducted in Quebec (2005), including a population-based survey of patients' experience of care (N=4,417) and a survey of PHC clinics.

OUTCOME MEASURES

Patient-reported CPS delivery rates and CPS coverage scores. Multiple logistic regressions used to assess factors associated with higher probability of receiving CPS.

RESULTS

CPS delivery rates were higher among patients with a regular source of PHC. Higher CPS score was associated with having a public (OR 1.79; 95% CI 1.35-2.37) or mixed (OR 1.22; 95% CI 1.01-1.48) type of organization as source of PHC compared to a private one, and having had a high number of visits to the regular source of PHC in the past two years (≤6: OR 1.83; 95% CI 1.41-2.38) compared to a single visit.

CONCLUSION

Public and mixed PHC organizations seem to perform better. CPS delivery is strongly associated with having a regular source of care.

A comparison of standardised patients, record abstraction and clinical vignettes for the purpose of measuring clinical practice

BACKGROUND

A recent review found standardised patient (SP) methodology to be the gold standard method for evaluating clinical care. Alternative methods include record abstraction and computerised clinical vignettes. SPs were compared to clinical records and to computerised vignettes in order to assess whether record abstraction and vignettes are accurate measures of clinical care provided within optometry.

METHODS

A total of 111 community optometrists in the south east of the UK consented to be visited by unannounced actors for an eye examination. The SPs received extensive training to enable accurate reporting of the content of the eye examinations using checklists. Clinical records were requested from optometrists who chose to receive feedback following the SP visits. The SP checklists were used as a guide to extract relevant information from the clinical records. An opportunity was made available to all UK qualified optometrists, through publicity in the College of Optometrists' monthly newsletter, the Association of Optometrists' monthly newsletter (Blink) and the UK optometry e-mail discussion list to complete three computerised clinical vignettes by performing a virtual eye examination using an on-line optometric record card. The average differences were calculated between the record abstraction and vignette results compared to the standardised patient encounter findings for different domains of an eye examination. Chi-square analyses were performed on the tests which were of the greatest clinical significance for each scenario.

RESULTS

The average overall difference for information gathered from record abstraction compared to the standardised patient encounter ranged from +2 to -26% (positive values indicate items that were recorded on the clinical records but not reported by the SP). For history and symptoms, the average difference ranged from -9 to -26%; for the proportion of tests performed during the examination this value ranged from +2 to -24% and for management issues the difference ranged from -1 to -4%. The average overall difference for the vignette data compared to the standardised patient encounter ranged from 0 to +26% (positive values indicate items that were not carried out in a clinical setting, as recorded by the SP, but were described by optometrists who completed the vignette as tests they would have carried out). For history and symptoms, the average difference for the vignette data ranged from +2 to +26%; from 0 to +20% for tests performed during the eye examination and from 0 to +11% for management.

CONCLUSION

Different methods of measuring clinical care capture different elements of clinical practice and are prone to different biases. This three-way comparison indicates that clinical records tend to under-estimate actual care provided, while vignette scores tend to over-estimate clinical performance. Low participation rates mean that the participating optometrists could be described as a 'self-selected sample'. This is a limitation of the research and the conclusions should be considered with this in mind. The significance of these findings for future research and for litigation and disciplinary cases is discussed.

British and Polish general practitioners' opinions on the importance of preventive medicine

The delivery of preventive services in GPs' surgeries falls below recommended levels. Different attempts are performed to change the situation. The introduction in the UK in 2004 of a new GP contract based on a QOF fundamentally changed the way that primary practitioners are paid. Success of this intervention caused international interest in using financial incentives as a method of improving general practice. Polish primary care is still under development. In contemplating reform and the possible introduction of some of these British solutions in Poland, it would be valuable to compare what British and Polish GPs' think about prevention and see how their opinions can be affected by context in which they work. The aim of the survey involving 65 British and 65 Polish GPs was to compare what are their views of health promotion. The questionnaire solicited information on doctors' demographic characteristics, attitudes toward prevention, time they think they spend on preventive procedures and perceptions of the importance of 13 selected preventive procedures. Most British and Polish GPs answered that prevention is very important and they spend 10-30% of each consultation on it. Almost all doctors rated blood pressure measurement as important. For British GPs BMI, blood glucose, lipid profile, anti-alcohol and anti-smoking advice were important, in contrast to Polish doctors. In the presence of clear clinical guidelines, age and gender do not affect doctors' opinion on the importance of preventive procedures. Procedures in which GPs are directly involved are more important for them.

The effectiveness of a posted information package on the beliefs and behavior of musculoskeletal practitioners: the UK Chiropractors, Osteopaths, and Musculoskeletal Physiotherapists Low Back Pain ManagemENT (COMPLeMENT) randomized trial

STUDY DESIGN

Randomized controlled trial.

OBJECTIVE

To investigate the effect of a printed information package on the low back pain (LBP)-related beliefs and reported behavior of musculoskeletal practitioners (chiropractors, osteopaths, and musculoskeletal physiotherapists) across the United Kingdom.

SUMMARY OF BACKGROUND DATA

A substantial proportion of musculoskeletal practitioners in United Kingdom does not follow current LBP guideline recommendations.

METHODS

In total, 1758 practitioners were randomly allocated to either of the 2 study arms. One arm was posted a printed information package containing guideline recommendations for the management of LBP (n = 876) and the other received no intervention (n = 882). The primary outcome measure consisted of 3 "quality indicators" (activity, work, and bed-rest) relating to a vignette of a patient with LBP, in which responses were dichotomized into either "guideline-inconsistent" or "guideline-consistent." The secondary outcome was the practitioners' LBP-related beliefs, measured using the Health Care Providers Pain and Impairment Relationship Scale. Outcomes were measured at baseline and at 6 months.

RESULTS

Follow-up at 6 months was 89%. The changes in reported behavior on the quality indicators were as follows: activity, odds ratio (OR) 1.29 (95% confidence interval, 1.03-1.61) and number needed to be treated (NNT), 19 (15-28); work, OR 1.35 (1.07-1.70) and NNT 19 (14-29); and bed-rest, OR 1.31 (0.97-1.76) and NNT 47 (33-103). The composite NNT for a change from guideline-inconsistent to guideline-consistent behavior on at least 1 of the 3 quality indicators was 10 (9-14). LBP-related beliefs were significantly improved in those who were sent the information package (P = 0.002), but only to a small degree (mean difference, 0.884 scale points; 95% confidence interval, 0.319-1.448).

CONCLUSION

Printed educational material can shift LBP-related beliefs and reported behaviors of musculoskeletal practitioners, toward practice that is more in line with guideline recommendations.

Variation in cardiologists' propensity to test and treat: is it associated with regional variation in utilization?

BACKGROUND

Regional variation in healthcare utilization, including cardiac testing and procedures, is well documented. Some factors underlying such variation are understood, including resource supply. However, less is known about how physician behaviors and attitudes may influence variation in utilization across regions.

METHODS AND RESULTS

We performed a survey of a national sample of cardiologists using patients vignettes to ascertain physicians' self-reported propensity to test and treat patients with cardiovascular problems, computing a Cardiac Intensity Score for each physician based on his/her responses intended to measure the physician's propensity to recommend high-tech and/or invasive tests and treatments. In addition, we asked under what circumstances they would order a cardiac catheterization "for other than purely clinical reasons." For some survey items, there was substantial variation in physician responses. We found that the Cardiac Intensity Score was associated with 2 measures of population based healthcare utilization measured within geographic regions, with a stronger association with general healthcare spending than with delivery of cardiac services. Although nearly all physicians denied ordering a potentially unnecessary cardiac catheterization for financial reasons, some physicians acknowledged ordering the test for other reasons, including meeting patient and referring physician expectations, meeting peer expectations, and malpractice concerns. More than 27% of respondents reported ordering a cardiac catheterization if a colleague would in the same situation frequently or sometimes, and nearly 24% reported doing so out of fear of malpractice. These 2 factors were significantly associated with the propensity to test and treat, but only fear of malpractice was associated with regional utilization.

CONCLUSIONS

Variability in cardiologists' propensity to test and treat partly underlies regional variation in utilization of general health and cardiology services. The factor most closely associated with this propensity was fear of malpractice suits. This factor may be an appropriate target of intervention.

Statistical considerations in a systematic review of proxy measures of clinical behaviour

Background

Studies included in a related systematic review used a variety of statistical methods to summarise clinical behaviour and to compare proxy (or indirect) and direct (observed) methods of measuring it. The objective of the present review was to assess the validity of these statistical methods and make appropriate recommendations.

Methods

Electronic bibliographic databases were searched to identify studies meeting specified inclusion criteria. Potentially relevant studies were screened for inclusion independently by two reviewers. This was followed by systematic abstraction and categorization of statistical methods, as well as critical assessment of these methods.

Results

Fifteen reports (of 11 studies) met the inclusion criteria. Thirteen analysed individual clinical actions separately and presented a variety of summary statistics: sensitivity was available in eight reports and specificity in six, but four reports treated different actions interchangeably. Seven reports combined several actions into summary measures of behaviour: five reports compared means on direct and proxy measures using analysis of variance or t-tests; four reported the Pearson correlation; none compared direct and proxy measures over the range of their values. Four reports comparing individual items used appropriate statistical methods, but reports that compared summary scores did not.

Conclusions

We recommend sensitivity and positive predictive value as statistics to assess agreement of direct and proxy measures of individual clinical actions. Summary measures should be reliable, repeatable, capture a single underlying aspect of behaviour, and map that construct onto a valid measurement scale. The relationship between the direct and proxy measures should be evaluated over the entire range of the direct measure and describe not only the mean of the proxy measure for any specific value of the direct measure, but also the range of variability of the proxy measure. The evidence about the relationship between direct and proxy methods of assessing clinical behaviour is weak.