Reducing practice variation through clinical pathways-Is it enough?

The impact of grey zones on the accuracy of agreement measures for ordinal tables

Background

In an inter-rater agreement study, if two raters tend to rate considering different aspects of the subject of interest or have different experience levels, a grey zone occurs among the levels of a square contingency table showing the inter-rater agreement. These grey zones distort the degree of agreement between raters and negatively impact the decisions based on the inter-rater agreement tables. In this sense, it is important to know how the existence of a grey zone impacts the inter-rater agreement coefficients to choose the most reliable agreement coefficient against the grey zones to reach out with more reliable decisions.

Methods

In this article, we propose two approaches to create grey zones in simulations setting and conduct an extensive Monte Carlo simulation study to figure out the impact of having grey zones on the weighted inter-rater agreement measures for ordinal tables over a comprehensive simulation space.

Results

The weighted inter-rater agreement coefficients are not reliable against the existence of grey zones. Increasing sample size and the number of categories in the agreement table decreases the accuracy of weighted inter-rater agreement measures when there is a grey zone. When the degree of agreement between the raters is high, the agreement measures are not significantly impacted by the existence of grey zones. However, if there is a medium to low degree of inter-rater agreement, all the weighted coefficients are more or less impacted.

Conclusions

It is observed in this study that the existence of grey zones has a significant negative impact on the accuracy of agreement measures especially for a low degree of true agreement and high sample and tables sizes. In general, Gwet’s AC2 and Brennan-Prediger’s κ with quadratic or ordinal weights are reliable against the grey zones.

Supplementary Information

The online version contains supplementary material available at (10.1186/s12874-021-01248-3).

Quality improvement initiative to improve pulmonary function in pediatric cystic fibrosis patients

BACKGROUND

Our Cystic Fibrosis (CF) Center initiated a Quality Improvement (QI) project in November 2017 with the goal of improving our patients' forced expiratory volume in 1 second (FEV1) percent predicted (pp) and continued for 1 year. Our specific aim was to increase the relative mean FEV1 pp by 5% in 12 months for CF patients 6 to 21 years old with FEV1 ≤ 80 pp.

METHODS

We identified patients with FEV1 ≤ 80 pp, developed cause and effect diagrams (fishbones) to identify contributing factors to FEV1 ≤ 80 pp, and created flowcharts to address barriers. The barriers to adherence that may result in FEV1 ≤ 80 pp were studied using a fishbone. A standardized approach across providers was implemented to individualize care for each patient. Each discipline developed a flowchart to address barriers to improving FEV1.

RESULTS

Forty patients were identified (43% male). Their mean age was 16.8 years (range 8.2-21.5 years). Mean FEV1 pp at baseline was 58.6 (range 30-80). The fishbone identified needs for continuing education for patients/families, and providing a treatment plan at each clinic visit. After 6 months of implementation, patients had an improvement in mean FEV1 pp by 6.4% (CI, 0.4%-12.9%). At 12 months, mean FEV1 pp had improved by 14% (CI, 6.5%-21.4%), which exceeded our goal of 5%.

CONCLUSION

Through this ongoing project, team members, patients, and families partnered to improve lung function in pediatric CF patients. Flowcharts facilitated a standardized approach across providers to develop individualized treatment plans for patients, which resulted in improved lung function.

Impact of Multidisciplinary Standardization of Care for Gastroschisis: Treatment, Outcomes, and Cost

BACKGROUND/PURPOSE

Elimination of unnecessary practice variation through standardization creates opportunities for improved outcomes and cost-effectiveness. A quality improvement (QI) initiative at our institution used evidence and consensus to standardize management of gastroschisis (GS) from birth to discharge.

METHODS

An interdisciplinary team utilized best practice evidence and expert opinion to standardize GS care. Following stakeholder engagement and education, care standardization was implemented in September 2014. A comparative cohort study was conducted on consecutive patients treated before (n=33) and after (n=24) standardization. Demographic, treatment, and outcome measures were collected from a prospective GS registry. Direct costs were estimated, and protocol compliance was audited.

RESULTS

BW, GA, and bowel injury severity were comparable between groups. Key practice changes were: closure technique (pre-88% primary fascial, post-83% umbilical cord flap; p<0.001), closure location (pre-97% OR, post-67% NICU; p<0.001), and GA avoidance (pre-0%, post-48%; p<0.001). Median post-closure ventilation days were shorter (pre-4, post-1; p<0.001), and SSI rates trended lower (pre-21%, post-8%; p=0.3) in the post-implementation group with no differences in TPN days or LOS. No significant difference was seen in average per-patient costs: pre-$85,725 ($29,974-221,061), post-$76,329 ($14,205-176,856).

CONCLUSION

Care standardization for GS enables practice transformation, cost-effective outcome improvement, and supports an organizational culture dedicated to continuous improvement.

LEVEL OF EVIDENCE

III.