Development and Testing of the Pediatric Respiratory Illness Measurement System (PRIMES) Quality Indicators

A multicenter randomized trial to compare automatic versus as-needed follow-up for children hospitalized with common infections: The FAAN-C trial protocol

INTRODUCTION

Physicians commonly recommend automatic primary care follow-up visits to children being discharged from the hospital. While automatic follow-up provides an opportunity to address postdischarge needs, the alternative is as-needed follow-up. With this strategy, families monitor their child's symptoms and decide if they need a follow-up visit in the days after discharge. In addition to being family centered, as-needed follow-up has the potential to reduce time and financial burdens on both families and the healthcare system. As-needed follow-up has been shown to be safe and effective for children hospitalized with bronchiolitis, but the extent to which hospitalized children with other common conditions might benefit from as-needed follow-up is unclear.

METHODS

The Follow-up Automatically versus As-Needed Comparison (FAAN-C, or "fancy") trial is a multicenter randomized controlled trial. Children who are hospitalized for pneumonia, urinary tract infection, skin and soft tissue infection, or acute gastroenteritis are eligible to participate. Participants are randomized to an as-needed versus automatic posthospitalization follow-up recommendation. The sample size estimate is 2674 participants and the primary outcome is all-cause hospital readmission within 14 days of discharge. Secondary outcomes are medical interventions and child health-related quality of life. Analyses will be conducted in an intention-to-treat manner, testing noninferiority of as-needed follow-up compared with automatic follow-up.

DISCUSSION

FAAN-C will elucidate the relative benefits of an as-needed versus automatic follow-up recommendation, informing one of the most common decisions faced by families of hospitalized children and their medical providers. Findings from FAAN-C will also have implications for national quality metrics and guidelines.

Trends in Low-Value Care Among Children's Hospitals

BACKGROUND AND OBJECTIVES

Longitudinal pediatric low-value care (LVC) trends are not well established. We used the Pediatric Health Information System LVC Calculator, which measures utilization of 30 nonevidenced-based services, to report 7-year LVC trends.

METHODS

This retrospective cohort study applied the LVC Calculator to emergency department (ED) and hospital encounters from January 1, 2016, to December 31, 2022. We used generalized estimating equation models accounting for hospital clustering to assess temporal changes in LVC.

RESULTS

There were 5 265 153 eligible ED encounters and 1 301 613 eligible hospitalizations. In 2022, of 21 LVC measures applicable to the ED cohort, the percentage of encounters with LVC had increased for 11 measures, decreased for 1, and remained unchanged for 9 as compared with 2016. Computed tomography for minor head injury had the largest increase (17%-23%; P < .001); bronchodilators for bronchiolitis decreased (22%-17%; P = .001). Of 26 hospitalization measures, LVC increased for 6 measures, decreased for 9, and was unchanged for 11. Inflammatory marker testing for pneumonia had the largest increase (23%-38%; P = .003); broad-spectrum antibiotic use for pneumonia had the largest decrease (60%-48%; P < .001). LVC remained unchanged or decreased for most medication and procedure measures, but remained unchanged or increased for most laboratory and imaging measures.

CONCLUSIONS

LVC improved for a minority of services between 2016 and 2022. Trends were more favorable for therapeutic (medications and procedures) than diagnostic measures (imaging and laboratory studies). These data may inform prioritization of deimplementation efforts.

Pediatric Respiratory Illnesses: An Update on Achievable Benchmarks of Care

BACKGROUND AND OBJECTIVES

Pediatric respiratory illnesses (PRI): asthma, bronchiolitis, pneumonia, croup, and influenza are leading causes of pediatric hospitalizations, and emergency department (ED) visits in the United States. There is a lack of standardized measures to assess the quality of hospital care delivered for these conditions. We aimed to develop a measure set for automated data extraction from administrative data sets and evaluate its performance including updated achievable benchmarks of care (ABC).

METHODS

A multidisciplinary subject-matter experts team selected quality measures from multiple sources. The measure set was applied to the Public Health Information System database (Children's Hospital Association, Lenexa, KS) to cohorts of ED visits and hospitalizations from 2017 to 2019. ABC for pertinent measures and performance gaps of mean values from the ABC were estimated. ABC were compared with previous reports.

RESULTS

The measure set: PRI report includes a total of 94 quality measures. The study cohort included 984 337 episodes of care, and 82.3% were discharged from the ED. Measures with low performance included bronchodilators (19.7%) and chest x-rays (14.4%) for bronchiolitis in the ED. These indicators were (34.6%) and (29.5%) in the hospitalized cohort. In pneumonia, there was a 57.3% use of narrow spectrum antibiotics. In general, compared with previous reports, there was improvement toward optimal performance for the ABCs.

CONCLUSIONS

The PRI report provides performance data including ABC and identifies performance gaps in the quality of care for common respiratory illnesses. Future directions include examining health inequities, and understanding and addressing the effects of the coronavirus disease 2019 pandemic on care quality.

National cross-sectional study on cost consciousness, cost accuracy, and national medical waste reduction initiative knowledge among pediatric hospitalists in the United States

BACKGROUND/OBJECTIVE

Despite initiatives to reduce waste and spending, there is a gap in physician knowledge regarding the cost of commonly ordered items. We examined the relationship between pediatric hospitalists' knowledge of national medical waste reduction initiatives, self-reported level of cost-consciousness (the degree in which cost affects practice), and cost accuracy (how close an estimate is to its hospital cost) at a national level.

METHODS

This cross-sectional study used a national, online survey sent to hospitalists at 49 children's hospitals to assess their knowledge of national medical waste reduction initiatives, self-reported cost consciousness, and cost estimates for commonly ordered laboratory studies, medications, and imaging studies. Actual unit costs for each hospital were obtained from the Pediatric Health Information System (PHIS). Cost accuracy was calculated as the percent difference between each respondent's estimate and unit costs, using cost-charge ratios (CCR).

RESULTS

The hospitalist response rate was 17.7% (327/1850), representing 40 hospitals. Overall, 33.1% of respondents had no knowledge of national medical waste reduction initiatives and 24.3% had no knowledge of local hospital costs. There was no significant relationship between cost accuracy and knowledge of national medical waste reduction initiatives or high self-reported cost consciousness. Hospitalists with the highest self-reported cost consciousness were the least accurate in estimating costs for commonly ordered laboratory studies, medications, or imaging studies. Respondents overestimated the cost of all items with the largest percent difference with medications. Hospitalists practicing over 15 years had the highest cost accuracy.

CONCLUSIONS

A large proportion of pediatric hospitalists lack knowledge on national waste reduction initiatives. Improving the cost-accuracy of pediatric hospitalists may not reduce health care costs as they overestimated many hospital costs. Median unit cost lists could be a resource for educating medical students and residents about health care costs.

Variation in Dexamethasone Dosing and Use Outcomes for Inpatient Croup

OBJECTIVES

Evaluate the association between dexamethasone dosing and outcomes for children hospitalized with croup.

METHODS

This study was nested within a multisite prospective cohort study of children aged 6 months to 6 years admitted to 1 of 5 US children's hospitals between July 2014 and June /2016. Multivariable linear and logistic mixed-effects regression models were used to examine the association between the number of dexamethasone doses (1 vs >1) and outcomes (length of stay [LOS], cost, and 30-day same-cause reuse). All multivariable analyses included a site-specific random effect to account for clustering within hospital and were adjusted for age, sex, race and ethnicity, presenting severity, medical complexity, insurance, caregiver education, and hospital. In cost analyses, we controlled for LOS.

RESULTS

Among 234 children hospitalized with croup, patient characteristics did not differ by number of doses. The proportion receiving >1 dose varied by hospital (range 27.9%-57.1%). In adjusted analyses, >1 dose was not associated with same-cause reuse (odds ratio 0.87 [95% confidence interval (CI): 0.26 to 2.95]) but was associated with 45% longer LOS (relative risk = 1.45 [95% CI: 1.30 to 1.62]). When we controlled for LOS, >1 dose was not associated with differential cost ($-31.2 [95% CI $-424.4 to $362.0]). Eighty-two (35%) children received dexamethasone before presentation.

CONCLUSIONS

We found significant interhospital variation in dexamethasone dosing and LOS. When we controlled for severity on presentation, >1 dexamethasone dose was associated with longer LOS but not reuse. Although incomplete adjustment for severity is one possible explanation, some providers may routinely keep children hospitalized to administer multiple dexamethasone doses.

Development and Use of a Calculator to Measure Pediatric Low-Value Care Delivered in US Children's Hospitals

IMPORTANCE

The scope of low-value care in children's hospitals is poorly understood.

OBJECTIVE

To develop and apply a calculator of hospital-based pediatric low-value care to estimate prevalence and cost of low-value services.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study developed and applied a calculator of hospital-based pediatric low-value care to estimate the prevalence and cost of low-value services among 1 011 950 encounters reported in 49 US children's hospitals contributing to the Pediatric Health Information System (PHIS) database. To develop the calculator, a multidisciplinary stakeholder group searched existing pediatric low-value care measures and used an iterative process to identify and operationalize relevant hospital-based measures in the PHIS database. Children with an eligible encounter in 2019 were included in the calculator-applied analysis. Two cohorts were analyzed: an emergency department cohort (with encounters resulting in emergency department discharge) and a hospitalized cohort.

EXPOSURES

Eligible condition-specific hospital encounters.

MAIN OUTCOMES AND MEASURES

The proportion and volume of encounters in which low-value services were delivered and their associated standardized costs. Measures were ranked by those outcomes.

RESULTS

There were 1 011 950 encounters eligible for 1 or more of 30 calculator-included measures in 2019; encounters were incurred by 816 098 unique patients with a median age of 3 years (IQR, 1-8 years). In the emergency department cohort, low-value services delivered in the greatest percentage of encounters were Group A streptococcal testing among children younger than 3 years with pharyngitis (3679 of 9785 [37.6%]), computed tomography scan for minor head injury (7541 of 42 602 [17.7%]), and bronchodilators for treatment of bronchiolitis (8899 of 55 616 [16.0%]). In the hospitalized cohort, low-value care was most prevalent for broad-spectrum antibiotics in the treatment of community-acquired pneumonia (3406 of 5658 [60.2%]), acid suppression therapy for infants with esophageal reflux (3814 of 7507 of [50.8%]), and blood cultures for uncomplicated community-acquired pneumonia (2277 of 5823 [39.1%]). Measured low-value services generated nearly $17 million in total standardized cost. The costliest services in the emergency department cohort were computed tomography scan for abdominal pain (approximately $1.8 million) and minor head injury (approximately $1.5 million) and chest radiography for asthma (approximately $1.1 million). The costliest services in the hospitalized cohort were receipt of 2 or more concurrent antipsychotics (approximately $2.4 million), and chest radiography for bronchiolitis ($801 680) and asthma ($625 866).

CONCLUSIONS AND RELEVANCE

This cross-sectional analysis found that low-value care for some pediatric services was prevalent and costly. Measuring receipt of low-value services across conditions informs prioritization of deimplementation efforts. Continued use of this calculator may establish trends in low-value care delivery.

Predictors of Quality Improvement in Pediatric Asthma Care

BACKGROUND

Little is known about what hospital and emergency department (ED) factors predict performance in pediatric quality improvement efforts.

OBJECTIVES

Identify site characteristics and implementation strategies associated with improvements in pediatric asthma care.

METHODS

In this secondary analysis, we used data from a national quality collaborative. Data on site factors were collected via survey of implementation leaders. Data on quality measures were collected via chart review of children with a primary diagnosis of asthma. ED measures included severity assessment at triage, corticosteroid administration within 60 minutes, avoidance of chest radiographs, and discharge from the hospital. Inpatient measures included early administration of bronchodilator via metered-dose inhaler, screening for tobacco exposure, and caregiver referral to smoking cessation resources. We used multilevel regression models to determine associations between site factors and changes in mean compliance across all measures.

RESULTS

Sixty-four EDs and 70 inpatient units participated. Baseline compliance was similar by site characteristics. We found significantly greater increases in compliance in EDs within nonteaching versus teaching hospitals (12% vs 5%), smaller versus larger hospitals (10% vs 4%), and rural and urban versus suburban settings (6%-7% vs 3%). In inpatient units, we also found significantly greater increases in compliance in nonteaching versus teaching hospitals (36% vs 17%) and community versus children's hospitals (23% vs 14%). Changes in compliance were not associated with organizational readiness or number of audit and feedback sessions or improvement cycles.

CONCLUSIONS

Specific hospital and ED characteristics are associated with improvements in pediatric asthma care. Identifying setting-specific barriers may facilitate more targeted implementation support.

Associations Between Quality Measures and Outcomes for Children Hospitalized With Bronchiolitis

OBJECTIVES

To use adherence to the Pediatric Respiratory Illness Measurement System (PRIMES) indicators to evaluate the strength of associations for individual indicators with length of stay (LOS) and cost for bronchiolitis.

METHODS

We prospectively enrolled children with bronchiolitis at 5 children's hospitals between July 1, 2014, and June 30, 2016. We examined associations between adherence to each individual PRIMES indicator for bronchiolitis and LOS and cost. Sixteen indicators were included, 9 "overuse" indicators for care that should not occur and 7 "underuse" indicators for care that should occur. We performed mixed effects linear regression to examine the association between adherence to each individual indicator and LOS (hours) and cost (dollars). All models controlled for patient demographics, patient complexity, and hospital.

RESULTS

We enrolled 699 participants. The mean age was 8 months; 56% were male, 38% were white, and 63% had public insurance. Three indicators were significantly associated with shorter LOS and lower cost. All 3 indicators were overuse indicators and related to laboratory testing: no blood cultures (adjusted mean difference in LOS: -24.3 hours; adjusted mean cost difference: -$731, P < .001), no complete blood cell counts (LOS: -17.8 hours; cost: -$399, P < .05), and no respiratory syncytial virus testing (LOS: -16.6 hours; cost: -$272, P < .05). Two underuse indicators were associated with higher cost: documentation of oral intake at discharge ($671, P < .01) and documentation of hospital follow-up ($538, P < .05).

CONCLUSIONS

A subset of PRIMES quality indicators for bronchiolitis are strongly associated with improved outcomes and can serve as important measures for future quality improvement efforts.

Comparison of As-Needed and Scheduled Posthospitalization Follow-up for Children Hospitalized for Bronchiolitis: The Bronchiolitis Follow-up Intervention Trial (BeneFIT) Randomized Clinical Trial

IMPORTANCE

Posthospitalization follow-up visits are prescribed frequently for children with bronchiolitis. The rationale for this practice is unclear, but prior work has indicated that families value these visits for the reassurance provided. The overall risks and benefits of scheduled visits have not been evaluated.

OBJECTIVE

To assess whether an as-needed posthospitalization follow-up visit is noninferior to a scheduled posthospitalization follow-up visit with respect to reducing anxiety among parents of children hospitalized for bronchiolitis.

DESIGN, SETTING, AND PARTICIPANTS

This open-label, noninferiority randomized clinical trial, performed between January 1, 2018, and April 31, 2019, assessed children younger than 24 months of age hospitalized for bronchiolitis at 2 children's hospitals (Primary Children's Hospital, Salt Lake City, Utah, and Lucile Packard Children's Hospital, Palo Alto, California) and 2 community hospitals (Intermountain Riverton Hospital, Riverton, Utah, and Packard El Camino Hospital, Mountain View, California). Data analysis was performed in an intention-to-treat manner.

INTERVENTIONS

Randomization (1:1) to a scheduled (n = 151) vs an as-needed (n = 153) posthospitalization follow-up visit.

MAIN OUTCOME AND MEASURES

The primary outcome was parental anxiety 7 days after hospital discharge, measured using the anxiety portion of the Hospital Anxiety and Depression Scale, which ranged from 0 to 28 points, with higher scores indicating greater anxiety. Fourteen prespecified secondary outcomes were assessed.

RESULTS

Among 304 children randomized (median age, 8 months; interquartile range, 3-14 months; 179 [59%] male), the primary outcome was available for 269 patients (88%). A total of 106 children (81%) in the scheduled follow-up group attended a scheduled posthospitalization visit compared with 26 children (19%) in the as-needed group (absolute difference, 62%; 95% CI, 53%-71%). The mean (SD) 7-day parental anxiety score was 3.9 (3.5) among the as-needed posthospitalization follow-up group and 4.2 (3.5) among the scheduled group (absolute difference, -0.3 points; 95% CI, -1.0 to 0.4 points), with the upper bound of the 95% CI within the prespecified noninferiority margin of 1.1 points. Aside from a decreased mean number of clinic visits (absolute difference, -0.6 visits per patient; 95% CI, -0.4 to -0.8 visits per patient) among the as-needed group, there were no significant between-group differences in secondary outcomes, including readmissions (any hospital readmission before symptom resolution: absolute difference, -1.6%; 95% CI, -5.7% to 2.5%) and symptom duration (time from discharge to cough resolution: absolute difference, -0.6 days; 95% CI, -2.4 to 1.2 days; time from discharge to child reported "back to normal": absolute difference, -0.8 days; 95% CI, -2.7 to 1.0 days; and time from discharge to symptom resolution: absolute difference, -0.6 days; 95% CI, -2.5 to 1.3 days).

CONCLUSIONS AND RELEVANCE

Among parents of children hospitalized for bronchiolitis, an as-needed posthospitalization follow-up visit is noninferior to a scheduled posthospitalization follow-up visit with respect to reducing parental anxiety. These findings support as-needed follow-up as an effective posthospitalization follow-up strategy.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT03354325.

Vaccination Status and Adherence to Quality Measures for Acute Respiratory Tract Illnesses

OBJECTIVES

To assess the relationship between vaccination status and clinician adherence to quality measures for children with acute respiratory tract illnesses.

METHODS

We conducted a multicenter prospective cohort study of children aged 0 to 16 years who presented with 1 of 4 acute respiratory tract illness diagnoses (community-acquired pneumonia, croup, asthma, and bronchiolitis) between July 2014 and June 2016. The predictor variable was provider-documented up-to-date (UTD) vaccination status. Our primary outcome was clinician adherence to quality measures by using the validated Pediatric Respiratory Illness Measurement System (PRIMES). Across all conditions, we examined overall PRIMES composite scores and overuse (including indicators for care that should not be provided, eg, C-reactive protein testing in community-acquired pneumonia) and underuse (including indicators for care that should be provided, eg, dexamethasone in croup) composite subscores. We examined differences in length of stay, costs, and readmissions by vaccination status using adjusted linear and logistic regression models.

RESULTS

Of the 2302 participants included in the analysis, 92% were documented as UTD. The adjusted mean difference in overall PRIMES scores by UTD status was not significant (adjusted mean difference -0.3; 95% confidence interval: -1.9 to 1.3), whereas the adjusted mean difference was significant for both overuse (-4.6; 95% confidence interval: -7.5 to -1.6) and underuse (2.8; 95% confidence interval: 0.9 to 4.8) composite subscores. There were no significant adjusted differences in mean length of stay, cost, and readmissions by vaccination status.

CONCLUSIONS

We identified lower adherence to overuse quality indicators and higher adherence to underuse quality indicators for children not UTD, which suggests that clinicians "do more" for hospitalized children who are not UTD.

Process Metrics and Outcomes to Inform Quality Improvement in Pediatric Hospital Medicine

This article provides an overview of the selection, development, and use of process and outcome measures for pediatric hospital medicine quality improvement initiatives. It reviews commonly used categories of process and outcome measures and provides a list of common sources and repositories of previously validated measures. It also provides a blueprint for the development of novel measures. The relative merits of various data collection methods are discussed (eg, medical record abstraction, administrative, surveys), along with guiding principles for disseminating the results of quality improvement evaluations on a local and national level.

Pediatric Respiratory Illness Measurement System (PRIMES) Scores and Outcomes

BACKGROUND AND OBJECTIVES

The Pediatric Respiratory Illness Measurement System (PRIMES) generates condition-specific composite quality scores for asthma, bronchiolitis, croup, and pneumonia in hospital-based settings. We sought to determine if higher PRIMES composite scores are associated with improved health-related quality of life, decreased length of stay (LOS), and decreased reuse.

METHODS

We conducted a prospective cohort study of 2334 children in 5 children's hospitals between July 2014 and June 2016. Surveys administered on admission and 2 to 6 weeks postdischarge assessed the Pediatric Quality of Life Inventory (PedsQL). Using medical records data, 3 PRIMES scores were calculated (0-100 scale; higher scores = improved adherence) for each condition: an overall composite (including all quality indicators for the condition), an overuse composite (including only indicators for care that should not be provided [eg, chest radiographs for bronchiolitis]), and an underuse composite (including only indicators for care that should be provided [eg, dexamethasone for croup]). Multivariable models assessed relationships between PRIMES composite scores and (1) PedsQL improvement, (2) LOS, and (3) 30-day reuse.

RESULTS

For every 10-point increase in PRIMES overuse composite scores, LOS decreased by 8.8 hours (95% confidence interval [CI] -11.6 to -6.1) for bronchiolitis, 3.1 hours (95% CI -5.5 to -1.0) for asthma, and 2.0 hours (95% CI -3.9 to -0.1) for croup. Bronchiolitis overall composite scores were also associated with shorter LOS. PRIMES composites were not associated with PedsQL improvement or reuse.

CONCLUSIONS

Better performance on some PRIMES condition-specific composite measures is associated with decreased LOS, with scores on overuse quality indicators being a primary driver of this relationship.

Feasibility testing of the Core set of quality Indicators for Paediatric Primary Care in Europe, COSI-PPC-EU

There is a need to measure and improve the quality of paediatric primary care in Europe where major differences in the delivery and outcomes of child health care exist. A collaborative panel of paediatric senior experts developed a Core Set of Indicators for Paediatric Primary Care in Europe by compiling 42 quality indicators in a modified consensus process following the RAND/UCLA appropriateness method. The aim of this study was to explore the feasibility of the quality indicator set in European paediatric primary care practices. Seventy-nine practices from eight countries participated in a detailed online interview. The practices rated the applicability, relevance, reliability and acceptance of the 42 quality indicator as well as the availability, technical feasibility and effort to retrieve the needed data from their medical records. Most quality indicators were considered applicable, available, reliable, acceptable and relevant for monitoring quality of care in paediatric primary care. Respondents rated feasibility and effort to retrieve the data lowest because of difficulties collecting the data from the medical records.Conclusion: European paediatric primary care practices generally agree with the proposed quality indicator set. They document most of the parameters. However, the collection of specific needed values from available routine patient-data is considered technically difficult and time-consuming. What is Known? • Paediatric primary care systems in Europe show striking differences in their performance. Pre-existing sets of quality indicators are predominantly limited to national populations, specific diseases and hospital care. • A Core Set of 42 quality indicators for paediatric primary care in Europe was developed by European paediatricians using a systematic literature review and a consensus process following a modified RAND/UCLA appropriateness method. What is New? • Paediatric primary care providers in Europe agree with the idea to use COSI-PPC-EU to monitor and improve the quality of care. The set was considered applicable, available, reliable, acceptable, and relevant for quality improvement. • The score for feasibility and effort to retrieve the data was low, because of technical reasons; the electronical or paper-based medical documentation in most cases does not allow convenient access to all necessary data.

Quality indicators for responsible use of medicines: a systematic review

OBJECTIVE

All healthcare systems require valid ways to evaluate service delivery. The objective of this study was to identify existing content validated quality indicators (QIs) for responsible use of medicines (RUM) and classify them using multiple frameworks to identify gaps in current quality measurements.

DESIGN

Systematic review without meta-analysis.

SETTING

All care settings.

SEARCH STRATEGY

CINAHL, Embase, Global Health, International Pharmaceutical Abstract, MEDLINE, PubMed and Web of Science databases were searched up to April 2018. An internet search was also conducted. Articles were included if they described medication-related QIs developed using consensus methods. Government agency websites listing QIs for RUM were also included.

ANALYSIS

Several multidimensional frameworks were selected to assess the scope of QI coverage. These included Donabedian's framework (structure, process and outcome), the Anatomical Therapeutic Chemical (ATC) classification system and a validated classification for causes of drug-related problems (c-DRPs; drug selection, drug form, dose selection, treatment duration, drug use process, logistics, monitoring, adverse drug reactions and others).

RESULTS

2431 content validated QIs were identified from 131 articles and 5 websites. Using Donabedian's framework, the majority of QIs were process indicators. Based on the ATC code, the largest number of QIs pertained to medicines for nervous system (ATC code: N), followed by anti-infectives for systemic use (J) and cardiovascular system (C). The most common c-DRPs pertained to 'drug selection', followed by 'monitoring' and 'drug use process'.

CONCLUSIONS

This study was the first systematic review classifying QIs for RUM using multiple frameworks. The list of the identified QIs can be used as a database for evaluating the achievement of RUM. Although many QIs were identified, this approach allowed for the identification of gaps in quality measurement of RUM. In order to more effectively evaluate the extent to which RUM has been achieved, further development of QIs may be required.