Improving the reporting quality of reliability generalization meta-analyses: The REGEMA checklist

A comparison of univariate and meta-analytic structural equation modeling approaches to reliability generalization applied to the Maslach Burnout Inventory

Introduction

Reliability is a property of tests scores that varies from sample to sample. One way of generalizing reliability of a test is to perform a meta-analysis on some reliability estimator. In 2011, a reliability generalization meta-analysis on the Maslach Burnout Inventory (MBI) was conducted, concluding that average alpha values for the MBI dimensions ranged from 0.71 to 0.88. In the present study, we aimed to update the average reliability values of the MBI by conducting a literature search from 2010 until now and comparing to statistical procedures of meta-analysis: the Univariate approach, that were used in the previous study, and a novel meta-analytic approach based on structural equation modeling.

Method

An estimation of average reliability was done based on 69 independent primary reliability coefficients for the Univariate approach. The average reliability was based on 9 independent studies in the case of the Meta-analytic Structural Equation Modeling (MASEM) approach. Given that MASEM has the additional capability of testing the internal structure of a test, we also fitted several models.

Results

The data was well-suited to the bifactor model, revealing the dominance of the general factor over the domain-specific ones. Acceptable overall alpha and omega coefficients were achieved for the two of the MBI dimensions, having depersonalization reliability estimates below recommendations.

Discussion

In general, the MBI can be viewed as a highly interconnected three-factor scale, being its appropriate for research purposes.

A reliability generalization meta-analysis of the family resilience assessment scale (FRAS)

PROBLEM

Practitioners and researchers frequently rely on the Family Resilience Assessment Scale (FRAS) to assess family resilience, and previous research reported varying reliability statistics for the measurements with the scale. The present study aims to generalize the reliability of the FRAS based on Cronbach's alpha coefficients reported in the selected studies.

ELIGIBILITY CRITERIA

We selected relevant research on various databases, including Web of Science, PubMed, ProQuest, Scopus, YÖK Thesis Center, DergiPark, and TR Index.

SAMPLE

Satisfying our inclusion criteria, fifty-five studies were included in the present study.

RESULTS

We calculated the reliability generalization coefficients for the FRAS total score to be 0.951 (95% CI [0.942, 0.958]) and 0.949 for Family Communication and Problem Solving, 0.792 for Utilizing Social and Economic Resources, 0.861 for Maintaining a Positive Outlook, 0.635 for Family Connectedness, 0.873 for Family Spirituality, and 0.702 for Ability to Make Meaning of Adversity.

CONCLUSIONS

In a nutshell, our findings secure substantial insights into the reliability of the FRAS and its subscales for prospective researchers and practitioners. In this study, generalized Cronbach's alpha values imply average, good, and acceptable reliability for the FRAS subscales and total score, except for Family Connectedness.

IMPLICATIONS

The distinct contribution of our research may be to reemphasize the significance of avoiding reliability induction and to raise awareness among prospective researchers of evaluating the reliability of any measurement they would obtain.

Sleep effort and its measurement: A scoping review

Insomnia disorder is characterized by disruption in sleep continuity and an overall dissatisfaction with sleep. A relevant feature of insomnia is sleep effort, which refers to both cognitive and behavioural conscious attempts to initiate sleep. The Glasgow Sleep Effort Scale is a self-report tool developed to assess this construct. The objective of the current scoping review was to map how sleep effort has been discussed in the literature and operationalized through its respective measure. Medline/PubMed, Scopus, Web of Science and PsycInfo databases were used to search for potential studies. The search query used in databases was the specific name of the self-reported tool itself (Glasgow Sleep Effort Scale) and "sleep effort" term. This scoping review followed JBI guidelines. To be included, records pertaining to any type of study that mentioned the Glasgow Sleep Effort Scale were considered. No language constraint was used. At the end, 166 initial records were retrieved. From those, 46 records met eligibility criteria and were analysed. Among the main findings, it was observed that the Glasgow Sleep Effort Scale has been increasingly used in recent years, with a notable observed upward trend, especially in the last 2 years. In addition to the original measure, only three published adapted versions of the instrument were identified. This suggests that there is limited research on adapting the scale for different populations or contexts. Sleep effort has been increasingly studied in the last few years. Nonetheless, more research on the Glasgow Sleep Effort Scale tool is recommended, including cross-cultural adaptations.

Reliability generalization meta-analysis of orthorexia nervosa using the ORTO-11/12/15/R scale in all populations and language versions

BACKGROUND

The ORTO scale was developed in 2004 as a self-report questionnaire to assess symptoms of orthorexia nervosa (ON). ON is an unhealthy preoccupation with eating healthy food. The scale aims to measure obsessive attitudes and behaviors related to the selection, purchase, preparation, and consumption of pure, healthy food. Since its development, the ORTO-15 has been adapted into several shorter versions. The objective was to conduct a reliability generalization meta-analysis of the ORTO scale and its variant versions in all populations and languages.

METHODS

A systematic literature search was conducted to identify studies reporting the internal consistency of ORTO. Random-effect models were used to evaluate summary statistics of reliability coefficients, weighting the coefficients by the inverse variance using the restricted maximum likelihood method. The heterogeneity among the reliability coefficients was evaluated and assessed using numerous statistical metrics. The tau (τ), tau2 (τ2), I2, H2, R2, df, and the Q-statistic are among those obtained. Meta-regression analyses were used to examine moderators such as age and sex.

RESULTS

Twenty-one studies (k = 21) involving 11,167 participants (n = 11,167) were analyzed. The overall effect estimate on internal consistency was 0.59 (95% CI 0.49-0.68), with a minimum reliability coefficient of 0.23 and a maximum reliability coefficient of 0.83. The heterogeneity statistics were found to have an I2 of 99.31%, which suggested high heterogeneity owing to a decrease in the confidence interval (95% CI) and an increase in variability. Sensitivity analysis revealed that a few studies strongly influenced the overall estimate. Egger's test suggested possible publication bias. Neither age nor sex significantly moderated reliability via meta-regression.

CONCLUSIONS

The ORTO scale has a relatively low pooled reliability coefficient. Alternative ON assessment tools with enhanced psychometric properties are needed. Clinicians should not base diagnoses or treatment decisions on ORTO alone. Comprehensive psychiatric assessment is essential for accurate ON evaluation.

A Reliability Generalization Meta-Analysis of the Mother-To-Infant Bonding Scale

The Mother-to-Infant Bonding Scale (MIBS) is among the most popular measurement tools to evaluate caregiver-infant attachment. We carried out a meta-analysis study to explore the generalizability of the reliability coefficients for the MIBS in different studies. The literature review yielded a total of 702 studies investigating caregiver-infant attachment. After removing duplicate studies, we also excluded compilations, meta-analyses, qualitative studies, those using different measurement tools, studies published in a language other than English, citations, and those whose full texts could not be accessed. Eventually, we considered a total of 26 studies with 33 Cronbach's alpha coefficients that satisfied the inclusion criteria. We normalized the alpha coefficients using Bonett's transformation, and the analyses were performed using a 95% confidence interval. The findings revealed a Cronbach's alpha (n = 33) coefficient of 0.73 (CI = 0.68-0.77); hence, the present reliability generalization study provides evidence that the reliability scores produced after measurements with the MIBS in previous studies are acceptable across samples. Overall, further studies may reliably utilize the MIBS to evaluate mother-infant attachment.

Evaluating the reliability of the athlete sleep behavior questionnaire (ASBQ): a meta-analysis of Cronbach's alpha and intraclass correlation coefficient

BACKGROUND

The Athlete Sleep Behavior Questionnaire (ASBQ) was designed to identify maladaptive sleep practices among athletes. The aim of this meta-analysis was to evaluate the internal consistency and the test-retest reliability coefficients of the ASBQ.

METHODS

A systematic search across 10 databases from inception of the ASBQ to August 2023 was performed. Publications that reported estimates of internal consistency and/or test-retest reliability of the ASBQ were included. A random-effects model was employed to estimate the overall reliability measures of the ASBQ.

RESULTS

Meta-analytic results demonstrated a good level of internal consistency within the ASBQ, evidenced by a Cronbach's alpha of 0.73 (95% CI: 0.63 to 0.80). This suggests a modest correlation among the questionnaire items, supporting its reliability as an effective measure of sleep behavior. In terms of test-retest reliability, our meta-analysis revealed a very good degree of consistency (ICC = 0.88; 95% CI: 0.87 to 0.89), suggesting that the ASBQ can serve as an instrument for monitoring and evaluating changes in athletes' sleep behavior over time. No evidence of publication bias was identified.

CONCLUSION

While the ASBQ demonstrates a moderate level of internal consistency, its test-retest reliability suggests that it can serve as an instrument for longitudinal assessments of athletes' sleep behavior. Future studies focusing on refining the ASBQ to optimize its internal consistency and validate its applicability across diverse athletic populations are warranted.

A Reliability Generalization Meta-Analysis of the Child and Adolescent Mindfulness Measure (CAMM)

We carried out the present meta-analysis study to secure the reliability generalization of the Child and Adolescent Mindfulness Scale (CAMM) based on its reliability findings in different studies. The study only considered full-text papers in English that were indexed in Web of Science, PubMed, ProQuest, and Scopus and presented a Cronbach's alpha value of the measurements with the CAMM (209 alpha values out of 155 papers). Using the random effects model and the Bonnet transformation, the relevant analyses yielded the transformed Cronbach's alpha value of the CAMM to be 0.80 (CI = 0.79-0.81). The present meta-analysis study concluded that the CAMM is rather reliable to be utilized in child and adolescent populations. Besides, moderator analyses to scrutinize the alpha values yielded significant differences in favor of the origin continent of the research and the CAMM-short version (25-item/10-item). Yet, it was not the case with the publication type and the adaptation status of the scale. In a nutshell, future research may confidently employ the 10-item short form of the CAMM to assess mindfulness among children and adolescents.

Reliability generalization meta-analysis of the Athens Insomnia Scale and its translations: Examining internal consistency and test-retest validity

The Athens Insomnia Scale (AIS) is a widely used self-report measure to evaluate insomnia symptoms based on the International Classification of Diseases criteria version 10 (ICD-10). Despite its extensive use in clinical and research settings, the reliability of the AIS has not been systematically investigated. This systematic review reports a reliability generalization meta-analysis study to assess the internal consistency and the test-retest reliability of the AIS across various populations and settings. A systematic literature search was conducted to identify studies reporting Cronbach's alpha and test-retest coefficients for the AIS. Pooled estimates of reliability, along with moderator analyses, were calculated. The AIS has an excellent internal consistency of 0.84 (95% CI: 0.81 to 0.86), and re-test reliability of 0.86 (95% CI: 0.80 to 0.92). The significant heterogeneity levels support the recommendation that future studies using the AIS include and discuss reliability estimates based on their own data.

Epworth sleepiness scale: A meta-analytic study on the internal consistency

OBJECTIVE/BACKGROUND

The Epworth Sleepiness Scale (ESS) is one of the most used self-reported instruments to assess sleepiness. Thus, several adaptations into different Languages have been performed worldwide over the years. The scale has produced disparate psychometric properties when applied in different settings. In the current study, our aim was to perform a Reliability Generalization meta-analysis of the Cronbach᾽s alphas of all published studies on ESS, specifically with a psychometric focus.

PATIENTS/METHODS

Three reference databases (Scopus, PubMed and Web of Science) were searched since 1991 to October 2022 and all the records on psychometric or validation studies that reported Cronbach's alphas, from clinical and nonclinical groups, were included. In total, data from 46 publications (63 estimates) were extracted, comprising 92,503 participants.

RESULTS

Using a Random-Effects Model, the cumulative Cronbach's alpha for the 63 estimates was about 0.82 (CI: 0.798, 0.832) which can be considered as a good measure. However, and as expected, it was observed a high level of heterogeneity (I2 = 98.96%). Moderation analyses considering setting, date, continent, risk of bias, sex, age and language were performed in order to account for the heterogeneity. Even so, only the variables study setting and continent were significant, and had little importance in explaining the heterogeneity.

CONCLUSIONS

The ESS is a reliable tool to measure sleepiness; however, further studies are needed to investigate what variables might explain the observed variability. Moreover, it will be important to include empirical studies beyond psychometric ones.

Internal consistency and structural validity of the nomophobia questionnaire (NMP-Q) and its translations: A systematic review with meta-analysis

Background

A psychological condition known as NOMOPHOBIA causes individuals to fear losing ability to use or reach their mobile phones. The NMP-Q (nomophobia questionnaire) is a commonly used survey for assessing symptoms related to nomophobia.

Materials and methods

We performed a meta-analysis using reliability generalization (RG) on the NMP-Q. Thirteen studies involving 15,929 participants have reported original reliability estimates of the NMP-Q determined through a comprehensive and methodical examination of the available literature.

Results

For the total scores, the pooled internal consistency reliability was 0.93 [0.91; 0.95] and for the subscales it ranged from 0.83 to 0.91. Specifically, 0.91 [0.88; 0.93], 0.84 [0.80; 0.88], 0.83 [0.78; 0.88, and 0.83 [0.80; 0.85] for the subscales. Subscale 1 = not being able to communicate; subscale 2 = losing connectedness; subscale 3 = not being able to access information; and subscale 4 = giving up convenience", respectively. All reported effect sizes are Cronbach's alphas. Structural validity supported a solution of four-factors.

Conclusions

NMP-Q has an excellent internal consistency. Structural validity of four factors appears to be vigorous in fitting NMP-Q items. Our recommendation is to require future studies using NMP-Q to provide a reliability estimate based on their own data.

Measurement of Prosocial Tendencies: Meta-Analysis of the Generalization of the Reliability of the Instrument

The Prosocial Tendencies Measure (PTM) and its revised version (PTM-R) are used internationally to measure prosocial behaviors in different life situations. To obtain accumulated evidence of the report and the reliability of its scores, a meta-analysis of the reliability of internal consistency was performed. The databases of Web of Science (WoS) and Scopus were reviewed and all the studies that applied it from 2002 to 2021 were selected. Results: Only 47.9% of the studies presented the index of reliability of PTM and PTM-R. The meta-analytic results of the reliability report of the subscales that the PTM and the PTM-R have in common were: Public 0.78 (95% CI: 0.76-0.80), Anonymous 0.80 (95% CI: 0.79-0.82), Dire 0.74 (95% CI: 0.71-0.76), and Compliant 0.71 (95% CI: 0.72-0.78). Each one of them presents high levels of heterogeneity derived from the gender of the participants (percentage of women), the continent of the population, the validation design, the incentive to participate, and the form of application. It is concluded that both versions present acceptable reliabilities to measure prosocial behavior in different groups and situations, as adolescents and young people, but their clinical use is discouraged.

Insomnia Severity Index: A reliability generalisation meta-analysis

The aim of the current study was to conduct a reliability generalisation (RG) meta-analysis of Cronbach's alpha for the Insomnia Severity Index (ISI). A systematic search of three databases (PubMed, Scopus, and Web of Science) from inception to 12 March 2021 was performed. Publications that reported Cronbach's alpha for the total ISI score were included. Only psychometric-focussed studies were considered. Meta-analysis was carried out using a random-effects model to derive a pooled estimate of Cronbach's alphas. The number of participants in the included publications ranged from 25 to 12,056, with 33 studies (42 estimates) comprising internal consistency coefficients, and a combined sample size of N = 29,688. The age range of the included publications was from 13.4 to 74.3 years. Data extraction implied 33 publications out of 706 found through the database search. Cronbach's alphas ranged from 0.65 to 0.92. The majority of the reported coefficients were ≥0.7 and presented a low risk of bias (n = 32). The pooled alpha coefficient was 0.83 (IC [0.81-0.85]; SE = 0.009) with high heterogeneity among the included publications (I²  = 97%). Subgroup analyses including moderators such as continent, setting, risk of bias, and age did not affect significantly the overall result. In general, the cumulative estimate of Cronbach's alpha for the ISI is good. However, this finding should be interpreted with caution since there is a high heterogeneity level and some of the studies might not have checked the assumptions underlying Cronbach's alphas.

Complications Associated with Autologous Costal Cartilage Used in Rhinoplasty: An Updated Meta-Analysis

OBJECTIVE

To estimate the complications using autologous costal cartilage as grafts in rhinoplasty objectively and systematically with newly published literature.

METHODS

The literature was searched systematically; included studies were published between July of 1990 and April of 2020. Meta-analysis was performed using a random-effects model.

RESULTS

Twenty studies involving 1648 patients were included for meta-analysis. The pooled rates of complications were 3.05% of warping (95% CI 1.36-5.19%), 1.2% of resorption (95% CI 0.26-2.56%), 1.45% of infection (95% CI 0.34-3.06%), and 1.53% of contour irregularity (95% CI 0.53-2.88%). The revision rate was 2.25% (95% CI 0.96-3.9%). Regarding of donor-site morbidities, the rate of hypertrophic chest scar was 2.08% (95% CI 0.31-4.83%), and the rate of pneumothorax was 0% (95% CI 0-0.46%). The pooled rates of complications were 9.06% (95% CI 6.13-12.43%) at the recipient site when complications at the recipient site did not include revision surgery, 1.47% (95% CI 0.17-3.56%) at the donor site, and 15.13% (95%CI 11.03-19.69%) overall. The recipient-site adverse event rate was 12.44% (95% CI 8.98-16.33%).

CONCLUSIONS

Warping was found the most common complication after rhinoplasty with autologous costal cartilage. Revision after rhinoplasty using autologous costal cartilage was increased in these years. Donor-site complications increased the complication rate after rhinoplasty using autologous rib cartilage by 22%.

LEVEL OF EVIDENCE III

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266.

Florida Obsessive-Compulsive Inventory and Children's Florida Obsessive Compulsive Inventory: A reliability generalization meta-analysis

BACKGROUND

The Florida Obsessive Compulsive Inventory (FOCI) and its pediatric version, the Children's Florida Obsessive Compulsive Inventory (C-FOCI), are instruments for evaluating obsessive-compulsive symptomatology.

METHOD

A reliability generalization meta-analysis was conducted to estimate an average reliability of the scores and to identify study characteristics that explained the heterogeneity among scores. Using Kuder-Richardson 20 (KR-20) and Cronbach's α, a total of 23 and 20 independent samples were included in the meta-analysis for the FOCI and C-FOCI.

RESULTS

We found an average KR-20 of 0.826 for the FOCI's Symptom Checklist and an α of 0.882 FOCI's Symptom Severity. An average KR-20 of 0.740 was found for the C-FOCI's Symptom Checklist, while an average α of 0.794 was found for the C-FOCI's Symptom Severity. Moderator analyses showed that the source of the coefficients (i.e., whether they were reported by the authors of the primary study or estimated by the meta-analysts) was an important variable for the FOCI Symptom Severity, and that the focus of the study (i.e., whether it was psychometric or applied) and the sample size were relevant for the C-FOCI Symptom Checklist.

CONCLUSIONS

Considering that the FOCI and C-FOCI are scales characterized by their brevity and ease of use, and the reliabilities obtained here, both scales are well suited for screening purposes.

The Adult Prosocialness Behavior Scale: A reliability generalization meta-analysis

The Adult Prosocialness Behavior Scale (APBS) is most often used to measure adult prosociality. We conducted a reliability generalization meta-analysis to compute the average APBS reliability and examine the heterogeneity among reliability estimations and the influence of moderator variables. An exhaustive search identified 74 articles that applied the APBS with 16 items assessed on a 5-point Likert-type scale. Of these, 58 had reliability coefficients with the current data, and 76 reliability estimates were provided. Random- and mixed-effects models were used. The average reliability coefficient was .903 for Cronbach’s alpha, .896 for McDonald’s omega, and .674 for test–retest. Moderator analyses were used to create a predictive model in which the target population and study language accounted for 48.7% of the total variability among Cronbach’s alpha coefficients. Although the APBS has shown satisfactory internal consistency, it can vary as a function of several factors.

Reliability Generalization Meta-Analysis of Internet Gaming Disorder Scale

The objective of this study was to carry out a reliability generalization meta-analysis of the Internet Gaming Disorder Scale (IGDS) to find out if it presents adequate values that justify its application in its four versions: original and abbreviated with dichotomous or ordinal response. A systematic review including 26 studies that apply this instrument from 2015 to June 2022 was conducted. For each version, a reliability generalization meta-analysis was performed with the random coefficients (RC) and varying coefficients (VC) model. Results showed reliability levels > 0.80 in the ordinal versions (IGDS9P and IGDS27P) and in the dichotomous 27-item version (IGDS27D), while the dichotomous 9-item version (IGDS9D) produced levels > 0.70. High heterogeneity was found in all versions (I2 > 95%; Rb > 90%). Cronbach’s α means with both models (RC-VC) did not differ significantly except for the IGDS9D version. It is concluded that, considering the dependence of the results on sample size, abbreviated versions do not guarantee that reliability remains acceptable, and dichotomous versions may provide limited but acceptable reliability. Finally, IGDS27P version is recommended in contexts where high precision is required.

How (Not) to Measure Loneliness: A Review of the Eight Most Commonly Used Scales

Loneliness affects well-being and has long-term negative impacts on physical and mental health, educational outcomes, and employability. Because of those current and long-term impacts, loneliness is a significant issue for which we need reliable and appropriate measurement scales. In the current paper, psychometric properties of the eight most commonly used loneliness scales are reviewed both descriptively and meta-analytically. Results suggest that for many of the scales, the psychometric properties are promising. However, for some psychometric features, especially test-retest reliability and measurement invariance, evidence is rather scarce. Most striking, however, is the fact that all of the scales included items that do not measure loneliness. Surprisingly, for many (sub)scales, this was even the case for about half of the items. Because our measures are the foundation of our research work, it is crucial to improve the way loneliness is being measured.

California Critical Thinking Disposition Inventory: Reliability Generalization Meta-Analysis

The aims of this reliability generalization study were to provide the overall alpha values of the California critical thinking disposition inventory (CCTDI) total score and subscales scores and investigate the characteristics of the studies that may be associated with the variability in the reliability values of the CCTDI total score and subscales scores. This study was carried out with 98 alpha values from 87 unique studies for total CCTDI scores. In the random effects model, total CCTDI was found to be reliable across samples with alpha value of 0.83. Also, the general alpha coefficients were 0.65, 0.56, 0.64, 0.66, 0.74, 0.72, and 0.61 for TS-scale, OM-scale, A-scale, S-scale, SC-scale, I-scale, and M-scale, respectively. Examination of study characteristics indicated that sample type was a significant predictor for alpha value of total CCTDI and all subscales. Samples with university students reported larger Cronbach’s alpha estimates for total CCTDI and all subscales. Also, while language was found to moderate the general alpha coefficient of total CCTDI, OM-scale, A-scale, S-scale, and I-scale, it was not a significant moderator on the general alpha value of TS-scale, SC-scale, and M-scale. Total CCTDI and all subscales showed higher Cronbach’s alpha values for the English-language administrations. Besides, country of the study was a significant moderator on the general alpha coefficient of total CCTDI, S-scale, and I-scale. However, subgroup of discipline was not a significant moderator on the general alpha coefficient of total CCTDI and its subscales. The mean of the test scores significantly explained %5 of the variance of alpha values of the total CCTDI. SD of the test scores significantly explained %10, %55, and %54 of the variance of alpha values of the total CCTDI, A-scale, and S-scale, respectively. It was found that gender and ethnicity significantly moderated the alpha values for M-scale.

Basic Empathy Scale: A Systematic Review and Reliability Generalization Meta-Analysis

The Basic Empathy Scale (BES) has been internationally used to measure empathy. A systematic review including 74 articles that implement the instrument since its development in 2006 was carried out. Moreover, an evidence validity analysis and a reliability generalization meta-analysis were performed to examine if the scale presented the appropriate values to justify its application. Results from the systematic review showed that the use of the BES is increasing, although the research areas in which it is being implemented are currently being broadened. The validity analyses indicated that both the type of factor analysis and reliability are reported in validation studies much more than the consequences of testing are. Regarding the meta-analysis results, the mean of Cronbach's α for cognitive empathy was 0.81 (95% CI: 0.77-0.85), with high levels of heterogeneity (I2 = 98.81%). Regarding affective empathy, the mean of Cronbach's α was 0.81 (95% CI: 0.76-0.84), with high levels of heterogeneity. It was concluded that BES is appropriate to be used in general population groups, although not recommended for clinical diagnosis; and there is a moderate to high heterogeneity in the mean of Cronbach's α. The practical implications of the results in mean estimation and heterogeneity are discussed.

A Reliability Generalization Meta-analysis of the Padua Inventory-Revised (PI-R)

Background/Objective: The Padua Inventory-Revised (PI-R) is a widely applied instrument to measure obsessive-compulsive symptoms in clinical and nonclinical samples. We conducted a reliability generalization meta-analysis on the PI-R. Method: An exhaustive literature search yielded 118 empirical studies that had applied the PI-R, from which 30 studies (33 samples) reported an original reliability estimate. Results: Assuming a random-effects model, the average internal consistency reliability (Cronbach's alpha) was .92 (95% CI [.91, .93]) for the total scores, and ranged from .74 to .89 for the subscales. Assuming mixed-effects models, moderator analyses showed a positive statistically significant association between the standard deviation of the total scores and the reliability coefficients (p = .002; R² = .38). Conclusions: In terms of reliability, the PI-R scale was found to be adequate for both research and clinical purposes, although exhibiting large heterogeneity across studies. Future empirical studies using the PI-R should be required to provide at least one reliability estimate based on their own data.

Introduction to PRISMA 2020 and implications for research synthesis methodologists

Systematic reviews should be reported completely and accurately so that users can interpret and appraise the review findings appropriately. In this commentary, we describe key features of a major update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses statement-PRISMA 2020-and discuss its implications for research synthesis methodologists.

Reliability Generalization Study of the Person-Centered Care Assessment Tool

The so-called Person-Centered Care (PCC) model identifies three fundamental principles: changing the focus of attention from the disease to the person, individualizing care, and promoting empowerment. The Person-Centered Care Assessment Tool (P-CAT) has gained wide acceptance as a measure of PCC in recent years due to its brevity and simplicity, as well as its ease of application and interpretation. The objective of this study is to carry out a reliability generalization meta-analysis to estimate the internal consistency of the P-CAT and analyze possible factors that may affect it, such as the year of publication, the care context, the application method, and certain sociodemographic properties of the study sample. The mean value of α for the 25 samples of the 23 studies in the meta-analysis was 0.81 (95% CI: 0.79-0.84), with high heterogeneity (squared-I = 85.83%). The only variable that had a statistically significant relationship with the reliability coefficient was the mean age of the sample. The results show that the P-CAT gives acceptably consistent scores when its use is oriented toward the description and investigation of groups, although it may be affected by variables such as the age of participants.

A meta-review of transparency and reproducibility-related reporting practices in published meta-analyses on clinical psychological interventions (2000-2020)

Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data and reanalysed applying new and more refined analysis techniques. We attempted to empirically assess the prevalence of transparency and reproducibility-related reporting practices in published meta-analyses from clinical psychology by examining a random sample of 100 meta-analyses. Our purpose was to identify the key points that could be improved, with the aim of providing some recommendations for carrying out reproducible meta-analyses. We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. A structured coding form to assess transparency indicators was created based on previous studies and existing meta-analysis guidelines. We found major issues concerning: completely reproducible search procedures report, specification of the exact method to compute effect sizes, choice of weighting factors and estimators, lack of availability of the raw statistics used to compute the effect size and of interoperability of available data, and practically total absence of analysis script code sharing. Based on our findings, we conclude with recommendations intended to improve the transparency, openness, and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.