Assessing context suitability (generalizability, external validity, applicability or transferability) of findings in evidence syntheses in healthcare-An integrative review of methodological guidance

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Assessing Electronic Health Literacy in Individuals With the Post-COVID-19 Condition Using the German Revised eHealth Literacy Scale: Validation Study

BACKGROUND

The eHealth Literacy Scale (eHEALS) is a widely used instrument for measuring eHealth literacy (eHL). However, little is known so far about whether the instrument is valid for the assessment of eHL in persons who are affected by the post-COVID-19 condition. This is particularly important as people with the post-COVID-19 condition are frequently affected by false information from the internet.

OBJECTIVE

The objective of our study was to evaluate the validity and reliability of the German Revised eHealth Literacy Scale (GR-eHEALS) in individuals with the post-COVID-19 condition.

METHODS

A cross-sectional study was conducted from January to May 2022. The self-assessment survey consisted of the GR-eHEALS, health status- and internet use-related variables, sociodemographic data, and (post)-COVID-19-related medical data. Confirmatory factor analysis (CFA), correlational analyses, and tests of measurement invariance were deployed.

RESULTS

In total, 330 participants were included in the statistical analyses. CFA revealed that the 2-factor model reached an excellent model fit (comparative fit index=1.00, Tucker-Lewis index=0.99, root mean square error of approximation=0.036, standardized root mean square residual=0.038). Convergent validity was confirmed by significant positive correlations between eHL and knowledge of internet-based health promotion programs, experience in using these programs, and the duration of private internet use. In addition, a significantly negative relationship of eHL with internet anxiety supported convergent validity. Further, significant relationships of eHL with mental health status and internal health locus of control confirmed the criterion validity of the instrument. However, relationships of eHL with physical health status and quality of life could not be confirmed. The 2-factor model was fully measurement invariant regarding gender. Regarding age and educational level, partial measurement invariance was confirmed. The subscales as well as the overall GR-eHEALS reached good-to-excellent reliability (Cronbach α≥.86).

CONCLUSIONS

The GR-eHEALS is a reliable and largely valid instrument for assessing eHL in individuals with the post-COVID-19 condition. Measurement invariance regarding gender was fully confirmed and allows the interpretation of group differences. Regarding age and educational level, group differences should be interpreted with caution. Given the high likelihood that individuals with the post-COVID-19 condition will be confronted with misinformation on the Internet, eHL is a core competency that is highly relevant in this context, in both research and clinical practice. Therefore, future research should also explore alternative instruments to capture eHL to overcome shortcomings in the validity of the GR-eHEALS.

Assessing Healthcare Simulation Facilitation: A Scoping Review of Available Tools, Validity Evidence, and Context Suitability for Faculty Development in Low-Resource Settings

SUMMARY STATEMENT

Assessment tools support simulation facilitation skill development by guiding practice, structuring feedback, and promoting reflective learning among educators. This scoping review followed a systematic process to identify facilitation assessment tools used in postlicensure healthcare simulation. Secondary objectives included mapping of the validity evidence to support their use and a critical appraisal of their suitability for simulation faculty development in low-resource settings. Database searching, gray literature searching, and stakeholder engagement identified 11,568 sources for screening, of which 72 met criteria for full text review. Thirty sources met inclusion; 16 unique tools were identified. Tools exclusively originated from simulation practice in high-resource settings and predominantly focused on debriefing. Many tools have limited validity evidence supporting their use. In particular, the validity evidence supporting the extrapolation and implications of assessment is lacking. No current tool has high context suitability for use in low-resource settings.

The use of positive deviance approach to improve health service delivery and quality of care: a scoping review

BACKGROUND

Quality has been a persistent challenge in the healthcare system, particularly in resource-limited settings. As a result, the utilization of innovative approaches is required to help countries in their efforts to enhance the quality of healthcare. The positive deviance (PD) approach is an innovative approach that can be utilized to improve healthcare quality. The approach assumes that solutions to problems are already available within the community and identifying and sharing those solutions can help others to resolve existing issues. Therefore, this scoping review aimed to synthesize the evidence regarding the use of the PD approach in healthcare system service delivery and quality improvement programs.

METHODS

Articles were retrieved from six international databases. The last date for article search was June 02, 2023, and no date restriction was applied. All articles were assessed for inclusion through a title and/or abstract read. Then, articles that passed the title and abstract review were screened by reading their full texts. In case of duplication, only the full-text published articles were retained. A descriptive mapping and evidence synthesis was done to present data with the guide of the Preferred Reporting Items for Systematic Reviews and Meta-analysis extension for Scoping Reviews checklist and the results are presented in text, table, and figure formats.

RESULTS

A total of 125 articles were included in this scoping review. More than half, 66 (52.8%), of the articles were from the United States, 11(8.8%) from multinational studies, 10 (8%) from Canada, 8 (6.4%) from the United Kingdom and the remaining, 30 (24%) are from other nations around the world. The scoping review indicates that several types of study designs can be applied in utilizing the PD approach for healthcare service and quality improvement programs. However, although validated performance measures are utilized to identify positive deviants (PDs) in many of the articles, some of the selection criteria utilized by authors lack clarity and are subject to potential bias. In addition, several limitations have been mentioned in the articles including issues in operationalizing PD, focus on leaders and senior managers and limited staff involvement, bias, lack of comparison, limited setting, and issues in generalizability/transferability of results from prospects perspective. Nevertheless, the limitations identified are potentially manageable and can be contextually resolved depending on the nature of the study. Furthermore, PD has been successfully employed in healthcare service and quality improvement programs including in increasing surgical care quality, hand hygiene practice, and reducing healthcare-associated infections.

CONCLUSION

The scoping review findings have indicated that healthcare systems have been able to enhance quality, reduce errors, and improve patient outcomes by identifying lessons from those who exhibit exceptional practices and implementing successful strategies in their practice. All the outcomes of PD-based research, however, are dependent on the first step of identifying true PDs. Hence, it is critical that PDs are identified using objective and validated measures of performance as failure to identify true PDs can subsequently lead to failure in identifying best practices for learning and dissemination to other contextually similar settings.

Consensus on the definition and assessment of external validity of randomized controlled trials: A Delphi study

External validity is an important parameter that needs to be considered for decision making in health research, but no widely accepted measurement tool for the assessment of external validity of randomized controlled trials (RCTs) exists. One of the most limiting factors for creating such a tool is probably the substantial heterogeneity and lack of consensus in this field. The objective of this study was to reach consensus on a definition of external validity and on criteria to assess the external validity of RCTs included in systematic reviews. A three-round online Delphi study was conducted. The development of the Delphi survey was based on findings from a previous systematic review. Potential panelists were identified through a comprehensive web search. Consensus was reached when at least 67% of the panelists agreed to a proposal. Eighty-four panelists from different countries and various disciplines participated in at least one round of this study. Consensus was reached on the definition of external validity ("External validity is the extent to which results of trials provide an acceptable basis for generalization to other circumstances such as variations in populations, settings, interventions, outcomes, or other relevant contextual factors"), and on 14 criteria to assess the external validity of RCTs in systematic reviews. The results of this Delphi study provide a consensus-based reference standard for future tool development. Future research should focus on adapting, pilot testing, and validating these criteria to develop measurement tools for the assessment of external validity.

Evaluating the prerequisites for adapting a paediatric nighttime telemedicine and medication delivery service to a setting with high malarial burden: A cross-sectional pre-implementation study

OBJECTIVE

We sought to evaluate the prerequisites (demand, interest, feasibility) for adapting a paediatric nighttime telemedicine and medication delivery service (TMDS) to Ghana.

METHODS

A cross-sectional survey of households and associated healthcare providers was conducted in urban and rural Ghana. Households were identified through randomised geospatial sampling; households with at least one child <10 years were enrolled. Household surveys collected information relating to demographics, household resources, standardised case scenarios, recent paediatric health events, satisfaction with healthcare access, and interest in TMDS intervention models. Providers were identified by households and enrolled. Provider surveys collected provider type, hours of operation, services, and opinions of a TMDS model.

RESULTS

A total of 511 (263 urban, 248 rural) households and 18 providers (10 urban, 8 rural) were surveyed. A total of 262 health events involving children <10 years were reported, of which 47% occurred at night. Care was sought for >70% of health events presenting at night; however, care-seeking was delayed until morning or later for >75% of these events; 54% of households expressed dissatisfaction with their current access to paediatric care at night; 99% of households expressed that a nighttime TMDS service for children would be directly useful to their families. Correspondingly, 17 of 18 providers stated that a TMDS was needed in their community; >99% of households had access to a cellular phone. All households expressed willingness to use their phones to call a TMDS and allow a TMDS provider into their homes at night. Willingness to pay and provider-recommended price points varied by setting.

CONCLUSIONS

Prerequisites for adapting a TMDS to Ghana were met. A nighttime paediatric TMDS service was found to be needed, appealing, and feasible in Ghana. These data motivate the adaptation of a TMDS to urban and rural Ghana.

Underreporting contextual factors preclude the applicability appraisal in primary care randomized controlled trials

OBJECTIVES

To assess applicability reporting in randomized controlled trials (RCTs) conducted in primary care (PC).

STUDY DESIGN AND SETTING

We used a random sample of PC RCTs published between 2000 and 2020 to assess applicability. We extracted data related to setting, population, intervention (including implementation), comparator, outcomes, and context. Based on data availability, we assessed whether the five predefined applicability questions were adequately addressed by each PC RCT.

RESULTS

Adequately described elements that were reported frequently (>50%) included the responsible organization for intervention provision (97, 93.3%), study population characteristics (94, 90.4%), intervention implementation including monitoring and evaluation (92, 88.5%), intervention components (89, 85.6%), time frame (82, 78.8%), baseline prevalence (58, 55.8%), and the type of setting and location (53, 51%). Elements that were often underreported included contextual factors, that is, evidence of differential effects across sociodemographic or other groupings (2, 1.9%), intervention components tailored for specific settings (7, 6.7%), health system structure (32, 30.8%), factors affecting implementation (40, 38.5%) and organization structure (50, 48.1%). The proportion of trials that adequately addressed each applicability question ranged between 1% and 20.2%, while none RCT could address all of them.

CONCLUSION

Underreporting contextual factors jeopardize the appraisal of applicability in PC RCTs.

The Methodological Quality of Studies Investigating the Acute Effects of Exercise During Hypoxia Over the Past 40 years: A Systematic Review

Exercise under hypoxia and the physiological impact compared to normoxia or hypoxia has gained attention in the last decades. However, methodological quality assessment of articles in this area is lacking in the literature. Therefore, this article aimed to evaluate the methodologic quality of trials studying exercise under hypoxia. An electronic search was conducted until December 2021. The search was conducted in PubMed, CENTRAL, and PEDro using the PICO model. (P) Participants had to be healthy, (I) exercise under normobaric or hypobaric hypoxia had to be (C) compared to exercise in normoxia or hypoxia on (O) any physiological outcome. The 11-item PEDro scale was used to assess the methodological quality (internal validity) of the studies. A linear regression model was used to evaluate the evolution of trials in this area, using the total PEDro score of the rated trials. A total of n = 81 studies met the inclusion criteria and were processed in this study. With a mean score of 5.1 ± 0.9 between the years 1982 and 2021, the mean methodological quality can be described as “fair.” Only one study reached the highest score of 8/10, and n = 2 studies reached the lowest observed value of 3/10. The linear regression showed an increase of the PEDro score of 0.1 points per decade. A positive and small tendency toward increased methodologic quality was observed. The current results demonstrate that a positive and small tendency can be seen for the increase in the methodological quality in the field of exercise science under hypoxia. A “good” methodological quality, reaching a PEDro score of 6 points can be expected in the year 2063, using a linear regression model analysis. To accelerate this process, future research should ensure that methodological quality criteria are already included during the planning phase of a study.

Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

Background

Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods

A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results

38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence).

Conclusions

Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial registration

Prospective registration at Open Science Framework (OSF): 10.17605/OSF.IO/PTG4D.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01561-5.

The use of cognitive task analysis in clinical and health services research — a systematic review

Background

At times, clinical case complexity and different types of uncertainty present challenges to less experienced clinicians or the naive application of clinical guidelines where this may not be appropriate. Cognitive task analysis (CTA) methods are used to elicit, document and transfer tacit knowledge about how experts make decisions.

Methods

We conducted a methodological review to describe the use of CTA methods in understanding expert clinical decision-making. We searched MEDLINE, EMBASE and PsycINFO from inception to 2019 for primary research studies which described the use of CTA methods to understand how qualified clinicians made clinical decisions in real-world clinical settings.

Results

We included 81 articles (80 unique studies) from 13 countries, published from 1993 to 2019, most commonly from surgical and critical care settings. The most common aims were to understand expert decision-making in particular clinical scenarios, using expert decision-making in the development of training programmes, understanding whether decision support tools were warranted and understanding procedural variability and error identification or reduction. Critical decision method (CDM) and CTA interviews were most frequently used, with hierarchical task analysis, task knowledge structures, think-aloud protocols and other methods less commonly used. Studies used interviews, observation, think-aloud exercises, surveys, focus groups and a range of more CTA-specific methodologies such as the systematic human error reduction and prediction approach. Researchers used CTA methods to investigate routine/typical (n = 64), challenging (n = 13) or more uncommon, rare events and anomalies (n = 3).

Conclusions

In conclusion, the elicitation of expert tacit knowledge using CTA has seen increasing use in clinical specialties working under challenging time pressures, complexity and uncertainty. CTA methods have great potential in the development, refinement, modification or adaptation of complex interventions, clinical protocols and practice guidelines.

Registration

PROSPERO ID CRD42019128418.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-01002-6.

Assessing transferability in systematic reviews of health economic evaluations – a review of methodological guidance

Objective

For assessing cost-effectiveness, Health Technology Assessment (HTA) organisations may use primary economic evaluations (P-HEs) or Systematic Reviews of Health Economic evaluations (SR-HEs). A prerequisite for meaningful results of SR-HEs is that the results from existing P-HEs are transferable to the decision context (e.g, HTA jurisdiction). A particularly pertinent issue is the high variability of costs and resource needs across jurisdictions. Our objective was to review the methods documents of HTA organisations and compare their recommendations on considering transferability in SR-HE.

Methods

We systematically hand searched the webpages of 158 HTA organisations for relevant methods documents from 8th January to 31st March 2019. Two independent reviewers performed searches and selected documents according to pre-defined criteria. One reviewer extracted data in standardised and piloted tables and a second reviewer checked them for accuracy. We synthesised data using tabulations and in a narrative way.

Results

We identified 155 potentially relevant documents from 63 HTA organisations. Of these, 7 were included in the synthesis. The included organisations have different aims when preparing a SR-HE (e.g. to determine the need for conducting their own P-HE). The recommendations vary regarding the underlying terminology (e.g. transferability/generalisability), the assessment approaches (e.g. structure), the assessment criteria and the integration in the review process.

Conclusion

Only few HTA organisations address the assessment of transferability in their methodological recommendations for SR-HEs. Transferability considerations are related to different purposes. The assessment concepts and criteria are heterogeneous. Developing standards to consider transferability in SR-HEs is desirable.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01536-6.