1
|
Zhao L, Shen C, Liu M, Zhang J, Cheng L, Li Y, Yuan L, Zhang J, Tian J. Comparison of Reporting and Transparency in Published Protocols and Publications in Umbrella Reviews: Scoping Review. J Med Internet Res 2023; 25:e43299. [PMID: 37531172 PMCID: PMC10433027 DOI: 10.2196/43299] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2022] [Revised: 02/15/2023] [Accepted: 05/05/2023] [Indexed: 08/03/2023] Open
Abstract
BACKGROUND Inconsistencies between a protocol and its umbrella review (UR) may mislead readers about the importance of findings or lead to false-positive results. Furthermore, not documenting and explaining inconsistencies in the UR could reduce its transparency. To our knowledge, no study has examined the methodological consistency of the protocols with their URs and assessed the transparency of the URs when generating evidence. OBJECTIVE This study aimed to investigate the inconsistency of protocols with their URs in the methodology and assess the transparency of the URs. METHODS We searched medical-related electronic databases from their inception to January 1, 2022. We investigated inconsistencies between protocols and their publications and transparencies in the search strategy, inclusion criteria, methods of screening and data extraction, quality assessment, and statistical analysis. RESULTS We included 31 protocols and 35 publications. For the search strategy, 39 inconsistencies between the protocols and their publications were found in 26 of the 35 (74%) URs, and 16 of these inconsistencies were indicated and explained. There were 84 inconsistencies between the protocols and their URs regarding the inclusion criteria in 31 of the 35 (89%) URs, and 29 of the inconsistencies were indicated and explained. Deviations from their protocols were found in 12 of the 32 (38%) URs reporting the methods of screening, 14 of the 30 (47%) URs reporting the methods of data extraction, and 11 of the 32 (34%) URs reporting the methods for quality assessment. Of the 35 URs, 6 (17%) were inconsistent with their protocols in terms of the tools for quality assessment; one-half (3/6, 50%) of them indicated and explained the deviations. As for the statistical analysis, 31 of the 35 (89%) URs generated 61 inconsistencies between the publications and their protocols, and 16 inconsistencies were indicated and explained. CONCLUSIONS There was a high prevalence of inconsistencies between protocols and publications of URs, and more than one-half of the inconsistencies were not indicated and explained in the publications. Therefore, how to promote the transparency of URs will be a major part of future work.
Collapse
Affiliation(s)
- Liang Zhao
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Caiyi Shen
- The First Clinical Medical College, Lanzhou University, Lanzhou, China
| | - Ming Liu
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Jiaoyan Zhang
- Evidence-Based Nursing Center, School of Nursing, Lanzhou University, Lanzhou, China
| | - Luying Cheng
- Evidence-Based Nursing Center, School of Nursing, Lanzhou University, Lanzhou, China
- Zigong First People's Hospital, Zigong, China
| | - Yuanyuan Li
- Evidence-Based Nursing Center, School of Nursing, Lanzhou University, Lanzhou, China
| | - Lanbin Yuan
- The First Clinical Medical College, Lanzhou University, Lanzhou, China
| | - Junhua Zhang
- Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine, Tianjin, China
| | - Jinhui Tian
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| |
Collapse
|
2
|
Webbe J, Allin B, Knight M, Modi N, Gale C. How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development. Trials 2023; 24:345. [PMID: 37217933 PMCID: PMC10201748 DOI: 10.1186/s13063-023-07285-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Accepted: 03/29/2023] [Indexed: 05/24/2023] Open
Abstract
BACKGROUND Core outcomes sets are increasingly used to define research outcomes that are most important for a condition. Different consensus methods are used in the development of core outcomes sets; the most common is the Delphi process. Delphi methodology is increasingly standardised for core outcomes set development, but uncertainties remain. We aimed to empirically test how the use of different summary statistics and consensus criteria impact Delphi process results. METHODS Results from two unrelated child health Delphi processes were analysed. Outcomes were ranked by mean, median, or rate of exceedance, and then pairwise comparisons were undertaken to analyse whether the rankings were similar. The correlation coefficient for each comparison was calculated, and Bland-Altman plots produced. Youden's index was used to assess how well the outcomes ranked highest by each summary statistic matched the final core outcomes sets. Consensus criteria identified in a review of published Delphi processes were applied to the results of the two child-health Delphi processes. The size of the consensus sets produced by different criteria was compared, and Youden's index was used to assess how well the outcomes that met different criteria matched the final core outcomes sets. RESULTS Pairwise comparisons of different summary statistics produced similar correlation coefficients. Bland-Altman plots showed that comparisons involving ranked medians had wider variation in the ranking. No difference in Youden's index for the summary statistics was found. Different consensus criteria produced widely different sets of consensus outcomes (range: 5-44 included outcomes). They also showed differing abilities to identify core outcomes (Youden's index range: 0.32-0.92). The choice of consensus criteria had a large impact on Delphi results. DISCUSSION The use of different summary statistics is unlikely to affect how outcomes are ranked during a Delphi process: mean, median, and rates of exceedance produce similar results. Different consensus criteria have a large impact on resultant consensus outcomes and potentially on subsequent core outcomes sets: our results confirm the importance of adhering to pre-specified consensus criteria.
Collapse
Affiliation(s)
- James Webbe
- Section of Neonatal Medicine, School of Public Health, Chelsea and Westminster Hospital Campus, Imperial College London, 369 Fulham Road, London, SW10 9NX, UK.
| | - Benjamin Allin
- National Perinatal Epidemiology Unit, University of Oxford, Oxford, UK
| | - Marian Knight
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
| | - Neena Modi
- Section of Neonatal Medicine, School of Public Health, Chelsea and Westminster Hospital Campus, Imperial College London, 369 Fulham Road, London, SW10 9NX, UK
| | - Chris Gale
- Section of Neonatal Medicine, School of Public Health, Chelsea and Westminster Hospital Campus, Imperial College London, 369 Fulham Road, London, SW10 9NX, UK
| |
Collapse
|
3
|
Lei R, Shen Q, Yang B, Hou T, Liu H, Luo X, Li Y, Zhang J, Norris SL, Chen Y. Core Outcome Sets in Child Health: A Systematic Review. JAMA Pediatr 2022; 176:1131-1141. [PMID: 36094597 DOI: 10.1001/jamapediatrics.2022.3181] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
Abstract
IMPORTANCE Developing core outcome sets is essential to ensure that results of clinical trials are comparable and useful. A number of core outcome sets in pediatrics have been published, but a comprehensive in-depth understanding of core outcome sets in this field is lacking. OBJECTIVE To systematically identify core outcome sets in child health, collate the diseases to which core outcome sets have been applied, describe the methods used for development and stakeholder participation, and evaluate the methodological quality of existing core outcome sets. EVIDENCE REVIEW MEDLINE, SCOPUS, Cochrane Library, and CINAHL were searched using relevant search terms, such as clinical trials, core outcome, and children, along with relevant websites, such as Core Outcome Measures in Effectiveness Trials (COMET). Four researchers worked in teams of 2, performed literature screening and data extraction, and evaluated the methodological quality of core outcome sets using the Core Outcome Set-Standards for Development (COS-STAD). FINDINGS A total of 77 pediatric core outcome sets were identified, mainly developed by organizations or researchers in Europe, North America, and Australia and mostly from the UK (22 [29%]) and the US (22 [29%]). A total of 77 conditions were addressed; the most frequent International Classification of Diseases, 11th Revision category was diseases of the digestive system (14 [18%]). Most of the outcomes in pediatric core outcome sets were unordered (34 [44%]) or presented in custom classifications (29 [38%]). Core outcome sets used 1 or more of 8 development methods; the most frequent combination of methods was systematic review/literature review/scoping review, together with the Delphi approach and consensus for decision-making (10 [14%]). Among the 6 main types of stakeholders, clinical experts were the most frequently involved (74 [100%]), while industry representatives were rarely involved (4 [5%]). Only 6 core outcome sets (8%) met the 12 criteria of COS-STAD. CONCLUSIONS AND RELEVANCE Future quality of pediatric core outcome sets should be improved based on the standards proposed by the COMET initiative, while core outcome sets methodology and reporting standards should be extended to pediatric populations to help improve the quality of core outcome sets in child health. In addition, the COMET outcome taxonomy should also add items applicable to children.
Collapse
Affiliation(s)
- Ruobing Lei
- Chevidence Lab of Child and Adolescent Health, Children's Hospital of Chongqing Medical University, Chongqing, China.,National Clinical Research Center for Child Health and Disorders, Chongqing, China.,Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China.,China International Science and Technology Cooperation Base of Child Development and Critical Disorders, Chongqing, China.,Chongqing Key Laboratory of Pediatrics, Chongqing, China
| | - Quan Shen
- Chevidence Lab of Child and Adolescent Health, Children's Hospital of Chongqing Medical University, Chongqing, China.,National Clinical Research Center for Child Health and Disorders, Chongqing, China.,Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China.,China International Science and Technology Cooperation Base of Child Development and Critical Disorders, Chongqing, China.,Chongqing Key Laboratory of Pediatrics, Chongqing, China
| | - Bo Yang
- Shapingba District Center for Disease Control and Prevention of Chongqing, Chongqing, China
| | - Tianchun Hou
- Chevidence Lab of Child and Adolescent Health, Children's Hospital of Chongqing Medical University, Chongqing, China.,National Clinical Research Center for Child Health and Disorders, Chongqing, China.,Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China.,China International Science and Technology Cooperation Base of Child Development and Critical Disorders, Chongqing, China.,Chongqing Key Laboratory of Pediatrics, Chongqing, China
| | - Hui Liu
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Xufei Luo
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Yuehuan Li
- Department of Cardiac Surgery, Beijing Anzhen Hospital, Capital Medical University, Beijing, China
| | - Junhua Zhang
- Evidence-Based Medicine Center, Tianjin University of Traditional Chinese Medicine, Tianjin, China
| | | | - Yaolong Chen
- Chevidence Lab of Child and Adolescent Health, Children's Hospital of Chongqing Medical University, Chongqing, China.,Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China.,WHO Collaborating Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
| |
Collapse
|
4
|
Lei R, Li Y, Chen Y. Re: Katharina Beyer, Lisa Moris, Michael Lardas, et al. Updating and Integrating Core Outcome Sets for Localised, Locally Advanced, Metastatic, and Nonmetastatic Castration-resistant Prostate Cancer: An Update from the PIONEER Consortium. Eur Urol 2022;81:503–14. Eur Urol 2022; 82:e67. [DOI: 10.1016/j.eururo.2022.04.036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2022] [Accepted: 04/25/2022] [Indexed: 12/01/2022]
|
5
|
The standards of obstetrics and gynecology core outcome sets: A scoping review. Integr Med Res 2021; 11:100776. [PMID: 34745879 PMCID: PMC8551850 DOI: 10.1016/j.imr.2021.100776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2021] [Revised: 09/07/2021] [Accepted: 09/09/2021] [Indexed: 11/23/2022] Open
Abstract
Background Core outcome sets (COSs) are the minimum outcomes which should be measured and reported by researchers investigating a specific condition. The definition of standards of COSs vary across different health-related areas. This investigated the characteristics of COSs regarding obstetrics and gynecology (OG) and examined the reports and designs of standards of OG COSs. Methods A comprehensive search was conduced on the COMET database on December 20, 2019 to identify systematic reviews on COSs. Two reviewers independently evaluated whether the reported OG COS met the reporting requirements as stipulated in the Core Outcome Set-STAndards for Reporting (COS-STAR) statement checklist and the minimum design recommendations as outlined in the Core Outcome Set-STAndards for Development (COS-STAD) checklist. Results Forty-four OG COSs related to 26 topics were identified. None of them met all the 25 standards of COS-STAR statement which representing 18 items considered essential for transparent and complete reporting list for all COS studies (range: 6.0-24.0, median: 14.0). The compliance rates to 16 standards of methods and result sections ranged from 27.3%–68.2%. Total COS-STAR compliance items for OG COSs with the prior protocol was significantly higher than without prior protocol (MD = 3.846, 95% CI: 0.835–6.858, P = 0.012). None of the OG COSs met all the 12 criteria in the COS-STAD minimum standards (range: 3.0-11.0, median: 5.0). The compliance rates for all three standards of stakeholders involved and all four standards of the consensus process were lower than 60%. Conclusions Methodological and reporting standards of OG COSs should be improved.
Collapse
|