What feedback do reviewers give when reviewing qualitative manuscripts? A focused mapping review and synthesis

Librarians and information specialists as methodological peer-reviewers: a case-study of the International Journal of Health Governance

BACKGROUND

Objectives of this study were to analyze the impact of including librarians and information specialist as methodological peer-reviewers. We sought to determine if and how librarians' comments differed from subject peer-reviewers'; whether there were differences in the implementation of their recommendations; how this impacted editorial decision-making; and the perceived utility of librarian peer-review by librarians and authors.

METHODS

We used a mixed method approach, conducting a qualitative analysis of reviewer reports, author replies and editors' decisions of submissions to the International Journal of Health Governance. Our content analysis categorized 16 thematic areas, so that methodological and subject peer-reviewers' comments, decisions and rejection rates could be compared. Categories were based on the standard areas covered in peer-review (e.g., title, originality, etc.) as well as additional in-depth categories relating to the methodology (e.g., search strategy, reporting guidelines, etc.). We developed and used criteria to judge reviewers' perspectives and code their comments. We conducted two online multiple-choice surveys which were qualitatively analyzed: one of methodological peer-reviewers' perceptions of peer-reviewing, the other of published authors' views on the suggested revisions.

RESULTS

Methodological peer-reviewers assessed 13 literature reviews submitted between September 2020 and March 2023. 55 reviewer reports were collected: 25 from methodological peer-reviewers, 30 from subject peer-reviewers (mean: 4.2 reviews per manuscript). Methodological peer-reviewers made more comments on methodologies, with authors more likely to implement their changes (52 of 65 changes, vs. 51 of 82 by subject peer-reviewers); they were also more likely to reject submissions (seven vs. four times, respectively). Where there were differences in recommendations to editors, journal editors were more likely to follow methodological peer-reviewers (nine vs. three times, respectively). The survey of published authors (87.5% response rate) revealed four of seven found comments on methodologies helpful. Librarians' survey responses (66.5% response rate) revealed those who conducted peer-reviews felt they improved quality of publications.

CONCLUSIONS

Librarians can enhance evidence synthesis publications by ensuring methodologies have been conducted and reported appropriately. Their recommendations helped authors revise submissions and facilitated editorial decision-making. Further research could determine if sharing reviews with subject peer-reviewers and journal editors could benefit them in better understanding of evidence synthesis methodologies.

A framework for selecting data generation strategies in qualitative health research studies

BACKGROUND

Qualitative health research has the potential to answer important applied health research questions to inform nutrition and dietetics practice, education and policy. Qualitative health research is a distinct subdiscipline of qualitative inquiry that purposefully draws upon the context of healthcare and emphasises health and wellness.

METHODS

Qualitative health research is defined by two parameters: (1) the focus of the study and (2) the methods used. When considering the methods to be used, decisions are required about the type of data to be generated (e.g., transcripts, images and notes) and the process involved in data generation (e.g., interviews, elicitation strategies and observations) to answer the research question(s). Drawing upon examples from nutrition and dietetics literature, this paper provides a framework to support decision-making for nutrition and dietetics researchers and clinician researchers designing conducting qualitative health research.

RESULTS

The guiding questions of the framework include: What types of data will be generated? Who is involved in data generation? Where will data generation occur? When will data generation occur? How will data be recorded and managed? and How will participants' and researchers' emotional safety be promoted?

CONCLUSION

Questions about the types of data, those involved, where and when, as well as how safety can be maintained in data generation, not only support a more robust design and description of data generation methods but also keep the person at the centre of the research.

[DNVF Memorandum: Health Services Research in the Last Year of Life]

This memorandum outlines current issues concerning health services research on seriously ill and dying people in the last year of their lives as well as support available for their relatives. Patients in the last phase of life can belong to different disease groups, they may have special characteristics (e. g., people with cognitive and complex impairments, economic disadvantage or migration background) and be in certain phases of life (e. g., parents of minor children, (old) age). The need for a designated memorandum on health services research in the last year of life results from the special situation of those affected and from the special features of health services in this phase of life. With reference to these special features, this memorandum describes methodological and ethical specifics as well as current issues in health services research and how these can be adequately addressed using quantitative, qualitative and mixed methods. It has been developed by the palliative medicine section of the German Network for Health Services Research (DNVF) according to the guidelines for DNVF memoranda.

Qualitative health research in the fields of developmental medicine and child neurology

This invited review introduces the principles of qualitative health research (QHR) to the fields of developmental medicine and child neurology to facilitate the conduct of applied qualitative research. It provides practical guidance on how to write a study purpose statement aligned with the foci of QHR and then articulate an overarching research question using the Emphasis-Purposeful sample-Phenomenon of interest-Context framework. Guidance for health researchers on how to select a study design that aligns with the practice, education, or policy goals of applied QHR is provided. This is followed by strategies to guide decision-making with respect to purposeful sampling, selecting data collection methods, and identifying the most appropriate analytic approach to code and synthesize the data. Findings from QHR studies can be used conceptually or instrumentally to provide new insights or inform decisions within the discipline of developmental medicine and child neurology. While qualitive findings are increasingly valued in the field, designing studies that demonstrate methodological congruence is one strategy to improve the overall quality and trustworthiness of discipline specific QHR.

Measuring the developmental function of peer review: a multi-dimensional, cross-disciplinary analysis of peer review reports from 740 academic journals

Reviewers do not only help editors to screen manuscripts for publication in academic journals; they also serve to increase the rigor and value of manuscripts by constructive feedback. However, measuring this developmental function of peer review is difficult as it requires fine-grained data on reports and journals without any optimal benchmark. To fill this gap, we adapted a recently proposed quality assessment tool and tested it on a sample of 1.3 million reports submitted to 740 Elsevier journals in 2018-2020. Results showed that the developmental standards of peer review are shared across areas of research, yet with remarkable differences. Reports submitted to social science and economics journals show the highest developmental standards. Reports from junior reviewers, women and reviewers from Western Europe are generally more developmental than those from senior, men and reviewers working in academic institutions outside Western regions. Our findings suggest that increasing the standards of peer review at journals requires effort to assess interventions and measure practices with context-specific and multi-dimensional frameworks.

Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts’ content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers’ comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers’ comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers’ comments. Reviewers’ comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

The case for using mixed methods for designing, implementing, and disseminating evidence-based interventions for public health practice

A well-recognized gap exists between findings from public health research and their use in public health practice to improve outcomes. The World Health Organization (WHO) has identified implementation research as vital to improving the adaptation and dissemination of public health interventions into real-world settings. Implementation research encompasses multimethod evaluation approaches; 'mixed methods', the planned integration of qualitative and quantitative methods, is a key tool. We argue that mixed methods designs are crucial for design and evaluation of public health interventions, provide illustrative case studies, discuss key analytic approaches in mixed methods design, identify resources for mixed methods research and advocate for more training. On behalf of the International Network for Epidemiology in Policy (INEP), an international non-profit organization of 24 member societies that promotes the development of equitable, evidence-based health policies, we call for increased collaboration between qualitative and quantitative research teams to improve the design and evaluation of public health interventions.

A modest proposal to the peer review process: a collaborative and interdisciplinary approach in the assessment of scholarly communication

The purpose of the traditional peer review process (TPR) is to provide a more constructive and scientifically rigorous critical review of scholarly research that builds scientific rigor and validity within diverse academic disciplines. Peer review has received criticism as the demand for publications in a variety of competitive journals has significantly increased while the number of individuals who are both willing and qualified to conduct thorough reviews is significantly declining. The purpose of this topic piece is to examine the overall efficacy of the peer review process and provide recommendations toward a more collaborative, transparent (i.e. “open”), and interdisciplinary communication process.

Challenges in qualitative social pharmacy research: Reflections based on a conference workshop

A methodological debate within social pharmacy is ongoing regarding how to apply a qualitative approach. This paper emanates from a workshop at the Nordic Social Pharmacy Conference in 2019, named 'How do we know it's good? A workshop on quality criteria in qualitative social and clinical pharmacy research', that addressed this debate. The aim of this paper is twofold (1) to present the main key points raised during the workshop and (2) based on these inputs to contribute to the ongoing discussion on qualitative methodology within social pharmacy research. This paper starts with what was discussed at the workshop and further elaborated are some of the challenges with conducting qualitative research within social pharmacy. These include methodological and disciplinary competence and insecurity, reflections on the consequences of that many social pharmacy researchers come from a natural science background and how this (possibly) shapes the practice of qualitative research within the field. For example, how concepts like transparency and saturation, together with checklists and quality criteria are understood and used. Finally, we make suggestions for the next step for qualitative research in social pharmacy.

Lessons from an International Initiative to Set and Share Good Practice on Human Health in Environmental Impact Assessment

Environmental Impact Assessment (EIA) is applied to infrastructure and other large projects. The European Union EIA Directive (2011/92/EU as amended by 2014/52/EU) requires EIAs to consider the effects that a project might have on human health. The International Association for Impact Assessment and the European Public Health Association prepared a reference paper on public health in EIA to enable the health sector to contribute to this international requirement. We present lessons from this joint action. We review literature on policy analysis, impact assessment and Health Impact Assessment (HIA). We use findings from this review and from the consultation on the reference paper to consider how population and human health should be defined; how the health sector can participate in the EIA process; the relationship between EIA and HIA; what counts as evidence; when an effect should be considered ‘likely’ and ‘significant’; how changes in health should be reported; the risks from a business-as-usual coverage of human health in EIA; and finally competencies for conducting an assessment of human health. This article is relevant for health authorities seeking to ensure that infrastructure, and other aspects of development, are not deleterious to, but indeed improve, human health.

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

The global Covid-19 pandemic has had a considerable impact on the scientific enterprise, including scholarly publication and peer-review practices. Several studies have assessed these impacts, showing among others that medical journals have strongly accelerated their review processes for Covid-19-related content. This has raised questions and concerns regarding the quality of the review process and the standards to which manuscripts are held for publication. To address these questions, this study sets out to assess qualitative differences in review reports and editorial decision letters for Covid-19 related, articles not related to Covid-19 published during the 2020 pandemic, and articles published before the pandemic. It employs the open peer-review model at the British Medical Journal and eLife to study the content of review reports, editorial decisions, author responses, and open reader comments. It finds no clear differences between the review processes of articles not related to Covid-19 published during or before the pandemic. However, it does find notable diversity between Covid-19 and non-Covid-19-related articles, including fewer requests for additional experiments, more cooperative comments, and different suggestions to address too strong claims. In general, the findings suggest that both reviewers and journal editors implicitly and explicitly use different quality criteria to assess Covid-19-related manuscripts, hence transforming science’s main evaluation mechanism for their underlying studies and potentially affecting their public dissemination.