Medical education research study quality instrument: an objective instrument susceptible to subjectivity

BACKGROUND

The medical education research study quality instrument (MERSQI) was designed to appraise medical education research quality based on study design criteria. As with many such tools, application of the results may have unintended consequences. This study applied the MERSQI to published medical education research identified in a bibliometric analysis.

METHODS

A bibliometric analysis identified highly cited articles in medical education that two authors independently evaluated using the MERSQI. After screening duplicate or non-research articles, the authors reviewed 21 articles with the quality instrument. Initially, five articles were reviewed independently and results were compared to ensure agreed upon understanding of the instrument items. The remainder of the articles were independently reviewed. Overall scores for the articles were analyzed with a paired samples t-test and individual item ratings were analyzed for inter-rater reliability.

RESULTS

There was a significant difference in mean MERSQI score between reviewers. Inter-rater reliability for MERSQI items labeled response rate, validity and outcomes were considered unacceptable.

CONCLUSIONS

Based on these results there is evidence that MERSQI items can be significantly influenced by interpretation, which lead to a difference in scoring. The MERSQI is a useful guide for identifying research methodologies. However, it should not be used to make judgments on the overall quality of medical education research methodology in its current format. The authors make specific recommendations for how the instrument could be revised for greater clarity and accuracy.

We Need to Do Better: A Scoping Review of Wellness Programs In Surgery Residency

OBJECTIVE

Burnout, depression, and fatigue are common among surgical residents. Most published wellness studies in surgery only focus on a cross-sectional view of attitudes and perceptions around wellness in training. While much of this literature calls for interventions and presents strategies for improving resident well-being, there is a paucity of published wellness initiatives, and even fewer with programmatic evaluation.

DESIGN

A scoping review was designed to address: (1) What wellness initiatives are used in surgery residency programs? (2) Which wellness domains do these programs address? and (3) How are program outcomes evaluated? A formal literature search was conducted using PubMed, Embase, and Scopus databases to identify English-language studies conducted in the United States that described wellness-focused initiatives for surgery residents. Two authors independently screened all abstracts and full texts for inclusion. Data were extracted including wellness domain(s) and outcomes evaluation methods with associated Kirkpatrick level(s) (1-reaction, 2-learning, 3-behavior, 4-results). Study quality was examined using the medical education research study quality index (MERSQI) score.

RESULTS

A total of 2237 abstracts were screened with 115 full texts reviewed for eligibility. Fifty-one studies were included in the final analysis, representing 39 distinct wellness programs. The most common domains of wellness addressed were emotional (19/39, 48.7%), occupational (17/39, 43.6%), and physical (16/39, 41.0%). Of the 51 studies reviewed, 8 (15.7%) did not conduct any program evaluation, 27 (52.9%) evaluated level 1, 30 (58.8%) evaluated level 2, 3 (5.9%) evaluated level 3, and none evaluated level 4 outcomes. The mean MERSQI score was 9.16 (SD 1.8).

CONCLUSIONS

Wellness is an established problem in surgical training. This review reveals a small number of published wellness interventions and even fewer that incorporate programmatic evaluation at the level of behavior and results change. Effective change will require rigorous and deliberate programming that addresses multiple domains and evaluation levels.

Surgical endoscopy education research: how are we doing?

BACKGROUND

Surgical endoscopy (SE), the official journal of the Society of American Gastrointestinal and Endoscopic Surgeons and the European Association for Endoscopic Surgery, is an important source of new evidence pertaining to surgical education in the field. However, qualitative deficiencies in medical education research have prompted medical education leaders to advocate for increased methodological rigor. The purpose of this study is to review the quality of education-focused research published through SE.

METHODS

A PubMed search examining all SE articles categorized as education-related research from 2010 to 2019 was conducted; studies not meeting inclusion criteria were excluded. Remaining publications were independently reviewed, classified, and scored by 7 raters using the medical education research study quality instrument (MERSQI). Intraclass correlation was calculated and data were examined with descriptive statistics.

RESULTS

A total of 227 studies met inclusion criteria. There was no significant difference in number of publications by year (average 25.88 [SD 5.6]); 60% were conducted outside of the United States, and 47% (n = 106) were funded. The average MERSQI was 12.5 (SD 2). Most studies used two-group non-random (42%, n = 96) or post/cross-sectional designs (29%, n = 65). Thirty-six (16%) were randomized controlled trials. Multi-institutional studies comprised 24% (n = 54). Of the manuscripts, 96% (n = 217) reported at least one measure of validity evidence and 28% (n = 67) described three levels of validity evidence. Studies primarily reported changes in skills or knowledge (45%, n = 103) or satisfaction or general facts (44%, n = 99), while patient-related outcomes encompassed 3% (n = 6) of studies. ICC between raters was 0.93 (CI 0.90-0.93, p < 0.001).

CONCLUSIONS

Based on publications to date, this journal's peer review process appears to facilitate the dissemination of education-related studies of moderate to good quality. However, there were uncovered deficits, ranging from validity evidence to study designs and level of outcomes. This journal's breadth of viewership offers a potential venue to advance education-related research.

Tracking Surgical Education Survey Research Through the APDS Listserv

OBJECTIVE

Survey-based studies are cornerstones in medical education research. The Association of Program Directors in Surgery (APDS) listserv offers a method to contact program directors (PD) and residents for such research. To facilitate research beneficial to the APDS, improve the quality of survey-based research and minimize survey fatigue, the APDS research committee (ARC) developed a survey review process to grant access to the listserv for research. This study was conducted to determine the impact of the review process on the quality of survey-based research and eventual publication.

DESIGN

This log was systematically reviewed identifying publications resulting from accepted surveys. Publications were categorically analyzed to determine the components of their survey tool methodology, response rate (RR), and medical education research study quality instrument (MERSQI) score.

SETTING

The ARC used a 2-reviewer peer-review process for survey distribution requests. The request was either accepted, rejected, or returned for revision. Accepted surveys were distributed through the listserv with an ARC attestation of approval.

PARTICIPANTS

A log of all survey requests maintained from 2014 to 2017 and subsequent publications.

RESULTS

Thirty-five requests were accepted (40%), 30 were reviewed discovering 10 surveys that led to 12 publications (publication rate of 33%). The average RR was 60% (SD = 29%). Detailed explanations of survey development strategies were reported in 5 (42%), consisting of methods building validity evidence such as expert consensus, modified Delphi method, and pilot group sampling. Half of study participants were PD (50%). MERSQI scores averaged 10 (SD = 1.6).

CONCLUSION

Based on those survey research published to date, the ARC survey peer-review process has enabled most accepted surveys to achieve adequate RR. Although the pool of accepted requests is small, it does highlight areas of improvement. With further refinement of the process, including questioning the survey development methods, the process and listserv can be a powerful tool for further research.