Best practice versus actual practice: an audit of survey pretesting practices reported in a sample of medical education journals

Deloading Practices in Strength and Physique Sports: A Cross-sectional Survey

BACKGROUND

This study explored the deloading practices of competitive strength and physique athletes. A 55-item anonymised web-based survey was distributed to a convenience-based, cross-sectional sample of competitive strength and physique athletes (n = 246; males = 181 [73.6%], females = 65 [26.4%]; age = 29.5 ± 8.6 years) who had 8.2 ± 6.2 years of resistance training and 3.8 ± 3.1 years of competition experience.

RESULTS

All athletes deloaded within training with energy and fatigue management being the main reasons to do so. The typical duration of a deload was 6.4 ± 1.7 days, integrated into the training programme every 5.6 ± 2.3 weeks. Deloading was undertaken using a proactive, pre-planned strategy (or in combination with an autoregulated approach) and undertaken when performance stalled or during periods of increased muscle soreness or joint aches. Athletes reported that training volume would decrease (through a reduction in both repetitions per set and sets per week), but training frequency would remain unchanged during deloads. Additionally, athletes reported that training intensity (load lifted) would decrease, and effort would be reduced (facilitated through an increase in repetitions in reserve). Athletes would generally maintain the same exercise selection during deloading. For athletes that supplemented deloading with additional recovery modalities (n = 118; 48%), the most reported strategies were massage, static stretching and foam rolling.

CONCLUSION

Results from this research might assist strength and physique athletes and coaches to plan their deloading. Future research should empirically investigate the findings from this study to further evaluate the potential utility of deloading in strength and physique sports.

Nursing experiences in antimicrobial optimisation in the intensive care unit: A convergent analysis of a national survey

BACKGROUND

Recent evidence highlights the need for an interdisciplinary approach to antimicrobial stewardship (AMS). Nursing involvement in optimising antimicrobials in the intensive care unit (ICU) remains understudied.

OBJECTIVE

The objective of this study was to explore nurses' perceptions and experiences of antimicrobial optimisation or stewardship in ICUs in Australia.

METHODS

An anonymous web-based survey was deployed nationally in early 2021 through two ICU nursing networks. Associations between survey responses were analysed descriptively and by using nonparametric tests (with statistical significance established at p ≤ 0.05). Free-text survey responses underwent qualitative thematic analysis. Interpretation and reporting of quantitative and qualitative data were integrated.

RESULTS

A total of 226 ICU nurses completed the survey. The majority (197/226; 87%) responded that lack of education limits engagement in AMS. Only 13% (30/226) reported the presence of AMS education and training for nurses in their ICUs. Only about half (108/226; 48%) of the nurses were confident to question prescribers when they considered that the antimicrobial prescribed was unnecessary, with nurses in senior roles more likely to do so than nurses providing bedside care (p < 0.05). Gaps in education (including unfamiliarity with AMS roles), noninclusive antimicrobial discussions, moral distress, and potential workload burden were seen as potential barriers/challenges to engagement.

CONCLUSION

The multifactorial barriers identified that inhibit nurses from performing AMS tasks could be addressed by strengthening interprofessional education at all levels and by applying practical AMS interventions that are inclusive for nursing participation. A purposeful culture change that fosters psychological safety and collaborative practice is paramount to supporting nurses in these roles.

Web-Based Survey Piloting Process as a Model for Developing and Testing Past Contraceptive Use and Pregnancy History: Cystic Fibrosis Case Study

BACKGROUND

Individuals with complex, chronic diseases are now living longer, making reproductive health an important topic to address in the health care setting. Self-respondent surveys are a feasible way to collect past contraceptive use and pregnancy history to assess contraceptive safety and effectiveness. Because sensitive topics, such as contraception and pregnancy outcomes, can vary across social groups or cultures, piloting questions and evaluating survey administration procedures in the target population are necessary for precise and reliable responses before wide distribution.

OBJECTIVE

This study aimed to develop a precise and reliable survey instrument and related procedures among individuals with cystic fibrosis regarding contraceptive use and obstetrical history.

METHODS

We piloted and tested web-based questions related to contraceptive use and pregnancy history among 50 participants with and those without cystic fibrosis aged 18 to 45 years using a 3-tier process. Findings from each tier informed changes to the questionnaire before testing in the subsequent tier. Tier 1 used cognitive pretesting to assess question understanding and the need for memory prompts. In tier 2, we used test-retest self- and interviewer-administered approaches to assess question reliability, evaluate response missingness, and determine confidence between 2 types of survey administration approaches. In tier 3, we tested the questionnaire for clarity, time to complete, and whether additional prompts were necessary.

RESULTS

In tier 1, respondents suggested improvements to the web-based survey questions and to the written and visual prompts for better recall regarding past contraceptive use. In tier 2, the test-retest reliability between self- and interviewer-administrative procedures of "ever use" contraceptive method questions was similar, with percent absolute agreement ranging between 84% and 100%. When the survey was self-administered, the percentage of missing responses was higher and respondent confidence about month and year when contraceptive methods were used was lower. Most respondents reported that they preferred the self-administered survey because it was more convenient and faster to complete.

CONCLUSIONS

Our 3-tier process to pilot web-based survey questions related to contraceptive and obstetrical history in our complex disease population helped us tailor content and format questions before wide dissemination to our target population. Results from this pilot study informed the subsequent larger study design to include a 10% respondent test-retest self- and interviewer-administered quality control component to better inform imputation procedures of missing data.

Research Methodologies in Health Professions Education Publications: Breadth and Rigor

PURPOSE

Research methodologies represent assumptions about knowledge and ways of knowing. Diverse research methodologies and methodological standards for rigor are essential in shaping the collective set of knowledge in health professions education (HPE). Given this relationship between methodologies and knowledge, it is important to understand the breadth of research methodologies and their rigor in HPE research publications. However, there are limited studies examining these questions. This study synthesized current trends in methodologies and rigor in HPE papers to inform how evidence is gathered and collectively shapes knowledge in HPE.

METHOD

This descriptive quantitative study used stepwise stratified cluster random sampling to analyze 90 papers from 15 HPE journals published in 2018 and 2019. Using a research design codebook, the authors conducted group coding processes for fidelity, response process validity, and rater agreement; an index quantifying methodological rigor was developed and applied for each paper.

RESULTS

Over half of research methodologies were quantitative (51%), followed by qualitative (28%), and mixed methods (20%). No quantitative and mixed methods papers reported an epistemological approach. All qualitative papers that reported an epistemological approach (48%) used social constructivism. Most papers included participants from North America (49%) and Europe (20%). The majority of papers did not specify participant sampling strategies (56%) or a rationale for sample size (80%). Among those reported, most studies (81%) collected data within 1 year.The average rigor score of the papers was 56% (SD = 17). Rigor scores varied by journal categories and research methodologies. Rigor scores differed between general HPE journals and discipline-specific journals. Qualitative papers had significantly higher rigor scores than quantitative and mixed methods papers.

CONCLUSIONS

This review of methodological breadth and rigor in HPE papers raises awareness in addressing methodological gaps and calls for future research on how the authors shape the nature of knowledge in HPE.

Are sexual health survey items understood as intended by African and Asian migrants to Australia? Methods, results and recommendations for qualitative pretesting

INTRODUCTION

More research and policy action are needed to improve migrant health in areas such as sexual health and blood-borne viruses (SHBBV). While Knowledge, Attitudes and Practice Surveys (KAPS) can inform planning, there are no SHBBV KAPS suitable for use across culturally and linguistically diverse contexts. This study pretests one instrument among people born in Sub-Saharan Africa, South-East and North-East Asia living in Australia.

METHODS

Employees of multicultural organisations were trained to collect data over three rounds using a hybrid qualitative pretesting method. Two researchers independently coded data. Researchers made revisions to survey items after each round. Responses to feedback questions in the final survey were analysed.

RESULTS

Sixty-two participants pretested the survey. Issues were identified in all three rounds of pretesting. Of the 77 final survey respondents who responded to a survey experience question, 21% agreed and 3% strongly agreed with the statement 'I found it hard to understand some questions/words'.

CONCLUSION

It is essential to pretest SHBBV surveys in migrant contexts. We offer the following pretesting guidance: (1) large samples are needed in heterogeneous populations; (2) intersectionality must be considered; (3) it may be necessary to pretest English language surveys in the participants' first language; (4) bilingual/bicultural workers must be adequately trained to collect data; (5) results need to be interpreted in the context of other factors, including ethics and research aims; and (6) pretesting should occur over multiple rounds.

Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030

Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study’s findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.

What Is an Innovation Article? A Systematic Overview of Innovation in Health Professions Education Journals

PURPOSE

Innovation articles have their own submission category and guidelines in health professions education (HPE) journals, which suggests innovation might be a unique genre of scholarship. Yet, the requirements for innovation submissions vary among journals, suggesting ambiguity about the core content of this type of scholarship. To reduce this ambiguity, the researchers conducted a systematic overview to identify key features of innovation articles and evaluate their consistency in use across journals. Findings from this review may have implications for further development of innovation scholarship within HPE.

METHOD

In this systematic overview, conducted in 2020, the researchers identified 13 HPE journals with innovation-type articles and used content analysis to identify key features from author guidelines and publications describing what editors look for in innovation articles. The researchers then audited a sample of 39 innovation articles (3/journal) published in 2019 to determine presence and consistency of 12 innovation features within and across HPE journals. Audit findings informed the researchers' evaluation of innovation as a genre in HPE.

RESULTS

Findings show variability of innovation feature presence within and across journals. On average, articles included 7.8 of the 12 innovation features (SD 2.1, range 3-11). The most common features were description of: how the innovation was implemented (92%), a problem (90%), what was new or novel (79%), and data or outcomes (77%). On average, 5.5 (SD 1.5) out of 12 innovation features were consistently used in articles within each journal.

CONCLUSIONS

The authors identified common features of innovation article types based on journal guidelines, but there was variability in presence and consistency of these features, suggesting HPE innovations are in an emerging state of genre development. The authors discuss potential reasons for variability within this article type and highlight the need for further discussion among authors, editors, and reviewers to improve clarity.

A secondary analysis to develop a scale for measuring unemployed workers' experiences of Australian employment services

BACKGROUND

Research consistently shows that Australian employment services are failing those they are intended to serve. Based on findings in other human service areas, a valid and reliable instrument to measure unemployed workers' experiences may provide an opportunity for improvement in this sector.

OBJECTIVE

To establish a basis for developing a suitable rating scale.

METHODS

An exploratory factor analysis combined with qualitative cross check for face validity of an existing large survey of Australian unemployed workers.

RESULTS

Six factors appear to be important elements of service delivery: (F1) useful and competent, (F2) client-centred, (F3) receptive to feedback, (F4) trustworthy, (F5) fair, and (F6) friendlyCONCLUSIONS:While each of these factors have been either described explicitly or referred to implicitly in previous studies, this study is the first to attempt to combine these factors and is a precursor to establishing a valid and reliable rating scale for use by unemployed workers in evaluating their employment service providers. At a time when Australia is exploring new approaches to employment services, such a scale using a robust set of factors may allow for the improvement of employment services and thus be held accountable to a significant stakeholder group whom they aim to serve -unemployed workers.

An Examination of Resident Perspectives on Survey Participation and Methodology: Implications for Educational Practice and Research

BACKGROUND

In medical education, self-administered questionnaires are used to gather information for needs assessments, innovation projects, program evaluations, and research studies. Despite the importance of survey methodology, response rates have declined for years, especially for physicians.

OBJECTIVE

This study explored residents' experiences with survey participation and perceptions of survey design and implementation.

METHODS

In 2019, residents at a large Midwestern academic medical center were recruited via email to participate in mixed specialty focus groups (FGs). Narrative comments were recorded, transcribed, and then analyzed via conventional content analysis, utilizing cognitive sociology as a conceptual framework. Themes and subthemes were generated iteratively.

RESULTS

Postgraduate year 1-4 residents (n = 33) from internal medicine, surgery, and neurology participated in 7 FGs (3-7 participants/group) from April-May 2019. Eight themes were generated during content analysis: Negative emotions, professionalism, accuracy, impact, survey design/implementation, biases, survey fatigue, and anonymity. Residents questioned the accuracy of survey data, given the tendency for self-selection to drive survey participation. Residents wanted survey participation to be meaningful and reported non-participation for a variety of reasons, including doubts over impact. Satisficing and breakoffs were commonly reported.

CONCLUSIONS

Though residency program cultures differ across institutions, the findings from this study, including potential barriers to survey participation, should be relevant to anyone in graduate medical education using survey methodology for programmatic data collection, accreditation, and research purposes.

Questionnaire surveys - sources of error and implications for design, reporting and appraisal

Questionnaire surveys of dentists are a commonly used research method which can yield important results. Nevertheless, in order to give reliable information, surveys must be carefully designed to avoid bias. The challenge of planning, designing and implementing a questionnaire study is often underestimated. While headline response rate is commonly used as an indication of survey quality, there are four potential areas of survey error. These are coverage, sampling, measurement and response. These four areas of error are discussed, with examples from the current literature, to assist readers in critically appraising questionnaire studies. We make 12 key recommendations to researchers to help avoid bias in their research; approaches to response enhancement are also considered. Reporting guidelines for questionnaire studies are suggested which improve transparency and assist in understanding of research methodologies and results. We have termed these guidelines: Reporting Items for Surveys and Questionnaires (RISQ). In conclusion, there is no single solution that can guarantee a successful study. Conversely, failure to pay careful attention to any one of many crucial aspects of survey design can be the downfall of an otherwise robust study.