Automated Item Generation: impact of item variants on performance and standard setting

BACKGROUND

Automated Item Generation (AIG) uses computer software to create multiple items from a single question model. There is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each.

METHODS

Pre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four 'isomorphic' 50-item MCQ tests using AIG software. Isomorphic questions use the same question template with minor alterations to test the same learning outcome. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting).

RESULTS

Two thousand two hundred eighteen students from 12 UK medical schools participated, with each school using one of the four papers. The average facility of the four papers ranged from 0.55-0.61, and the cut score ranged from 0.58-0.61. Twenty item models had a facility difference > 0.15 and 10 item models had a difference in standard setting of > 0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility.

CONCLUSIONS

Item facility varied to a greater extent than the standard set. This difference may relate to variants causing greater disruption of clinical reasoning strategies in novice learners compared to experts, but is confounded by the possibility that the performance differences may be explained at school level and therefore warrants further study.

Proceedings of the 2021 American Burn Association State and Future of Burn Science Meeting

Periodically, the American Burn Association (ABA) has convened a State of the Science meeting on various topics representing multiple disciplines within burn care and research. In 2021 at the request of the ABA President, meeting development was guided by the ABA's Burn Science Advisory Panel (BSAP) and a subgroup of meeting chairs. The goal of the meeting was to produce both an evaluation of the current literature and ongoing studies, and to produce a research agenda and/or define subject matter-relevant next steps to advance the field(s). Members of the BSAP defined the topics to be addressed and subsequently solicited for nominations of expert speakers and topic leaders from the ABA's Research Committee. Current background literature for each topic was compiled by the meeting chairs and the library then enhanced by the invited topic and breakout discussion leaders. The meeting was held in New Orleans, LA on November 2nd and 3rd and was formatted to allow for 12 different topics, each with two subtopics, to be addressed. Topic leaders provided a brief overview of each topic to approximately 100 attendees, followed by expert-lead breakout sessions for each topic that allowed for focused discussion among subject matter experts and interested participants. The breakout and topic group leaders worked with the participants to determine research needs and associated next steps including white papers, reviews and in some cases collaborative grant proposals. Here, summaries from each topic area will be presented to highlight the main foci of discussion and associated conclusions.

The influence of candidates' physical attributes on patient ratings in simulated assessments of clinical practice

BACKGROUND

We have previously shown that clinical examiners' scoring is not negatively impacted when a candidate has a tattoo, unnatural hair colour, or a regional accent. We investigated whether these physical attributes in exam candidates impact patient scoring.

METHODS

Simulated/real patients were randomly assigned to watch five videos of simulated candidate performances of a cranial nerve examination: clear fail, borderline, good, 'clear pass' without an attribute, and 'clear pass' with one of the attributes (tattoo, purple hair, accent). Participants scored domains of communication and professionalism. We compared scores for the clear pass candidates with and without attributes.

RESULTS

One hundred and eighty three patients participated. The total scores for the candidates with tattoos and purple hair were higher than the candidate with no physical attribute (p < 0.001). For the candidate with a Liverpool English accent no difference was identified (p = 0.120).

CONCLUSIONS

The presence of certain physical attributes (tattoos or purple hair) was associated with higher scores given by patients to candidates in a simulated physical examination station.

Burn Survivors' Near-Death Experiences: A Qualitative Examination

Persons who come close to death but survive catastrophic accidents sometimes report very vivid experiences during times when their survival was in doubt, when they were believed to be dead, and during resuscitation efforts. This qualitative study builds upon existing research on near-death experiences (NDEs) by focusing on the oral accounts from a sample of individuals with large and life-threatening burns. The NDE accounts were obtained from burn survivors attending the Phoenix Society's World Burn Congress and are similar to reports by notable researchers ( Greyson, 2003 ; Moody, 1975 ; Ring, 1980 ) while reflecting the uniqueness of the individual survivor's experiences. Six major themes are reported. Counselors and health professionals need to be aware of and educated about NDEs as these experiences can have profound effects upon the individual. Patients who have had NDEs may need to discuss them but fear professionals will reject their stories as being crazy.

Near-death experiences, posttraumatic growth, and life satisfaction among burn survivors

Survivors of large burns may face positive and negative psychological after-effects from close-to-death injuries. This study is the first to examine their near-death experiences (NDEs) and posttraumatic growth (PTG) and life satisfaction afterwards. With an available sample of 92 burn survivors, half met the criteria for an NDE using an objective scale. Those who indicated religion was a source of strength and comfort had high scores on life satisfaction, PTG, and the NDE Scale. Individuals with larger burns reported greater PTG than those with smaller total body surface area burned (TBSA). There were no significant differences on life satisfaction, PTG, or NDEs when examined by gender or years since the burn injury. Elements of the NDE most frequently reported were: An altered sense of time, a sense of being out of the physical body, a feeling of peace, vivid sensations, and sense of being in an "other worldly" environment. Social workers and other health providers need to be comfortable helping burn survivors discuss any NDEs and process these through survivors' spirituality and religious belief systems as they recover.

Understanding the student veterans' college experience: an exploratory study

OBJECTIVE

Students with active duty military experience are a unique and growing population on college campuses in the United States. This study explores student veterans' perceptions of their transition to and experience in higher education.

METHOD

This mixed methods study used a sample of 10 active military and reserve component student veterans to explore their perceptions of their personal strengths, challenges, factors impacting participation in university resource programs, and suggestions for ideal resources to support their academic success.

RESULTS

Content analysis yielded primary themes such as the strength of self-discipline, the challenge of social interactions, and the desire for programs that connect student-veterans and assist with social integration.

CONCLUSION

Implications for education, retention, and transition from active duty are discussed.

Describing compassionate care: the burn survivor's perspective

Compassion is vital in burn care. Its delivery could be considered a professional duty and a characteristic of care that affects patient satisfaction. However, the description of compassionate care is underexplored in the burn care literature. This study investigates the concept of compassionate care and how it is described from the perspective of the burn survivor. A qualitative design with two focus groups at the Phoenix Society for Burn Survivors' World Burn Congress was used to examine views of compassionate care directly through 31 burn survivors' accounts to better understand the concept and its delivery within the context of burn care. Discussions were recorded and transcripts were analyzed for prominent themes and descriptive components. Participants were primarily Caucasian (77%), female (60%), with an average age of 47.6 years and an average TBSA burn of approximately 49% sustained approximately 12 years ago. Qualitative data analysis yielded primary themes of: 1) respect the person (subthemes were: establishing an empathic connection, restoring control through choice, providing individualized care, and going above and beyond), 2) communication (subthemes: interpersonal and informational), and 3) provision of competent care. The three primary themes were components of compassionate care; it was not defined by a single characteristic, behavior, or skill but might be best understood as the convergence of the three themes. Implications of findings and barriers to the provision of compassionate care are also discussed.

What's in a story? A text analysis of burn survivors' web-posted narratives

Story-telling has been found to be beneficial following trauma, suggesting a potential intervention for burn survivors who frequently make use of? telling their story? as part of their recovery. This study is the first to examine the word content of burn survivors' Web-posted narratives to explore their perceptions of the event, supportive resources, their post-burn well-being, and re-integration using a comparison group and a text data analysis software developed by the widely recognized James Pennebaker. Suggestions for using expressive writing or story-telling as a guided psychosocial intervention with burn survivors are made.

Adult burn survivors' views of peer support: a qualitative study

This qualitative study examined 30 burn survivors' perceptions of the value of peer support in their own psychosocial rehabilitation. Little research is available that investigates the role of peer support in post-burn recovery in terms of perceived benefits and costs. Findings revealed strong positive views regarding the helpfulness of peer support. Burn survivors reported that peer supporters provided a sense of belonging and affiliation and gave hope and confidence. Two-thirds of the sample had served as peer supporters themselves after receiving their injuries, suggesting that mutual aid does involve reaching out to others. At the same time, survivors spoke of possible costs in helping others. Involving peer supporters in the psychological rehabilitation of burn survivors may be an important complement to the medical team.

Helping others heal: burn survivors and peer support

Burns require psychosocial healing in addition to physical recovery as survivors face challenges such as scarring and altered appearance. An adjunct to interventions provided by social workers and other professionals is peer support from fellow burn survivors. But how do burn survivors view this intervention? This study examined a sample of burn survivors to learn about their views of peer support and the services of a national burn survivor organization. Outcome variables were explored relative to survivors' views of peer support to gauge its influence on psychosocial recovery. Findings suggest peer support is a promising resource in burn rehabilitation warranting further investigation.

Hospital social workers and indirect trauma exposure: an exploratory study of contributing factors

This article explores the predictive ability of empathy (measured by the Interpersonal Reactivity Index), emotional separation (measured by the Maintenance of Emotional Separation Scale), occupational stress (measured by the Work-Related Strain Inventory), and social support (measured by the Multidimensional Scale of Perceived Social Support) on secondary traumatic stress (STS) (measured by the Secondary Traumatic Stress Scale) in hospital social workers. This cross-sectional study used a sample of 121 trauma center social workers who were predominantly master's-level prepared women with an average of 15.8 years' experience. Emotional separation and occupational stress were the strongest predictors of STS, explaining 49 percent of the variance, which suggests that hospital social workers need to be able to emotionally differentiate during work with patients and families and manage organizational stressors to minimize indirect trauma reactions.

Age and gender differences in value orientation among American adolescents

This study investigated the development of a caring value orientation in four age groups. The sample included 1,247 students in grades 6, 8, 10, and 12 who were asked to complete a questionnaire, containing 14 contrasting word pairs and phrases, designed to measure preference for more caring, personal values. Gender differences were found across age groups and geographic regions. These findings are discussed in terms of theories that hypothesize gender differences in worldview.

The promise of automated wellness systems

Do automated wellness programs hold out the promise of improving the health of individuals and populations through the establishment of baseline wellness checkpoints? Nurse case managers have a strong interest in this question. The automation of wellness programs and the integration of this information into the continuum of a care information system model offers ongoing, interactive accountability of individual health status. The crowning glory will be the ultimate integration of the individual's personal health game plan with the provider's continuum of care offering. The strength of the nursing profession lies in its traditional role of promoting health through preventive measures (public health nursing) and managing the ill patient's progress toward his/her optimal health status. Therefore, nurses and nurse case managers should embrace wellness programs and wellness data availability. Because of their experience in using and understanding wellness data, nurses are in a unique position. Not only will they be able to assimilate rapidly the use of wellness data into their own practice, they also will be able to guide other professionals in its use. The table in this articles provides examples of the impact that automated wellness program data could have at various stages of the clinical decision-making process.

Information systems tools available to the case manager

Regardless of the setting, the role of the nurse as a manager of care continues to grow and develop. The nurse case manager needs to have vital information readily available to facilitate critical thinking, analysis, and decision-making on the appropriateness of care provided to patients and to ensure that optimal outcomes are achieved. While there are a variety of clinical information systems available, it is evident that the complexity of the case manager's information needs presents challenges for software developers. Accordingly, nurse case managers must take an active role in defining and communicating their needs. The purpose of this article (the first of two) is to present a general overview of the information needs of the nurse case manager and the computerized information system tools available (and emerging) to meet these needs. A follow-up article in the next issue of Nursing Case Management will focus on a typical clinical information system selection process; outline specific ways the nurse case manager can effectively influence that process; and provide a checklist that will enable the nurse case manager to assist the institution in selecting and implementing the most appropriate system.