Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education

Construct: High-stakes assessments measure several constructs, such as knowledge, competencies, and skills. In this case, validity evidence for test scores' uses and interpretations is of utmost importance, because of the consequences for everyone involved in their development and implementation. Background: Educational assessment requires an appropriate understanding and use of validity frameworks; however, health professions educators still struggle with the conceptual challenges of validity, and frequently validity analyses have a narrow focus. Important obstacles are the plurality of validity frameworks and the difficulty of grounding these abstract concepts in practice. Approach: We reviewed the validity frameworks literature to identify the main elements of frequently used models (Messick and Kane's) and proposed linking frameworks including Russell's recent overarching proposal. Examples are provided with commonly used assessment instruments in health professions education. Findings: Several elements in these frameworks can be integrated into a common approach, matching and aligning Messick's sources of validity with Kane's four inference types. Conclusions: This proposal to contribute evidence for assessment inferences may provide guidance to understanding the use of validity evidence in applied settings. The evolving field of validity research provides opportunities for its integration and practical use in health professions education.

Moral reasoning and moral competence as predictors of cooperative behavior in a social dilemma

The level of moral development may be crucial to understand behavior when people have to choose between prioritizing individual gains or pursuing general social benefits. This study evaluated whether two different psychological constructs, moral reasoning and moral competence, are associated with cooperative behavior in the context of the prisoner's dilemma game, a two-person social dilemma where individuals choose between cooperation or defection. One hundred and eighty-nine Mexican university students completed the Defining Issues Test (DIT-2; measuring moral reasoning) and the Moral Competence Test (MCT) and played an online version of the prisoner's dilemma game, once against each participant in a group of 6-10 players. Our results indicate that cooperative behavior is strongly affected by the outcomes in previous rounds: Except when both participants cooperated, the probability of cooperation with other participants in subsequent rounds decreased. Both the DIT-2 and MCT independently moderated this effect of previous experiences, particularly in the case of sucker-outcomes. Individuals with high scores on both tests were not affected when in previous rounds the other player defected while they cooperated. Our findings suggest that more sophisticated moral reasoning and moral competence promote the maintenance of cooperative behaviors despite facing adverse situations.

Temporal lobe interictal spikes disrupt encoding and retrieval of verbal memory: A subregion analysis

OBJECTIVE

The medial temporal lobe (MTL) encodes and recalls memories and can be a predominant site for interictal spikes (IS) in patients with focal epilepsy. It is unclear whether memory deficits are due to IS in the MTL producing a transient decline. Here, we investigated whether IS in the MTL subregions and lateral temporal cortex impact episodic memory encoding and recall.

METHODS

Seventy-eight participants undergoing presurgical evaluation for medically refractory focal epilepsy with depth electrodes placed in the temporal lobe participated in a verbal free recall task. IS were manually annotated during the pre-encoding, encoding, and recall epochs. We examined the effect of IS on word recall using mixed-effects logistic regression.

RESULTS

IS in the left hippocampus (odds ratio [OR] = .73, 95% confidence interval [CI] = .63-.84, p < .001) and left middle temporal gyrus (OR = .46, 95% CI = .27-.78, p < .05) during word encoding decreased subsequent recall performance. Within the left hippocampus, this effect was specific for area CA1 (OR = .76, 95% CI = .66-.88, p < .01) and dentate gyrus (OR = .74, 95% CI = .62-.89, p < .05). IS in other MTL subregions or inferior and superior temporal gyrus and IS occurring during the prestimulus window did not affect word encoding (p > .05). IS during retrieval in right hippocampal (OR = .22, 95% CI = .08-.63, p = .01) and parahippocampal regions (OR = .24, 95% CI = .07-.8, p < .05) reduced the probability of recalling a word.

SIGNIFICANCE

IS in medial and lateral temporal cortex contribute to transient memory decline during verbal episodic memory.

The Feeling Is Mutual: Clarity of Haptics-Mediated Social Perception Is Not Associated With the Recognition of the Other, Only With Recognition of Each Other

The enactive theory of perception hypothesizes that perceptual access to objects depends on the mastery of sensorimotor contingencies, that is, on the know-how of the regular ways in which changes in sensations depend on changes in movements. This hypothesis can be extended into the social domain: perception of other minds is constituted by mastery of self-other contingencies, that is, by the know-how of the regular ways in which changes in others' movements depend on changes in one's movements. We investigated this proposal using the perceptual crossing paradigm, in which pairs of players are required to locate each other in an invisible one-dimensional virtual space by using a minimal haptic interface. We recorded and analyzed the real-time embodied social interaction of 10 pairs of adult participants. The results reveal a process of implicit perceptual learning: on average, clarity of perceiving the other's presence increased over trials and then stabilized. However, a clearer perception of the other was not associated with correctness of recognition as such, but with both players correctly recognizing each other. Furthermore, the moments of correct mutual recognition tended to happen within seconds. The fact that changes in social experience can only be explained by the successful performance at the level of the dyad, and that this veridical mutual perception tends toward synchronization, lead us to hypothesize that integration of neural activity across both players played a role.

Factores relacionados con la elección de una especialidad en médicos residentes mexicanos

Objective

This study explored choice factors in four specialties in Mexico.

Method

Mixed methods design. Qualitative phase: four focus groups with first-year residents, to obtain information of how specialty choice was done. With this information a web-based cross-sectional questionnaire was applied to residents registered in the 1st year of Postgraduate Studies Division, UNAM.

Results

32 residents participated in qualitative phase and for the quantitative phase, the survey was answered by 35 surgical, 28 gynecology, 61 internal medicine and 62 pediatric residents. The specialty choice decision was done during the last years of the medical career. The majority of the resident choice was a subspecialty after the general residency. The type of patient was more crucial to choose pediatrics while a good academic program was for internal medicine. Negative models and bullying were decisive to rule out surgery as well as a not well-known hospital was to rule out pediatrics.

Conclusion

The specialty choice is done during undergraduate training, with the intention of doing a subspecialty. Demographic and personality traits were identified.

A Dominance Variant Under the Multi-Unidimensional Pairwise-Preference Framework: Model Formulation and Markov Chain Monte Carlo Estimation

Forced-choice questionnaires have been proposed as a way to control some response biases associated with traditional questionnaire formats (e.g., Likert-type scales). Whereas classical scoring methods have issues of ipsativity, item response theory (IRT) methods have been claimed to accurately account for the latent trait structure of these instruments. In this article, the authors propose the multi-unidimensional pairwise preference two-parameter logistic (MUPP-2PL) model, a variant within Stark, Chernyshenko, and Drasgow's MUPP framework for items that are assumed to fit a dominance model. They also introduce a Markov Chain Monte Carlo (MCMC) procedure for estimating the model's parameters. The authors present the results of a simulation study, which shows appropriate goodness of recovery in all studied conditions. A comparison of the newly proposed model with a Brown and Maydeu's Thurstonian IRT model led us to the conclusion that both models are theoretically very similar and that the Bayesian estimation procedure of the MUPP-2PL may provide a slightly better recovery of the latent space correlations and a more reliable assessment of the latent trait estimation errors. An application of the model to a real data set shows convergence between the two estimation procedures. However, there is also evidence that the MCMC may be advantageous regarding the item parameters and the latent trait correlations.

Traditional scores versus IRT estimates on forced-choice tests based on a dominance model

BACKGROUND

Forced-choice tests (FCTs) were proposed to minimize response biases associated with Likert format items. It remains unclear whether scores based on traditional methods for scoring FCTs are appropriate for between-subjects comparisons. Recently, Hontangas et al. (2015) explored the extent to which traditional scoring of FCTs relates to the true scores and IRT estimates. The authors found certain conditions under which traditional scores (TS) can be used with FCTs when the underlying IRT model was an unfolding model. In this study, we examine to what extent the results are preserved when the underlying process becomes a dominance model.

METHOD

The independent variables analyzed in a simulation study are: forced-choice format, number of blocks, discrimination of items, polarity of items, variability of intra-block difficulty, range of difficulty, and correlation between dimensions.

RESULTS

A similar pattern of results was observed for both models; however, correlations between TS and true thetas are higher and the differences between TS and IRT estimates are less discrepant when a dominance model involved.

CONCLUSIONS

A dominance model produces a linear relationship between TS and true scores, and the subjects with extreme thetas are better measured.

A typology of uncertainty derived from an analysis of critical incidents in medical residents: A mixed methods study

BACKGROUND

Medical uncertainty is inherently related to the practice of the physician and generally affects his or her patient care, job satisfaction, continuing education, as well as the overall goals of the health care system. In this paper, some new types of uncertainty, which extend existing typologies, are identified and the contexts and strategies to deal with them are studied.

METHODS

We carried out a mixed-methods study, consisting of a qualitative and a quantitative phase. For the qualitative study, 128 residents reported critical incidents in their clinical practice and described how they coped with the uncertainty in the situation. Each critical incident was analyzed and the most salient situations, 45 in total, were retained. In the quantitative phase, a distinct group of 120 medical residents indicated for each of these situations whether they have been involved in the described situations and, if so, which coping strategy they applied. The analysis examines the relation between characteristics of the situation and the coping strategies.

RESULTS

From the qualitative study, a new typology of uncertainty was derived which distinguishes between technical, conceptual, communicational, systemic, and ethical uncertainty. The quantitative analysis showed that, independently of the type of uncertainty, critical incidents are most frequently resolved by consulting senior physicians (49 % overall), which underscores the importance of the hierarchical relationships in the hospital. The insights gained by this study are combined into an integrative model of uncertainty in medical residencies, which combines the type and perceived level of uncertainty, the strategies employed to deal with it, and context elements such as the actors present in the situation. The model considers the final resolution at each of three levels: the patient, the health system, and the physician's personal level.

CONCLUSIONS

This study gives insight into how medical residents make decisions under different types of uncertainty, giving account of the context in which the interactions take place and of the strategies used to resolve the incidents. These insights may guide the development of organizational policies that reduce uncertainty and stress in residents during their clinical training.

Comparing Traditional and IRT Scoring of Forced-Choice Tests

This article explores how traditional scores obtained from different forced-choice (FC) formats relate to their true scores and item response theory (IRT) estimates. Three FC formats are considered from a block of items, and respondents are asked to (a) pick the item that describes them most (PICK), (b) choose the two items that describe them the most and the least (MOLE), or (c) rank all the items in the order of their descriptiveness of the respondents (RANK). The multi-unidimensional pairwise-preference (MUPP) model, which is extended to more than two items per block and different FC formats, is applied to obtain the responses to each item block. Traditional and IRT (i.e., expected a posteriori) scores are computed from each data set and compared. The aim is to clarify the conditions under which simpler traditional scoring procedures for FC formats may be used in place of the more appropriate IRT estimates for the purpose of inter-individual comparisons. Six independent variables are considered: response format, number of items per block, correlation between the dimensions, item discrimination level, and sign-heterogeneity and variability of item difficulty parameters. Results show that the RANK response format outperforms the other formats for both the IRT estimates and traditional scores, although it is only slightly better than the MOLE format. The highest correlations between true and traditional scores are found when the test has a large number of blocks, dimensions assessed are independent, items have high discrimination and highly dispersed location parameters, and the test contains blocks formed by positive and negative items.

[The use of seatbelts and child seats in drivers and passengers of motor vehicles in four metropolitan areas in Mexico]

OBJECTIVES

To estimate the rate of seatbelt and child seat use in drivers and passengers of motor vehicles in four metropolitan areas in Mexico (Guadalajara, León, Monterrey and Mexico City). To evaluate the impact of the Mexican Initiative for Road Safety (IMESEVI) in this respect.

MATERIAL AND METHODS

Data were collected at the start of IMESEVI (June 2008) and one year after the program's implementation (October 2009) in the four participating metropolitan areas. In particular, the use of seatbelts and child seats was observed in occupants of automobiles, station wagons, and light trucks. The sample included 28,412 (pre) and 52,274 (post) individuals, of which 1,454 (pre) and 1,679 (post) were younger than five years old. The data analysis was based on a hierarchical logistic model.

RESULTS

Globally, the probability of using either safety device is 46% (95% CI: 43-49%) at baseline and 52% (95% CI: 48-55%) at the post measurement, with large differences, though, among the four participating metropolitan areas. Factors that significantly affect their use include the individual's position in the vehicle, the type and age of the vehicle, and the individual's sex. Child seat use is very limited. At baseline, about 17% (95% CI: 11-25%) of children below five years old travelled in a special seat, with this number increasing to 26% (95% CI: 19-34%) after the implementation of IMESEVI. Child seat use for children above four years is virtually nonexistent.

CONCLUSIONS

Continued efforts are required to raise the public awareness of the importance of using safety devices, especially for passengers in the back of the car as well as with respect to the use of adapted seats for small children.

[Helmet use by motorcyclists in four metropolitan areas of Mexico]

OBJECTIVES

To estimate the prevalence of helmet use among motorcyclists in four metropolitan areas of Mexico (Guadalajara, León, Monterrey, and Mexico City).

MATERIAL AND METHODS

In October 2009, helmet use was observed in 26,046 drivers and 3,971 passengers of motorcycles at several zones of busy traffic. The data were analyzed by means of a hierarchical logistic regression model.

RESULTS

The adjusted probabilities of helmet use strongly differed among the four metropolitan areas: Mexico City: 79-91%; León: 99%; Guadalajara: 54-58%; Monterrey: 73-95%. The probability is lower in passengers (Odds Ratio [OR]: 0.15, with a 95%-confidence interval [CI] of 0.14-0.17) and in drivers who carry some passenger (OR: 0.49; 95% CI: 0.45-0.54, as compared to drivers without passengers), and higher in users of motorcycles for commercial use (OR: 1.76; 95% CI: 1.59-1.96, as compared to private use). Moreover, an estimated 44% of the motorcyclists used a type of helmet that was not officially approved and/or they did not properly adjust the device.

CONCLUSIONS

This study shows that in Guadalajara, Monterrey, and Mexico City it is appropriate to start initiatives to promote helmet use in motorcyclists, particularly for passengers and drivers who transport one or more passengers.

[If you drink, don't drive: drunk drivers in Guadalajara and León, Mexico]

OBJECTIVES

To estimate the percentage of drivers of motor vehicles under the influence of alcohol circulating during Thursday, Friday, and Saturday night in the metropolitan areas of Guadalajara (municipalities Zapopan and Guadalajara) and León, Mexico, together with their blood alcohol content. To evaluate the impact of the Mexican Initiative for Road Safety (IMESEVI, Phase I) in this respect.

MATERIAL AND METHODS

Drivers who were stopped at sobriety checkpoints during the three above-mentioned nights were given an alcohol test together with a brief questionnaire that asked for sociodemographic and other background variables. The study made use of a pre-post design, with data collections at the start of IMESEVI (June 2008) and a year and a half later (February 2010) in the metropolitan areas of Guadalajara and León. A random sample of 1,299 (pre) and 2,226 (post) drivers participated in the study. The data analysis was based on a hierarchical logistic model for ordinal dependent variables.

RESULTS

At baseline, the breathalyzer showed a positive result in between 17% (Zapopan) and 29% (León) of the drivers, while between 1-3% exceeded the legal maximum of 0.08 g/dl for blood alcohol content in Mexico. The night of observation, the presence of passengers in the vehicle, as well as the driver's civil state and sex affected the probability of a positive result. At the post measurement, the probability of a positive result decreased to about a third of the baseline measurement.

CONCLUSIONS

In view of the evidence that links alcohol with the probability of being involved in an accident, drinking and driving prevention programs may contribute significantly to better road safety in Mexico.

Modelo para la promoción de la salud en comunidades rurales a través del desarrollo de agencia personal y empoderamiento intrínseco

Con el propósito de modificar conductas alimentarias y sanitarias en los habitantes de comunidades marginadas, a través del desarrollo de conocimientos y habilidades psicosociales encaminadas a promover agencia personal y empoderamiento intrínseco, se desarrolló e instrumentó en México un modelo piloto basado en el programa “Yo quiero, yo puedo… mejorar mi salud y ejercer mis derechos”. La evaluación de dicho programa mostró un efecto sobre conocimientos, comunicación asertiva, agencia personal y equidad de género en el personal de los almacenes que surte las tiendas rurales, así como en conocimientos, comunicación asertiva, toma de decisiones y agencia personal en la población meta. Las habilidades psicosociales, los conocimientos y la agencia personal ofrecen oportunidades para superar la pobreza.

Longitudinal study of a school based HIV/AIDS early prevention program for Mexican adolescents

A quasi-experimental study is reported with four measurement occasions to evaluate longer-term effects of a life-skills and HIV/AIDS school-based prevention program. Trained teachers administered the program promoting precursors of safer sex behavior to 2064 Mexican high-school students at an age before most were sexually active. The variables included in the study (knowledge about HIV/AIDS prevention, attitudes towards use of condoms, subjective norms, intentions to use condoms and life skills as decision-making skills, partner communication and individual responsibility) have been reported as precursors of protective sexual behavior. The results demonstrate the stability of training effects and a positive impact on these precursors over 1 year of follow-up.

Effectiveness of a Mexican Health Education Program in a Poverty-Stricken Rural Area of Guatemala

In this article, the authors discuss the transfer to Guatemala of an integral health education program, originally developed for indigenous women in southern Mexico. The program was implemented with some 400 indigenous women in rural Guatemala living under dire poverty, and was carried out through a closely supervised cascade process in which specially trained local women conducted workshops to their fellow countrywomen. The program aimed at imparting knowledge as well as enabling changes in behavior with respect to everyday life issues, including nutrition, hygiene, sanitation, and sexual and reproductive health. Evaluation of the impact through questionnaires and direct observations showed significant increases in knowledge and actual behaviors.

Molecular epidemiology of human enteric caliciviruses in The Netherlands

In The Netherlands, illness due to Norwalk-like viruses (NLVs) of the family Caliciviridae is quite common. NLVs cause >80% of the outbreaks of gastroenteritis reported to municipal public health centers and at least 5% of the cases of gastroenteritis for which a general practitioner is consulted. In addition, up to 18% of community cases of gastroenteritis in the 1998/1999 winter season have been attributed to NLVs. Patterns of disease activity differ remarkably, with "normal years, when outbreaks occur that are caused by different types of NLV, and "epidemic years, when outbreaks appear to be caused by a single strain. This observation suggests selection of antigenic variants with increased virulence or altered modes of transmission. In addition, since caliciviruses related to the NLVs from humans have been detected in stool specimens from calves at 45% of the dairy farms in The Netherlands, the possibility of spillover of epidemic strains from an animal reservoir to humans should be considered.

A Generic Disjunctive/Conjunctive Decomposition Model for n-ary Relations

This paper discusses a generic decomposition model that represents an arbitrary n-ary relation as a disjunctive or conjunctive combination of a number of n-ary component relations of a prespecified type. An important subclass of order-preserving decompositions is defined and its properties are derived. The generic model is shown to subsume various known models as special cases, including the models of Boolean factor analysis, hierarchical classes analysis, and disjunctive/conjunctive nonmetric factor analysis. Moreover, it also subsumes a broad range of new models as exemplified with a novel model for multidimensional parallelogram analysis and novel three-way extensions of nonmetric factor analysis. Copyright 1999 Academic Press.