In-course and career outcomes predicted by medical school selection procedures based on personal qualities

This paper reports a 40-year follow-up of 57 graduates from the initial intake to an Australian medical school who were selected on the basis of either academic criteria alone or desirable personal qualities as assessed by non-cognitive tests and an interview (with a third small group satisfying both criteria). Both students and teaching staff have remained blind to the basis for selection until the present day. Analysis of their under- and post-graduate careers indicates that 'academic' entrants were more likely to complete an intercalated BMedSci degree and to become specialists, while 'personal qualities' entrants were more likely to graduate with honours, become GPs, and win higher degrees after graduation. However, gender more significantly predicted these outcomes, with female results similar to 'personal qualities' entrants and males' similar to 'academic.' The results are interpreted with reference to the aims and structure of the Newcastle medicine curriculum.

Why is it so hard to consider personal qualities when selecting medical students?

Having 'good' doctors is important to everybody. How to select medical students better has been discussed repeatedly for more than seventy years, implying that prevailing methods could be improved. A significant body of research exists about selection methods and their application in medicine. Yet most medical schools world-wide continue to use prior academic performance and cognitive ability as their major criteria for selection, with minor or no consideration of personal qualities and interpersonal skills (possibly assuming they will develop naturally during training and practice). We describe the main methods available for assessing personal qualities of applicants to medical school and have attempted to identify some reasons and systemic disincentives working against their adoption.

Item Response Theory Analysis of the Big Five Questionnaire for Children-Short Form (BFC-SF): A Self-Report Measure of Personality in Children Aged 11-12 Years

Prior investigations indicate that the five core personality dimensions (the "Big Five") are measurable by middle childhood. The aim of this research was to examine the psychometric properties of a short-form self-report measure of the Big Five personality dimensions in children that would be suitable for administration online in large population-based studies. Twenty-five questionnaire items in English, derived from the 65-item Big Five Questionnaire for Children in Italian (Barbaranelli, Caprara, Rabasca, & Pastorelli, 2003), were completed online by 27,415 Australian children in Year 6 (mean age 11.92 years). An item response theory approach evaluated the psychometric properties and resolved a 20-item short-form questionnaire. Exploratory and confirmatory factor analyses supported the Big Five structure. Construct validity was demonstrated via correlations between Big Five scores and the Strengths and Difficulties Questionnaire subscales (Goodman, 2001). The 20 items provide a brief, reliable, and valid child self-report measure of the Big Five personality dimensions.

Reliability of personality and values tests: The effects of "high stakes" selection conditions, and of four years in medical school

This study investigated if scores on tests of personal qualities are affected by whether they will determine selection decisions ("high stakes") or not; and whether they are stable for individuals and groups across a four-year medical course. Two tests, one assessing values and one assessing components of personality, were administered either at the same time as a medical university entrance exam (first cohort; N = 216), or after entry was confirmed (second cohort; N = 142). Both cohorts took the tests again after four years of medical school. Analysis of variance was used to compare group mean scores and interactions, and correlation coefficients to measure temporal reliability. The high stakes cohort initially presented themselves in a significantly more positive light on the personality test. After four years of medical school scores on both tests changed significantly, towards more communitarian values and less empathic attitudes. Thus, personality scores were affected by both the conditions under which the initial tests were conducted and by the passage of time, but values only by the passage of time. Before and after scores were significantly correlated.

The 2015 Middle Childhood Survey (MCS) of mental health and well-being at age 11 years in an Australian population cohort

PURPOSE

The Middle Childhood Survey (MCS) was designed as a computerised self-report assessment of children's mental health and well-being at approximately 11 years of age, conducted with a population cohort of 87 026 children being studied longitudinally within the New South Wales (NSW) Child Development Study.

PARTICIPANTS

School Principals provided written consent for teachers to administer the MCS in class to year 6 students at 829 NSW schools (35.0% of eligible schools). Parent or child opt-outs from participation were received for 4.3% of children, and MCS data obtained from 27 808 children (mean age 11.5 years, SD 0.5; 49.5% female), representing 85.9% of students at participating schools.

FINDINGS TO DATE

Demographic characteristics of participating schools and children are representative of the NSW population. Children completed items measuring Social Integration, Prosocial Behaviour, Peer Relationship Problems, Supportive Relationships (at Home, School and in the Community), Empathy, Emotional Symptoms, Conduct Problems, Aggression, Attention, Inhibitory Control, Hyperactivity-Inattention, Total Difficulties (internalising and externalising psychopathology), Perceptual Sensitivity, Psychotic-Like Experiences, Personality, Self-esteem, Daytime Sleepiness and Connection to Nature. Distributions of responses on each item and construct demarcate competencies and vulnerabilities within the population: most children report mental health and well-being, but the population distribution spanned the full range of possible scores on every construct.

FUTURE PLANS

Multiagency, intergenerational linkage of the MCS data with health, education, child protection, justice and early childhood development records took place late in 2016. Linked data were used to elucidate patterns of risk and protection across early and middle child development, and these data will provide a foundation for future record linkages in the cohort that will track mental and physical health, social and educational/occupational outcomes into adolescence and early adulthood.

Have we got the selection process right? The validity of selection tools for predicting academic performance in the first year of undergraduate medicine

This article was migrated. The article was marked as recommended. Content: There remains much debate over the 'best' method for selecting students in to medicine. This study aimed to assess the predictive validity of four different selection tools with academic performance outcomes in first-year undergraduate medical students. Methods: Regression analyses were conducted between admission scores on previous academic performance - the Australian Tertiary Admission Rank (ATAR), the Undergraduate Medicine and Health Sciences Admission Test (UMAT), Multiple-Mini Interview (MMI) and the Personal Qualities Assessment (PQA) with student performance in first-year assessments of Multiple Choice Questions, Short Answer Questions, Objective Structured Clinical Examinations (OSCE) and Problem-Based Learning (PBL) Tutor ratings in four cohorts of students (N = 604, 90%). Results: All four selection tools were found to have significant predictive associations with one or more measures of student performance in Year One of undergraduate medicine. UMAT, ATAR and MMI scores consistently predicted first year performance on a number of outcomes. ATAR was the only selection tool to predict the likelihood of making satisfactory progress overall. Conclusions: All four selection tools play a contributing role in predicting academic performance in first year medical students. Further research is required to assess the validity of selection tools in predicting performance in the later years of medicine.

New South Wales Child Development Study (NSW-CDS): an Australian multiagency, multigenerational, longitudinal record linkage study

PURPOSE

The initial aim of this multiagency, multigenerational record linkage study is to identify childhood profiles of developmental vulnerability and resilience, and to identify the determinants of these profiles. The eventual aim is to identify risk and protective factors for later childhood-onset and adolescent-onset mental health problems, and other adverse social outcomes, using subsequent waves of record linkage. The research will assist in informing the development of public policy and intervention guidelines to help prevent or mitigate adverse long-term health and social outcomes.

PARTICIPANTS

The study comprises a population cohort of 87,026 children in the Australian State of New South Wales (NSW). The cohort was defined by entry into the first year of full-time schooling in NSW in 2009, at which time class teachers completed the Australian Early Development Census (AEDC) on each child (with 99.7% coverage in NSW). The AEDC data have been linked to the children's birth, health, school and child protection records for the period from birth to school entry, and to the health and criminal records of their parents, as well as mortality databases.

FINDINGS TO DATE

Descriptive data summarising sex, geographic and socioeconomic distributions, and linkage rates for the various administrative databases are presented. Child data are summarised, and the mental health and criminal records data of the children's parents are provided.

FUTURE PLANS

In 2015, at age 11 years, a self-report mental health survey was administered to the cohort in collaboration with government, independent and Catholic primary school sectors. A second record linkage, spanning birth to age 11 years, will be undertaken to link this survey data with the aforementioned administrative databases. This will enable a further identification of putative risk and protective factors for adverse mental health and other outcomes in adolescence, which can then be tested in subsequent record linkages.

Potential predictors of psychological distress and well-being in medical students: a cross-sectional pilot study

PURPOSE

Research has consistently found that the proportion of medical students who experience high levels of psychological distress is significantly greater than that found in the general population. The aim of our research was to assess the levels of psychological distress more extensively than has been done before, and to determine likely predictors of distress and well-being.

SUBJECTS AND METHODS

In 2013, students from an Australian undergraduate medical school (n=127) completed a questionnaire that recorded general demographics, hours per week spent studying, in paid work, volunteer work, and physical exercise; past and current physical and mental health, social support, substance use, measures of psychological distress (Kessler Psychological Distress Scale, depression, anxiety, stress, burnout); and personality traits.

RESULTS

Females were found to have higher levels of psychological distress than males. However, in regression analysis, the effect of sex was reduced to nonsignificance when other variables were included as predictors of psychological distress. The most consistent significant predictors of our 20 indicators of psychological distress were social support and the personality traits of emotional resilience and self-control.

CONCLUSION

The findings suggest that emotional resilience skills training embedded into the medical school curriculum could reduce psychological distress among medical students.

Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study

BACKGROUND

Over the past 70 years, there has been a recurring debate in the literature and in the popular press about how best to select medical students. This implies that we are still not getting it right: either some students are unsuited to medicine or the graduating doctors are considered unsatisfactory, or both.

AIM

To determine whether particular variables at the point of selection might distinguish those more likely to become satisfactory professional doctors, by following a complete intake cohort of students throughout medical school and analysing all the data used for the students' selection, their performance on a range of other potential selection tests, academic and clinical assessments throughout their studies, and records of professional behaviour covering the entire five years of the course.

METHODS

A longitudinal database captured the following anonymised information for every student (n = 146) admitted in 2007 to the Hull York Medical School (HYMS) in the UK: demographic data (age, sex, citizenship); performance in each component of the selection procedure; performance in some other possible selection instruments (cognitive and non-cognitive psychometric tests); professional behaviour in tutorials and in other clinical settings; academic performance, clinical and communication skills at summative assessments throughout; professional behaviour lapses monitored routinely as part of the fitness-to-practise procedures. Correlations were sought between predictor variables and criterion variables chosen to demonstrate the full range of course outcomes from failure to complete the course to graduation with honours, and to reveal clinical and professional strengths and weaknesses.

RESULTS

Student demography was found to be an important predictor of outcomes, with females, younger students and British citizens performing better overall. The selection variable "HYMS academic score", based on prior academic performance, was a significant predictor of components of Year 4 written and Year 5 clinical examinations. Some cognitive subtest scores from the UK Clinical Aptitude Test (UKCAT) and the UKCAT total score were also significant predictors of the same components, and a unique predictor of the Year 5 written examination. A number of the non-cognitive tests were significant independent predictors of Years 4 and 5 clinical performance, and of lapses in professional behaviour. First- and second-year tutor ratings were significant predictors of all outcomes, both desirable and undesirable. Performance in Years 1 and 2 written exams did not predict performance in Year 4 but did generally predict Year 5 written and clinical performance.

CONCLUSIONS

Measures of a range of relevant selection attributes and personal qualities can predict intermediate and end of course achievements in academic, clinical and professional behaviour domains. In this study HYMS academic score, some UKCAT subtest scores and the total UKCAT score, and some non-cognitive tests completed at the outset of studies, together predicted outcomes most comprehensively. Tutor evaluation of students early in the course also identified the more and less successful students in the three domains of academic, clinical and professional performance. These results may be helpful in informing the future development of selection tools.

Trial use of the Personal Qualities Assessment (PQA) in the entrance examination of a Japanese medical university: similarities to the results in western countries

BACKGROUND

The Personal Qualities Assessment (PQA), developed by the University of Newcastle, Australia to assess the aptitude of future medical professionals, has been used in Western countries.

PURPOSES

The objective was to investigate whether the PQA is appropriate for Japanese medical school applicants.

METHODS

Two of the PQA tests, Libertarian-Dual-Communitarian moral orientations (Mojac) and Narcissism, Aloofness, Confidence, and Empathy (NACE), were translated into Japanese, and administered at the Tokyo Women's Medical University entrance examinations from 2007 to 2009.

RESULTS

The distributions of the applicants' Mojac and NACE scores were close to the normal distribution, and the mean scores did not exhibit a large difference from those in Western countries. The only significant difference was that the mean score of the NACE test was slightly lower than the Western norm.

CONCLUSIONS

The translated PQA tests may be appropriate for use with Japanese applicants, though further research considering cultural differences is required.

Comparing the cognitive, personality and moral characteristics of high school and graduate medical entrants to the Tehran University of Medical Sciences in Iran

BACKGROUND

Tehran University of Medical Sciences has two streams of medical student admission: an established high school entry (HSE) route and an experimental graduate entry (GE) route.

AIM

To compare the cognitive skills, personality traits and moral characteristics of HSE and GE students admitted to this university.

METHODS

The personal qualities assessment tool (PQA; www.pqa.net.au ) was translated from English to Persian and then back-translated. Afterwards 35 individuals from the GE and 109 individuals from the 2007 to 2008 HSE completed the test. The results were compared by t-test and Chi-square.

RESULTS

The HSE students showed significantly higher ability in the cognitive skills tests (p < 0.001). They were also more libertarian (p = 0.022), but had lower ability to confront stress and unpleasant events (p < 0.001), and had lower self-awareness and self-control (p < 0.001).

CONCLUSION

On the basis of their personal qualities, the GE students had more self-control and strength when coping with stress than the HSE students, but the latter had superior cognitive abilities. Hence it may be useful to include cognitive tests in GE students' entry exam and include tests of personal qualities to exclude those with unsuitable characteristics.

Emotional Intelligence and Personality as Predictors of Psychological Well-Being

Research studies have reported elevated rates of psychological distress (e.g., depression) in practicing lawyers yet little research has examined predictors of such problems in law students. Specific personality traits have been shown to be predictors of a range of psychological problems. We administered a battery of tests to a cohort of 1st-year law students ( n = 150) and measured the Big Five personality traits and emotional intelligence (EI) to examine their relationships to psychological well-being as indicated by coping styles, satisfaction with life, performance-based self-esteem (PBSE), Global Severity Index (GSI) scores from the Brief Symptom Inventory (BSI), depression, and alcohol use. We found that whereas EI was significantly related to three of the five well-being variables, the Big Five personality factor of neuroticism was found to be a stronger predictor of well-being. The findings suggest that EI does not account for additional variance in well-being over personality.

Can personal qualities of medical students predict in-course examination success and professional behaviour? An exploratory prospective cohort study

BACKGROUND

Over two-thirds of UK medical schools are augmenting their selection procedures for medical students by using the United Kingdom Clinical Aptitude Test (UKCAT), which employs tests of cognitive and non-cognitive personal qualities, but clear evidence of the tests' predictive validity is lacking. This study explores whether academic performance and professional behaviours that are important in a health professional context can be predicted by these measures, when taken before or very early in the medical course.

METHODS

This prospective cohort study follows the progress of the entire student cohort who entered Hull York Medical School in September 2007, having taken the UKCAT cognitive tests in 2006 and the non-cognitive tests a year later. This paper reports on the students' first and second academic years of study. The main outcome measures were regular, repeated tutor assessment of individual students' interpersonal skills and professional behaviour, and annual examination performance in the three domains of recall and application of knowledge, evaluation of data, and communication and practical clinical skills. The relationships between non-cognitive test scores, cognitive test scores, tutor assessments and examination results were explored using the Pearson product-moment correlations for each group of data; the data for students obtaining the top and bottom 20% of the summative examination results were compared using Analysis of Variance.

RESULTS

Personal qualities measured by non-cognitive tests showed a number of statistically significant relationships with ratings of behaviour made by tutors, with performance in each year's objective structured clinical examinations (OSCEs), and with themed written summative examination marks in each year. Cognitive ability scores were also significantly related to each year's examination results, but seldom to professional behaviours. The top 20% of examination achievers could be differentiated from the bottom 20% on both non-cognitive and cognitive measures.

CONCLUSIONS

This study shows numerous significant relationships between both cognitive and non-cognitive test scores, academic examination scores and indicators of professional behaviours in medical students. This suggests that measurement of non-cognitive personal qualities in applicants to medical school could make a useful contribution to selection and admission decisions. Further research is required in larger representative groups, and with more refined predictor measures and behavioural assessment methods, to establish beyond doubt the incremental validity of such measures over conventional cognitive assessments.

Using personal qualities assessment to measure the moral orientation and personal qualities of medical students in a non-Western culture

How to select candidates with appropriate personal qualities for medical school is an important issue. This study examined the psychometric properties and group differences of the Personal Qualities Assessment (PQA) to test the feasibility of using it as a tool to assess the medical school applicants in a non-Western culture. Seven hundred forty-six medical students in Taiwan completed two psychometric measures: Mojac to assess moral orientation and NACE to assess four aspects of interpersonal relationships. Thirty-one students completed the tests twice to establish test-retest reliability. A subsample of 127 students also completed a measure of the "Big Five" personality traits to examine the construct validity of these scales. Both Mojac and NACE had acceptable internal consistency and test-retest reliability. Conceptually, coherent and significant relationships were observed between test components and between the NACE and Big Five. NACE but not Mojac varied significantly between different sociodemographic groups. Both tests demonstrated acceptable psychometric properties. However, the predictive validity of PQA requires future studies.

Predictive validity of the personal qualities assessment for selection of medical students in Scotland

BACKGROUND

The Personal Qualities Assessment (PQA) was developed to enhance medical student selection by measuring a range of non-cognitive attributes in the applicants to medical school. Applicants to the five Scottish medical schools were invited to pilot the test in 2001 and 2002.

AIMS

To evaluate the predictive validity of PQA for selecting medical students.

METHODS

A longitudinal cohort study was conducted in which PQA scores were compared with senior year medical school performance.

RESULTS

Consent to access performance markers was obtained from 626 students (61.6% of 1017 entrants in 2002-2003). Linkable Foundation Year (4th) rankings were available for 411 (66%) students and objective structured clinical examination (OSCE) rankings for 335 (54%) of those consenting. Both samples were representative of the original cohort. No significant correlations were detected between separate elements of the PQA assessment and student performance. However, using the algorithm advocated by Powis et al. those defined as 'non-extreme' (<±1.5 SD from the cohort mean scores; SD, standard deviation) character types on the involved-detached and on the libertarian-communitarian moral orientation scales were ranked higher in OSCEs (average of 7.5% or 25 out of 335, p = 0.049).

CONCLUSIONS

This study was limited by high attrition and basic outcome markers which are insensitive to relevant non-cognitive characteristics. However, it is the largest currently available study of predictive validity for the PQA assessment. There was one finding of significance: that those students who were identified by PQA as 'not extreme' on the two personal characteristics scales performed better in an OSCE measure of professionalism. Futures studies are required since psychometric testing for both cognitive and non-cognitive attributes are increasingly used in admission process and these should include more and better measures of professionalism against which to correlate non-cognitive traits.

A comprehensive model for the selection of medical students

BACKGROUND

Medical schools have a need to select their students from an excess of applicants. Selection procedures have evolved piecemeal: Academic thresholds have risen, written tests have been incorporated and interview protocols are developed.

AIM

To develop and offer for critical review and, ultimately, present for adoption by medical schools, an evidence-based and defensible model for medical student selection.

METHODS

We have described here a comprehensive model for selecting medical students which is grounded on the theoretical and empirical selection and assessment literature, and has been shaped by our own research and experience.

RESULTS

The model includes the following selection criteria: Informed self-selection, academic achievement, general cognitive ability (GCA) and aspects of personality and interpersonal skills. A psychometrically robust procedure by which cognitive and non-cognitive test scores can be used to make selection decisions is described. Using de-identified data (n = 1000) from actual selection procedures, we demonstrate how the model and the procedure can be used in practice.

CONCLUSION

The model presented is based on a currently best-practice approach and uses measures and methods that maximise the probability of making accurate, fair and defensible selection decisions.

Graduate entry to medicine: widening psychological diversity

BACKGROUND

At Nottingham University more than 95% of entrants to the traditional 5-year medical course are school leavers. Since 2003 we have admitted graduate entrants (GEM) to a shortened (4-year) course to 'widen access to students from more disadvantaged backgrounds'. We have recently shown that the GEM course widens academic and socio-demographic diversity of the medical student population. This study explored whether GEM students also bring psychological diversity and whether this could be beneficial.

METHODS

We studied: a) 217 and 96 applicants to the Nottingham 5- and 4-year courses respectively, applying in the 2002-3 UCAS cycle, and, b) 246 school leavers starting the 5-year course and 39 graduate entrants to the 4-year course in October 2003. The psychological profiles of the two groups of applicants and two groups of entrants were compared using their performance in the Goldberg 'Big 5' Personality test, the Personal Qualities Assessment (PQA; measuring interpersonal traits and interpersonal values), and the Lovibond and Lovibond measure of depression, anxiety and stress. For the comparison of the Entrants we excluded the 33 school leavers and seven graduates who took the tests as Applicants.Statistical analyses were undertaken using SPSS software (version 16.0).

RESULTS

Graduate applicants compared to school leaver applicants were significantly more conscientious, more confident, more self controlled, more communitarian in moral orientation and less anxious. Only one of these differences was preserved in the entrants with graduates being less anxious. However, the graduate entrants were significantly less empathetic and conscientious than the school leavers.

CONCLUSION

This study has shown that school leaver and graduate entrants to medical school differ in some psychological characteristics. However, if confirmed in other studies and if they were manifest in the extreme, not all the traits brought by graduates would be desirable for someone aiming for a medical career.

Selection of medical students according to their moral orientation

INTRODUCTION

Consideration has been given to the use of tests of moral reasoning in the selection procedure for medical students. We argue that moral orientation, rather than moral reasoning, might be more efficacious in minimising the likelihood of inappropriate ethical behaviour in medicine. A conceptualisation and measure of moral orientation are presented, together with findings from 11 samples of medical school applicants and students.

AIM

To provide empirical evidence for the reliability and validity of a measure of moral orientation and to explore gender, age, cultural and educational influences on moral orientation.

METHODS

A questionnaire designed to measure a libertarian-dual-communitarian dimension of moral orientation was completed by 7864 medical school applicants and students in Australia, Israel, Fiji, New Zealand, Scotland and England and by 84 Australian psychology students between 1997 and 2001.

RESULTS

Older respondents produced marginally higher (more communitarian) moral orientation scores, as did women compared to men. Minor but significant (P <0.05) cultural differences were found. The Israeli samples produced higher mean moral orientation scores, while the Australian psychology student sample produced a lower (more libertarian) mean score relative to all other samples. No significant change in moral orientation score was observed after 1 year in a sample of Australian medical school students (n=59), although some differences observed between 5 cohorts of Australian medical students (Years 1-5; n=234) did reach significance. Moral orientation scores were found to be significantly correlated with a number of personality measures, providing evidence of construct validity. In all samples moral orientation significantly predicted the moral decisions made in response to the hypothetical dilemmas embedded in the measurement instrument. Discussion The results provide support for the conceptualisation of a libertarian-dual-communitarian dimension of moral orientation and demonstrate the psychometric properties of the measurement instrument. A number of questions concerning the use of such tests in selection procedures are considered.

Assessment of personal qualities in relation to admission to medical school

BACKGROUND

Recently there has been much scrutiny of the medical school admissions process by universities, the General Medical Council and the public. Improved objectivity, fairness and effectiveness of selection procedures are desirable. The ultimate outcome sought is the graduation of competent doctors who reflect the values of and are in tune with the communities they serve.

METHODS

Applicants to the Scottish medical schools sat a battery of psychometric tests to measure cognitive ability, personality traits and moral/ethical reasoning (Personal Qualities Assessment, PQA). Analysis determined the potential impact of the latter variables, and those of educational background and socioeconomic class (assessed by residential 'deprivation category'), upon success in gaining a place to study medicine.

RESULTS

Cognitive ability did not vary significantly as a function of gender or educational background, although there was a trend for it to be lower in individuals from more deprived backgrounds. Women as a group were more empathic, with a greater communitarian orientation, than men. There was no significant difference between individuals attending independent and state-funded schools in respect of any of the qualities measured by the PQA. Applicants from deprived backgrounds and those attending state-funded schools would not be disadvantaged by an admissions process based on the PQA.

CONCLUSION

The incorporation of an assessment tool such as the PQA may have positive implications for widening access and the objective selection of suitable medical students, resulting in the training of doctors who are more representative of the community at large. A longterm follow-up of the professional careers of those medical students who completed the PQA will be undertaken.

Is it possible to assess the "ethics" of medical school applicants?

Questions surrounding the assessment of medical school applicants' morality are difficult but they are nevertheless important for medical schools to consider. It is probably inappropriate to attempt to assess medical school applicants' ethical knowledge, moral reasoning, or beliefs about ethical issues as these all may be developed during the process of education. Attitudes towards ethical issues and ethical sensitivity, however, might be tested in the context of testing for personality attributes. Before any "ethics" testing is introduced as part of screening for admission to medical school it would require validation. We suggest a number of ways in which this might be achieved.