Aptitude, achievement and competence in medicine: a latent variable path model

How do physicians become medical experts? A test of three competing theories: distinct domains, independent influence and encapsulation models

The distinction between basic sciences and clinical knowledge which has led to a theoretical debate on how medical expertise is developed has implications for medical school and lifelong medical education. This longitudinal, population based observational study was conducted to test the fit of three theories-knowledge encapsulation, independent influence, distinct domains-of the development of medical expertise employing structural equation modelling. Data were collected from 548 physicians (292 men-53.3%; 256 women-46.7%; mean age = 24.2 years on admission) who had graduated from medical school 2009-2014. They included (1) Admissions data of undergraduate grade point average and Medical College Admission Test sub-test scores, (2) Course performance data from years 1, 2, and 3 of medical school, and (3) Performance on the NBME exams (i.e., Step 1, Step 2 CK, and Step 3). Statistical fit indices (Goodness of Fit Index-GFI; standardized root mean squared residual-SRMR; root mean squared error of approximation-RSMEA) and comparative fit [Formula: see text] of three theories of cognitive development of medical expertise were used to assess model fit. There is support for the knowledge encapsulation three factor model of clinical competency (GFI = 0.973, SRMR = 0.043, RSMEA = 0.063) which had superior fit indices to both the independent influence and distinct domains theories ([Formula: see text] vs [Formula: see text] [[Formula: see text]] vs [Formula: see text] [[Formula: see text]], respectively). The findings support a theory where basic sciences and medical aptitude are direct, correlated influences on clinical competency that encapsulates basic knowledge.

The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications

Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = -0.18, -0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration.

Structural equation modeling identifies markers of damage and function in the aging male Fischer 344 rat

The male Fischer 344 rat is an established model to study progressive renal dysfunction that is similar, but not identical, to chronic kidney disease (CKD) in humans. These studies were designed to assess age-dependent alterations in renal structure and function at late-life timepoints, 16-24 months. Elevations in BUN and plasma creatinine were not significant until 24 months, however, elevations in the more sensitive markers of function, plasma cystatin C and proteinuria, were detectable at 16 and 18 months, respectively. Interestingly, cystatin C levels were not corrected by caloric restriction. Urinary Kim-1, a marker of CKD, was elevated as early as 16 months. Klotho gene expression was significantly decreased at 24 months, but not at earlier timepoints. Alterations in renal structure, glomerulosclerosis and tubulointerstitial fibrosis, were noted at 16 months, with little change from 18 to 24 months. Tubulointerstitial inflammation was increased at 16 months, and remained similar from 18 to 24 months. A SEM (structural equation modeling) model of age-related renal dysfunction suggests that proteinuria is a marker of renal damage, while urinary Kim-1 is a marker of both damage and function. Taken together, these results demonstrate that age-dependent nephropathy begins as early as 16 months and progresses rapidly over the next 8 months.

Medical University admission test: a confirmatory factor analysis of the results

BACKGROUND

The Graz Admission Test has been applied since the academic year 2006/2007. The validity of the Test was demonstrated by a significant improvement of study success and a significant reduction of dropout rate. The purpose of this study was a detailed analysis of the internal correlation structure of the various components of the Graz Admission Test. In particular, the question investigated was whether or not the various test parts constitute a suitable construct which might be designated as "Basic Knowledge in Natural Science."

METHODS

This study is an observational investigation, analyzing the results of the Graz Admission Test for the study of human medicine and dentistry. A total of 4741 applicants were included in the analysis. Principal component factor analysis (PCFA) as well as techniques from structural equation modeling, specifically confirmatory factor analysis (CFA), were employed to detect potential underlying latent variables governing the behavior of the measured variables.

RESULTS

PCFA showed good clustering of the science test parts, including also text comprehension. A putative latent variable "Basic Knowledge in Natural Science," investigated by CFA, was indeed shown to govern the response behavior of the applicants in biology, chemistry, physics, and mathematics as well as text comprehension. The analysis of the correlation structure of the various test parts confirmed that the science test parts together with text comprehension constitute a satisfactory instrument for measuring a latent construct variable "Basic Knowledge in Natural Science."

CONCLUSIONS

The present results suggest the fundamental importance of basic science knowledge for results obtained in the framework of the admission process for medical universities.

Effects of environmental levels of cadmium, lead and mercury on human renal function evaluated by structural equation modeling

A relationship between exposure to heavy metals, including lead and cadmium, and renal dysfunction has long been suggested. However, modeling of the potential additive, or synergistic, impact of metals on renal dysfunction has proven to be challenging. In these studies, we used structural equation modeling (SEM), to investigate the relationship between heavy metal burden (serum and urine levels of lead, cadmium and mercury) and renal function using data from the NHANES database. We were able to generate a model with goodness of fit indices consistent with a well-fitting model. This model demonstrated that lead and cadmium had a negative relationship with renal function, while mercury did not contribute to renal dysfunction. Interestingly, a linear relationship between lead and loss of renal function was observed, while the maximal impact of cadmium occurred at or above serum cadmium levels of 0.8 μg/L. The interaction of lead and cadmium in loss of renal function was also observed in the model. These data highlight the use of SEM to model interaction between environmental contaminants and pathophysiology, which has important implications in mechanistic and regulatory toxicology.

The UKCAT-12 study: educational attainment, aptitude test performance, demographic and socio-economic contextual factors as predictors of first year outcome in a cross-sectional collaborative study of 12 UK medical schools

BACKGROUND

Most UK medical schools use aptitude tests during student selection, but large-scale studies of predictive validity are rare. This study assesses the United Kingdom Clinical Aptitude Test (UKCAT), and its four sub-scales, along with measures of educational attainment, individual and contextual socio-economic background factors, as predictors of performance in the first year of medical school training.

METHODS

A prospective study of 4,811 students in 12 UK medical schools taking the UKCAT from 2006 to 2008 as a part of the medical school application, for whom first year medical school examination results were available in 2008 to 2010.

RESULTS

UKCAT scores and educational attainment measures (General Certificate of Education (GCE): A-levels, and so on; or Scottish Qualifications Authority (SQA): Scottish Highers, and so on) were significant predictors of outcome. UKCAT predicted outcome better in female students than male students, and better in mature than non-mature students. Incremental validity of UKCAT taking educational attainment into account was significant, but small. Medical school performance was also affected by sex (male students performing less well), ethnicity (non-White students performing less well), and a contextual measure of secondary schooling, students from secondary schools with greater average attainment at A-level (irrespective of public or private sector) performing less well. Multilevel modeling showed no differences between medical schools in predictive ability of the various measures. UKCAT sub-scales predicted similarly, except that Verbal Reasoning correlated positively with performance on Theory examinations, but negatively with Skills assessments.

CONCLUSIONS

This collaborative study in 12 medical schools shows the power of large-scale studies of medical education for answering previously unanswerable but important questions about medical student selection, education and training. UKCAT has predictive validity as a predictor of medical school outcome, particularly in mature applicants to medical school. UKCAT offers small but significant incremental validity which is operationally valuable where medical schools are making selection decisions based on incomplete measures of educational attainment. The study confirms the validity of using all the existing measures of educational attainment in full at the time of selection decision-making. Contextual measures provide little additional predictive value, except that students from high attaining secondary schools perform less well, an effect previously shown for UK universities in general.

Structural equation modeling highlights the potential of Kim-1 as a biomarker for chronic kidney disease

BACKGROUND

Chronic kidney disease (CKD) is a major public health problem, and despite continued research in the field, there is still a need to identify both biomarkers of risk and progression, as well as potential therapeutic targets. Structural equation modeling (SEM) is a family of statistical techniques that has been utilized in the fields of sociology and psychology for many years; however, its utilization in the biological sciences is relatively novel. SEM's ability to investigate complex relationships in an efficient, single model could be utilized to understand the progression of CKD, as well as to develop a predictive model to assess kidney status in the patient.

METHODS

Fischer 344 rats were fed either an ad libitum diet or a calorically restricted diet, and a time-course study of kidney structure and function was performed. EQS, a SEM software package, was utilized to generate five CKD models of the Fisher 344 rat and identify relationships between measured variables and estimates of kidney damage and kidney function.

RESULTS

All models identified strong relationships between a biomarker for CKD, kidney injury molecule-1 (Kim-1) and kidney damage, in the Fischer 344 rat CKD model. Models also indicate a strong relationship between age and renal damage and dysfunction.

CONCLUSION

SEM can be used to model CKD and could be useful to examine biomarkers in CKD patients.

Selected medical students achieve better than lottery-admitted students during clerkships

OBJECTIVES

A recent controlled study by our group showed that the dropout rate in the first 2 years of study of medical students selected for entry by the assessment of a combination of non-cognitive and cognitive abilities was 2.6 times lower than that of a control group of students admitted by lottery. The aim of the present study was to compare the performance of these two groups in the clinical phase.

METHODS

A prospective cohort study was performed to compare the performance of 389 medical students admitted by selection with that of 938 students admitted by weighted lottery between 2001 and 2004. Follow-up of these cohorts lasted 5.5-8.5 years. The main outcome measures were the mean grade obtained on the first five discipline-specific clerkships by all cohorts and the mean grade achieved on all 10 clerkships by the cohorts of 2001 and 2002.

RESULTS

Selected students obtained a significantly higher mean grade during their first five clerkships than lottery-admitted students (mean ± standard error [SE] 7.95 ± 0.03, 95% confidence interval [CI] 7.90-8.00 versus mean ± SE 7.84 ± 0.02, 95% CI 7.81-7.87; p < 0.001). This difference reflected the fact that selected students achieved a grade of ≥ 8.0 1.5 times more often than lottery-admitted students. An analysis of all mean grades awarded on 10 clerkships revealed the same results. Moreover, the longer follow-up period over the clerkships showed that the relative risk for dropout was twice as low in the selected student group as in the lottery-admitted student group.

CONCLUSIONS

The selected group received significantly higher mean grades on their first five clerkships, which could not be attributed to factors other than the selection procedure. Although the risk for dropout before the clinical phase increased somewhat in both groups, the actual dropout rate proved to be twice as low in the selected group.

Relationships between admissions requirements and pre-clinical and clinical performance in a distributed veterinary curriculum

The purpose of this study was twofold: first, to assess the relationships between knowledge-based admission requirements and pre-clinical and clinical performance in a distributed model of veterinary education that uses problem-based learning as the main instruction method in the first two years of the curriculum; second, to compare pre-clinical and clinical performance with performance on the Program for the Assessment of Veterinary Education Equivalence (PAVE) exam. Admissions data including overall GPA, prerequisite GPA, Graduate Record Examination (GRE) score on the Analytical, Analytical Writing, Quantitative, and Verbal sections), veterinary school performance data (GPA for pre-clinical and clinical years), and performance PAVE (taken at the end of second year) were analyzed for two classes (N = 155, 85.8% women and 14.2% men). Overall GPA, prerequisite GPA, and GRE Quantitative and Analytical scores were the best predictors for pre-clinical (years 1 and 2) performance (R = 0.49, 23.5% of the variance), GRE Analytical score was the best predictor for year 3 (pre-clinical and clinical) performance (R = 0.25, 6.3% of the variance), GRE Quantitative score was the best predictor for PAVE performance (R = 0.27, 7.5% of the variance), and GRE Analytical score was the best predictor for clinical performance (year 4; R = 0.21, 4.4% of the variance). PAVE scores correlated with GRE Quantitative scores (r = 0.27, p <.01) and veterinary school performance, with higher correlations in the pre-clinical years (rs = 0.67-0.36, p < .01), providing evidence of convergent validity for the PAVE exam.