Computer-supported collaborative learning in the medical workplace: Students' experiences on formative peer feedback of a critical appraisal of a topic paper

BACKGROUND

Medical workplace learning consists largely of individual activities, since workplace settings do not lend themselves readily to group learning. An electronic Learning Management with System Computer-Supported Collaborative Learning (CSCL) could enable learners at different workplace locations to discuss personal clinical experiences at a distance to enhance learning.

AIM

To explore whether CSCL-enabled structured asynchronous discussions on an authentic task has additional value for learning in the medical workplace.

METHODS

Between January 2008 and June 2010, we conducted an exploratory evaluation study among senior medical students that were engaged in clinical electives. Students wrote a Critical Appraisal of a Topic paper about a clinical problem they had encountered and discussed it in discipline homogeneous subgroups on an asynchronous forum in a CSCL environment. A mixed method design was used to explore students' perceptions of the CSCL arrangement with respect to their preparation and participation, the design and knowledge gains. We analysed the messages recorded during the discussions to investigate which types of interactions occurred.

RESULTS

Students perceived knowledge improvement of their papers. The discussions were mostly task-focused. The students considered an instruction session and a manual necessary to prepare for CSCL. A high amount of sent messages and a high activity in discussion seem to influence scores on perceptions: 'participation' and 'knowledge gain' positively.

CONCLUSION

CSCL appears to offer a suitable environment for peers to provide formative feedback on a Critical Appraisal of a Topic paper during workplace learning. The CSCL environment enabled students to collaborate in asynchronous discussions, which positively influenced their learning.

Script concordance testing: more cases or more questions?

Script concordance test (SCT) is a case based assessment format of clinical reasoning in which questions are nested into several cases. Recent results using Q4 format suggest that nested questions contribute more to reliability of measure than cases. The present study aims at documenting variance components associated with SCT cases and nested questions and to determine what are the optimal number and combinations of cases and nested questions. Data from SCT in three different fields are presented. G study and D study methodology are used to estimate variance component and to determine optimal number and combinations of cases and questions. Questions nested into cases contributed a large amount of score variance (more than 70%). D studies with varying samples show that, depending on the reliability of the test, an optimal number of 2-4 questions nested into 15-25 cases represents the best combination. Nested questions contribute to a significant portion of score variance, with the implication that formulation of up to 5 questions per case is an efficient way to optimize the reliability of SCT scores.

On the reliability of a dental OSCE, using SEM: effect of different days

AIM

The first aim was to study the reliability of a dental objective structured clinical examination (OSCE) administered over multiple days, and the second was to assess the number of test stations required for a sufficiently reliable decision in three score interpretation perspectives of a dental OSCE administered over multiple days.

MATERIALS AND METHODS

In four OSCE administrations, 463 students of the year 2005 and 2006 took the summative OSCE after a dental course in comprehensive dentistry. The OSCE had 16-18 5-min stations (scores 1-10), and was administered per OSCE on four different days of 1 week. ANOVA was used to test for examinee performance variation across days. Generalizability theory was used for reliability analyses. Reliability was studied from three interpretation perspectives: for relative (norm) decisions, for absolute (domain) and pass-fail (mastery) decisions. As an indicator of reproducibility of test scores in this dental OSCE, the standard error of measurement (SEM) was used. The benchmark of SEM was set at <0.51. This is corresponding to a 95% confidence interval (CI) of <1 on the original scoring scale that ranged from 1 to 10.

RESULTS

The mean weighted total OSCE score was 7.14 on a 10-point scale. With the pass-fail score set at 6.2 for the four OSCE, 90% of the 463 students passed. There was no significant increase in scores over the different days the OSCE was administered. 'Wished' variance owing to students was 6.3%. Variance owing to interaction between student and stations and residual error was 66.3%, more than two times larger than variance owing to stations' difficulty (27.4%). The SEM norm was 0.42 with a CI of +/-0.83 and the SEM domain was 0.50, with a CI of +/-0.98. In order to make reliable relative decisions (SEM <0.51), the use of minimal 12 stations is necessary, and for reliable absolute and pass-fail decisions, the use of minimal 17 stations is necessary in this dental OSCE.

CONCLUSIONS

It appeared reliable, when testing large numbers of students, to administer the OSCE on different days. In order to make reliable decisions for this dental OSCE, minimum 17 stations are needed. Clearly, wide sampling of stations is at the heart of obtaining reliable scores in OSCE, also in dental education.

Language background and OSCE performance: a study of potential bias

AIM

To investigate whether Non-Native dental students perceive drawbacks in their education and examination experience because of their lack of language proficiency and to test whether prolonging Objective Structured Clinical Examination (OSCE) test-station duration improves their performance.

MATERIALS AND METHODS

Dental students (n = 345) completed a questionnaire about their native country, their language background and the possible drawback they perceived in dental education and examination because of their language proficiency. Students were marked as 'native', when they were born in the Netherlands with Dutch as native language or 'Non-Native' when they were born outside the Netherlands, raised with a non-Dutch native language, or raised bilingually. A sample of 108 students was assessed by an OSCE testing a periodontal course with nine test-stations. Test-station topics were: (1) history taking, (2) measuring attachment level, (3) educating patients, (4) tracing a radiograph, (5) root-planing, (6) writing a prescription, (7) diagnostics and prognostics, (8) differential diagnostics and (9) writing a referral letter. The first five test-stations mentioned were of 5-min duration. The other four test-stations were provided in two modes: either with a short (5 min) or longer (10 min) version. Every student took at random two long and two short test-stations.

RESULTS

In the group of 345 questionnaire responders, Non-Native students (n = 116) perceived significantly more drawback in education and examination than Native students (n = 229) (P < 0.001). When Non-Native students speak Dutch at home, around 38% of them reported perceived drawbacks in education, whereas when they speak their native language at home, around 60% reported perceived drawbacks in education (P = 0.005). In the periodontal OSCE (n = 108), the Native group (n = 70) had significantly higher total scores than the Non-Native (n = 38), (P = 0.009, d = 0.53). The Non-Native group had significantly lower mean scores in the communication station 'educating patients' (P = 0.034, d = 0.42). Prolonged test-station duration from 5 to 10 min had no positive effect in all experimental test-stations in the Native and Non-Native group. Female students in the Native group out-performed male in a communication test-stations. Female students in Native and Non-Native groups were found to be more successful in 'tracing bone loss on radiographs'.

CONCLUSION

Non-Native students perceived a drawback in dental education and examination because of their language proficiency in Dutch, which is confirmed by their actual OSCE performance. Prolonging the time for a test-station did not improve OSCE performance of Non-Native students. It is recommended that students with problems in language ability need additional tuition and practice.

Assessment of Advanced Life Support competence when combining different test methods—Reliability and validity

Robust assessment of Advanced Life Support (ALS) competence is paramount to the credibility of ALS-provider certification and for estimating the learning outcome and retention of ALS competence following the courses. The European Resuscitation Council (ERC) provides two sets of MCQs and four Cardiac Arrest Simulation Test (CASTest) scenarios for the assessments according to guidelines 2005.

AIMS

To analyse the reliability and validity of the individual sub-tests provided by ERC and to find a combination of MCQ and CASTest that provides a reliable and valid single effect measure of ALS competence.

METHODS

Two groups of participants were included in this randomised, controlled experimental study: a group of newly graduated doctors, who had not taken the ALS course (N=17) and a group of students, who had passed the ALS course 9 months before the study (N=16). Reliability in terms of inter-rater agreement and generalisability across skills scenarios were estimated. Validity was studied in terms of equality of test difficulty and ability to discriminate performance between the groups.

RESULTS

Inter-rater agreement on checklist scores were generally high, Intraclass Correlation Coefficients between 0.766 and 0.977. Inter-rater agreements on pass/fail decisions were not perfect. The one MCQ test was significantly more difficult than the other. There were no significant differences between CASTests. Generalisability theory was use to identify a composite of MCQ and CASTest scenarios that possessed high reliability, equality of test sets, and ability to discriminate between the two groups of supposedly different ALS competence.

CONCLUSIONS

ERC sub-tests of ALS competence possess sufficient reliability and validity. A combined ALS score with equal weighting of one MCQ and one CASTest can be used as a single measurement of ALS competence.

Regulatory end-point assessment of the consultation competence of family practice trainees in Kuwait

BACKGROUND

No single approach to the regulatory assessment of global consultation competence has been shown to possess the required levels of validity, reliability and feasibility.

OBJECTIVE

To evaluate the approach adopted in Kuwait to the regulatory end-point assessment of the global consultation competence of family practice trainees with particular reference to validity, reliability and feasibility.

METHODS

Family practice trainees in Kuwait were individually and directly observed for 3 hours in consultation with a minimum of 10 patients by a pair of examiners. Performance was judged against the explicit criteria of consultation competence as contained in the Leicester Assessment Package (LAP).

RESULTS

The marks independently allocated by the pairs of examiners to 126 trainees between 1994 and 2001 were within five percentage points on 91% of occasions. A reliability coefficient of 0.82 was achieved when two examiners independently marked candidates consulting with 10 real patients; this rose to 0.95 at the critical 50% pass-fail margin. The main sources of variance contributing to the reliability of marks allocated were candidate performance (42%) and the interaction of candidate performance across cases, i.e., case specificity (30%). The clinical challenges presented by the patients were judged by both examiners to be sufficient to enable performance to be assessed across the seven LAP consultation categories as follows: behaviour and relationship with patients (100% of consultations), interviewing/history taking (100%), record keeping (99%), patient management (99%), problem solving (98%), physical examination (95%), and anticipatory care (86%). Each assessment involved a pair of examiners and lasted approximately 3.5 hours.

CONCLUSION

The Kuwait clinical examination achieves high content validity and authenticity as it uses direct observation of performance, validated and explicit criteria against which performance is judged, and real patient challenges. It can discriminate between different levels of consultation performance and satisfies the recognized reliability threshold for regulatory examinations (0.82 vs 0.80). Accordingly, we recommend the use of such an approach in the regulatory end-point assessment of the global consultation competence of trainees in family practice. Such an approach is more valid, and is likely to be more feasible, than simulated surgeries or the short-case OSCE format.

The role of a computerised case-based testing procedure in practice performance assessment

INTRODUCTION

For postgraduate training of doctors there is a need for valid and reliable instruments to assess their daily performance. Various instruments have been suggested, some of which use incognito simulated patients (SPs). These methods are resource intensive. Computerised Case-based testing (CCT) is logistically simpler and may still predict performance well. The research question was to evaluate the predictive validity of CCT for performance.

METHODS

Seventeen rheumatologists were each visited by eight incognito SPs presenting various rheumatological complaints, and scoring the performance of the rheumatologists using a predefined checklist. From this checklist a panel of experts identified essential items. In addition the rheumatologists sat a CCT test containing 55 cases with a total of 121 items.

RESULTS

Negative correlations were found between the SP scores and the CCT scores. This was unexpected. Therefore, background variables on experience were used to compare both methods. The correlation between these and CCT were high and positive and with the SP scores high and negative. This pattern did not differ when using the essential items of the checklist. Reliabilities of the SP scores were markedly high.

DISCUSSION

Although CCT was not predictive of SP scores, it was related to working experience. There are good reasons to assume that although SP-scores were more authentic, they were less valid than CCT scores, mainly because they focussed more on thoroughness than on efficiency in data gathering. The results underpin the assumption that for valid performance assessment the most important issue is what information about the candidate is collected and now how authentic the method is.

Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test

BACKGROUND

The script concordance (SC) test was conceived to measure knowledge organization, the presence of links between items of knowledge which allow for interpretation of data in clinical decision making situations. Earlier studies have shown that the SC test has good psychometric qualities and overcomes some of the limitations of simulation clinical testing. This study explores the predictive validity of the test.

OBJECTIVES

To verify whether scores obtained by students at the end of clerkship predict their clinical reasoning performance at the end of residency.

DESIGN

Comparison of scores obtained on a SC test taken at the end of clerkship with those obtained 2 years later at the end of residency on two clinical reasoning assessments of known validity, called the short-answer management problems (SAMPs) and the simulated office orals (SOOs), and an objective structured clinical examination (OSCE) aimed at assessing hands-on skills and clinical reasoning. Data were treated by Pearson correlation analysis.

SUBJECTS AND SETTING

A cohort of 24 students from a medical school in Quebec was followed up to the end of their residency in family medicine, completed in several schools across Quebec.

RESULTS

The observed Pearson correlation coefficients of the SC test were statistically significant (0.451, P=0.013; 0.447; P=0.015) when compared with the SAMPs and the SOOs, respectively. They were not statistically significant (0.340, P=0.052) when compared with the OSCE.

CONCLUSION

The authors assumed that the richness of knowledge organization, as indicated by SC test scores, would predict part of the performance on the measures of clinical reasoning (SAMP and SOO), but would predict less well performance on the OSCE which measures both clinical skills and clinical reasoning. Data found in the study are coherent with this hypothesis. This is evidence in favour of the construct validity of the SC test. It also indicates that, in the context of certification assessment, if a candidate has shown good organization of clinical knowledge at an early point in training, it can be expected that he/she will show good organization at subsequent measurements of this kind of knowledge. This appears to be true even if the later measures bear on a wider clinical domain.

Standardized or real patients to test clinical competence? The long case revisited

BACKGROUND

In undergraduate clinical examinations, the use of real patients as long cases is being replaced by objective structured clinical examinations (OSCEs) which use simulated scenarios, although we lack published psychometric data on long cases to support the move from real to simulated patients.

AIM

To assess candidate performance across two history-taking long cases to estimate the number of cases required for a reliable assessment. Results are compared with psychometric data from an OSCE.

SETTING

A final-year qualifying undergraduate clinical examination.

METHOD

Two observed history-taking long cases were included, alongside an OSCE. Candidates interviewed two unstandardized real patients. The history-taking part (14 minutes) was observed, uninterrupted, by examiner(s) who assessed data gathering, interviewing, and diagnostic and management skills. The presentation (7 minutes) was unstructured; the examiner(s) intervened as appropriate. Marks were expressed as a percentage of the total possible score and analysed using generalizability theory to estimate intercase reliability.

RESULTS

Two examiner pairs independently rated both long cases for 79 (36.7%) of the 214 candidates. Projections based on generalizability theory showed that 10 20-minute cases would give reliabilities of 0.84 for single-marked and 0.88 for double-marked candidates, compared with a projected reliability of 0.73 for the same 214 candidates taking the OSCE.

CONCLUSION

If history-taking long cases are observed, three-and-a-half hours of testing time using 10 unstandardized patients would produce a reliable test. Long cases therefore are, in terms of reliability, no worse and no better than OSCEs in assessing clinical competence.

Composite undergraduate clinical examinations: how should the components be combined to maximize reliability?

BACKGROUND

Clinical examinations increasingly consist of composite tests to assess all aspects of the curriculum recommended by the General Medical Council.

SETTING

A final undergraduate medical school examination for 214 students.

AIM

To estimate the overall reliability of a composite examination, the correlations between the tests, and the effect of differences in test length, number of items and weighting of the results on the reliability.

METHOD

The examination consisted of four written and two clinical tests: multiple-choice questions (MCQ) test, extended matching questions (EMQ), short-answer questions (SAQ), essays, an objective structured clinical examination (OSCE) and history-taking long cases. Multivariate generalizability theory was used to estimate the composite reliability of the examination and the effects of item weighting and test length.

RESULTS

The composite reliability of the examination was 0.77, if all tests contributed equally. Correlations between examination components varied, suggesting that different theoretically interpretable parameters of competence were being tested. Weighting tests according to items per test or total test time gave improved reliabilities of 0.93 and 0.81, respectively. Double weighting of the clinical component marginally affected the reliability (0.76).

CONCLUSION

This composite final examination achieved an overall reliability sufficient for high-stakes decisions on student clinical competence. However, examination structure must be carefully planned and results combined with caution. Weighting according to number of items or test length significantly affected reliability. The components testing different aspects of knowledge and clinical skills must be carefully balanced to ensure both content validity and parity between items and test length.

Assessment of clinical competence

Tests of clinical competence, which allow decisions to be made about medical qualification and fitness to practise, must be designed with respect to key issues including blueprinting, validity, reliability, and standard setting, as well as clarity about their formative or summative function. Multiple choice questions, essays, and oral examinations could be used to test factual recall and applied knowledge, but more sophisticated methods are needed to assess clincial performance, including directly observed long and short cases, objective structured clinical examinations, and the use of standardised patients. The goal of assessment in medical education remains the development of reliable measurements of student performance which, as well as having predictive value for subsequent clinical competence, also have a formative, educational role.