Paper versus electronic feedback in high stakes assessment

Tablet computers have emerged as increasingly useful tools in medical education, particularly for assessment. However, it is not fully established whether tablet computers influence the quality and/or quantity of feedback provided in high stakes assessments. It is also unclear how electronically-recorded feedback relates to student performance. Our primary aim was to determine whether differences existed in feedback depending on the tool used to record it.

METHODS

We compared quantitative and qualitative feedback between paper-scoring sheets versus iPads™ across two consecutive years of a final year MBChB (UK medical degree) Objective Structured Clinical Examination. Quality of comments (using a validated five-point rating scale), number of examiner comments and number of words were compared across both methods of recording assessment performance using chi-squared analysis and independent t-test. We also explored relationships between student performance (checklist and global scoring) and feedback.

RESULTS

Data from 190 students (2850 paper scored interactions) in 2015 and 193 (2895 iPad™ scored interactions) in 2016 were analysed. Overall, a greater number of comments were given with iPad™ compared to written (42% versus 20%; p < 0.001) but the quality of feedback did not differ significantly. For both written and electronic feedback, students with low global scores were more likely to receive comments (p < 0.001).

CONCLUSION

The use of iPads™ in high stakes assessment increases the quantity of feedback compared to traditional paper scoring sheets. The quantity and quality of feedback for poorer performing candidates (by global score) were also better with iPad™ feedback.

Factors predicting success in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination: a summary for OMFS

The Intercollegiate Membership of the Royal College of Surgeons (MRCS, parts A and B) is a mandatory examination for entry into higher surgical training in the UK. We investigated which factors predict success in both the written (Part A) and clinical (Part B) parts of the examination, and provide a summary for oral and maxillofacial surgeons (OMFS). All UK graduates who attempted both parts between 2007 and 2016 were included. There was a positive correlation between the scores in parts A and B (r=0.41, p<0.01). For Part A, men (odds ratio (OR) 2.78; 95% CI 1.83 to 4.19), white candidates (OR 1.70; 95% CI 1.52 to 1.89), and younger graduates (under 29years of age), were more likely to pass (OR 2.60; 95% CI 1.81 to 3.63). Foundation year one (FY1) doctors had higher pass rates than all other grades (e.g. core surgical trainee 2 compared with FY1 OR 0.50; 95% CI 0.32 to 0.77). The number of attempts at Part A and the final score, as well as ethnicity and stage of training, were independent predictors of success in Part B. Candidates who did well in Part A were more likely to do well in Part B. Several independent predictors of success were identified, but only the stage of training and ethnicity were common predictors in both parts. Higher scores obtained by younger candidates might be relevant to OMFS trainees who take Part A after studying medicine as a second degree.

Predictors of success in the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination

Introduction The Intercollegiate Membership of the Royal College of Surgeons (MRCS, Parts A and B) is designed to safeguard patients and ensure high standards for trainees wishing to pursue a surgical career. We investigated which factors predict Part A and B success. Methods UK graduates who had attempted both parts of MRCS from September 2007 to February 2016 were included. Pearson correlation coefficients were used to examine the linear relationship between each part of MRCS and logistic regression analysis to identify potential independent predictors of MRCS pass/fail outcome. Results A positive correlation between Part A and B score was identified (r = 0.41, P < 0.01). In Part A, males had higher pass rates than females (odds ratio, OR, 2.78, 95% confidence interval, CI, 1.83-4.19), white candidates were more likely to pass than black minority and ethnic doctors (OR 1.70, 95% CI 1.52-1.89), foundation year 1 doctors had higher pass rates than all other grades (e.g. core surgical year 2 trainees vs. foundation year 1 doctors, OR 0.50, 95% CI 0.32-0.77) and younger graduates were more likely to pass than older graduates (OR 2.60, 95% CI 1.81-3.63). In addition to ethnicity and stage of training, Part A performance (number of attempts and score) was an independent predictor for Part B. Conclusions Part A performance is related to Part B score. Many independent predictors of MRCS success were identified, but only stage of training and ethnicity were found to be common predictors of both parts. This article summarises the findings of our most recent MRCS study which was published in The Surgeon (online) in October 2017. ¹.

Impact of performance in a mandatory postgraduate surgical examination on selection into specialty training

Background

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination is undertaken by large numbers of trainees in the UK and internationally as a mandatory step within surgical training. Unlike some high-stakes medical examinations, the MRCS is yet to be validated. A quantitative study was undertaken to assess its predictive validity by investigating the relationship between MRCS (Parts A and B) and national selection interview scores for general and vascular surgery in the UK.

Methods

Pearson correlation coefficients were used to examine the linear relationship between each assessment, and linear regression analyses were employed to identify potential independent predictors of the national selection score. All UK medical graduates who attempted the interview in 2011-2015 were included.

Results

Some 84·4 per cent of the candidates (1231 of 1458) were matched with MRCS data. There was a significant positive correlation between the first attempt score at Part B of the MRCS examination and the national selection score (r = 0·38, P < 0·001). In multivariable analysis, 17 per cent of variance in the national selection first attempt score was explained by the Part B MRCS score and number of attempts (change in R ² value of 0·10 and 0·07 respectively; P < 0·001). Candidates who required more than two attempts at Part B were predicted to score 8·1 per cent less than equally matched candidates who passed at their first attempt.

Conclusion

This study supports validity of the MRCS examination, and indicates its predictive value regarding entry into specialist training.

Tablet computers in assessing performance in a high stakes exam: opinion matters

Background Tablet computers have emerged as a tool to capture, process and store data in examinations, yet evidence relating to their acceptability and usefulness in assessment is limited. Methods We performed an observational study to explore opinions and attitudes relating to tablet computer use in recording performance in a final year objective structured clinical examination at a single UK medical school. Examiners completed a short questionnaire encompassing background, forced-choice and open questions. Forced choice questions were analysed using descriptive statistics and open questions by framework analysis. Results Ninety-two (97% response rate) examiners completed the questionnaire of whom 85% had previous use of tablet computers. Ninety per cent felt checklist mark allocation was 'very/quite easy', while approximately half considered recording 'free-type' comments was 'easy/very easy'. Greater overall efficiency of marking and resource savings were considered the main advantages of tablet computers, while concerns relating to technological failure and ability to record free type comments were raised. Discussion In a context where examiners were familiar with tablet computers, they were preferred to paper checklists, although concerns were raised. This study adds to the limited literature underpinning the use of electronic devices as acceptable tools in objective structured clinical examinations.

Effectiveness of manual therapy versus surgery in pain processing due to carpal tunnel syndrome: A randomized clinical trial

BACKGROUND

People with carpal tunnel syndrome (CTS) exhibit widespread pressure pain and thermal pain hypersensitivity as a manifestation of central sensitization. The aim of our study was to compare the effectiveness of manual therapy versus surgery for improving pain and nociceptive gain processing in people with CTS.

METHODS

The trial was conducted at a local regional Hospital in Madrid, Spain from August 2014 to February 2015. In this randomized parallel-group, blinded, clinical trial, 100 women with CTS were randomly allocated to either manual therapy (n = 50), who received three sessions (once/week) of manual therapies including desensitization manoeuvres of the central nervous system, or surgical intervention (n = 50) group. Outcomes including pressure pain thresholds (PPT), thermal pain thresholds (HPT or CPT), and pain intensity which were assessed at baseline, and 3, 6, 9 and 12 months after the intervention by an assessor unaware of group assignment. Analysis was by intention to treat with mixed ANCOVAs adjusted for baseline scores.

RESULTS

At 12 months, 95 women completed the follow-up. Patients receiving manual therapy exhibited higher increases in PPT over the carpal tunnel at 3, 6 and 9 months (all, p < 0.01) and higher decrease of pain intensity at 3 month follow-up (p < 0.001) than those receiving surgery. No significant differences were observed between groups for the remaining outcomes.

CONCLUSIONS

Manual therapy and surgery have similar effects on decreasing widespread pressure pain sensitivity and pain intensity in women with CTS. Neither manual therapy nor surgery resulted in changes in thermal pain sensitivity.

SIGNIFICANCE

The current study found that manual therapy and surgery exhibited similar effects on decreasing widespread pressure pain sensitivity and pain intensity in women with carpal tunnel syndrome at medium- and long-term follow-ups investigating changes in nociceptive gain processing after treatment in carpal tunnel syndrome.

Management of COPD in Primary Care in North-East Scotland

Introduction

We wished to obtain a snapshot of current service provision and how this could best be developed approximately one year on from the introduction of the National Institute for Clinical Excellence (NICE) guidelines for the management of chronic obstructive pulmonary disease (COPD) and the inclusion of COPD care in the New GMS Contract Quality and Outcomes Framework (QOF).

Methodology

A questionnaire-based survey sent to every general practice (n = 84) in Grampian.

Results

Responses were received from 75 of 84 practices (89%). Questionnaires were returned by both general practitioners (GPs) and practice nurses in 45 practices (54%). All responding practices reported that they had COPD registers. 60/75 (80%) of practices reported having a dedicated COPD clinic; 70/75 (93%) had a spirometer. Areas identified for service development were: quality assuring training in COPD care and spirometry; expanding pulmonary rehabilitation provision (86%), delivering this service locally (54%) and in primary care (75%); standardising referral, assessment and communication about provision of home oxygen; training in pulse oximetry (71%).

Conclusion

This data has important implications for the validity of the quality indicators (QOF) under the new GMS contract. Our respondents identified areas where the new GMS contract QOF could be improved, as well as providing useful suggestions for service development. Respondents recognised that not all clinical services can be effectively delivered by general practice with data supporting the development of intermediate care services for people with COPD.

Covert contrast and covert errors in persistent velar fronting

Acoustic and articulatory studies demonstrate covert contrast in perceptually neutralised phonemic contrasts in both typical children and children with speech disorders. These covert contrasts are thought to be relatively common and symptomatic of phonetic speech disorders. However, clinicians in the speech therapy clinic have had no easy way of identifying this covertness. This study uses ultrasound tongue imaging to compare tongue contours for /t/and /k/in seven children with persistent velar fronting. We present a method of overlaying tongue contours to identify covert contrast at the articulatory level. Results show that all seven children, contrary to expectations, produced both /t/and /k/with near-identical tongue shapes showing no evidence of covert contrast. However, further analysis of one of the participants showed highly variable tongue shapes for /t/and /k/, including retroflex productions of both. Although not phonologically conditioned, this covert error is evidence of speech disorder at the phonetic level.

Multiple phonetically trained-listener comparisons of speech before and after articulatory intervention in two children with repaired submucous cleft palate

Cleft Palate (CP) assessments based on phonetic transcription are the "gold standard" therapy outcome measure, despite reliability difficulties. Here we propose a novel perceptual evaluation, applied to ultrasound-visual biofeedback (U-VBF) therapy and therapy using visual articulatory models (VAMs) for two children with repaired submucous CP. Three comparisons were made: post VAM, post U-VBF and overall pre- and post-therapy. Twenty-two phonetically-trained listeners were asked to determine whether pre- or post-therapy recordings sounded "closer to the English target", using their own implicit phonological knowledge (prompted via orthographic representation). Results are compared with segment-oriented percent target consonant correct (PTCC) derived from phonetic transcriptions. Listener judgements and PTCC suggest that both children made improvements using both VAM and U-VBF. Statistical analysis showed listener agreement in each comparison, though agreement was weak. However, perceptual evaluation offers a straightforward method of evaluating the effectiveness of interventions and can be used by phonetically trained or lay listeners.

Using ultrasound visual biofeedback to treat persistent primary speech sound disorders

Growing evidence suggests that speech intervention using visual biofeedback may benefit people for whom visual skills are stronger than auditory skills (for example, the hearing-impaired population), especially when the target articulation is hard to describe or see. Diagnostic ultrasound can be used to image the tongue and has recently become more compact and affordable leading to renewed interest in it as a practical, non-invasive visual biofeedback tool. In this study, we evaluate its effectiveness in treating children with persistent speech sound disorders that have been unresponsive to traditional therapy approaches. A case series of seven different children (aged 6-11) with persistent speech sound disorders were evaluated. For each child, high-speed ultrasound (121 fps), audio and lip video recordings were made while probing each child's specific errors at five different time points (before, during and after intervention). After intervention, all the children made significant progress on targeted segments, evidenced by both perceptual measures and changes in tongue-shape.

Are educational interventions to prevent catheter-related bloodstream infections in intensive care unit cost-effective?

BACKGROUND

There is increasing interest in evidence-based educational interventions in central venous catheter care. It is unclear how effective these are at reducing the risk of bloodstream infections from the use of intravascular catheters (catheter-BSIs) and the associated costs and health benefits.

AIM

To estimate the additional costs and health benefits from introducing such interventions and the costs associated with catheter-BSIs.

METHODS

A comprehensive epidemiological and economic review was performed to develop the parameters for an economic model to assess the cost-effectiveness of introducing an educational intervention compared with clinical practice without the intervention. The model follows the clinical pathway of cohorts of patients from their admission to an intensive care unit (ICU), where some may acquire catheter-BSI, and estimates the associated costs, mortality and life expectancy.

FINDINGS

The additional cost per catheter-BSI episode was £3940. The results of this model demonstrate that introducing an additional educational intervention to prevent catheter-BSI improved patient life expectancy and reduced overall costs.

CONCLUSION

Introducing evidence-based education is likely to reduce the incidence of catheter-BSI and the model results suggest that the cost of introducing the interventions will be outweighed by savings related to reduced ICU bed occupancy costs.

Tongue reading: comparing the interpretation of visual information from inside the mouth, from electropalatographic and ultrasound displays of speech sounds

Speakers possess a natural capacity for lip reading; analogous to this, there may be an intuitive ability to "tongue-read." Although the ability of untrained participants to perceive aspects of the speech signal has been explored for some visual representations of the vocal tract (e.g. talking heads), it is not yet known to what extent there is a natural ability to interpret speech information presented through two clinical phonetic tools: EPG and ultrasound. This study aimed to determine whether there is any intuitive ability to interpret the images produced by these systems, and to determine whether one tool is more conducive to this than the other. Twenty adults viewed real-time and slow motion EPG and ultrasound silent movies of 10 different linguo-palatal consonants and 4 vowels. Participants selected which segment they perceived from four forced-choice options. Overall, participants scored above chance in the EPG and ultrasound conditions, suggesting that these images can be interpreted intuitively to some degree. This was the case for consonants in both the conditions and for vowels in the EPG condition.

Emotional recognition in autism spectrum conditions from voices and faces

The present study reports on a new vocal emotion recognition task and assesses whether people with autism spectrum conditions (ASC) perform differently from typically developed individuals on tests of emotional identification from both the face and the voice. The new test of vocal emotion contained trials in which the vocal emotion of the sentence were congruent, incongruent, or neutral with respect to the semantic content. We also included a condition in which there was no semantic content (an 'mmm' was uttered using an emotional tone). Performance was compared between 11 adults with ASC and 14 typically developed adults. Identification of emotion from sentences in which the vocal emotion and the meaning of sentence were congruent was similar in people with ASC and a typically developed comparison group. However, the comparison group was more accurate at identifying the emotion in the voice from incongruent and neutral trials, and also from trials with no semantic content. The results of the vocal emotion task were correlated with performance on a face emotion recognition task. In decoding emotion from spoken utterances, individuals with ASC relied more on verbal semantics than did typically developed individuals, presumably as a strategy to compensate for their difficulties in using prosodic cues to recognize emotions.