The Cochran-Weiss-Shanteau performance index as an indicator of upper limb risk assessment expertise

Prescribing hand strengthening exercise for patients with rheumatoid arthritis; clinical cues influencing occupational therapists' and physiotherapists' judgements

OBJECTIVE

To explore the clinical judgements of therapists in prescribing the intensity of hand strengthening exercise in rheumatoid arthritis (RA).

METHODS

Phase I: Eleven therapists knowledgeable in treating patients with RA subjectively identified seven clinical cues. These were incorporated into 54 hypothetical patient case scenarios.

PHASE II

Therapists with ≥2 years post-registration experience and current or recent experience in treating patients with RA were asked to assess 69 case scenarios in total (54 + 15 repeats) and judge what intensity of hand strengthening exercise they would prescribe using the OMNI-Resistance Exercise Scale of perceived exertion. Using responses to the repeated cases, the Cochran-Weiss-Shanteau index of expertise was used to identify therapists who prescribed more consistently. Multiple regression was used to determine which clinical cues were most strongly associated with the intensity of exercise prescribed. A sub-group analysis explored differences between consistent and inconsistent prescribers.

RESULTS

Fifty-three therapists took part. Thirty completed all 69 case scenarios. Across all therapists, the three most important clinical cues associated with lower intensity of exercise prescribed were (1) Patient's reported pain intensity whilst practising the exercise (β = -1.150, p < 0.001), (2) Disease activity (β = -0.425, p < 0.001) and (3) average hand pain over the last week (β = -0.353 p < 0.001). Twelve therapists were categorised as consistent prescribers. This group relied on fewer clinical cues (three vs. seven) when judging what intensity of exercise to prescribe.

CONCLUSION

This study provides insights into how therapists prescribe hand exercises. Intensity of hand strengthening exercise was influenced by three key clinical cues, including pain intensity and disease activity.

An Empirical Approach to Identifying Subject Matter Experts for the Development of Situational Judgment Tests

Abstract. The development of a scoring key for the situational judgment test often requires subject matter experts (SMEs) to identify the best responses for a hypothetical situation. And yet, there is no gold standard for identifying the SMEs. This paper describes an empirical and context-free approach: the Cochran–Weiss–Shanteau (CWS) method, which does not rely on external criteria such as tenure or credential. We first describe the theory behind the empirical approach of expertise. We also outline the CWS method and provide an R script for calculating the CWS index. Next, we demonstrate how the CWS index can be used for improving interrater agreement and the efficiency of SME selection. Finally, we examined the nomological network of the CWS index. We found that the CWS index was associated with reflective thinking and intuition avoidance.

The role of cue utilisation in reducing the workload in a train control task

Skilled performance has been characterised, in part, by the capacity to accurately identify and respond to patterns as cues in the environment. The outcome is a reduction in cognitive load and a greater residual capacity to undertake concurrent tasks. The present study was designed to examine the relationship between cue utilisation and temporal pattern recognition in the context of a simulated, rail control task. Sixty-one university students undertook an assessment of cue utilisation and engaged in a rail control simulation. The appearance and movement of trains followed a consistent but implicit (undisclosed) pattern. Throughout the second half of the rail task, a secondary task was included. The results indicated that participants with relatively higher cue utilisation were more likely to identify the implicit pattern of rail movements, were more accurate and responded more rapidly under increased workload conditions. The results suggest that a propensity to identify patterns as cues may provide an opportunity to reduce cognitive demands, thereby facilitating performance in a novel task. Implications for selection and system design are discussed. Practitioner Summary: This study was designed to explain differences in the way in which people learn, particularly when tasks involve recurring patterns. Using simulated rail control, the results indicated that participants who display behaviour that is indicative of the utilisation of cues also recognise patterns in the movement of simulated trains. This enables them to manage trains more effectively, even while undertaking other tasks.

Decloaking the privileged expert

When seeking help or advice, one naturally wishes to get that aid from someone who does the task well, from an expert. Determining whether someone merits that label is not trivial, and the judgment can have important consequences. Experts in most domains, from athletes to plumbers, need to exhibit a high level of performance to maintain their status. But there is another group of professionals that is never examined. The expert status of such experts is conferred via criteria such as education or experience. While their credentials may be challenged, their work itself is not seen to bear upon their status. We refer to this group as privileged experts. These experts make assessments or predictions whose correctness is rarely evaluated. Even if prophecy fails, errors are forgiven. We account for this non-evidentiary perspective by positing a widely-held misconception that expertise generalizes. We outline a task-specific, performance-based, limited-term certification scheme that can supplant privilege.

Decloaking the privileged expert

When seeking help or advice, one naturally wishes to get that aid from someone who does the task well, from an expert. Determining whether someone merits that label is not trivial, and the judgment can have important consequences. Experts in most domains, from athletes to plumbers, need to exhibit a high level of performance to maintain their status. But there is another group of professionals that is never examined. The expert status of such experts is conferred via criteria such as education or experience. While their credentials may be challenged, their work itself is not seen to bear upon their status. We refer to this group as privileged experts. These experts make assessments or predictions whose correctness is rarely evaluated. Even if prophecy fails, errors are forgiven. We account for this non-evidentiary perspective by positing a widely-held misconception that expertise generalizes. We outline a task-specific, performance-based, limited-term certification scheme that can supplant privilege.

Is length of experience an appropriate criterion to identify level of expertise?

Clinical experience plays an important role in the development of expertise, particularly when coupled with reflection on practice. There is debate, however, regarding the amount of clinical experience that is required to become an expert. Various lengths of practice have been suggested as suitable for determining expertise, ranging from five years to 15 years. This study aimed to investigate the association between length of experience and therapists' level of expertise in the field of cerebral palsy with upper limb hypertonicity using an empirical procedure named Cochrane-Weiss-Shanteau (CWS). The methodology involved re-analysis of quantitative data collected in two previous studies. In Study 1, 18 experienced occupational therapists made hypothetical clinical decisions related to 110 case vignettes, while in Study 2, 29 therapists considered 60 case vignettes drawn randomly from those used in Study 1. A CWS index was calculated for each participant's case decisions. Then, in each study, Spearman's rho was calculated to identify the correlations between the duration of experience and level of expertise. There was no significant association between these two variables in both studies. These analyses corroborated previous findings of no association between length of experience and judgemental performance. Therefore, length of experience may not be an appropriate criterion for determining level of expertise in relation to cerebral palsy practice.

Criteria for performance evaluation

Using a cognitive task (mental calculation) and a perceptual-motor task (stylized golf putting), we examined differential proficiency using the CWS index and several other quantitative measures of performance. The CWS index (Weiss & Shanteau, 2003) is a coherence criterion that looks only at internal properties of the data without incorporating an external standard. In Experiment 1, college students (n = 20) carried out 2- and 3-digit addition and multiplication problems under time pressure. In Experiment 2, experienced golfers (n = 12), also college students, putted toward a target from nine different locations. Within each experiment, we analyzed the same responses using different methods. For the arithmetic tasks, accuracy information (mean absolute deviation from the correct answer, MAD) using a coherence criterion was available; for golf, accuracy information using a correspondence criterion (mean deviation from the target, also MAD) was available. We ranked the performances of the participants according to each measure, then compared the orders using Spearman’s rs. For mental calculation, the CWS order correlated moderately (rs =.46) with that of MAD. However, a different coherence criterion, degree of model fit, did not correlate with either CWS or accuracy. For putting, the ranking generated by CWS correlated .68 with that generated by MAD. Consensual answers were also available for both experiments, and the rankings they generated correlated highly with those of MAD. The coherence vs. correspondence distinction did not map well onto criteria for performance evaluation.