Is It All About the Form? Norm- vs Criterion-Referenced Ratings and Faculty Inter-Rater Reliability

Background: Little research to date has examined the quality of data obtained from resident performance evaluations. This study sought to address this need and compared inter-rater reliability obtained from norm-referenced and criterion-referenced evaluation scaling approaches for faculty completing resident performance evaluations. Methods: Resident performance evaluation data were examined from 2 institutions (3 programs, 2 internal medicine and 1 surgery; 426 residents in total), with 4 evaluation forms: 2 criterion-referenced (1 with an additional norm-referenced item) and 2 norm-referenced. Faculty inter-rater reliability was calculated with intraclass correlation coefficients (ICCs) (1,10) for each competency area within the form. ICCs were transformed to z-scores, and 95% CIs were computed. Reliabilities for each evaluation form and competency, averages within competency, and averages within scaling type were examined. Results: Inter-rater reliability averages were higher for all competencies that used criterion-referenced scaling relative to those that used norm-referenced scaling. Aggregate scores of all independent categories (competencies and the items assessing overall competence) for criterion-referenced scaling demonstrated higher reliability (z=1.37, CI 1.26-1.48) than norm-referenced scaling (z=0.88, CI 0.77-0.99). Moreover, examination of the distributions of composite scores (average of all competencies and raters for each individual being rated) suggested that the criterion-referenced evaluations better represented the performance continuum. Conclusion: Criterion-referenced evaluation approaches appear to provide superior inter-rater reliability relative to norm-referenced evaluation scaling approaches. Although more research is needed to identify resident evaluation best practices, using criterion-referenced scaling may provide more valid data than norm-referenced scaling.

Virtual non-technical skills assessment training is an effective, scalable approach for novice raters

OBJECTIVES

The COVID-19 pandemic has forced a creative transition to virtual platforms due to physical distancing and travel restrictions. We designed and tested a highly scalable virtual training curriculum for novice raters using the Oxford NOTECHS non-technical skills rating system.

DESIGN

A three-day training course comprising virtual didactics, virtually facilitated simulations, and independent live observations was implemented. NOTECHS scores were submitted for eleven standardized video simulations and four live operations. Intraclass correlation coefficients (ICCs) were calculated for total NOTECHS scores and subcomponent scores. Raters previously trained in-person with the same standardized videos served as a comparator group for equivalence testing.

SETTING

All study activities were conducted in a large academic tertiary referral center in Singapore as part of an ongoing surgical safety initiative.

PARTICIPANTS

Seven staff members underwent training (three virtually and four in-person).  None had prior surgical experience or non-technical skills assessment training.

RESULTS

ICCs for total NOTECHS scores were 0.85 (95% CI, 0.73-0.98) for virtually trained raters and 0.83 for those trained in-person (95% CI, 0.68-0.99).  Scores were equivalent between groups within a 10% margin.

CONCLUSIONS

Non-technical skills assessment can be reliably taught in a highly scalable virtual format. Virtual NOTECHS training is a valuable tool for educational and quality improvement initiatives during the COVID-19 pandemic and for centers that lack ready access to onsite non-technical skills training expertise.

Intra-examiner and inter-examiner reliability of rehabilitative ultrasound imaging for lumbar multifidus and anterolateral abdominal muscles in females with recurrent low back pain: an observational, cross-sectional study

Aim: To examine the reliability of rehabilitative ultrasound imaging performed to measure the thickness of the transverse abdominis, internal oblique, external oblique, and lumbar multifidus muscles in females with recurrent low back pain. Material and methods: A sample of 15 women was recruited. Two independent examiners recorded the thickness of their deep abdominal and spinal muscles by rehabilitative ultrasound imaging. Imaging scans of the transverse abdominis, internal oblique, and external oblique muscles were performed in the supine position and in the midaxillary line, between the lower edge of the ribcage and the iliac crest. Imaging of the lumbar multifidus was done in the prone position and at the level of the L5/S1 zygapophyseal joints. Imaging scans were performed bilaterally in rest and contraction, three times by the first examiner (at baseline, after two hours, and one week later) and once by the second examiner. Results: Good to excellent within-session intra-rater (ICC = 0.76, 0.97), good to excellent between-session intra-rater reliability (ICC = 0.73, 0.93), and good to excellent inter-rater reliability (ICC = 0.73, 0.98) were obtained. Conclusions: The results showed that rehabilitative ultrasound imaging can be used as an excellent reliable instrument by one or two examiners to measure the thickness of the transverse abdominis, internal oblique, external oblique and lumbar multifidus muscles in females with recurrent low back pain.

Measuring Kinematic Viscosity of Engine Oils: A Comparison of Data Obtained from Four Different Devices

The aim of this paper is to compare the results of kinematic viscosity of lubricating oils measurements at 40 °C, obtained with three different rapid evaluation devices, and the standardized method using an Ubbelohde Capillary viscometer. The following instruments were selected to measure: a mid-FTIR spectrophotometer, a microchannel viscometer, and a Stabinger viscometer. The study material comprised 42 fresh engine oils, all of which are commercially available. The main data analysis tools used in the study were multiple regression, Mahala Nobis distance, post-hoc analysis, and the Wilcoxon signed-rank test with the Bonferroni correction. Consistent outcomes were obtained for the Stabinger viscometer only, whereas the microchannel viscometer and the mid-FTIR spectrophotometer were not as precise as the reference method. It was also found that the results obtained with the use of the mid-FTIR spectrophotometer were burdened with a very large measurement error. Therefore, a very careful approach is suggested when choosing these instruments. The study fills an important gap in empirical research in the context of the reliability of measurement results obtained using various research techniques.

Intra- and inter-investigator reliabilities of oral moisture measured using an oral moisture-checking device

BACKGROUND

Oral mucosal moisture determined using oral moisture-checking devices is used as a mouth dryness evaluation method. Such devices are capable of evaluating the state of mouth dryness in a simple manner and have applicability in a wide range of subjects; however, their intra- and inter-investigator reliabilities have not yet been clarified.

OBJECTIVE

This study aims to investigate the intra- and inter-investigator reliabilities of measuring oral moisture using an oral moisture-checking device for a wide range of age groups.

METHODS

Intra- and inter-investigator reliabilities were investigated in 28 young subjects and 19 older subjects aged ≥65 years. Three trained investigators independently measured oral mucosal moisture values using an oral moisture-checking device. Intra-investigator reliability was assessed using the coefficient of variation (CV) and intraclass correlation coefficient (ICC) (1.1), and inter-investigator reliability was assessed using ICC (2.1).

RESULTS

Mean CV was 0.015 and 0.016, mean ICC (1.1) was 0.806 and 0.877, and ICC (2.1) was 0.873 and 0.829 in the young and older subjects, respectively.

CONCLUSION

In young subjects, the mean values of ICC (1.1) and ICC (2.1) of the oral moisture-checking device were 0.806 and 0.873, respectively, whereas in older subjects, these values were 0.877 and 0.829, respectively. Thus, this confirms that the examination of oral mucosal moisture using the oral moisture-checking device has sufficient intra- and inter-investigator reliabilities for a wide range of age groups.

Quantitative tooth mobility evaluation based on intraoral scanner measurements

BACKGROUND

Tooth mobility assessment is subjective and current techniques require the translation of a continuous variable to a categorical variable based on the perception of the examiner. The aim of this study was to evaluate the reliability of a novel technique to assess tooth mobility.

METHODS

Three experienced periodontists were asked to push tooth #16 into a buccal position to in a typodont model with different mobility (M1-M2). Tooth position was obtained using an intraoral scanner and files were compared in metrology software. Mobility was calculated at three reference points at the cervical (C), middle (M), and occlusal (O) regions of the buccal surface of the tooth to determine the linear deviation in the three axes (x, y, and z). Reliability was determined by intraclass-correlation coefficient, differences between M1 and M2 determined by t test, and the analysis of variance (ANOVA) was used to compare the data at the C-M-O regions.

RESULTS

Excellent reliability was assessed by Cronbach alpha >0.9 on the x-y-z axes for both mobility tested, except for M1-C X (0.85), M1-M Y (0.89), and M2-M Z (0.89). The correlation between the examiners demonstrated excellent (˃0.90) or good (0.75˃ x ˂0.90) consistency, except for M1-C Y (0.73; examiner 1 to 2) and M1-M X (0.69; examiners 1 to 3). Significant changes were detected in all axes at the three reference points comparing M1 and M2, and a similar proportional change was observed between O-M-C reference points for M1 and M2.

CONCLUSION

A novel technique to assess tooth mobility based on intraoral scanner measurements provided reliable data in an in vitro experiment.

Evaluator Training and Reliability for SMA Global Nusinersen Trials1

Background:

Training methodology was established to optimize reliability of outcome measures in the nusinersen clinical trials. The Children’s Hospital of Philadelphia Infant Test of Neuromuscular Disorders (CHOP INTEND), Hammersmith Functional Motor Scale Expanded (HFMSE), and Revised Upper Limb (RULM) were primary or secondary outcomes.

Methods:

Video review, quarterly conference calls, and item scoring checks supported evaluator competence. Baseline and screening along with video review established intra and inter-rater reliability.

Results:

Inter and intra-rater reliability were both excellent. Intraclass correlation coefficients (ICC) ranged between 0.906–0.994 across initial training meetings and 0.824–0.996 across annual retraining meetings. This was similar for CHOP INTEND (ICC = 0.824–0.951), HFMSE (ICC = 0.981–0.996), and RULM (ICC = 0.966–0.990). Intra-rater reliability for the CHOP INTEND, HFMSE, and RULM were ICC = 0.895 (95% CI: 0.852–0.926; n = 116), ICC = 0.959 (95% CI: 0.942–0.971; n = 125), and ICC = 0.948 (95% CI: 0.927–0.963; n = 126) respectively.

Conclusions:

Rigorous evaluator training ensures reliability of assessment of subjects with spinal muscular atrophy (SMA) in multicenter international trials.

Inter-rater reliability of diagnostic criteria for sacroiliac joint-, disc- and facet joint pain

BACKGROUND/OBJECTIVE

Several diagnostic criteria sets are described in the literature to identify low back pain subtypes, but very little is known about the inter-rater reliability of these criteria. We conducted a study to determine the reliability of diagnostic tests that point towards SI joint-, disc- or facet joint pain.

METHODS

Inter-rater reliability study alongside three randomized clinical trials. Multidisciplinary pain center of general hospital. Patients aged 18 or more with medical history and physical examination suggestive of sacroiliac joint-, disc- and facet joint pain on lumbar level. Making use of nowadays most common used diagnostic criteria, a physical examination is taken independently by three physicians (two pain physicians and one orthopedic surgeon). Inter-rater reliability (Kappa (κ) measure of agreement) and significance (p) between raters are presented. Strengths of agreement, indicated with κ values above 0,20, are presented in order of agreement.

RESULTS

One hundred patients were included. None of the parameters from the physical investigation had κ values of more than 0.21 (fair) in all pairs of raters. Between two raters (C and D), there was an almost perfect agreement on three parameters, more specifically ``Abnormal sensory and motor examination, hyperactive or diminished reflexes'', ``Sitting exam shows no reflex, motor or sensory signs in the legs'' and ``Straight leg raising (Laségue) negative between 30 and 70 degrees of flexion''. The ``Drop test positive'' parameters had moderate strength of agreement between raters A and D and fair strength between raters A and B. The ``Digital interspinous pressure test positive'' had moderate strength of agreement between raters C and D and fair strength of agreement between raters A and B as well as raters B and C. Three other parameters had a fair strength of agreement between two raters, all other parameters had a slight or poor strength of agreement. Inter-rater reliability, confidence intervals and significance of pooled items for SI joint-, disc- and facet joint pain are represented; κ values for the pooled parameters of the physical examination suggestive of SI joint pain stayed below 0.20 between all raters. The same applies for the pooled parameters of the physical examination suggestive of facet joint or disc pain.

CONCLUSIONS

The poor reliability of the diagnostic parameters seriously limits their predictive validity, and as such their use in patients with low back pain for more than 3 months.

Reliability of clinical impact grading by healthcare professionals of common prescribing error and optimisation cases in critical care patients

Objective

To identify between and within profession-rater reliability of clinical impact grading for common critical care prescribing error and optimisation cases. To identify representative clinical impact grades for each individual case.

Design

Electronic questionnaire.

Setting

5 UK NHS Trusts.

Participants

30 Critical care healthcare professionals (doctors, pharmacists and nurses).

Intervention

Participants graded severity of clinical impact (5-point categorical scale) of 50 error and 55 optimisation cases.

Main Outcome Measures

Case between and within profession-rater reliability and modal clinical impact grading.

Methods

Between and within profession rater reliability analysis used linear mixed model and intraclass correlation, respectively.

Results

The majority of error and optimisation cases (both 76%) had a modal clinical severity grade of moderate or higher. Error cases: doctors graded clinical impact significantly lower than pharmacists (-0.25; P < 0.001) and nurses (-0.53; P < 0.001), with nurses significantly higher than pharmacists (0.28; P < 0.001). Optimisation cases: doctors graded clinical impact significantly lower than nurses and pharmacists (-0.39 and -0.5; P < 0.001, respectively). Within profession reliability grading was excellent for pharmacists (0.88 and 0.89; P < 0.001) and doctors (0.79 and 0.83; P < 0.001) but only fair to good for nurses (0.43 and 0.74; P < 0.001), for optimisation and error cases, respectively.

Conclusions

Representative clinical impact grades for over 100 common prescribing error and optimisation cases are reported for potential clinical practice and research application. The between professional variability highlights the importance of multidisciplinary perspectives in assessment of medication error and optimisation cases in clinical practice and research.

Comparing reliabilities of strip and conventional patch testing

BACKGROUND

The standardized protocol for performing the strip patch test has proven to be valid, but evidence on its reliability is still missing.

OBJECTIVE

To estimate the parallel-test reliability of the strip patch test as compared with the conventional patch test.

METHODS

In this multicentre, prospective, randomized, investigator-blinded reliability study, 132 subjects were enrolled. Simultaneous duplicate strip and conventional patch tests were performed with the Finn Chambers^® on Scanpor^® tape test system and the patch test preparations nickel sulfate 5% pet., potassium dichromate 0.5% pet., and lanolin alcohol 30% pet. Reliability was estimated by the use of Cohen's kappa coefficient.

RESULTS

Parallel-test reliability values of the three standard patch test preparations turned out to be acceptable, with slight advantages for the strip patch test. The differences in reliability were 9% (95%CI: -8% to 26%) for nickel sulfate and 23% (95%CI: -16% to 63%) for potassium dichromate, both favouring the strip patch test.

CONCLUSIONS

The standardized strip patch test method for the detection of allergic contact sensitization in patients with suspected allergic contact dermatitis is reliable. Its application in routine clinical practice can be recommended, especially if the conventional patch test result is presumably false negative.

A Reliability Generalization Meta-analysis of the Leyton Obsessional Inventory Child Version Survey Form

BACKGROUND

The Leyton Obsessional Inventory Child Version Survey Form (LOI-CV-SF) is a scale created in order to detect obsessive-compulsive symptoms and their effects on the daily lives of children and adolescents from ages 10 to 18 and for the screening of non-clinic infant population. With the purpose of estimating the reliability of the test scores, and how it varies in relation to the characteristics of the studies, a meta-analysis of reliability generalization was carried out.

METHODS

A meta-analysis was accomplished. An exhaustive research allowed to select 13 studies that reported some reliability estimate of the test scores and, through the KR-21 formula, this number was increased to 43 estimations of internal consistency reliability. Taking these coefficients, an estimation of the average reliability and analyses of the variables that could be affecting the heterogeneity of reliability coefficients was accomplished.

RESULTS

On average, the reliability by internal consistency of the symptom scale scores was 0.79 (95%CI:0.76 and 0.82), with minimum and maximum values of 0.52 and 0.97, respectively. A large heterogeneity was found (I2 = 96%).The predictive model identified the standard deviation of tests scores as the most relevant variable. The analysis of moderator variables revealed that, mainly, the standard deviation of the test scores is the most statistically related to the reliability, presenting a positive relationship with it.

CONCLUSIONS

The results of the LOI-CV-SF showed a satisfactory average reliability for research purposes, but not for clinical practice purposes.

Telemedicine Versus Face-to-Face Evaluations by Respiratory Therapists of Mechanically Ventilated Neonates and Children: A Pilot Study

BACKGROUND

Mechanical ventilation is one of the most important therapeutic interventions in neonatal and pediatric ICUs. Telemedicine has been shown to reliably extend pediatric intensivist expertise to facilities where expertise is limited. If reliable, telemedicine may extend the reach of pediatric respiratory therapists (RTs) to facilities where expertise does not exist or free up existing RT resources for important face-to-face activities in facilities where expertise is limited. The aim of this study was to determine how well respiratory assessments for ventilated neonates and children correlated when performed simultaneously by 2 RTs face-to-face and via telemedicine.

METHODS

We conducted a pilot study including 40 assessments by 16 RTs on 11 subjects (5 neonatal ICU; 6 pediatric ICU). Anonymously completed intake forms by 2 different RTs concurrently assessing 14 ventilator-derived and patient-based respiratory variables were used to determine correlations.

RESULTS

Forty paired assessments were performed. Median telemedicine assessment time was 8 min. The Pearson correlation coefficient (r) was used to determine agreement between continuous data, and the Cohen kappa statistics were used for binary variables. Pressure control, PEEP, breathing frequency, and FIO2 perfectly correlated (r = 1, all P < .001) as did the presence of a CO2 monitor and need for increased ventilatory support (kappa = 1). The Pearson correlation coefficient for VT, minute ventilation, mean airway pressure, and oxygen saturation ranged from 0.84 to 0.97 (all P < .001). kappa = 0.41 (95% CI 0.02-0.80) for patient-triggered breaths, and kappa = 0.57 (95% CI 0.19-0.94) for breathing frequency higher than set frequency. kappa = -0.25 (95% CI -0.46 to -0.04) for need for suctioning.

CONCLUSIONS

Telemedicine technology was acceptable to RTs. Telemedicine evaluations highly correlated with face-to-face for 10 of 14 aspects of standard bedside respiratory assessment. Poor correlation was noted for more complex, patient-generated parameters, highlighting the importance of further investigation incorporating a virtual stethoscope.

Is visual evaluation of aneurysm coiling a reliable study end point? Systematic review and meta-analysis

BACKGROUND AND PURPOSE

Angiographic occlusion as a surrogate marker of satisfactory aneurysm treatment is commonly used in clinical trials although some pitfalls have to be considered. To investigate the inter-rater reliability of visual rating of aneurysm occlusion as study end point, we performed a systematic review and meta-analysis.

METHODS

Electronic databases (MEDLINE, EMBASE, PubMed, and the Cochrane Library) were searched up to June 2014. Assessment of risk for bias was based on the Quality Appraisal Tool for Studies of Diagnostic Reliability and the Guidelines for Reporting Reliability and Agreement studies. Inter-rater reliability estimates were pooled across studies using meta-analysis, and the influence of several factors (eg, imaging methods, grading scales, and occlusion rate) was tested with meta-regression.

RESULTS

From 1193 titles, 644 abstracts and 87 full-text versions were reviewed. Twenty-six articles met the inclusion criteria and provided 77 reliability estimates. Twenty-one different rating scales were used, and statistical analysis varied. Mean inter-rater agreement of the pooled studies was substantial (κ=0.65; 95% confidence interval, 0.60-0.69). Reliability varied significantly as a function of imaging methods, grading scales, occlusion rates, and their interaction. Observer agreement substantially increased with increasing occlusion rate in digital subtraction angiography but not in MR angiography. Reliability was higher in studies using 2- or 3-value grading scales than in studies with 4-value grading scales.

CONCLUSIONS

There is significant heterogeneity between studies evaluating the reliability of visual evaluation of aneurysm coiling. On the basis of our analysis, we found that the combination of magnetic resonance angiography, 3-value grading scale, and 2 trained raters seems most promising for usage as surrogate study end points.

Test-retest reliability of transcarpal sensory NCV method for diagnosis of carpal tunnel syndrome

Background:

Carpal Tunnel Syndrome (CTS) is the most frequent entrapment neuropathy affecting the upper extremity. There are a variety of electrodiagnostic methods available for documenting median neuropathy in CTS. In some studies, determining the sensory NCV across the palm-wrist segment has been introduced as the most sensitive diagnostic procedure for CTS. The aim of this study was to investigate the test-retest reliability of transcarpal median sensory NCV method for the diagnosis of CTS.

Materials and Methods:

Twenty-three patients with clinical symptoms of CTS were tested two times by two different practitioners in one session and again by the first practitioner after one week. Stimulation of the median nerve was performed in the wrist and palm, with a conduction distance maximum of 7 cm, reliabilities of median nerves sensory nerve action potential latencies with stimulation at wrist and palm (W-SNAP, P-SNAP) and its transcarpal NCV were assessed with intraclass correlation coefficient (ICC).

Results:

Comparison of the obtained values, which were done by two practitioners in one session showed ICC of W-SNAP latency, P-SNAP latency and transcarpal NCV of 0.93, 0.88 and 0.87, respectively and values that were done by one practitioner in two sessions with one-week interval showed ICC of 0.60, 0.50 and 0.47, respectively.

Conclusion:

Our findings suggest excellent interpractitioner test-retest reliability of transcarpal median sensory NCV method for diagnosing CTS.

High reliability in respiratory rate assessment in children with respiratory symptomatology in a rural area in Mozambique

Early recognition of severe medical conditions is often based on clinical scores and vital sign measurements such as the respiratory rate (RR) count. We designed this study to determine the reliability of RR assessment counted three times during a full minute by independent observers in children in a developing country setting. A total of 55 participants were enrolled in the study. Participant ages ranged from 10 days to 7 years (median 22 months). Agreement for RR count was high (intraclass correlation coefficient of 0.95; 95% confidence interval: 0.93-0.97). Agreement for presence of tachypnea was also high (Kappa coefficient of 0.83, p < 0.001). However, a single reading would have misclassified 5-11% of the participants as non-tachypneic. Repeated RR counts offer reliable results if done during a full minute. Patients not fulfilling tachypnea criterion but with a high RR count should have the measurement repeated.

Reliability of the jones patellofemoral osteoarthritis score for radiological assessment of patellofemoral degenerative changes: comparing a physiotherapist to doctors

PURPOSE

Radiological measurement scores are used by orthopaedic physiotherapists in triage and surveillance of patients with degenerative changes of the tibiofemoral and patellofemoral joints. The purpose of this study was to estimate the intra- and inter-observer reliability of a radiological measurement score to assess patellofemoral joint osteoarthritis, the Jones Patellofemoral Osteoarthritis Score (JPOS), with an orthopaedic physiotherapist and three grades of junior surgeons/doctors.

METHODS

A total of 390 skyline X-rays of the patellofemoral joint were reviewed on 2 occasions, 4 weeks apart, by an orthopaedic physiotherapist, 2 orthopaedic surgeons, and an orthopaedic doctor. Intra- and inter-observer reliability of the JPOS was calculated using a weighted kappa analysis.

RESULTS

The results indicated that while there was good overall intra-observer agreement for this measurement score for all assessors, inter-observer reliability was generally poor between the orthopaedic physiotherapist and the orthopaedic surgeons/doctor.

CONCLUSION

Our findings call into question the use of the JPOS by orthopaedic physiotherapists working in triage or orthopaedic surveillance programmes with other orthopaedic clinicians.

Standardizing foot-type classification using arch index values

PURPOSE

The lack of a reliable classification standard for foot type makes drawing conclusions from existing research and clinical decisions difficult, since different foot types may move and respond to treatment differently. The purpose of this study was to determine interrater agreement for foot-type classification based on photo-box-derived arch index values.

METHOD

For this correlational study with two raters, a sample of 11 healthy volunteers with normal to obese body mass indices was recruited from both a community weight-loss programme and a programme in physical therapy. Arch index was calculated using AutoCAD software from footprint photographs obtained via mirrored photo-box. Classification as high-arched, normal, or low-arched foot type was based on arch index values. Reliability of the arch index was determined with intra-class correlations; agreement on foot-type classification was determined using quadratic weighted kappa (κw).

RESULTS

Average arch index was 0.215 for one tester and 0.219 for the second tester, with an overall range of 0.017 to 0.370. Both testers classified 6 feet as low-arched, 9 feet as normal, and 7 feet as high-arched. Interrater reliability for the arch index was ICC=0.90; interrater agreement for foot-type classification was κw=0.923.

CONCLUSIONS

Classification of foot type based on arch index values derived from plantar footprint photographs obtained via mirrored photo-box showed excellent reliability in people with varying BMI. Foot-type classification may help clinicians and researchers subdivide sample populations to better differentiate mobility, gait, or treatment effects among foot types.

[Reliability of a questionnaire used in the long-term follow up of the opinion and perceptions of family medicine by medical students of the University of Zaragoza (Spain)]

AIM

Family medicine deals with certain aspects and perspectives that are often left behind in the training of other levels of care, thus the need for medical students to make contact with Primary Care is of increasing importance. The aim of this study is to evaluate the reliability of the questionnaire of the UNIMEDIFAM group (FIS PI070975) for the long-term outcome of expectations and knowledge about family medicine.

DESIGN

Reliability of a questionnaire.

SETTING

University of Zaragoza.

PARTICIPANTS

371 students from the 1st, 3rd, 5th, and 6th years.

METHOD

The internal consistency of the questionnaire was determined using Cronbach's alpha coefficient, and the stability using the test-retest.

RESULTS

On scale A of the questionnaire we found that three of 19 items were unstable, in 8 of 26 on scale B, and 3 of 38 items on scale C. The Cronbach's alpha value of scale A was 0.722, 0.861 on scale B, and 0,663 on scale C.

CONCLUSIONS

The 3 scales are within the appropriate values, except scale C, which is very close. The findings of this validation process can serve as a reference that may be extrapolated to the UNIMEDIFAM national questionnaire.

A shortened version of the Western ontario rotator cuff disability index: development and measurement properties

PURPOSE

The objective of this study was to develop and examine the measurement properties of a shortened version of the Western Ontario Rotator Cuff Index (WORC), the SHORTWORC, in individuals with rotator-cuff pathology.

METHODS

The study occurred in two stages, both using secondary analysis of existing data sets. The first stage used cross-sectional data from candidates for rotator-cuff surgery to develop the SHORTWORC. The second stage examined various measurement properties of the SHORTWORC by analyzing scores from the WORC, the American Shoulder and Elbow Surgeons questionnaire, and the Constant-Murley score obtained from patients before and after rotator-cuff surgery. Approaches to validating the SHORTWORC included calculating the standard error of measurement (SEM) at an instant in time, performing a confirmatory factor analysis, correlating findings among the questionnaires, and examining differences between men and women. Sensitivity to change was investigated using standardized response mean and relative efficiency.

RESULTS

Data for 712 patients were used to develop the SHORTWORC, the final version of which consisted of 7 questions. Data for 166 patients (86 men, 80 women; mean age 57±11 years) were used for validation. The SEM based on internal consistency (SEMIC) was calculated as 7.43 SHORTWORC points. The SHORTWORC had similar convergent validity (r=0.72-0.82) and sensitivity to change (SRM=1.20 vs. 1.25, p>0.05) to the longer version. The relative efficiency of the SHORTWORC was 3.19 times that of the WORC (95% CI, 1.50-71.51) in discriminating men's from women's level of disability.

CONCLUSIONS

The SHORTWORC has indicators of validity, relative efficiency, and sensitivity to change comparable to those of the original version but has a smaller response burden.

Purpose: The objective of this study was to develop and examine the measurement properties of a shortened version of the Western Ontario Rotator Cuff Index (WORC), the SHORTWORC, in individuals with rotator-cuff pathology. Methods: The study occurred in two stages, both using secondary analysis of existing data sets. The first stage used cross-sectional data from candidates for rotator-cuff surgery to develop the SHORTWORC. The second stage examined various measurement properties of the SHORTWORC by analyzing scores from the WORC, the American Shoulder and Elbow Surgeons questionnaire, and the Constant–Murley score obtained from patients before and after rotator-cuff surgery. Approaches to validating the SHORTWORC included calculating the standard error of measurement (SEM) at an instant in time, performing a confirmatory factor analysis, correlating findings among the questionnaires, and examining differences between men and women. Sensitivity to change was investigated using standardized response mean and relative efficiency. Results: Data for 712 patients were used to develop the SHORTWORC, the final version of which consisted of 7 questions. Data for 166 patients (86 men, 80 women; mean age 57±11 years) were used for validation. The SEM based on internal consistency (SEM_IC) was calculated as 7.43 SHORTWORC points. The SHORTWORC had similar convergent validity (r=0.72−0.82) and sensitivity to change (SRM=1.20 vs. 1.25, p>0.05) to the longer version. The relative efficiency of the SHORTWORC was 3.19 times that of the WORC (95% CI, 1.50–71.51) in discriminating men's from women's level of disability. Conclusions: The SHORTWORC has indicators of validity, relative efficiency, and sensitivity to change comparable to those of the original version but has a smaller response burden.

Inconsistent reports of sexual intercourse among South African high school students

PURPOSE

This study aims to describe patterns of inconsistent reports of sexual intercourse among a sample of South African adolescents.

METHODS

Consistency of reported lifetime sexual intercourse was assessed using five semiannual waves of data. Odds ratios related inconsistent reporting to demographic variables and potential indicators of general and risk-behavior-specific reliability problems.

RESULTS

Of the sexually active participants in the sample, nearly 40% reported being virgins after sexual activity had been reported at an earlier assessment. Inconsistent reporting could not be predicted by gender or race or by general indicators of poor reliability (inconsistent reporting of gender and birth year). However individuals with inconsistent reports of sexual intercourse were more likely to be inconsistent reporters of substance use.

CONCLUSIONS

These results suggest that researchers need to undertake efforts to deal specifically with inconsistent risk behavior data. These may include modification of data collection procedures and use of statistical methodologies that can account for response inconsistencies.

A bedside clinical prediction rule for detecting moderate or severe aortic stenosis

OBJECTIVE

To evaluate a bedside clinical prediction rule for detecting moderate or severe aortic stenosis.

DESIGN

Cross-sectional study with independent comparison to a diagnostic reference standard, doppler echocardiography.

SETTING

Urban university hospital.

PARTICIPANTS

Consecutive hospital inpatients (n = 124) who had been referred for echocardiography.

MEASUREMENTS AND MAIN RESULTS

Participants were examined by a third-year general internal medicine resident and a staff general internist. We hypothesized in advance that absence of a murmur over the right clavicle would rule out aortic stenosis, while the presence of three or four associated findings (slow carotid artery upstroke, reduced carotid artery volume, maximal murmur intensity at the second right intercostal space, and reduced intensity of the second heart sound) would rule in aortic stenosis. Study physicians were unaware of echocardiographic findings. The outcome was echocardiographic moderate or severe aortic stenosis, defined as a valve area of 1.2 cm2 or less, or a peak instantaneous gradient of 25 mm Hg or greater. Absence of a murmur over the right clavicle ruled out aortic stenosis (likelihood ratio [LR] 0.10; 95% confidence interval [CI] 0.01, 0.44). The presence of three or four associated findings ruled in aortic stenosis (LR 40; 95% CI 6.6, 240). If a murmur was present over the right clavicle, but no more than two associated findings were present, then the examination was indeterminate (LR 1.8; 95% CI 0.93, 2.9).

CONCLUSION

A clinical prediction rule, using simple bedside maneuvers, accurately ruled in and ruled out aortic stenosis.