Reliability of a special device for measuring the cross-sectional area of the patellar tendon by ultrasonography

The objective of this study was to evaluate the intra- and inter-rater reliability and agreement between conditions with and without a special device (SD) in the evaluation of the patellar tendon (PT) cross-sectional area (CSA). Forty trained adult volunteers participated in the study. With the knee positioned at 90°, the ultrasound probe was placed in the transverse plane at 25, 50, and 75% of the PT length. Two raters and one analyzer obtained the images. We use a two-way ANOVA with a significance level of α = 0.05. No significant differences were found between raters or conditions. Intra-rater reliability ranged from moderate to good. Inter-rater reliability without the SD ranged from low to good, improving from moderate to good when the SD was used. Evaluation of the PT ends showed a lower coefficient of variation with the SD. We observed a moderate correlation at the ends and a strong correlation in the middle between conditions. The mean difference in the three positions is small (~ 0.013 cm2/ ~ 1.7%) with an upper limit of 43.2% and a lower limit of 32.5%. Therefore, we conclude that the use of the SD can be employed for evaluating the PT ends, while for the central region, it becomes optional.

Improving the reliability of measurements in orthopaedics and sports medicine

A large space still exists for improving the measurements used in orthopaedics and sports medicine, especially as we face rapid technological progress in devices used for diagnostic or patient monitoring purposes. For a specific measure to be valuable and applicable in clinical practice, its reliability must be established. Reliability refers to the extent to which measurements can be replicated, and three types of reliability can be distinguished: inter-rater, intra-rater, and test-retest. The present article aims to provide insights into reliability as one of the most important and relevant properties of measurement tools. It covers essential knowledge about the methods used in orthopaedics and sports medicine for reliability studies. From design to interpretation, this article guides readers through the reliability study process. It addresses crucial issues such as the number of raters needed, sample size calculation, and breaks between particular trials. Different statistical methods and tests are presented for determining reliability depending on the type of gathered data, with particular attention to the commonly used intraclass correlation coefficient.

B-Mode Ultrasonography Is a Reliable and Valid Alternative to Magnetic Resonance Imaging for Measuring Patellar Tendon Cross-Sectional Area

This study investigated the validity and reliability of measuring patellar tendon (PT) cross-sectional area (CSA) using magnetic resonance imaging (MRI) and ultrasound (US) imaging. Nineteen healthy participants (10 women, 9 men) participated in three imaging sessions of the PT, once via MRI and twice via US, with image acquisition conducted by two raters, one experienced (rater 2) and one inexperienced (rater 1). All PT segmentations were analyzed by both raters. The validity of US-derived estimates of PT CSA against MRI estimates was analyzed using linear regression. Within-day reliability of US and MRI measurements and between-day reliability of US measurements were quantified using typical error (TE) and intra-class correlation coefficients (ICC_3,1). There was good agreement between US- and MRI-derived estimations of PT CSA (standard errors of the estimate of 3.3 mm² for rater 1 and 2.6 mm² for rater 2; Pearson's r = 0.97 and 0.98 for raters 1 and 2, respectively). Within-session reliability for estimations of total PT CSA from US and MRI were excellent (ICC_3,1 >0.95, coefficient of variation [CV] <4.1%, TE = 1.3-3.6 mm². Between-day reliability for US was excellent (ICC_3,1 >0.97, CV <2.7%, TE = 1.6-2.3 mm²), with little difference between raters. These findings suggest that MRI and US both provide reliable estimates of PT CSA and that US can provide a valid measure of PT CSA.

Reliability of a Wearable Motion System for Clinical Evaluation of Dynamic Lumbar Spine Function

Background

Low back pain is the leading cause of disability worldwide. Subjective assessments are often used to assess extent of functional limitations and treatment response. However, these measures have poor sensitivity and are influenced by the patient's perception of their condition. Currently, there are no objective tools to effectively assess the extent of an individual's functional disability and inform clinical decision-making.

Objective

The purpose of this study was to evaluate the reliability of a wearable motion system based on Inertial Measurement Unit (IMU) sensors for use in quantifying low back function.

Methods

Low back motion assessments were conducted by 3 novice raters on 20 participants using an IMU-based motion system. These assessments were conducted over 3 days with 2 days of rest in between tests. A total of 37 kinematic parameters were extracted from the low back motion assessment in all three anatomical planes. Intra-rater and inter-rater reliability were assessed using Intraclass Correlation Coefficients (ICCs) calculated from repeated measures, mixed-effects regression models.

Results

Lumbar spine-specific kinematic parameters showed moderate to excellent reliability across all kinematic parameters. The ICC values ranged between 0.84-0.93 for intra-rater reliability and 0.66 - 0.83 for inter-rater reliability. In particular, velocity measures showed higher reliabilities than other kinematic variables.

Conclusion

The IMU-based wearable motion system is a valid and reliable tool to objectively assess low back function. This study demonstrated that lumbar spine-specific kinematic metrics have the potential to provide good, repeatable metrics to assess clinical function over time.

Development and reliability of a new system for bedside evaluation of non-volitional knee extension force

PURPOSE

Neuromuscular electrical stimulation (NMES) is a widely-used technique for diagnostic and therapeutic purposes. Here we developed and tested the reliability of a new NMES-dynamometer system for bedside evaluation of knee extensor muscle function.

MATERIALS AND METHODS

Thirty-two healthy participants (16 men, 16 women; 27±5 years) completed two testing sessions, 7 days apart. On day 1, a single experienced rater, who repeated the evaluation on day 2 with two other raters, completed a standardized testing procedure. Participants were placed supine, with knees flexed and legs connected to the dynamometer. Maximal voluntary knee extensor isometric force (MVF) and supramaximal twitch force (TwF) were obtained.

RESULTS

High intra-rater intraclass correlation coefficients were observed for both MVF (0.91) and TwF (0.94). MVF and TwF standard error of measurements (8.2%, 5.9%) and minimal detectable changes (16%, 11.6%) were low compared to mean values. High intraclass correlation coefficients were also observed for inter-rater comparisons of MVF (0.89) and TwF (0.86). Standard errors of measurements (MVF: 8.7%, TwF: 5.5%) and minimal detectable changes (MVF: 17.2%, TwF: 10.8%) were similar to intra-rater comparisons.

CONCLUSION

The good reliability of the novel NMES-dynamometer system suggests it as an appropriate tool for the bedside evaluation of knee extensor muscle function.

Reliability of knee extensor neuromuscular structure and function and functional tests' performance

INTRODUCTION

The aim of this study was to evaluate the intra and inter-rater and inter-analyzer reliability of neuromuscular variables and functional tests.

METHODS

Cross-sectional crossover design. Two independent raters and analyzers evaluated twenty-two healthy subjects. Knee-extensor strength was assessed from three maximal voluntary isometric contractions. Muscle activation was obtained from the vastus lateralis (VL), rectus femoris (RF), and vastus medialis (VM) muscles. VL and RF muscles' architecture [fascicle length (FL), pennation angle (PA), muscle thickness (MT)] was obtained at rest by ultrasound. The time from five sit-to-stand (STS) trials, and the distance from the 6-min walk test (6MWT) were obtained. Intraclass correlation coefficient was determined and classified as strong (r = 0.75-1.00), moderate (r = 0.40-0.74), and weak (r < 0.40).

RESULTS

Strong intra-rater reliability values were observed for strength (r = 0.97), muscle activation [VL (r = 0.91); RF (r = 0.92); VM (r = 0.80)], VL [FL (r = 0.90); PA (r = 0.94); MT (r = 0.99)] and RF [MT (r = 0.85)] muscle architecture, STS (r = 0.95), and 6MWT (r = 0.98). Inter-rater reliability also presented strong values for strength (r = 0.97), muscle activation [VL (r = 0.94); RF (r = 0.79); VM (r = 0.78)], muscle architecture VL [PA (r = 0.81) and MT (r = 0.88)] and RF [MT (r = 0.80)], STS (r = 0.93), and 6MWT (r = 0.98). A moderate correlation VL muscle architecture [FL (r = 0.69)]. Inter-analyzer muscle architecture reliability presented strong VL [FL (r = 0.77); PA (r = 0.76); MT (r = 0.91)] and RF [MT (r = 0.99)].

CONCLUSION

The high intra and inter-rater and inter-analyzer reliability values for most variables is evidence that they can be used for clinical evaluation. Muscle architecture might need a longer training period by different raters and analyzers to increase reliability.

Concurrent validity and reliability of measuring range of motion during the cervical flexion rotation test with a novel digital goniometer

BACKGROUND

Headache is a common and costly health problem. Although the pathogenesis of headache is heterogeneous, reported contributing factors are dysfunctions of the upper cervical spine. The flexion rotation test (FRT) is a commonly used diagnostic test to detect upper cervical movement impairment. A digital goniometer may support precise measurement of movement impairment in the upper cervical spine. However, its reliability and validity is not assessed, yet. The aim of this study was to investigate the reliability and validity of the digital goniometer compared to an ultrasound-based movement analysis system.

METHODS

Two separate cross-sectional studies were conducted using the digital goniometer EasyAngle (Meloq AB, Stockholm, Sweden) for a) investigating the concurrent validity of upper cervical range of motion (ROM) during the FRT and b) determining the inter- and intra-rater reliability in the target population of patients with head and neck pain. Sixty-two participants, 39 with and 23 without head and neck pain, were recruited for the concurrent validity study. For the reliability study, a total of 50 participants were recruited. Intraclass correlation coefficients (ICC) and Bland Altmann plots were used to assess validity and ICC values, Bland Altmann plots as well as Kappa coefficients were used for estimating intra-rater and inter-rater reliability.

RESULTS

Concurrent validity was strong with an ICC (2,1) of 0.97 for ROM to either side (95%CI = 0.95-0.98). Bland Altman Plots revealed a mean difference between measurement systems of 0.5° for the left and 0.11° for the right side. The inter-rater ICC (2,1) was 0.66 (95%CI 0.47-0.79, p <  0.001, SEM 6.6°), indicating good reliability. The limits of agreement were between 10.25° and - 11.89°, the mean difference between both raters was - 0.82°. Intra-rater reliability for the measurement of ROM during the FRT was between 0.96 (ICC 3,1) for rater 1 and 0.94 (ICC 3,1) for rater 2.

CONCLUSIONS

The digital goniometer demonstrated strong concurrent validity and good to strong reliability and can be used in clinical practice to accurately determine movement impairment in the upper cervical spine.

TRIAL REGISTRATION

German Registry of Clinical Trials DRKS00013051 .

Echo intensity reliability between two rectus femoris probe sites

Introduction

The ultrasound technique has been extensively used to measure echo intensity, with the goal of measuring muscle quality, muscle damage, or to detect neuromuscular disorders. However, it is not clear how reliable the technique is when comparing different days, raters, and analysts, or if the reliability is affected by the muscle site where the image is obtained from. The goal of this study was to compare the intra-rater, inter-rater, and inter-analyst reliability of ultrasound measurements obtained from two different sites at the rectus femoris muscle.

Methods

Muscle echo intensity was quantified from ultrasound images acquired at 50% [RF₅₀] and at 70% [RF₇₀] of the thigh length in 32 healthy subjects.

Results

Echo intensity values were higher (p = 0.0001) at RF₅₀ (61.08 ± 12.04) compared to RF₇₀ (57.32 ± 12.58). Reliability was high in both RF₅₀ and RF₇₀ for all comparisons: intra-rater (ICC = 0.89 and 0.94), inter-rater (ICC = 0.89 and 0.89), and inter-analyst (ICC = 0.98 and 0.99), respectively. However, there were differences (p < 0.05) between raters and analysts when obtaining/analyzing echo intensity values in both rectus femoris sites.

Conclusions

The differences in echo intensity values between positions suggest that rectus femoris's structure is not homogeneous, and therefore measurements from different muscle regions should not be used interchangeably. Both sites showed a high reliability, meaning that the measure is accurate if performed by the same experienced rater in different days, if performed by different experienced raters in the same day, and if analyzed by different well-trained analysts, regardless of the evaluated muscle site.

Athletes at late stage rehabilitation have persisting deficits in plantar- and dorsiflexion, and inversion (but not eversion) after ankle sprain

OBJECTIVES

Document reliability and normative data for a novel device measuring weight-bearing ankle range of motion after ankle injury.

DESIGN

Cross-sectional Cohort, two occasions one day apart.

SETTING

Sports medicine hospital.

PARTICIPANTS

87 ankle-injured male athletes at a late stage of their rehabilitation and 25 uninjured subjects. The injured athletes had met all criteria to return to functional, on-field rehabilitation.

MAIN OUTCOME MEASURES

Reliability (Intra-Class correlation Coefficient (ICC), and Minimum Detectable Change as a percent of the grand mean), weight-bearing range of motion (degrees) of dorsiflexion, plantarflexion, inversion, and eversion.

RESULTS

Good (dorsiflexion = 0.82[0.76-0.87] and inversion = 0.81[0.75-0.86]) and excellent (plantarflexion = 0.93[0.90-0.95]) reliability was documented, however reliability for the eversion measure showed only fair reliability (0.61[0.49-0.70]). Reduced range of motion in the injured leg was seen in all 4 directions, however with different magnitudes: Large differences were plantarflexion (-8.5°, ES = 0.80), medium for dorsiflexion (-5.2°, 0.57), small for inversion (-4.8°, 0.36), and trivial for eversion (-1.7°, 0.15).

CONCLUSION

The device demonstrated clinically useful reliability for measuring these ranges of motion in a functional, weight-bearing position. PF ROM showed the greatest reduction in range in these athletes at a late stage of their rehabilitation.

Inter- and intra-rater reliability of vertebral fracture classifications in the Swedish fracture register

AIM

To investigate the inter- and intra-rater reliability of the vertebral fracture classifications used in the Swedish fracture register.

METHODS

Radiological images of consecutive patients with cervical spine fractures (n = 50) were classified by 5 raters with different experience levels at two occasions. An identical process was performed with thoracolumbar fractures (n = 50). Cohen’s kappa was used to calculate the inter- and intra-rater reliability.

RESULTS

The mean kappa coefficient for inter-rater reliability ranged between 0.54 and 0.79 for the cervical fracture classifications, between 0.51 and 0.72 for the thoracolumbar classifications (overall and for different sub classifications), and between 0.65 and 0.77 for the presence or absence of signs of ankylosing disorder in the fracture area. The mean kappa coefficient for intra-rater reliability ranged between 0.58 and 0.80 for the cervical fracture classifications, between 0.46 and 0.68 for the thoracolumbar fracture classifications (overall and for different sub classifications) and between 0.79 and 0.81 for the presence or absence of signs of ankylosing disorder in the fracture area.

CONCLUSION

The classifications used in the Swedish fracture register for vertebral fractures have an acceptable inter- and intra-rater reliability with a moderate strength of agreement.

Information from dynamic length changes improves reliability of static ultrasound fascicle length measurements

Purpose

Various strategies for improving reliability of fascicle identification on ultrasound images are used in practice, yet these strategies are untested for effectiveness. Studies suggest that the largest part of differences between fascicle lengths on one image are attributed to the error on the initial image. In this study, we compared reliability results between different strategies.

Methods

Static single-image recordings and image sequence recordings during passive ankle rotations of the medial gastrocnemius were collected. Images were tracked by three different raters. We compared results from uninformed fascicle identification (UFI) and results with information from dynamic length changes, or data-informed tracking (DIT). A second test compared tracking of image sequences of either fascicle shortening (initial-long condition) or fascicle lengthening (initial-short condition).

Results

Intra-class correlations (ICC) were higher for the DIT compared to the UFI, yet yielded similar standard error of measurement (SEM) values. Between the initial-long and initial-short conditions, similar ICC values, coefficients of multiple determination, mean squared errors, offset-corrected mean squared errors and fascicle length change values were found for the DIT, yet with higher SEM values and greater absolute fascicle length differences between raters on the first image in the initial-long condition and on the final image in the initial-short condition.

Conclusions

DIT improves reliability of fascicle length measurements, without lower SEM values. Fascicle length on the initial image has no effect on subsequent tracking results. Fascicles on ultrasound images should be identified by a single rater and care should be taken when comparing absolute fascicle lengths between studies.

How to assess intra- and inter-observer agreement with quantitative PET using variance component analysis: a proposal for standardisation

BACKGROUND

Quantitative measurement procedures need to be accurate and precise to justify their clinical use. Precision reflects deviation of groups of measurement from another, often expressed as proportions of agreement, standard errors of measurement, coefficients of variation, or the Bland-Altman plot. We suggest variance component analysis (VCA) to estimate the influence of errors due to single elements of a PET scan (scanner, time point, observer, etc.) to express the composite uncertainty of repeated measurements and obtain relevant repeatability coefficients (RCs) which have a unique relation to Bland-Altman plots. Here, we present this approach for assessment of intra- and inter-observer variation with PET/CT exemplified with data from two clinical studies.

METHODS

In study 1, 30 patients were scanned pre-operatively for the assessment of ovarian cancer, and their scans were assessed twice by the same observer to study intra-observer agreement. In study 2, 14 patients with glioma were scanned up to five times. Resulting 49 scans were assessed by three observers to examine inter-observer agreement. Outcome variables were SUVmax in study 1 and cerebral total hemispheric glycolysis (THG) in study 2.

RESULTS

In study 1, we found a RC of 2.46 equalling half the width of the Bland-Altman limits of agreement. In study 2, the RC for identical conditions (same scanner, patient, time point, and observer) was 2392; allowing for different scanners increased the RC to 2543. Inter-observer differences were negligible compared to differences owing to other factors; between observer 1 and 2: -10 (95 % CI: -352 to 332) and between observer 1 vs 3: 28 (95 % CI: -313 to 370).

CONCLUSIONS

VCA is an appealing approach for weighing different sources of variation against each other, summarised as RCs. The involved linear mixed effects models require carefully considered sample sizes to account for the challenge of sufficiently accurately estimating variance components.

Vibration sensibility of the median nerve in a population with chronic whiplash associated disorder: Intra- and inter-rater reliability study

Whiplash Associated Disorders (WAD) grade II are the most prevalent group of whiplash patients seen on a regular basis by musculoskeletal physiotherapists. Impairment of vibration sensibility may be an early indicator of nerve pathology and it has previously been demonstrated in individuals with chronic WAD symptoms utilising vibrameters. A less expensive option, such the tuning fork (TF) may assist with these measures, but research regarding its measurement properties is lacking.

OBJECTIVES

To investigate the intra- and inter-rater reliability of vibration sensibility of the median nerve in chronic WAD II (CWAD II).

METHODS

A double blinded, within day intra- and inter-rater reliability study was undertaken. A convenience sample of 26 individuals (8 males, 18 females, age mean 29.9 ± 10.0 years) with CWADII was recruited.

EXCLUSION CRITERIA

WAD I, III & indications of neuropathic pain. Vibration attenuation times were recorded from skin innervated by the median nerve (thenar eminence).

RESULTS

Descriptive statistics (mean scores) and reliability statistics [intraclass correlation coefficient (ICC2,1) and Bland and Altman limits of agreement] were undertaken with p = 0.05. Almost perfect intra-rater reliability (Intraclass Correlation Coefficiency (ICC): 0.972-0.955) and inter-rater reliability (ICC: 0.983) were identified. Confidence Intervals (CI) for inter-rater reliability were 95% CI: -1.461 to -0.056.

CONCLUSIONS

Almost perfect reliability scores across intra- and inter-rater reliability were found. This provides evidence that, with a standardised testing protocol the TF can be a highly reliable means of vibration sensibility testing. Future studies assessing the validity of the TF in different WAD populations may provide further information about the usefulness of this protocol.

Intra-rater and inter-rater reliability of ultrasonographic measurements of acromion-greater tuberosity distance in patients with post-stroke hemiplegia

BACKGROUND

Glenohumeral subluxation (GHS) is reported in up to 81% of patients with stroke. Ultrasonographic measurements of GHS by measuring the acromion-greater tuberosity (AGT) have been found to be reliable for experienced raters.

OBJECTIVES

The primary aim was to assess the intra-rater reliability of measurements of AGT distance in people with stroke following a short course of rater training. A secondary aim was to compare the inter-rater reliability of these measurements between novice and experienced raters.

METHODS

Patients with stroke (n = 16; 5 men, 11 women; 74 ± 10 years) with 1-sided weakness who gave informed consent were recruited. Ultrasonographic measurements were recorded at the bedside by two physiotherapists with patients seated upright in a hospital chair. Reliability was assessed by intra-class correlation coefficients (ICCs) and the standard error of measurements (SEM). Minimum detectable change (MDC90) scores were used to estimate the magnitude of change that is likely to exceed measurement error.

RESULTS

Mean ± SD AGT distances on the affected and unaffected sides for rater 1 were 2.2 ± 0.7 and 1.7 ± 0.4 cm, respectively. Corresponding values for rater 2 were 2.5 ± 0.6 and 2.0 ± 0.4 cm. Intra-class correlation coefficient values for the affected and unaffected shoulders for rater 1 were 0.96 and 0.91, respectively. Corresponding values for rater 2 were 0.95 and 0.90.SEM and MDC90 for both affected and unaffected shoulders were ≤ 0.2 cm. Inter-rater reliability coefficients were 0.86 (affected) and 0.76 (unaffected) shoulders.

CONCLUSION

Ultrasonographic measurement of AGT distance demonstrates excellent intra-rater reliability for a novice rater. Inter-rater reliability of ultrasonographic measurement of AGT also demonstrates good reliability between novice and experienced raters.

Inter- and intra-rater reliability of diffusion tensor imaging parameters in the normal pediatric spinal cord

AIM

To assess inter- and intra-rater reliability (agreement) between two region of interest (ROI) methods in pediatric spinal cord diffusion tensor imaging (DTI).

METHODS

Inner-Field-of-View DTI data previously acquired from ten pediatric healthy subjects (mean age = 12.10 years) was used to assess for reliability. ROIs were drawn by two neuroradiologists on each subject data twice within a 3-mo interval. ROIs were placed on axial B0 maps along the cervical spine using free-hand and fixed-size ROIs. Agreement analyses for fractional anisotropy (FA), axial diffusivity, radial diffusivity and mean diffusivity were performed using intra-class-correlation (ICC) and Cronbach's alpha statistical methods.

RESULTS

Inter- and intra-rater agreement between the two ROI methods showed moderate (ICC = 0.5) to strong (ICC = 0.84). There were significant differences between raters in the number of pixels selected using free-hand ROIs (P < 0.05). However, no significant differences were observed in DTI parameter values. FA showed highest variability in ICC values (0.10-0.87). Cronbach's alpha showed moderate-high values for raters and ROI methods.

CONCLUSION

The study showed that high reproducibility in spinal cord DTI can be achieved, and demonstrated the importance of setting detailed methodology for post-processing DTI data, specifically the placement of ROIs.

Inter-expert and intra-expert reliability in sleep spindle scoring

OBJECTIVES

To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings.

METHODS

The EEG dataset was comprised of 400 randomly selected 115s segments of stage 2 sleep from 110 sleeping subjects in the general population (57±8, range: 42-72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F1-scores, Cohen's kappa (κ), and intra-class correlation coefficient (ICC).

RESULTS

We found an average intra-expert F1-score agreement of 72±7% (κ: 0.66±0.07). The average inter-expert agreement was 61±6% (κ: 0.52±0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system.

CONCLUSIONS

We estimate that 2-3 experts are needed to build a spindle scoring dataset with 'substantial' reliability (κ: 0.61-0.8), and 4 or more experts are needed to build a dataset with 'almost perfect' reliability (κ: 0.81-1).

SIGNIFICANCE

Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.