On Jones et al.'s method for extending Bland-Altman plots to limits of agreement with the mean for multiple observers

Towards clinical implementation of automated segmentation of vestibular schwannomas: a reliability study comparing AI and human performance

PURPOSE

To evaluate the clinimetric reliability of automated vestibular schwannoma (VS) segmentations by a comparison with human inter-observer variability on T1-weighted contrast-enhanced MRI scans.

METHODS

This retrospective study employed MR images, including follow-up, from 1,015 patients (median age: 59, 511 men), resulting in 1,856 unique scans. Two nnU-Net models were trained using fivefold cross-validation to create a single-center segmentation model, along with a multi-center model using additional publicly available data. Geometric-based segmentation metrics (e.g. the Dice score) were used to evaluate model performance. To quantitatively assess the clinimetric reliability of the models, automated tumor volumes from a separate test set were compared to human inter-observer variability using the limits of agreement with the mean (LOAM) procedure. Additionally, new agreement limits that include automated annotations are calculated.

RESULTS

Both models performed comparable to current state-of-the-art VS segmentation models, with median Dice scores of 91.6% and 91.9% for the single and multi-center models, respectively. There is a stark difference in clinimetric performance between both models: automated tumor volumes of the multi-center model fell within human agreement limits in 73% of the cases, compared to 44% for the single-center model. Newly calculated agreement limits including the single-center model, resulted in very high and wide limits. For the multi-center model, the new agreement limits were comparable to human inter-observer variability.

CONCLUSION

Models with excellent geometric-based metrics do not necessarily imply high clinimetric reliability, demonstrating the need to clinimetrically evaluate models as part of the clinical implementation process. The multi-center model displayed high reliability, warranting its possible future use in clinical practice. However, caution should be exercised when employing the model for small tumors, as the reliability was found to be volume-dependent.

Fasciculation distribution in a healthy population assessed with diffusion tensor imaging

Fasciculations, a hallmark of motor neuron diseases, also occur in healthy individuals, highlighting the need to understand fasciculation intensity and distribution. Motor unit MRI (MUMRI) can assess fasciculations in large volumes but is not widely applied. We hypothesize that a more common MRI technique, diffusion tensor imaging (DTI), can also detect fasciculation when correcting for low signal-to-noise ratios and signal variability. We first systematically compared MUMRI and DTI in upper leg muscles of healthy subjects (n = 5). Secondly, we retrospectively determined fasciculation intensity and distribution in lower extremity muscles of 30 healthy subjects using DTI (n = 30). DTI and MUMRI had comparable sensitivity (75%) and precision (80%) to expert reviews. In our healthy cohort, fasciculations were more prevalent in the lower legs than upper legs (13.9 ± 11.5% vs. 9.8 ± 6.3%, p = 0.011), particularly in the soleus (9.3 ± 8.1%). This effect persisted after normalizing for muscle volume (7.2 ± 5.1%/dm3 vs. 2.9 ± 1.8%/dm3, p < 0.001). Lower leg fasciculations were larger compared to upper leg fasciculations (0.81 ± 0.31 cm3 vs. 0.54 ± 0.15 cm3, p < 0.001). Longitudinal analysis showed consistent fasciculation distribution over 8 months (n = 13, ICC = 0.803). In conclusion, muscle DTI detects fasciculations in all lower extremity muscles, enabling retrospective analysis of existing datasets and reducing the need for prospective MUMRI studies if muscle DTI is already acquired.

Defining tumor growth in vestibular schwannomas: a volumetric inter-observer variability study in contrast-enhanced T1-weighted MRI

PURPOSE

For patients with vestibular schwannomas (VS), a conservative observational approach is increasingly used. Therefore, the need for accurate and reliable volumetric tumor monitoring is important. Currently, a volumetric cutoff of 20% increase in tumor volume is widely used to define tumor growth in VS. The study investigates the tumor volume dependency on the limits of agreement (LoA) for volumetric measurements of VS by means of an inter-observer study.

METHODS

This retrospective study included 100 VS patients who underwent contrast-enhanced T1-weighted MRI. Five observers volumetrically annotated the images. Observer agreement and reliability was measured using the LoA, estimated using the limits of agreement with the mean (LOAM) method, and the intraclass correlation coefficient (ICC).

RESULTS

The 100 patients had a median average tumor volume of 903 mm3 (IQR: 193-3101). Patients were divided into four volumetric size categories based on tumor volume quartile. The smallest tumor volume quartile showed a LOAM relative to the mean of 26.8% (95% CI: 23.7-33.6), whereas for the largest tumor volume quartile this figure was found to be 7.3% (95% CI: 6.5-9.7) and when excluding peritumoral cysts: 4.8% (95% CI: 4.2-6.2).

CONCLUSION

Agreement limits within volumetric annotation of VS are affected by tumor volume, since the LoA improves with increasing tumor volume. As a result, for tumors larger than 200 mm3, growth can reliably be detected at an earlier stage, compared to the currently widely used cutoff of 20%. However, for very small tumors, growth should be assessed with higher agreement limits than previously thought.

Reproducibility and Repeatability in Focus: Evaluating LVEF Measurements with 3D Echocardiography by Medical Technologists

Three-dimensional echocardiography (3DE) is currently the preferred method for monitoring left ventricular ejection fraction (LVEF) in cancer patients receiving potentially cardiotoxic anti-neoplastic therapy. In Denmark, however, the traditional standard for LVEF monitoring has been rooted in nuclear medicine departments utilizing equilibrium radionuclide angiography (ERNA). Although ERNA remains a principal modality, there is an emerging trend towards the adoption of echocardiography for this purpose. Given this context, assessing the reproducibility of 3DE among non-specialized medical personnel is crucial for its clinical adoption in such departments. To assess the feasibility of 3DE for LVEF measurements by technologists, we evaluated the repeatability and reproducibility of two moderately experienced technologists. They performed 3DE on 12 volunteers over two sessions, with a collaborative review of the results from the first session before the second session. Two-way intraclass correlation values increased from 0.03 to 0.77 across the sessions. This increase in agreement was mainly due to the recognition of false low measurements. Our findings underscore the importance of incorporating reproducibility exercises in the context of 3DE, especially when operated by technologists. Additionally, routine control of the acquisitions by physicians is deemed necessary. Ensuring these hurdles are adequately managed enables the adoption of 3DE for LVEF measurements by technologists.

Comparison of Blood Pressure Measurement Results from Two Different Regions (Upper Arm and Ankle)

BACKGROUND

This study was conducted to determine the clinical concordance of non-invasive blood pressure (NIBP) measurements from the upper arm and ankle and to determine whether there is a difference between them.

METHODS

A prospective and cross-sectional research design was used in this study. The study population included 106 patients hospitalized in the surgical clinics of a training and research hospital. Non-invasive blood pressure measurements were performed from both upper arms and ankles in the supine position. In the data analysis, ANOVA and the Bland-Altman plot were employed.

RESULTS

Based on the results of NIBP measurements from the arm and ankle, the differences between diastolic blood pressure (DBP) and mean blood pressure (MBP) were acceptable and within the limits of concordance. The difference between the mean systolic blood pressure (SBP) was found to be significantly different and outside the limits of concordance. The difference between the mean SBP measured from the ankle and the upper arm was determined to be 18 mmHg.

CONCLUSION

It was determined that the results of SBP measured from the ankle were significantly higher and outside the limits of concordance compared to the upper arm, while the results of DBP and MBP measurements were within acceptable limits.

Clinical evaluation of AI-assisted muscle ultrasound for monitoring muscle wasting in ICU patients

Muscle ultrasound has been shown to be a valid and safe imaging modality to assess muscle wasting in critically ill patients in the intensive care unit (ICU). This typically involves manual delineation to measure the rectus femoris cross-sectional area (RFCSA), which is a subjective, time-consuming, and laborious task that requires significant expertise. We aimed to develop and evaluate an AI tool that performs automated recognition and measurement of RFCSA to support non-expert operators in measurement of the RFCSA using muscle ultrasound. Twenty patients were recruited between Feb 2023 and July 2023 and were randomized sequentially to operators using AI (n = 10) or non-AI (n = 10). Muscle loss during ICU stay was similar for both methods: 26 ± 15% for AI and 23 ± 11% for the non-AI, respectively (p = 0.13). In total 59 ultrasound examinations were carried out (30 without AI and 29 with AI). When assisted by our AI tool, the operators showed less variability between measurements with higher intraclass correlation coefficients (ICCs 0.999 95% CI 0.998-0.999 vs. 0.982 95% CI 0.962-0.993) and lower Bland Altman limits of agreement (± 1.9% vs. ± 6.6%) compared to not using the AI tool. The time spent on scans reduced significantly from a median of 19.6 min (IQR 16.9-21.7) to 9.4 min (IQR 7.2-11.7) compared to when using the AI tool (p < 0.001). AI-assisted muscle ultrasound removes the need for manual tracing, increases reproducibility and saves time. This system may aid monitoring muscle size in ICU patients assisting rehabilitation programmes.

The Trochlear Bisector as a New Landmark for Kinematic Alignment in Total Knee Arthroplasty: A Radiographic Study

Background: In recent years, there has been considerable interest in prosthetic alignment techniques for total knee arthroplasty (TKA), particularly in the so-called kinematic alignment, which aims to restore the knee's native alignment. However, implementing this technique requires specialized instruments and procedural steps that can be laborious. This study introduces the bisector of the trochlear groove as a reliable landmark for performing the distal femoral cut while maintaining parallelism with the native femoral joint line. Methods: Three orthopedic specialists assessed 110 X-ray images of full-leg, weight-bearing lower limbs obtained from healthy individuals between January 2021 and December 2022. The bisector of the trochlear groove was identified on the X-ray images, and the angle between this bisector and the femoral joint line was measured. The consistency of these measurements across repeated assessments and different examiners was evaluated. Results: The bisector of the trochlear groove was found to be perpendicular to the femoral joint line, with a mean angle of 89.4°. The inter-rater reliability was 68% within ±1.3° from the mean, while the intra-rater reliability was 82% within ±1.5° from the mean. Conclusions: These results suggest that by performing a femoral cut perpendicular to the bisector of the trochlear groove, surgeons can inherently restore the femoral joint line of the native knee in patients where the native joint line is no longer identifiable due to the effect of osteoarthritis. This method may offer a viable and straightforward alternative to the standard surgical technique currently practiced for kinematic alignment in TKA.

Is the Bland-Altman plot method useful without inferences for accuracy, precision, and agreement?

OBJECTIVE

This study aims to propose a comprehensive alternative to the Bland-Altman plot method, addressing its limitations and providing a statistical framework for evaluating the equivalences of measurement techniques. This involves introducing an innovative three-step approach for assessing accuracy, precision, and agreement between techniques, which enhances objectivity in equivalence assessment. Additionally, the development of an R package that is easy to use enables researchers to efficiently analyze and interpret technique equivalences.

METHODS

Inferential statistics support for equivalence between measurement techniques was proposed in three nested tests. These were based on structural regressions with the goal to assess the equivalence of structural means (accuracy), the equivalence of structural variances (precision), and concordance with the structural bisector line (agreement in measurements obtained from the same subject), using analytical methods and robust approach by bootstrapping. To promote better understanding, graphical outputs following Bland and Altman's principles were also implemented.

RESULTS

The performance of this method was shown and confronted by five data sets from previously published articles that used Bland and Altman's method. One case demonstrated strict equivalence, three cases showed partial equivalence, and one showed poor equivalence. The developed R package containing open codes and data are available for free and with installation instructions at Harvard Dataverse at https://doi.org/10.7910/DVN/AGJPZH.

CONCLUSION

Although easy to communicate, the widely cited and applied Bland and Altman plot method is often misinterpreted, since it lacks suitable inferential statistical support. Common alternatives, such as Pearson's correlation or ordinal least-square linear regression, also fail to locate the weakness of each measurement technique. It may be possible to test whether two techniques have full equivalence by preserving graphical communication, in accordance with Bland and Altman's principles, but also adding robust and suitable inferential statistics. Decomposing equivalence into three features (accuracy, precision, and agreement) helps to locate the sources of the problem when fixing a new technique.

Ultrasound assessment of testicular volume - An interobserver variability study

BACKGROUND

Ultrasonography measurement of the testicles and subsequent calculation of the testicular volume is recommended as a part of a standard scrotal ultrasound examination. The interobserver variability of testicular volume measurement has implications for surgical recommendations. Therefore, this study aimed to investigate the interobserver variability in the measurement of testicular volume.

METHODS

Interobserver variability was established by comparing testicular measurements performed by two observers on the same patient during the same clinical appointment. The observers were blinded to each other's measurements. Testicular volume was calculated using the Lambert formula: length x width x height x 0.71. A total of three observers, A, B and C, participated in the study. The observers had between 4 to 20 years' experience with scrotal ultrasound examinations.

RESULTS

In total, 24 patients' were included (48 testicles). The patient´s mean age was 43 years (range 19-75 years). The overall mean right testicular volume was 19.8 ml (range 7.3-31.6 ml), and the left was 20.1 ml (range 7.1-36.1 ml). The interclass correlation coefficient (ICC) between observer A and B was excellent (ICC= 0.98, CI:0.92-0.99), between observer A and C, was excellent (ICC=0.91, CI: 0.77-0.97) and between B and C good (ICC=0.82, CI:0.51-0.93).

CONCLUSION

Variability in estimating testicular volume is low, with interobserver agreement ranging from good to excellent. Ultrasound provides a highly reproducible tool to determine testicular volume.

Dataset on renal tumor diameter assessment by multiple observers in normal-dose and low-dose CT

Computed tomography-based active surveillance is increasingly used to manage small renal tumors, regardless of patient age. However, there is an unmet need for decreasing radiation exposure while maintaining the necessary accuracy and reproducibility in radiographic measurements, allowing for detecting even minor changes in renal mass size. In this article, we present supplementary data from a multiobserver investigation. We explored the accuracy and reproducibility of low-dose CT (75% dose reduction) compared to normal-dose CT in assessing maximum axial renal tumor diameter. Open-access CT datasets from the 2019 Kidney and Kidney Tumor Segmentation Challenge were used. A web-based platform for assessing observer performance was used by six radiologist observers to obtain and provide data on tumor diameters and accompanying viewing settings, in addition to key images of each measurement and an interactive module for exploring diameter measurements. These data can serve as a baseline and inform future studies investigating and validating lower-dose CT protocols for active surveillance of small renal masses.

QUANTITATIVE ASSESSMENT OF MYOCARDIAL FIBROSIS BY DIGITAL IMAGE ANALYSIS: an adjunctive tool for pathologist "ground truth": original article

AIMS

Myocardial fibrosis (MF) is a common pathological process in a wide range of cardiovascular diseases. Its quantity has diagnostic and prognostic relevance. We aimed to assess if the complementary use of an automated artificial intelligence software might improve the precision of the pathologist´s quantification of MF on endomyocardial biopsies (EMB).

METHODS AND RESULTS

Intraoperative EMB samples from 30 patients with severe aortic stenosis submitted to surgical aortic valve replacement were analysed. Tissue sections were stained with Masson´s trichrome for collagen/fibrosis and whole slide images (WSI) from the experimental glass slides were obtained at a resolution of 0.5μm using a digital microscopic scanner. Three experienced pathologists made a first quantification of MF excluding the subendocardium. After two weeks, an algorithm for Masson´s trichrome brightfield WSI (at QuPath software) was applied and the automatic quantification was revealed to the pathologists, who were asked to reassess MF, blinded to their first evaluation. The impact of the automatic algorithm on the inter-observer agreement was evaluated using Bland-Altman type methodology. Median values of MF on EMB were 8.33% [IQR 5.00-12.08%] and 13.60% [IQR 7.32-21.2%], respectively for the first pathologist´s and automatic algorithm quantification, being highly correlated (R²: 0.79; p<0.001). Inter-observer discordance was relevant, particularly for higher percentages of MF. The knowledge of the automatic quantification significantly improved the overall pathologist´s agreement, which became unaffected by the degree of MF severity.

CONCLUSIONS

The use of an automated artificial intelligence software for MF quantification on EMB samples improves the reproducibility of measurements by experienced pathologists. By improving the reliability of the quantification of myocardial tissue components, this adjunctive tool may facilitate the implementation of imaging-pathology correlation studies.

Inter- and intra-observer variability of computed tomography-based parenchymal- and ductal diameters in chronic pancreatitis: a multi-observer international study

PURPOSE

The need for incorporation of quantitative imaging biomarkers of pancreatic parenchymal and ductal structures has been highlighted in recent proposals for new scoring systems in chronic pancreatitis (CP). To quantify inter- and intra-observer variability in CT-based measurements of ductal- and gland diameters in CP patients.

MATERIALS AND METHODS

Prospectively acquired pancreatic CT examinations from 50 CP patients were reviewed by 12 radiologists and four pancreatologists from 10 institutions. Assessment entailed measuring maximum diameter in the axial plane of four structures: (1) pancreatic head (PDhead), (2) pancreatic body (PDbody), (3) main pancreatic duct in the pancreatic head (MPDhead), and (4) body (MPDbody). Agreement was assessed by the 95% limits of agreement with the mean (LOAM), representing how much a single measurement for a specific subject may plausibly deviate from the mean of all measurements on the specific subject. Bland-Altman limits of agreement (LoA) were generated for intra-observer pairs.

RESULTS

The 16 observers completed 6400 caliper placements comprising a first and second measurement session. The widest inter-observer LOAM was seen with PDhead (± 9.1 mm), followed by PDbody (± 5.1 mm), MPDhead (± 3.2 mm), and MPDbody (± 2.6 mm), whereas the mean intra-observer LoA width was ± 7.3, ± 5.1, ± 3.7, and ± 2.4 mm, respectively.

CONCLUSION

Substantial intra- and inter-observer variability was observed in pancreatic two-point measurements. This was especially pronounced for parenchymal and duct diameters of the pancreatic head. These findings challenge the implementation of two-point measurements as the foundation for quantitative imaging scoring systems in CP.

Reliability of radiographic assessment of growth modulation from anterior vertebral body tethering surgery in pediatric scoliosis

PURPOSE

To assess the reliability of vertebral height and angular measurements for anterior vertebral body tethering (AVBT).

METHODS

Eight observers measured PA radiographs of 15 idiopathic scoliosis patients treated with AVBT, pre-operative and 4-year follow-up. Vertebral wedging, disc wedging, convex vertebral body heights, and concave vertebral body heights of the 3 apical vertebrae were measured. For each observer, there were a total of 90 measurements for vertebral body height and 75 measurements for all wedging types At least 14 days elapsed between first and second round measurements.

RESULTS

From the pre-operative to the 4-year follow-up time-point, the total wedging angle over the 3 peri-apical levels fell from 30 ± 7° to 16 ± 6° (p < 0.001) and the difference between the convex and concave vertebral heights decreased from 9 ± 4 to 6 ± 3 mm (p < 0.001). Interobserver agreement for pre-operative vertebral body heights was good [ICC = 0.80; 95% CI (0.74-0.85)]. At 4-year follow-up there was a moderate agreement [ICC = 0.67 (0.59-0.74)]. There was a poor interobserver agreement for pre-operative wedging angle measurements [ICC = 0.41 (0.32-0.52)] and 4-year follow-up [ICC = 0.45 (0.36-0.56)]. The Limits of Agreement with the Mean (LOAM) for pre-op heights was ± 2.4 mm, similar to the follow-up ± 2.6 mm. When raters are averaged in random groups of two the agreement limits decrease to ± 1.8 mm pre-op and ± 1.6 mm at follow-up. Similarly for wedging angles, LOAM values among the 8 observers of ± 4.6° pre-op and ± 4.2° dropped to ± 2.7° for both pre-op and follow-up when random groups of two raters were averaged together. Intraobserver agreement ranged from good to excellent per individual (ICC = 0.84-0.94) for pre-operative vertebral body heights, but this decreased at 4-year follow-up (ICC = 0.52-0.88). Intraobserver agreement was low overall for wedging (pre-operative ICC = 0.41-0.71; 4-year follow-up ICC = 0.41-0.76).

CONCLUSION

Based on 8 individual observers, interobserver agreement ranged from good (pre-operative vertebral body heights) to moderate (4-year follow-up vertebral body heights) to poor (all wedging angles). To improve the reliability of the measurement of wedging angles, we recommend averaging the measurements of at least two observers.

Isolation and Barcoding of Trace Pollen-free DNA for Authentication of Honey

Adulteration and mislabeling of honey to mask its true origin have become a global concern. Pollen microscopy, the current gold standard for identifying honey's geographical and plant origins, is laborious, requires extensive training, and fails to identify filtered honey and honey spiked with pollen from a more favorable plant to disguise its origins. We successfully isolated pollen-free DNA from filtered honey using three types of adsorbents: (i) anti-dsDNA antibodies coupled to magnetic microspheres; (ii) anion-exchange adsorbent; and (iii) ceramic hydroxyapatite. The internal transcribed spacer 2 region of the captured pollen-free DNA was polymerase chain reaction-amplified and subjected to next-generation sequencing. Using an in-house bioinformatics pipeline, initial experiments showed that anion exchange had the greatest capacity to capture trace pollen-free DNA, and it was successfully applied to isolate DNA from five honey samples. Enrichment of trace pollen-free DNA from filtered honey samples opens a new approach for identifying the true origins of honey.

Ultra-low-dose non-contrast CT and CT angiography can be used interchangeably for assessing maximal abdominal aortic diameter

Background

Routine CT scans may increasingly be used to document normal aortic size and to detect incidental abdominal aortic aneurysms.

Purpose

To determine whether ultra-low-dose non-contrast CT (ULDNC-CT) can be used instead of the gold standard CT angiography (CTA) for assessment of maximal abdominal aortic diameter.

Materials and Methods

This retrospective study included 50 patients who underwent CTA and a normal-dose non-contrast CT for suspected renal artery stenosis. ULDNC-CT datasets were generated from the normal-dose non-contrast CT datasets using a simulation technique. Using the centerline technique, radiology consultants (n = 4) and residents (n = 3) determined maximal abdominal aortic diameter. The limits of agreement with the mean (LOAM) was used to access observer agreement. LOAM represents how much a measurement by a single observer may plausibly deviate from the mean of all observers on the specific subject.

Results

Observers completed 1400 measurements encompassing repeated CTA and ULDNC-CT measurements. The mean diameter was 24.0 and 25.0 mm for CTA and ULDNC-CT, respectively, yielding a significant but minor mean difference of 1.0 mm. The 95% LOAM reproducibility was similar for CTA and ULDNC-CT (2.3 vs 2.3 mm). In addition, the 95% LOAM and mean diameters were similar for CTA and ULDNC-CT when observers were grouped as consultants and residents.

Conclusions

Ultra-low-dose non-contrast CT exhibited similar accuracy and reproducibility of measurements compared with CTA for assessing maximal abdominal aortic diameter supporting that ULDNC-CT can be used interchangeably with CTA in the lower range of aortic sizes.

Sample size determination in method comparison and observer variability studies

The comparison of two quantitative measuring devices is often performed with the Limits of Agreement proposed by Bland and Altman in their seminal Lancet paper back in 1986. Sample size considerations were rare for such agreement analyses in the past, but recently several proposals have been made depending on how agreement is to be assessed and the number of replicates to be used. We have summarized recent developments and recommendations in various situations including a distinction between method comparison and observer variability studies. These include current state-of-the-art analysis of and reporting guidelines for agreement studies. General recommendations close the paper.

Bland–Altman Limits of Agreement from a Bayesian and Frequentist Perspective

Bland–Altman agreement analysis has gained widespread application across disciplines, last but not least in health sciences, since its inception in the 1980s. Bayesian analysis has been on the rise due to increased computational power over time, and Alari, Kim, and Wand have put Bland–Altman Limits of Agreement in a Bayesian framework (Meas. Phys. Educ. Exerc. Sci. 2021, 25, 137–148). We contrasted the prediction of a single future observation and the estimation of the Limits of Agreement from the frequentist and a Bayesian perspective by analyzing interrater data of two sequentially conducted, preclinical studies. The estimation of the Limits of Agreement θ1 and θ2 has wider applicability than the prediction of single future differences. While a frequentist confidence interval represents a range of nonrejectable values for null hypothesis significance testing of H0: θ1 ≤ −δ or θ2 ≥ δ against H1: θ1 > −δ and θ2 < δ, with a predefined benchmark value δ, Bayesian analysis allows for direct interpretation of both the posterior probability of the alternative hypothesis and the likelihood of parameter values. We discuss group-sequential testing and nonparametric alternatives briefly. Frequentist simplicity does not beat Bayesian interpretability due to improved computational resources, but the elicitation and implementation of prior information demand caution. Accounting for clustered data (e.g., repeated measurements per subject) is well-established in frequentist, but not yet in Bayesian Bland–Altman analysis.