Evaluation of previously embolized intracranial aneurysms: inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists

A comparison of treating physician versus independent core lab assessments of post-aneurysm treatment imaging outcomes: an analysis of prospectively collected data from a randomized trial

OBJECTIVE

Aneurysm occlusion has been used as surrogate marker of aneurysm treatment efficacy. Aneurysm occlusion scales are used to evaluate the outcome of endovascular aneurysm treatment and to monitor recurrence. These scales, however, require subjective interpretation of imaging data, which can reduce the utility and reliability of these scales and the validity of clinical studies regarding aneurysm occlusion rates. Use of a core lab with independent blinded reviewers has been implemented to enhance the validity of occlusion rate assessments in clinical trials. The degree of agreement between core labs and treating physicians has not been well studied with prospectively collected data.

METHODS

In this study, the authors analyzed data from the Hydrogel Endovascular Aneurysm Treatment (HEAT) trial to assess the interrater agreement between the treating physician and the blinded core lab. The HEAT trial included 600 patients across 46 sites with intracranial aneurysms treated with coiling. The treating site and the core lab independently reviewed immediate postoperative and follow-up imaging (3-12 and 18-24 months, respectively) using the Raymond-Roy occlusion classification (RROC) scale, Meyer scale, and recanalization survey. A post hoc analysis was performed to calculate interrater reliability using Cohen's kappa. Further analysis was performed to assess whether degree of agreement varied on the basis of various factors, including scale used, timing of imaging, size of the aneurysm, imaging modality, location of the aneurysm, dome-to-neck ratio, and rupture status.

RESULTS

Minimal interrater agreement was noted between the core lab reviewers and the treating physicians for assessing aneurysm occlusion using the RROC grading scale (k = 0.39, 95% CI 0.38-0.40) and Meyer scale (k = 0.23, 95% CI 0.14-0.38). The degree of agreement between groups was slightly better but still weak for assessing recanalization (k = 0.45, 95% CI 0.38-0.52). Factors that significantly improved degree of agreement were scales with fewer variables, greater time to follow-up, imaging modality (digital subtraction angiography), and wide-neck aneurysms.

CONCLUSIONS

Assessment of aneurysm treatment outcome with commonly used aneurysm occlusion scales suffers from risk of poor interrater agreement. This supports the use of independent core labs for validation of outcome data to minimize reporting bias. Use of outcome tools with fewer point categories is likely to provide better interrater reliability. Therefore, the outcome assessment tools are ideal for clinical outcome assessment provided that they are sensitive enough to detect a clinically significant change.

The clinical value of ceMRA versus DSA for follow-up of intracranial aneurysms treated by coil embolization: an assessment of occlusion classifications and impact on treatment decisions

OBJECTIVE

The aim of this study was a detailed analysis of the value of contrast-enhanced magnetic resonance angiography (ceMRA) compared to digital subtraction angiography (DSA) for follow-up imaging of intracranial aneurysms treated by coil embolization.

METHODS

Patients with coiled aneurysms and follow-up exams including both DSA and 3 T ceMRA were retrospectively identified. In blinded readings, both modalities were graded according to the modified Raymond-Roy classification (MRRC) and the Meyers scale. Additionally, readers were asked to make a decision regarding retreatment/follow-up based on the respective imaging findings.

RESULTS

The study comprised 92 patients harboring 102 coiled aneurysms. There was good intermethod agreement of DSA and ceMRA concerning both the MRRC (κ = 0.64) and the Meyers scale (κ = 0.74). Agreement regarding occlusion of < 90% of the aneurysm (Meyers grade ≥ 2) was very good (κ = 0.87). Regarding the detection of a remnant with contrast between the coil mass and the aneurysm wall (MRRC IIIb), there were 12 discrepant findings and agreement was good (κ = 0.70). Comparing treatment/follow-up decisions, the two methods agreed very well (κ = 0.92). In seven patients with discrepant treatment decisions, the authors concurred with DSA in four cases and with ceMRA in three cases when evaluating both modalities together. Interval aneurysm growth was found in more cases with ceMRA (n = 19) than with DSA (n = 16).

CONCLUSIONS

CeMRA is very unlikely to miss a relevant aneurysm remnant and thus could be suitable as the primary follow-up method. In case of remnant growth or recurrence, however, additional DSA might be required to guide treatment decisions.

KEY POINTS

• There is high accordance between ceMRA and DSA regarding the evaluation of intracranial aneurysms treated by endovascular coil embolization, but closer analysis also revealed relevant differences. • CeMRA could be suitable as the primary follow-up imaging modality, potentially eliminating the need for routine DSA. • DSA will still be required in case of aneurysm remnant growth or recurrence as detected by ceMRA.

MRA versus DSA for the follow-up imaging of intracranial aneurysms treated using endovascular techniques: a meta-analysis

BACKGROUND

Treated aneurysms must be followed over time to ensure durable occlusion, as more than 20% of endovascularly treated aneurysms recur. While digital subtraction angiography (DSA) remains the gold standard, magnetic resonance angiography (MRA) is attractive as a non-invasive follow-up technique. Two different MRA techniques have traditionally been used: time-of-flight (TOF) and contrast-enhanced (CE) MRA. We analysed data from studies comparing MRA techniques with DSA for the follow-up of aneurysms undergoing endovascular treatment. Subgroup analysis of stent-assisted coiling (SAC) and flow diversion (FD) techniques was completed.

METHODS

Comprehensive searches using the Embase, PubMed, and Cochrane databases were performed and updated to November 2018. Pooled sensitivity and specificity were calculated using aneurysm occlusion status as defined by the Raymond-Roy occlusion grading scale.

RESULTS

The literature search yielded 1579 unique titles. Forty-three studies were included. For TOF-MRA, sensitivity and specificity of all aneurysms undergoing endovascular therapy were 88% and 94%, respectively. For CE-MRA, the sensitivity and specificity were 88% and 96%, respectively. For SAC and FD techniques, sensitivity and specificity of TOF-MRA were 86% and 95%, respectively. CE-MRA had sensitivity and specificity of 90% and 92%.

CONCLUSION

MRA is a reliable modality for the follow-up of aneurysms treated using endovascular techniques. While the data are limited, MRA techniques can also be used to reliably follow patients undergoing FD and SAC. However, clinical factors must be used to optimize follow-up regimens for individual patients.

Post-operative radiograph assessment of children undergoing closed reduction and spica cast immobilization for developmental dysplasia of the hip: does experience matter?

PURPOSE

Closed reduction and spica cast immobilization are routinely used for young patients with developmental dysplasia of the hip with reducible hips. Our primary objective was to assess the interpretation quality of immediate post-operative pelvis radiographs after treatment.

METHODS

A series of 28 randomly selected patients (30 hips) with pre- and post-operative pelvis radiographs and post-operative magnetic resonance imaging were included. Each was presented twice with an interval of two weeks, in alternating orders. Raters with different experience and specialties from different institutions rated the quality of reduction (hip in or out) after treatment.

RESULTS

Thirteen surgeons and three radiologists evaluated 30 hips (28 patients). Agreement was not satisfactory (κ = 0.12). Experienced clinicians demonstrated similar agreement to inexperienced raters (κ = 0.04). Consistency at a two week interval was moderate (κ = 0.48, percent of agreement at 82%). The mean number of errors from the two ratings were 8.6 ± 2.5 and 8.9 ± 2.7, respectively (P = 0.72). There was no significant difference between surgeons with different levels of experience; radiologists did better than surgeons, but the difference was insignificant. Raters from different institutions had similar performance in poor judgment.

CONCLUSIONS

Our results show poor concordance between observers and ratings. Post-operative radiographs are unreliable for assessing the quality of hip reduction. The level of experience, subspecialty, and geographical origin do not impact the radiographic assessment. Based on the present findings, we recommend performing post-operative magnetic resonance imaging rather than anteroposterior pelvis radiograph to assess the hip. Compared to standard radiographs, magnetic resonance imaging allows more reliable interpretation while decreasing radiation exposure.