Inter-rater reliability between musculoskeletal radiologists and orthopedic surgeons on computed tomography imaging features of spinal metastases

Spine Instability Neoplastic Score (SINS) as a Predictive Tool in Conventional Radiotherapy: A Narrative Review

AIMS

This narrative review seeks to identify the SINS score application in the radiation oncology field.

METHODS

This literature review was performed searching papers on MEDLINE published from January 2010 to August 2022.

RESULTS

In terms of vertebral painful lesions and RT symptomatic responses, the SINS score could be an interesting aid in order to choose the right therapeutic approach. Lesions with higher level of instability, and therefore higher SINS score, could did not find any significant benefit from radiation therapy which is more effective on the tumor-related pain component. For SINS as a predictor of adverse event after RT or its changes after RT, we obtained contrasting results.

CONCLUSIONS

The reported few experiences showed ambiguous conclusions. Further prospective studies are needed.

A proposed novel traumatic brain injury classification system - an overview and inter-rater reliability validation on behalf of the Society of British Neurological Surgeons

INTRODUCTION

The measurement of traumatic brain injury (TBI) 'severity' has traditionally been based on the earliest Glasgow Coma Score (GCS) recorded, however, the underlying parenchymal pathology is highly heterogonous. This heterogeneity renders prediction of outcome on an individual patient level inaccurate and makes comparison between patients both in clinical practice and research difficult. The complexity of this heterogeneity has resulted in generic all encompassing 'traumatic brain injury protocols'. Early management and studies of neuro-protectants are often done irrespective of TBI type, yet it may well be that a specific treatment may be beneficial in a subset of TBI pathologies.

METHODS

A simple CT-based classification system rating the recognised types of blunt TBI (extradural, subdural, subarachnoid haemorrhage, contusions/intracerebral haematoma and diffuse axonal injury) as mild (1), moderate (2) or severe (3) is proposed. Hypoxic brain injury, a common secondary injury following TBI, is also included. Scores can be combined to reflect concomitant types of TBI and predominant location of injury is also recorded. To assess interrater reliability, 50 patient CT images were assessed by 5 independent clinicians of varying experience. Interrater reliability was calculated using overall agreement through Cronbach's alpha including confidence intervals for intra-class coefficients.

RESULTS

Interrater reliability scores showed strong agreement for same score and same injury for TBIs with blood on CT and Cronbach's alpha co-efficient (range 0.87-0.93) demonstrated excellent correlation between raters. Cronbach's alpha was not affected when individual raters were removed.

CONCLUSIONS

The proposed simple CT classification system has good inter-rater reliability and hence potentially could enable better individual prognostication and targeted treatments to be compared while also accounting for multiple intracranial injury types. Further studies are proposed and underway.

The Role of a Navigational Radiofrequency Ablation Device and Concurrent Vertebral Augmentation for Treatment of Difficult-to-Reach Spinal Metastases

Aims: The purpose of this study was to assess the effectiveness of a navigational radiofrequency ablation device with concurrent vertebral augmentation in the treatment of posterior vertebral body metastatic lesions, which are technically difficult to access. Primary outcomes of the study were evaluation of pain palliation and radiologic assessment of local tumor control. Materials and Methods: Thirty-five patients with 41 vertebral spinal metastases involving the posterior vertebral body underwent computed tomography-guided percutaneous targeted radiofrequency ablation, with a navigational radiofrequency ablation device, associated with vertebral augmentation. Twenty-one patients (60%) had 1 or 2 metastatic lesions (Group A) and fourteen (40%) patients had multiple (>2) vertebral lesions (Group B). Changes in pain severity were evaluated by visual analog scale (VAS). Metastatic lesions were evaluated in terms of radiological local control. Results: The procedure was technically successful in all the treated vertebrae. Among the symptomatic patients, the mean VAS score dropped from 5.7 (95% CI 4.9–6.5) before tRFA and to 0.9 (95% CI 0.4–1.3) after tRFA (p < 0.001). The mean decrease in VAS score between baseline and one week follow up was 4.8 (95% CI 4.2–5.4). VAS decrease over time between one week and one year following radiofrequency ablation was similar, suggesting that pain relief was immediate and durable. Neither patients with 1–2 vertebral metastases, nor those with multiple lesions, showed radiological signs of local progression or recurrence of the tumor in the index vertebrae during a median follow up of 19 months (4–46 months) and 10 months (4–37 months), respectively. Conclusion: Treatment of spinal metastases with a navigational radiofrequency ablation device and vertebral augmentation can be used to obtain local tumor control with immediate and durable pain relief, providing effective treatment in the multimodality management of difficult-to-reach spinal metastases.

Bridging the "last mile" gap between AI implementation and operation: "data awareness" that matters

Interest in the application of machine learning (ML) techniques to medicine is growing fast and wide because of their ability to endow decision support systems with so-called artificial intelligence, particularly in those medical disciplines that extensively rely on digital imaging. Nonetheless, achieving a pragmatic and ecological validation of medical AI systems in real-world settings is difficult, even when these systems exhibit very high accuracy in laboratory settings. This difficulty has been called the “last mile of implementation.” In this review of the concept, we claim that this metaphorical mile presents two chasms: the hiatus of human trust and the hiatus of machine experience. The former hiatus encompasses all that can hinder the concrete use of AI at the point of care, including availability and usability issues, but also the contradictory phenomena of cognitive ergonomics, such as automation bias (overreliance on technology) and prejudice against the machine (clearly the opposite). The latter hiatus, on the other hand, relates to the production and availability of a sufficient amount of reliable and accurate clinical data that is suitable to be the “experience” with which a machine can be trained. In briefly reviewing the existing literature, we focus on this latter hiatus of the last mile, as it has been largely neglected by both ML developers and doctors. In doing so, we argue that efforts to cross this chasm require data governance practices and a focus on data work, including the practices of data awareness and data hygiene. To address the challenge of bridging the chasms in the last mile of medical AI implementation, we discuss the six main socio-technical challenges that must be overcome in order to build robust bridges and deploy potentially effective AI in real-world clinical settings.

Metastatic Versus Osteoporotic Vertebral Fractures on MRI: A Blinded, Multicenter, and Multispecialty Observer Agreement Evaluation

BACKGROUND

MRI is assumed to be valid for distinguishing metastatic vertebral fractures (MVFs) from osteoporotic vertebral fractures (OVFs). This study assessed (1) concordance between the image-based diagnosis of MVF versus OVF and the reference (biopsy or follow-up of >6 months), (2) interobserver and intraobserver agreement on key imaging findings and the diagnosis of MVF versus OVF, and (3) whether disclosing a patient's history of cancer leads to variations in diagnosis, concordance, or agreement.

PATIENTS AND METHODS

This retrospective cohort study included clinical data and imaging from 203 patients with confirmed MVF or OVF provided to 25 clinicians (neurosurgeons, radiologists, orthopedic surgeons, and radiation oncologists). From January 2018 through October 2018, the clinicians interpreted images in conditions as close as possible to routine practice. Each specialist assessed data twice, with a minimum 6-week interval, blinded to assessments made by other clinicians and to their own previous assessments. The kappa statistic was used to assess interobserver and intraobserver agreement on key imaging findings, diagnosis (MVF vs OVF), and concordance with the reference. Subgroup analyses were based on clinicians' specialty, years of experience, and complexity of the hospital where they worked.

RESULTS

For diagnosis of MVF versus OVF, interobserver agreement was fair, whereas intraobserver agreement was substantial. Only the latter improved to almost perfect when a patient's history of cancer was disclosed. Interobserver agreement for key imaging findings was fair or moderate, whereas intraobserver agreement on key imaging findings was moderate or substantial. Concordance between the diagnosis of MVF versus OVF and the reference was moderate. Results were similar regardless of clinicians' specialty, experience, and hospital category.

CONCLUSIONS

When MRI is used to distinguish MVF versus OVF, interobserver agreement and concordance with the reference were moderate. These results cast doubt on the reliability of basing such a diagnosis on MRI in routine practice.

Exploring the Top-Cited and Most Influential Articles in Medical Education

INTRODUCTION

The citations received by a scientific publication have been used as a proxy measurement of scientific quality and in ranking researchers. Although these practices have been observed in several institutes, careful assessment of top-cited articles may provide more insight into exploring their characteristics. The aim of this study was to analyze the top-cited articles in medical education identified by Azer 2015 and explore the characteristics of these articles that can provide insight into their assessment.

METHODS

The most frequently cited articles identified by searching the Web of Science under the category "Education, Scientific Discipline" were included in the analysis. The following issues were further analyzed: 1) comparing the mean yearly citations received by articles published before the year 2000 and those published after, 2) assessing whether there was a correlation between the number of grants, number of authors, number of institutes, or number of countries involved and the number of citations received, 3) assessing the number of female representation in authorship, 4) assessing the representation of none-medical staff in the authorship, and 5) exploring any association between top authors identified and those who were awarded the Karolinska Institutet Prize for research in medical education.

RESULTS

Although there was no correlation between the number of citations and the number of years since publications, the mean number of citations received by articles published before the year 2000 varied from zero to 18.2 ± 16.6, whereas for those published after the year 2000 the mean varied from 2.0 ± 2.1 to 35.3 ± 26.8. No correlation was found between the number of citations obtained and number of grants, number of authors, number of institutes, or number of countries involved. Females comprised 50% of the total number of authors. However, no correlation was found between the number of females in authorship and the number of citations obtained. Similarly, the number of authors with non-medical degrees did not correlate with the number of citations obtained. An association was found between authoring more than one article in the list and receiving the Karolinska Institutet Prize for research in medical education.

DISCUSSION

The analysis shows that the higher number of citations received are not necessarily related to funding, working collaboratively with a big team from several institutes or from several countries. The finding that authors of more than one top-cited article received the Karolinska Institutet Prize for research in medical education indicates that this could be a parameter in selecting the candidates for the award.

Postoperative CT Evaluation After EVAR

Purpose: To compare the postoperative computed tomography angiography (CTA) assessment made by vascular surgeons and interventional radiologists after endovascular aneurysm repair (EVAR) at a tertiary vascular clinic to an outside core review facility. Methods: One hundred patients (mean age 78.7 years, range 88–55; 84 men) with consecutive, elective, routine CTA controls after EVAR were retrospectively studied. Consultant vascular surgeons or radiologists had evaluated all original scans and written the original report. All scans were then reevaluated by an independent core clinic. Findings were classified as vascular or extravascular and stratified as clinically significant or clinically nonsignificant by an independent external reviewer. Results: The number of vascular findings detected by the vascular clinic was 72 vs 69 by the core clinic. The vascular clinic reported more clinically significant findings (primarily stent compression or kinks) as well as endoleaks and their origin. The core clinic reported more pseudoaneurysms (24 vs 12). None of the patients with puncture complications needed reintervention. Interrater analysis of all findings between the 2 clinics showed good agreement when comparing endoleaks overall (without subclassification) and moderate agreement when assessing aneurysm growth. The core clinic reported extravascular findings in 58 patients; 37 of these were classified as clinically significant. The vascular clinic reported extravascular findings in 23 patients; 7 of these were clinically significant. The core clinic also reported 2 cases of suspected malignancies, which had not been reported by the vascular clinic. Conclusion: During routine CTA follow-up after EVAR, a significant number of vascular and nonvascular findings are detected. Whereas a highly dedicated vascular clinic identifies most vascular findings regardless of the specialty of the reader, some extravascular findings are missed. However, the frequency of clinically significant findings or findings that might warrant reintervention was low in this study.

Spinal instability as defined by the spinal instability neoplastic score is associated with radiotherapy failure in metastatic spinal disease

BACKGROUND CONTEXT

Although radiotherapy is effective in achieving pain relief in most patients, it is not completely understood why some patients respond well to radiotherapy and others do not. Our hypothesis was that metastatic bone pain, if predominantly caused by mechanical instability of the spine, responds less well to radiotherapy than metastatic bone pain caused by local tumor activity. Recently, the spinal instability neoplastic score (SINS) was proposed as a standardized referral tool for nonspine specialists to facilitate early diagnosis of spinal instability.

PURPOSE

To investigate the association between spinal instability as defined by the SINS and response to radiotherapy in patients with spinal metastases.

STUDY DESIGN

A retrospectively matched case-control study in an academic tertiary referral center, conducted according to the Strengthening the Reporting of Observational Studies in Epidemiology guidelines.

PATIENT SAMPLE

Thirty-eight patients with spinal metastases who were retreated after initial palliative radiotherapy from January 2009 to December 2010 were matched to 76 control patients who were not retreated.

OUTCOME MEASURES

Radiotherapy failure as defined by retreatment (radiotherapy, surgery, and conservative) after palliative radiotherapy for spinal metastases.

METHODS

Radiotherapy planning computed tomography scans were scored by a blinded spine surgeon according to the SINS criteria. The association between SINS and radiotherapy failure was estimated by univariate and multivariate conditional logistic regression analysis.

RESULTS

Median SINS was 10 (range 4-16) for cases and 7 (range 1-16) for controls. The SINS was significantly and independently associated with radiotherapy failure (adjusted odds ratio, 1.3; 95% confidence interval, 1.1-1.5; p=.01).

CONCLUSIONS

This study shows that a higher spinal instability score increases the risk of radiotherapy failure in patients with spinal metastases, independent of performance status, primary tumor, and symptoms. These results may support the hypothesis that metastatic spinal bone pain, predominantly caused by mechanical instability, responds less well to radiotherapy than pain mainly resulting from local tumor activity.

Reliability of the spinal instability neoplastic scale among radiologists: an assessment of instability secondary to spinal metastases

OBJECTIVE

The spinal instability neoplastic scale (SINS) is a new classification system for tumor-related spinal instability. The SINS may prove to be a valuable tool for radiologists to communicate with oncologists and surgeons in a standardized evidence-based manner. The objective of this study was to determine the inter- and intraobserver reliability and validity of the SINS among radiologists.

MATERIALS AND METHODS

Thirty-seven radiologists from 10 international sites used the SINS to categorize the degree of spinal instability in 30 patients with spinal tumors. To assess validity, we compared the SINS scores assigned by the radiologists with the SINS scores of 11 spine oncology surgeons (reference standard). Each total SINS score (range, 0-18 points) was converted into one of the following three clinical categories: 0-6 points, stable; 7-12 points, potentially unstable; and 13-18 points, unstable. In addition, each total SINS score was converted into a binary scale: 0-6 points was defined as stable, and 7-18 points was considered a current or possible instability for which surgical consultation is recommended.

RESULTS

Radiologists using the SINS binary scale showed excellent (κ = 0.88) validity, substantial (κ = 0.76) interobserver agreement, and excellent (κ = 0.82) intraobserver reproducibility. Radiologists rated all unstable cases and 621 of 629 (98.7%) potentially unstable cases with a SINS score of 7 or more points, thus appropriately initiating a referral for surgical assessment.

CONCLUSION

SINS is a reliable tool for radiologists rating tumor-related spinal instability. It accurately discriminates between stable and potentially unstable or unstable lesions and, therefore, can guide the need for surgical consultation.

Radiographic union score for hip substantially improves agreement between surgeons and radiologists

Background

Despite the prominence of hip fractures in orthopedic trauma, the assessment of fracture healing using radiographs remains subjective. The variability in the assessment of fracture healing has important implications for both clinical research and patient care. With little existing literature regarding reliable consensus on hip fracture healing, this study was conducted to determine inter-rater reliability between orthopedic surgeons and radiologists on healing assessments using sequential radiographs in patients with hip fractures. Secondary objectives included evaluating a checklist designed to assess hip fracture healing and determining whether agreement improved when reviewers were aware of the timing of the x-rays in relation to the patients’ surgery.

Methods

A panel of six reviewers (three orthopedic surgeons and three radiologists) independently assessed fracture healing using sequential radiographs from 100 patients with femoral neck fractures and 100 patients with intertrochanteric fractures. During their independent review they also completed a previously developed radiographic checklist (Radiographic Union Score for Hip (RUSH)). Inter and intra-rater reliability scores were calculated. Data from the current study was compared to the findings from a previously conducted study where the same reviewers, unaware of the timing of the x-rays, completed the RUSH score.

Results

The agreement between surgeons and radiologists for fracture healing was moderate for “general impression of fracture healing” in both femoral neck (ICC = 0.60, 95% CI: 0.42-0.71) and intertrochanteric fractures (0.50, 95% CI: 0.33-0.62). Using a standardized checklist (RUSH), agreement was almost perfect in both femoral neck (ICC = 0.85, 95% CI: 0.82-0.87) and intertrochanteric fractures (0.88, 95% CI: 0.86-0.90). We also found a high degree of correlation between healing and the total RUSH score using a Receiver Operating Characteristic (ROC) analysis, there was an area under the curve of 0.993 for femoral neck cases and 0.989 for intertrochanteric cases. Agreement within the radiologist group and within the surgeon group did not significantly differ in our analyses. In all cases, radiographs in which the time from surgery was known resulted in higher agreement scores compared to those from the previous study in which reviewers were unaware of the time the radiograph was obtained.

Conclusions

Agreement in hip fracture radiographic healing may be improved with the use of a standardized checklist and appears highly influenced by the timing of the radiograph. These findings should be considered when evaluating patient outcomes and in clinical studies involving patients with hip fractures. Future research initiatives are required to further evaluate the RUSH checklist.