Observer variation of 2-deoxy-2-[F-18]fluoro-D-glucose-positron emission tomography in mediastinal staging of non-small cell lung cancer as a function of experience, and its potential clinical impact

Interrater Agreement and Reliability of PERCIST and Visual Assessment When Using 18F-FDG-PET/CT for Response Monitoring of Metastatic Breast Cancer

Response evaluation at regular intervals is indicated for treatment of metastatic breast cancer (MBC). FDG-PET/CT has the potential to monitor treatment response accurately. Our purpose was to: (a) compare the interrater agreement and reliability of the semi-quantitative PERCIST criteria to qualitative visual assessment in response evaluation of MBC and (b) investigate the intrarater agreement when comparing visual assessment of each rater to their respective PERCIST assessment. We performed a retrospective study on FDG-PET/CT in women who received treatment for MBC. Three specialists in nuclear medicine categorized response evaluation by qualitative assessment and standardized one-lesion PERCIST assessment. The scans were categorized into complete metabolic response, partial metabolic response, stable metabolic disease, and progressive metabolic disease. 37 patients with 179 scans were included. Visual assessment categorization yielded moderate agreement with an overall proportion of agreement (PoA) between raters of 0.52 (95% CI 0.44-0.66) and a Fleiss kappa estimate of 0.54 (95% CI 0.46-0.62). PERCIST response categorization yielded substantial agreement with an overall PoA of 0.65 (95% CI 0.57-0.73) and a Fleiss kappa estimate of 0.68 (95% CI 0.60-0.75). The difference in PoA between overall estimates for PERCIST and visual assessment was 0.13 (95% CI 0.06-0.21; p = 0.001), that of kappa was 0.14 (95% CI 0.06-0.21; p < 0.001). The overall intrarater PoA was 0.80 (95% CI 0.75-0.84) with substantial agreement by a Fleiss kappa of 0.74 (95% CI 0.69-0.79). Semi-quantitative PERCIST assessment achieved significantly higher level of overall agreement and reliability compared with qualitative assessment among three raters. The achieved high levels of intrarater agreement indicated no obvious conflicting elements between the two methods. PERCIST assessment may, therefore, give more consistent interpretations between raters when using FDG-PET/CT for response evaluation in MBC.

Variation Between Multidisciplinary Tumor Boards in Clinical Staging and Treatment Recommendations for Patients With Locally Advanced Non-small Cell Lung Cancer

Background

Accurate diagnosis and staging are crucial to ensure uniform allocation to the optimal treatment methods for non-small cell lung cancer (NSCLC) patients, but may differ among multidisciplinary tumor boards (MDTs). Discordance between clinical and pathologic TNM stage is particularly important for patients with locally advanced NSCLC (stage IIIA) because it may influence their chance of allocation to curative-intent treatment. We therefore aimed to study agreement on staging and treatment to gain insight into MDT decision-making.

Research Question

What is the level of agreement on clinical staging and treatment recommendations among MDTs in stage IIIA NSCLC patients?

Study Design and Methods

Eleven MDTs each evaluated the same 10 pathologic stage IIIA NSCLC patients in their weekly meeting (n = 110). Patients were selected purposively for their challenging nature. All MDTs received exactly the same clinical information and images per patient. We tested agreement in cT stage, cN stage, cM stage (TNM 8th edition), and treatment proposal among MDTs using Randolph’s free-marginal multirater kappa.

Results

Considerable variation among the MDTs was seen in T staging (κ, 0.55 [95% CI, 0.34-0.75]), N staging (κ, 0.59 [95% CI, 0.35-0.83]), overall TNM staging (κ, 0.53 [95% CI, 0.35-0.72]), and treatment recommendations (κ, 0.44 [95% CI, 0.32-0.56]). Most variation in T stage was seen in patients with suspicion of invasion of surrounding structures, which influenced such treatment recommendations as induction therapy and type. For N stage, distinction between N1 and N2 disease was an important source of discordance among MDTs. Variation occurred between 2 patients even regarding M stage. A wide range of additional diagnostics was proposed by the MDTs.

Interpretation

This study demonstrated high variation in staging and treatment of patients with stage IIIA NSCLC among MDTs in different hospitals. Although some variation may be unavoidable in these challenging patients, we should strive for more uniformity.

Inter-observer agreement improves with PERCIST 1.0 as opposed to qualitative evaluation in non-small cell lung cancer patients evaluated with F-18-FDG PET/CT early in the course of chemo-radiotherapy

Background

The purpose of this study is to determine whether a qualitative approach or a semi-quantitative approach provides the most robust method for early response evaluation with 2′-deoxy-2′-[¹⁸F]fluoro-d-glucose (F-18-FDG) positron emission tomography combined with whole body computed tomography (PET/CT) in non-small cell lung cancer (NSCLC).

In this study eight Nuclear Medicine consultants analyzed F-18-FDG PET/CT scans from 35 patients with locally advanced NSCLC. Scans were performed at baseline and after 2 cycles of chemotherapy. Each observer used two different methods for evaluation: (1) PET response criteria in solid tumors (PERCIST) 1.0 and (2) a qualitative approach. Both methods allocate patients into one of four response categories (complete and partial metabolic response (CMR and PMR) and stable and progressive metabolic disease (SMD and PMD)). The inter-observer agreement was evaluated using Fleiss’ kappa for multiple raters, Cohens kappa for comparison of the two methods, and intraclass correlation coefficients (ICC) for comparison of lean body mass corrected standardized uptake value (SUL) peak measurements.

Results

The agreement between observers when determining the percentage change in SULpeak was “almost perfect”, with ICC = 0.959. There was a strong agreement among observers allocating patients to the different response categories with a Fleiss kappa of 0.76 (0.71–0.81). In 22 of the 35 patients, complete agreement was observed with PERCIST 1.0. The agreement was lower when using the qualitative method, moderate, having a Fleiss kappa of 0.60 (0.55–0.64). Complete agreement was achieved in only 10 of the 35 patients. The difference between the two methods was statistically significant (p < 0.005) (chi-squared).

Comparing the two methods for each individual observer showed Cohen’s kappa values ranging from 0.64 to 0.79, translating into a strong agreement between the two methods.

Conclusions

PERCIST 1.0 provides a higher overall agreement between observers than the qualitative approach in categorizing early treatment response in NSCLC patients. The inter-observer agreement is in fact strong when using PERCIST 1.0 even when the level of instruction is purposely kept to a minimum in order to mimic the everyday situation. The variability is largely owing to the subjective elements of the method.

Electronic supplementary material

The online version of this article (doi:10.1186/s13550-016-0223-6) contains supplementary material, which is available to authorized users.

Are discordant positron emission tomography and pathological assessments of the mediastinum in non-small cell lung cancer significant?

OBJECTIVE

Many patients with non-small cell lung cancer have positive mediastinal lymph nodes on preoperative positron emission tomography (PET) but do not have mediastinal involvement after surgery. The prognostic significance of this discordance was assessed.

METHODS

This Institutional Review Board-approved study evaluated patients treated with upfront surgery at Duke Cancer Institute (Durham, NC) for non-small cell lung cancer from 1995 to 2008. Those staged with PET with pN0-1 disease after negative invasive mediastinal assessment were included. Mediastinal lymph nodes were scored as positive or negative based on visual analysis of the preoperative PET. Clinical outcomes of the PET-positive and PET-negative cohorts were estimated using the Kaplan-Meier method and compared using a log-rank test. Prognostic factors were assessed using a multivariate analysis.

RESULTS

A total of 547 patients were assessed, of whom 105 (19%) were PET positive in the mediastinum. The median number of mediastinal lymph node stations sampled was 4 (range, 1-9). The 5-year risk of local recurrence was 26% in PET-positive versus 21% in PET-negative patients (P = .50). Patterns of local failure were similar between the 2 groups. Distant recurrence (35% vs 29%; P = .63) and overall survival (44% vs 54%; P = .52) were comparable for PET-positive and PET-negative patients. On multivariate analysis, a positive PET was not significant for local recurrence (hazard ratio [HR], 1; P = 1), distant recurrence (HR, 0.82; P = .42), or overall survival (HR, 1.08; P = .62).

CONCLUSIONS

Patients with positive mediastinal lymph nodes on preoperative PET, but negative on histologic analysis, are not at increased risk of disease recurrence. Pathologic staging remains the standard.

Comparison of the diagnostic accuracy of whole-body MRI and whole-body CT in stage III/IV malignant melanoma

BACKGROUND

Malignant melanoma (MM) is dramatically increasing in light-skinned populations worldwide. Staging and regular follow-up examinations are essential. The purpose of this study was to compare the diagnostic accuracy of whole-body MRI with the standard diagnostic algorithm (whole-body CT and brain MRI) in patients with stage III/IV MM.

PATIENTS AND METHODS

A group of 50 consecutively admitted patients with stage III/IV MM were included in the study. Whole-body CT and brain MRI scans were performed. Additionally, all patients underwent a whole-body MRI (1.5 Tesla Magnetom Avanto, Siemens Healthcare Sector, Erlangen). The findings were compared on a lesion-by-lesion basis as part of clinical routine follow-up.

RESULTS

33 patients received a follow-up CT and were evaluated. Overall, 824 lesions were detected. The sensitivity of whole-body MRI was observer-dependent. MRI was slightly less sensitive than CT according to the findings of the two most experienced observers (73.4 % vs. 78.2 %, p = 0.0744). CT was significantly more sensitive in the detection of small (1-5 mm) pulmonary nodules (2.9 % vs. 66.9 %, p < 0.0001). Yet overall, MRI was significantly more specific than CT (83.4 % vs. 50.4 %, p < 0.0001).

CONCLUSIONS

Whole-body MRI in compliance with standard requirements for the observers (high level of experience) should be considered as an appropriate alternative to CT without ionizing radiation, particularly for young patients with advanced MM.

From RECIST to PERCIST: Evolving Considerations for PET response criteria in solid tumors

The purpose of this article is to review the status and limitations of anatomic tumor response metrics including the World Health Organization (WHO) criteria, the Response Evaluation Criteria in Solid Tumors (RECIST), and RECIST 1.1. This article also reviews qualitative and quantitative approaches to metabolic tumor response assessment with (18)F-FDG PET and proposes a draft framework for PET Response Criteria in Solid Tumors (PERCIST), version 1.0.

METHODS

PubMed searches, including searches for the terms RECIST, positron, WHO, FDG, cancer (including specific types), treatment response, region of interest, and derivative references, were performed. Abstracts and articles judged most relevant to the goals of this report were reviewed with emphasis on limitations and strengths of the anatomic and PET approaches to treatment response assessment. On the basis of these data and the authors' experience, draft criteria were formulated for PET tumor response to treatment.

RESULTS

Approximately 3,000 potentially relevant references were screened. Anatomic imaging alone using standard WHO, RECIST, and RECIST 1.1 criteria is widely applied but still has limitations in response assessments. For example, despite effective treatment, changes in tumor size can be minimal in tumors such as lymphomas, sarcoma, hepatomas, mesothelioma, and gastrointestinal stromal tumor. CT tumor density, contrast enhancement, or MRI characteristics appear more informative than size but are not yet routinely applied. RECIST criteria may show progression of tumor more slowly than WHO criteria. RECIST 1.1 criteria (assessing a maximum of 5 tumor foci, vs. 10 in RECIST) result in a higher complete response rate than the original RECIST criteria, at least in lymph nodes. Variability appears greater in assessing progression than in assessing response. Qualitative and quantitative approaches to (18)F-FDG PET response assessment have been applied and require a consistent PET methodology to allow quantitative assessments. Statistically significant changes in tumor standardized uptake value (SUV) occur in careful test-retest studies of high-SUV tumors, with a change of 20% in SUV of a region 1 cm or larger in diameter; however, medically relevant beneficial changes are often associated with a 30% or greater decline. The more extensive the therapy, the greater the decline in SUV with most effective treatments. Important components of the proposed PERCIST criteria include assessing normal reference tissue values in a 3-cm-diameter region of interest in the liver, using a consistent PET protocol, using a fixed small region of interest about 1 cm(3) in volume (1.2-cm diameter) in the most active region of metabolically active tumors to minimize statistical variability, assessing tumor size, treating SUV lean measurements in the 1 (up to 5 optional) most metabolically active tumor focus as a continuous variable, requiring a 30% decline in SUV for "response," and deferring to RECIST 1.1 in cases that do not have (18)F-FDG avidity or are technically unsuitable. Criteria to define progression of tumor-absent new lesions are uncertain but are proposed.

CONCLUSION

Anatomic imaging alone using standard WHO, RECIST, and RECIST 1.1 criteria have limitations, particularly in assessing the activity of newer cancer therapies that stabilize disease, whereas (18)F-FDG PET appears particularly valuable in such cases. The proposed PERCIST 1.0 criteria should serve as a starting point for use in clinical trials and in structured quantitative clinical reporting. Undoubtedly, subsequent revisions and enhancements will be required as validation studies are undertaken in varying diseases and treatments.