Automated registration, segmentation, and measurement of metastatic melanoma tumors in serial CT scans

Translation of AI into oncology clinical practice

Artificial intelligence (AI) is a transformative technology that is capturing popular imagination and can revolutionize biomedicine. AI and machine learning (ML) algorithms have the potential to break through existing barriers in oncology research and practice such as automating workflow processes, personalizing care, and reducing healthcare disparities. Emerging applications of AI/ML in the literature include screening and early detection of cancer, disease diagnosis, response prediction, prognosis, and accelerated drug discovery. Despite this excitement, only few AI/ML models have been properly validated and fewer have become regulated products for routine clinical use. In this review, we highlight the main challenges impeding AI/ML clinical translation. We present different clinical use cases from the domains of radiology, radiation oncology, immunotherapy, and drug discovery in oncology. We dissect the unique challenges and opportunities associated with each of these cases. Finally, we summarize the general requirements for successful AI/ML implementation in the clinic, highlighting specific examples and points of emphasis including the importance of multidisciplinary collaboration of stakeholders, role of domain experts in AI augmentation, transparency of AI/ML models, and the establishment of a comprehensive quality assurance program to mitigate risks of training bias and data drifts, all culminating toward safer and more beneficial AI/ML applications in oncology labs and clinics.

Rethinking Clinical Trial Radiology Workflows and Student Training: Integrated Virtual Student Shadowing Experience, Education, and Evaluation

There is consistent demand for clinical exposure from students interested in radiology; however, the COVID-19 pandemic resulted in fewer available options and limited student access to radiology departments. Additionally, there is increased demand for radiologists to manage more complex quantification in reports on patients enrolled in clinical trials. We present an online educational curriculum that addresses both of these gaps by virtually immersing students (radiology preprocessors, or RPs) into radiologists' workflows where they identify and measure target lesions in advance of radiologists, streamlining report quantification. RPs switched to remote work at the beginning of the COVID-19 pandemic in our National Institutes of Health (NIH). We accommodated them by transitioning our curriculum on cross-sectional anatomy and advanced PACS tools to a publicly available online curriculum. We describe collaborations between multiple academic research centers and industry through contributions of academic content to this curriculum. Further, we describe how we objectively assess educational effectiveness with cross-sectional anatomical quizzes and decreasing RP miss rates as they gain experience. Our RP curriculum generated significant interest evidenced by a dozen academic and research institutes providing online presentations including radiology modality basics and quantification in clinical trials. We report a decrease in RP miss rate percentage, including one virtual RP over a period of 1 year. Results reflect training effectiveness through decreased discrepancies with radiologist reports and improved tumor identification over time. We present our RP curriculum and multicenter experience as a pilot experience in a clinical trial research setting. Students are able to obtain useful clinical radiology experience in a virtual learning environment by immersing themselves into a clinical radiologist's workflow. At the same time, they help radiologists improve patient care with more valuable quantitative reports, previously shown to improve radiologist efficiency. Students identify and measure lesions in clinical trials before radiologists, and then review their reports for self-evaluation based on included measurements from the radiologists. We consider our virtual approach as a supplement to student education while providing a model for how artificial intelligence will improve patient care with more consistent quantification while improving radiologist efficiency.

Augmented Radiologist Workflow Improves Report Value and Saves Time: A Potential Model for Implementation of Artificial Intelligence

RATIONALE AND OBJECTIVES

Our primary aim was to improve radiology reports by increasing concordance of target lesion measurements with oncology records using radiology preprocessors (RP). Faster notification of incidental actionable findings to referring clinicians and clinical radiologist exam interpretation time savings with RPs quantifying tumor burden were also assessed.

MATERIALS AND METHODS

In this prospective quality improvement initiative, RPs annotated lesions before radiologist interpretation of CT exams. Clinical radiologists then hyperlinked approved measurements into interactive reports during interpretations. RPs evaluated concordance with our tumor measurement radiologist, the determinant of tumor burden. Actionable finding detection and notification times were also deduced. Clinical radiologist interpretation times were calculated from established average CT chest, abdomen, and pelvis interpretation times.

RESULTS

RPs assessed 1287 body CT exams with 812 follow-up CT chest, abdomen, and pelvis studies; 95 (11.7%) of which had 241 verified target lesions. There was improved concordance (67.8% vs. 22.5%) of target lesion measurements. RPs detected 93.1% incidental actionable findings with faster clinician notification by a median time of 1 hour (range: 15 minutes-16 hours). Radiologist exam interpretation times decreased by 37%.

CONCLUSIONS

This workflow resulted in three-fold improved target lesion measurement concordance with oncology records, earlier detection and faster notification of incidental actionable findings to referring clinicians, and decreased exam interpretation times for clinical radiologists. These findings demonstrate potential roles for automation (such as AI) to improve report value, worklist prioritization, and patient care.

Comparative Effectiveness of Tumor Response Assessment Methods: Standard of Care Versus Computer-Assisted Response Evaluation

PURPOSE

To compare the effectiveness of metastatic tumor response evaluation with computed tomography using computer-assisted versus manual methods.

MATERIALS AND METHODS

In this institutional review board-approved, Health Insurance Portability and Accountability Act-compliant retrospective study, 11 readers from 10 different institutions independently categorized tumor response according to three different therapeutic response criteria by using paired baseline and initial post-therapy computed tomography studies from 20 randomly selected patients with metastatic renal cell carcinoma who were treated with sunitinib as part of a completed phase III multi-institutional study. Images were evaluated with a manual tumor response evaluation method (standard of care) and with computer-assisted response evaluation (CARE) that included stepwise guidance, interactive error identification and correction methods, automated tumor metric extraction, calculations, response categorization, and data and image archiving. A crossover design, patient randomization, and 2-week washout period were used to reduce recall bias. Comparative effectiveness metrics included error rate and mean patient evaluation time.

RESULTS

The standard-of-care method, on average, was associated with one or more errors in 30.5% (6.1 of 20) of patients, whereas CARE had a 0.0% (0.0 of 20) error rate ( P < .001). The most common errors were related to data transfer and arithmetic calculation. In patients with errors, the median number of error types was 1 (range, 1 to 3). Mean patient evaluation time with CARE was twice as fast as the standard-of-care method (6.4 minutes v 13.1 minutes; P < .001).

CONCLUSION

CARE reduced errors and time of evaluation, which indicated better overall effectiveness than manual tumor response evaluation methods that are the current standard of care.

Metastatic pulmonary malignant melanoma showing a ring-shaped and halo signs

Classically, metastatic tumours are solid, multiple, well-circumscribed, and rarely cavitary. This rare case of metastatic pulmonary malignant melanoma showed a ring-shaped ground-glass opacity and then the halo sign, depending on the disease progression.

Multimedia-enhanced Radiology Reports: Concept, Components, and Challenges

Multimedia-enhanced radiology report (MERR) development is defined and described from an informatics perspective, in which the MERR is seen as a superior information-communicating entity. Recent technical advances, such as the hyperlinking of report text directly to annotated images, improve MERR information content and accessibility compared with text-only reports. The MERR is analyzed by its components, which include hypertext, tables, graphs, embedded images, and their interconnections. The authors highlight the advantages of each component for improving the radiologist's communication of report content information and the user's ability to extract information. Requirements for MERR implementation (eg, integration of picture archiving and communication systems, radiology information systems, and electronic medical record systems) and the authors' initial experiences and challenges in MERR implementation at the National Institutes of Health are reviewed. The transition to MERRs has provided advantages over use of traditional text-only radiology reports because of the capacity to include hyperlinked report text that directs clinicians to image annotations, images, tables, and graphs. A framework is provided for thinking about the MERR from the user's perspective. Additional applications of emerging technologies (eg, artificial intelligence and machine learning) are described in the crafting of what the authors believe is the radiology report of the future. ^©RSNA, 2018.

ENABLE (Exportable Notation and Bookmark List Engine): an Interface to Manage Tumor Measurement Data from PACS to Cancer Databases

Oncologists evaluate therapeutic response in cancer trials based on tumor quantification following selected "target" lesions over time. At our cancer center, a majority of oncologists use Response Evaluation Criteria in Solid Tumors (RECIST) v1.1 quantifying tumor progression based on lesion measurements on imaging. Currently, our oncologists handwrite tumor measurements, followed by multiple manual data transfers; however, our Picture Archiving Communication System (PACS) (Carestream Health, Rochester, NY) has the ability to export tumor measurements, making it possible to manage tumor metadata digitally. We developed an interface, "Exportable Notation and Bookmark List Engine" (ENABLE), which produces prepopulated RECIST v1.1 worksheets and compiles cohort data and data models from PACS measurement data, thus eliminating handwriting and manual data transcription. We compared RECIST v1.1 data from eight patients (16 computed tomography exams) enrolled in an IRB-approved therapeutic trial with ENABLE outputs: 10 data fields with a total of 194 data points. All data in ENABLE's output matched with the existing data. Seven staff were taught how to use the interface with a 5-min explanatory instructional video. All were able to use ENABLE successfully without additional guidance. We additionally assessed 42 metastatic genitourinary cancer patients with available RECIST data within PACS to produce a best response waterfall plot. ENABLE manages tumor measurements and associated metadata exported from PACS, producing forms and data models compatible with cancer databases, obviating handwriting and the manual re-entry of data. Automation should reduce transcription errors and improve efficiency and the auditing process.

Resources Required for Semi-Automatic Volumetric Measurements in Metastatic Chordoma: Is Potentially Improved Tumor Burden Assessment Worth the Time Burden?

The Response Evaluation Criteria in Solid Tumors (RECIST) is the current standard for assessing therapy response in patients with malignant solid tumors; however, volumetric assessments are thought to be more representative of actual tumor size and hence superior in predicting patient outcomes. We segmented all primary and metastatic lesions in 21 chordoma patients for comparison to RECIST. Primary tumors were segmented on MR and validated by a neuroradiologist. Metastatic lesions were segmented on CT and validated by a general radiologist. We estimated times for a research assistant to segment all primary and metastatic chordoma lesions using semi-automated volumetric segmentation tools available within our PACS (v12.0, Carestream, Rochester, NY), as well as time required for radiologists to validate the segmentations. We also report success rates of semi-automatic segmentation in metastatic lesions on CT and time required to export data. Furthermore, we discuss the feasibility of volumetric segmentation workflow in research and clinical settings. The research assistant spent approximately 65 h segmenting 435 lesions in 21 patients. This resulted in 1349 total segmentations (average 2.89 min per lesion) and over 13,000 data points. Combined time for the neuroradiologist and general radiologist to validate segmentations was 45.7 min per patient. Exportation time for all patients totaled only 6 h, providing time-saving opportunities for data managers and oncologists. Perhaps cost-neutral resource reallocation can help acquire volumes paralleling our example workflow. Our results will provide researchers with benchmark resources required for volumetric assessments within PACS and help prepare institutions for future volumetric assessment criteria.

Radiology Reports With Hyperlinks Improve Target Lesion Selection and Measurement Concordance in Cancer Trials

OBJECTIVE

Radiology reports often lack the measurements of target lesions that are needed for oncology clinical trials. When available, the measurements in the radiology reports often do not match those in the records used to calculate therapeutic response. This study assessed the clinical value of hyperlinked tumor measurements in multimedia-enhanced radiology reports in the PACS and the inclusion of a radiologist assistant in the process of assessing tumor burden.

MATERIALS AND METHODS

We assessed 489 target lesions in 232 CT examinations of 71 patients with metastatic genitourinary cancer enrolled in two therapeutic trials. We analyzed target lesion selection and measurement concordance between oncology records (used to calculate therapeutic response) and two types of radiology reports in the PACS: multimedia-enhanced radiology reports and text-only reports. For statistical tests, we used the Wilcoxon signed rank, Wilcoxon rank sum test, and Fisher method to combine p values from the paired and unpaired results. The Fisher exact test was used to compare overall measurement concordance.

RESULTS

Concordance on target lesion selection was greater for multimedia-enhanced radiology reports (78%) than the text-only reports (52%) (p = 0.0050). There was also improved overall measurement concordance with the multimedia-enhanced radiology reports (68%) compared with the text-only reports (38%) (p < 0.0001).

CONCLUSION

Compared with text-only reports, hyperlinked multimedia-enhanced radiology reports improved concordance of target lesion selection and measurement with the measurements used to calculate therapeutic response.

A New Approach to Evaluate Drug Treatment Response of Ovarian Cancer Patients Based on Deformable Image Registration

Although Response Evaluation Criteria in Solid Tumors (RECIST) is the current clinical guideline to assess size change of solid tumors after therapeutic treatment, it has a relatively lower association to the clinical outcome of progression free survival (PFS) of the patients. In this paper, we presented a new approach to assess responses of ovarian cancer patients to new chemotherapy drugs in clinical trials. We first developed and applied a multi-resolution B-spline based deformable image registration method to register two sets of computed tomography (CT) image data acquired pre- and post-treatment. The B-spline difference maps generated from the co-registered CT images highlight the regions related to the volumetric growth or shrinkage of the metastatic tumors, and density changes related to variation of necrosis inside the solid tumors. Using a testing dataset involving 19 ovarian cancer patients, we compared patients' response to the treatment using the new image registration method and RECIST guideline. The results demonstrated that using the image registration method yielded higher association with the six-month PFS outcomes of the patients than using RECIST. The image registration results also provided a solid foundation of developing new computerized quantitative image feature analysis schemes in the future studies.

Solitary pulmonary metastasis from malignant melanoma of the bulbar conjunctiva presenting as a pulmonary ground glass nodule: Report of a case

We herein report a case of solitary pulmonary metastasis from malignant melanoma that presented as a pulmonary ground glass nodule. A 57-year-old man who had undergone resection of a malignant melanoma of the right bulbar conjunctiva at the age of 51 was referred to our hospital for management of ground glass opacity in his left lung. Because radiological examination suggested the nodule was an adenocarcinoma in situ, computed tomography (CT) follow-up was planned. CT examination performed nine months later showed that the nodule had grown from 6 mm to 8 mm. Moreover, CT performed one and a half years after first detection revealed that the nodule had grown up to 10 mm. The patient, therefore, underwent partial resection of the lung for diagnosis and treatment. Pathological examination of the resected specimen revealed atypical cells with melanin granules proliferating in a lepidic-like fashion. The cells were positive on S-100 staining, indicating a pulmonary metastasis from malignant melanoma. Thus, metastatic tumors from malignant melanoma can present as ground glass opacities.

Consistency and efficiency of CT analysis of metastatic disease: semiautomated lesion management application within a PACS

OBJECTIVE

The purpose of this study was to evaluate the success, consistency, and efficiency of a semiautomated lesion management application within a PACS in the analysis of metastatic lesions in serial CT examinations of cancer patients.

MATERIALS AND METHODS

Two observers using baseline and follow-up CT data independently reviewed 93 target lesions (17 lung, five liver, 71 lymph node) in 50 patients with either metastatic bladder or prostate cancer. The observers measured the longest axis (or short axis for lymph nodes) of each lesion and made Response Evaluation Criteria in Solid Tumors (RECIST) determinations using manual and lesion management application methods. The times required for examination review, RECIST calculations, and data input were recorded. The Wilcoxon signed rank test was used to assess time differences, and Bland-Altman analysis was used to assess interobserver agreement within the manual and lesion management application methods. Percentage success rates were also reported.

RESULTS

With the lesion management application, most lung and liver lesions were semiautomatically segmented. Comparison of the lesion management application and manual methods for all lesions showed a median time saving of 45% for observer 1 (p<0.05) and 28% for observer 2 (p=0.05) on follow-up scans versus 28% for observer 1 (p<0.05) and 9% for observer 2 (p=0.087) on baseline scans. Variability of measurements showed mean percentage change differences of only 8.9% for the lesion management application versus 26.4% for manual measurements.

CONCLUSION

With the lesion management application method, most lung and liver lesions were successfully segmented semiautomatically; the results were more consistent between observers; and assessment of tumor size was faster than with the manual method.

Automated liver lesion detection in CT images based on multi-level geometric features

PURPOSE

Due to the increasing number of liver cancer cases in clinical practice, there is a significant need for efficient tools for computer-assisted liver lesion analysis. A wide range of clinical applications, such as lesion characterization, quantification and follow-up, can be facilitated by automated liver lesion detection. Liver lesions vary significantly in size, shape, density and heterogeneity, which make them difficult to detect automatically. The goal of this work was to develop a method that can detect all types of liver lesions with high sensitivity and low false positive rate within a short run time.

METHODS

The proposed method identifies abnormal regions in liver CT images based on their intensity using a multi-level segmentation approach. The abnormal regions are analyzed from the inside-out using basic geometric features (such as asymmetry, compactness or volume). Using this multi-level shape characterization, the abnormal regions are classified into lesions and other region types (including vessel, liver boundary). The proposed analysis also allows defining the contour of each finding. The method was trained on a set of 55 cases involving 120 lesions and evaluated on a set of 30 images involving 59 (various types of) lesions, which were manually contoured by a physician.

RESULTS

The proposed algorithm demonstrated a high detection rate (92 %) at a low (1.7) false positive per case (precision 51 %), when the method was started from a manually contoured liver. The same level of false positive per case (1.6) and precision (51 %) was achieved at a somewhat lower detection rate (85 %), when the volume of interest was defined by a fully automated liver segmentation.

CONCLUSIONS

The proposed method can efficiently detect liver lesions irrespective of their size, shape, density and heterogeneity within half a minute. According to the evaluation, its accuracy is competitive with the actual state-of-the-art approaches, and the contour of the detected findings is acceptable in most of the cases. Future work shall focus on more precise lesion contouring so that the proposed method can be a solid basis for fully automated liver tumour burden estimation.