1
|
Koetzier LR, Wu J, Mastrodicasa D, Lutz A, Chung M, Koszek WA, Pratap J, Chaudhari AS, Rajpurkar P, Lungren MP, Willemink MJ. Generating Synthetic Data for Medical Imaging. Radiology 2024; 312:e232471. [PMID: 39254456 DOI: 10.1148/radiol.232471] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/11/2024]
Abstract
Artificial intelligence (AI) models for medical imaging tasks, such as classification or segmentation, require large and diverse datasets of images. However, due to privacy and ethical issues, as well as data sharing infrastructure barriers, these datasets are scarce and difficult to assemble. Synthetic medical imaging data generated by AI from existing data could address this challenge by augmenting and anonymizing real imaging data. In addition, synthetic data enable new applications, including modality translation, contrast synthesis, and professional training for radiologists. However, the use of synthetic data also poses technical and ethical challenges. These challenges include ensuring the realism and diversity of the synthesized images while keeping data unidentifiable, evaluating the performance and generalizability of models trained on synthetic data, and high computational costs. Since existing regulations are not sufficient to guarantee the safe and ethical use of synthetic images, it becomes evident that updated laws and more rigorous oversight are needed. Regulatory bodies, physicians, and AI developers should collaborate to develop, maintain, and continually refine best practices for synthetic data. This review aims to provide an overview of the current knowledge of synthetic data in medical imaging and highlights current key challenges in the field to guide future research and development.
Collapse
Affiliation(s)
- Lennart R Koetzier
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Jie Wu
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Domenico Mastrodicasa
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Aline Lutz
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Matthew Chung
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - W Adam Koszek
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Jayanth Pratap
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Akshay S Chaudhari
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Pranav Rajpurkar
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Matthew P Lungren
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| | - Martin J Willemink
- From the Delft University of Technology, Delft, the Netherlands (L.R.K.); Segmed, 3790 El Camino Real #810, Palo Alto, CA 94306 (J.W., A.L., M.C., W.A.K., J.P., M.J.W.); Department of Radiology, University of Washington, Seattle, Wash (D.M.); Department of Radiology, OncoRad/Tumor Imaging Metrics Core, Seattle, Wash (D.M.); Harvard University, Cambridge, Mass (J.P.); Department of Radiology, Stanford University School of Medicine, Palo Alto, Calif (A.S.C.); Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, Calif (A.S.C.); Department of Biomedical Informatics, Harvard Medical School, Boston, Mass (P.R.); Microsoft, Redmond, Wash (M.P.L.); and Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, Calif (M.P.L.)
| |
Collapse
|
2
|
Li X, Bellotti R, Bachtiary B, Hrbacek J, Weber DC, Lomax AJ, Buhmann JM, Zhang Y. A unified generation-registration framework for improved MR-based CT synthesis in proton therapy. Med Phys 2024. [PMID: 39137294 DOI: 10.1002/mp.17338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Revised: 06/11/2024] [Accepted: 07/06/2024] [Indexed: 08/15/2024] Open
Abstract
BACKGROUND The use of magnetic resonance (MR) imaging for proton therapy treatment planning is gaining attention as a highly effective method for guidance. At the core of this approach is the generation of computed tomography (CT) images from MR scans. However, the critical issue in this process is accurately aligning the MR and CT images, a task that becomes particularly challenging in frequently moving body areas, such as the head-and-neck. Misalignments in these images can result in blurred synthetic CT (sCT) images, adversely affecting the precision and effectiveness of the treatment planning. PURPOSE This study introduces a novel network that cohesively unifies image generation and registration processes to enhance the quality and anatomical fidelity of sCTs derived from better-aligned MR images. METHODS The approach synergizes a generation network (G) with a deformable registration network (R), optimizing them jointly in MR-to-CT synthesis. This goal is achieved by alternately minimizing the discrepancies between the generated/registered CT images and their corresponding reference CT counterparts. The generation network employs a UNet architecture, while the registration network leverages an implicit neural representation (INR) of the displacement vector fields (DVFs). We validated this method on a dataset comprising 60 head-and-neck patients, reserving 12 cases for holdout testing. RESULTS Compared to the baseline Pix2Pix method with MAE 124.95 ± $\pm$ 30.74 HU, the proposed technique demonstrated 80.98 ± $\pm$ 7.55 HU. The unified translation-registration network produced sharper and more anatomically congruent outputs, showing superior efficacy in converting MR images to sCTs. Additionally, from a dosimetric perspective, the plan recalculated on the resulting sCTs resulted in a remarkably reduced discrepancy to the reference proton plans. CONCLUSIONS This study conclusively demonstrates that a holistic MR-based CT synthesis approach, integrating both image-to-image translation and deformable registration, significantly improves the precision and quality of sCT generation, particularly for the challenging body area with varied anatomic changes between corresponding MR and CT.
Collapse
Affiliation(s)
- Xia Li
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
- Department of Computer Science, ETH Zürich, Zürich, Switzerland
| | - Renato Bellotti
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
- Department of Physics, ETH Zürich, Zürich, Switzerland
| | - Barbara Bachtiary
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
| | - Jan Hrbacek
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
| | - Damien C Weber
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
- Department of Radiation Oncology, University Hospital of Zürich, Zürich, Switzerland
- Department of Radiation Oncology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
| | - Antony J Lomax
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
- Department of Physics, ETH Zürich, Zürich, Switzerland
| | | | - Ye Zhang
- Center for Proton Therapy, Paul Scherrer Institut, Villigen PSI, Switzerland
| |
Collapse
|
3
|
Gao Y, Qiu RLJ, Xie H, Chang CW, Wang T, Ghavidel B, Roper J, Zhou J, Yang X. CT-based synthetic contrast-enhanced dual-energy CT generation using conditional denoising diffusion probabilistic model. Phys Med Biol 2024; 69:165015. [PMID: 39053511 PMCID: PMC11294926 DOI: 10.1088/1361-6560/ad67a1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2024] [Revised: 06/26/2024] [Accepted: 07/25/2024] [Indexed: 07/27/2024]
Abstract
Objective.The study aimed to generate synthetic contrast-enhanced Dual-energy CT (CE-DECT) images from non-contrast single-energy CT (SECT) scans, addressing the limitations posed by the scarcity of DECT scanners and the health risks associated with iodinated contrast agents, particularly for high-risk patients.Approach.A conditional denoising diffusion probabilistic model (C-DDPM) was utilized to create synthetic images. Imaging data were collected from 130 head-and-neck (HN) cancer patients who had undergone both non-contrast SECT and CE-DECT scans.Main Results.The performance of the C-DDPM was evaluated using Mean Absolute Error (MAE), Structural Similarity Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR). The results showed MAE values of 27.37±3.35 Hounsfield Units (HU) for high-energy CT (H-CT) and 24.57±3.35HU for low-energy CT (L-CT), SSIM values of 0.74±0.22 for H-CT and 0.78±0.22 for L-CT, and PSNR values of 18.51±4.55 decibels (dB) for H-CT and 18.91±4.55 dB for L-CT.Significance.The study demonstrates the efficacy of the deep learning model in producing high-quality synthetic CE-DECT images, which significantly benefits radiation therapy planning. This approach provides a valuable alternative imaging solution for facilities lacking DECT scanners and for patients who are unsuitable for iodine contrast imaging, thereby enhancing the reach and effectiveness of advanced imaging in cancer treatment planning.
Collapse
Affiliation(s)
- Yuan Gao
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Richard L J Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Huiqiao Xie
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, United States of America
| | - Chih-Wei Chang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, United States of America
| | - Beth Ghavidel
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Jun Zhou
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, United States of America
| |
Collapse
|
4
|
Pan S, Abouei E, Peng J, Qian J, Wynne JF, Wang T, Chang CW, Roper J, Nye JA, Mao H, Yang X. Full-dose whole-body PET synthesis from low-dose PET using high-efficiency denoising diffusion probabilistic model: PET consistency model. Med Phys 2024; 51:5468-5478. [PMID: 38588512 PMCID: PMC11321936 DOI: 10.1002/mp.17068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2023] [Revised: 03/26/2024] [Accepted: 03/26/2024] [Indexed: 04/10/2024] Open
Abstract
PURPOSE Positron Emission Tomography (PET) has been a commonly used imaging modality in broad clinical applications. One of the most important tradeoffs in PET imaging is between image quality and radiation dose: high image quality comes with high radiation exposure. Improving image quality is desirable for all clinical applications while minimizing radiation exposure is needed to reduce risk to patients. METHODS We introduce PET Consistency Model (PET-CM), an efficient diffusion-based method for generating high-quality full-dose PET images from low-dose PET images. It employs a two-step process, adding Gaussian noise to full-dose PET images in the forward diffusion, and then denoising them using a PET Shifted-window Vision Transformer (PET-VIT) network in the reverse diffusion. The PET-VIT network learns a consistency function that enables direct denoising of Gaussian noise into clean full-dose PET images. PET-CM achieves state-of-the-art image quality while requiring significantly less computation time than other methods. Evaluation with normalized mean absolute error (NMAE), peak signal-to-noise ratio (PSNR), multi-scale structure similarity index (SSIM), normalized cross-correlation (NCC), and clinical evaluation including Human Ranking Score (HRS) and Standardized Uptake Value (SUV) Error analysis shows its superiority in synthesizing full-dose PET images from low-dose inputs. RESULTS In experiments comparing eighth-dose to full-dose images, PET-CM demonstrated impressive performance with NMAE of 1.278 ± 0.122%, PSNR of 33.783 ± 0.824 dB, SSIM of 0.964 ± 0.009, NCC of 0.968 ± 0.011, HRS of 4.543, and SUV Error of 0.255 ± 0.318%, with an average generation time of 62 s per patient. This is a significant improvement compared to the state-of-the-art diffusion-based model with PET-CM reaching this result 12× faster. Similarly, in the quarter-dose to full-dose image experiments, PET-CM delivered competitive outcomes, achieving an NMAE of 0.973 ± 0.066%, PSNR of 36.172 ± 0.801 dB, SSIM of 0.984 ± 0.004, NCC of 0.990 ± 0.005, HRS of 4.428, and SUV Error of 0.151 ± 0.192% using the same generation process, which underlining its high quantitative and clinical precision in both denoising scenario. CONCLUSIONS We propose PET-CM, the first efficient diffusion-model-based method, for estimating full-dose PET images from low-dose images. PET-CM provides comparable quality to the state-of-the-art diffusion model with higher efficiency. By utilizing this approach, it becomes possible to maintain high-quality PET images suitable for clinical use while mitigating the risks associated with radiation. The code is availble at https://github.com/shaoyanpan/Full-dose-Whole-body-PET-Synthesis-from-Low-dose-PET-Using-Consistency-Model.
Collapse
Affiliation(s)
- Shaoyan Pan
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
- Department of Biomedical Informatics, Emory University, Atlanta, GA 30322, USA
| | - Elham Abouei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Junbo Peng
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Joshua Qian
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Jacob F Wynne
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY 10065, USA
| | - Chih-Wei Chang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Jonathon A Nye
- Radiology and Radiological Science, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Hui Mao
- Department of Radiology and Imaging Science, and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, USA
- Department of Biomedical Informatics, Emory University, Atlanta, GA 30322, USA
| |
Collapse
|
5
|
Chaudhary MFA, Gerard SE, Christensen GE, Cooper CB, Schroeder JD, Hoffman EA, Reinhardt JM. LungViT: Ensembling Cascade of Texture Sensitive Hierarchical Vision Transformers for Cross-Volume Chest CT Image-to-Image Translation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2448-2465. [PMID: 38373126 PMCID: PMC11227912 DOI: 10.1109/tmi.2024.3367321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/21/2024]
Abstract
Chest computed tomography (CT) at inspiration is often complemented by an expiratory CT to identify peripheral airways disease. Additionally, co-registered inspiratory-expiratory volumes can be used to derive various markers of lung function. Expiratory CT scans, however, may not be acquired due to dose or scan time considerations or may be inadequate due to motion or insufficient exhale; leading to a missed opportunity to evaluate underlying small airways disease. Here, we propose LungViT- a generative adversarial learning approach using hierarchical vision transformers for translating inspiratory CT intensities to corresponding expiratory CT intensities. LungViT addresses several limitations of the traditional generative models including slicewise discontinuities, limited size of generated volumes, and their inability to model texture transfer at volumetric level. We propose a shifted-window hierarchical vision transformer architecture with squeeze-and-excitation decoder blocks for modeling dependencies between features. We also propose a multiview texture similarity distance metric for texture and style transfer in 3D. To incorporate global information into the training process and refine the output of our model, we use ensemble cascading. LungViT is able to generate large 3D volumes of size 320×320×320 . We train and validate our model using a diverse cohort of 1500 subjects with varying disease severity. To assess model generalizability beyond the development set biases, we evaluate our model on an out-of-distribution external validation set of 200 subjects. Clinical validation on internal and external testing sets shows that synthetic volumes could be reliably adopted for deriving clinical endpoints of chronic obstructive pulmonary disease.
Collapse
|
6
|
Gao Y, Xie H, Chang CW, Peng J, Pan S, Qiu RLJ, Wang T, Ghavidel B, Roper J, Zhou J, Yang X. CT-based synthetic iodine map generation using conditional denoising diffusion probabilistic model. Med Phys 2024. [PMID: 38889368 DOI: 10.1002/mp.17258] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Revised: 04/17/2024] [Accepted: 06/03/2024] [Indexed: 06/20/2024] Open
Abstract
BACKGROUND Iodine maps, derived from image-processing of contrast-enhanced dual-energy computed tomography (DECT) scans, highlight the differences in tissue iodine intake. It finds multiple applications in radiology, including vascular imaging, pulmonary evaluation, kidney assessment, and cancer diagnosis. In radiation oncology, it can contribute to designing more accurate and personalized treatment plans. However, DECT scanners are not commonly available in radiation therapy centers. Additionally, the use of iodine contrast agents is not suitable for all patients, especially those allergic to iodine agents, posing further limitations to the accessibility of this technology. PURPOSE The purpose of this work is to generate synthetic iodine map images from non-contrast single-energy CT (SECT) images using conditional denoising diffusion probabilistic model (DDPM). METHODS One-hundered twenty-six head-and-neck patients' images were retrospectively investigated in this work. Each patient underwent non-contrast SECT and contrast DECT scans. Ground truth iodine maps were generated from contrast DECT scans using commercial software syngo.via installed in the clinic. A conditional DDPM was implemented in this work to synthesize iodine maps. Three-fold cross-validation was conducted, with each iteration selecting the data from 42 patients as the test dataset and the remainder as the training dataset. Pixel-to-pixel generative adversarial network (GAN) and CycleGAN served as reference methods for evaluating the proposed DDPM method. RESULTS The accuracy of the proposed DDPM was evaluated using three quantitative metrics: mean absolute error (MAE) (1.039 ± 0.345 mg/mL), structural similarity index measure (SSIM) (0.89 ± 0.10) and peak signal-to-noise ratio (PSNR) (25.4 ± 3.5 db) respectively. Compared to the reference methods, the proposed technique showcased superior performance across the evaluated metrics, further validated by the paired two-tailed t-tests. CONCLUSION The proposed conditional DDPM framework has demonstrated the feasibility of generating synthetic iodine map images from non-contrast SECT images. This method presents a potential clinical application, which is providing accurate iodine contrast map in instances where only non-contrast SECT is accessible.
Collapse
Affiliation(s)
- Yuan Gao
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Huiqiao Xie
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Chih-Wei Chang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Junbo Peng
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Shaoyan Pan
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Richard L J Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Beth Ghavidel
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Jun Zhou
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| |
Collapse
|
7
|
Khosravi B, Li F, Dapamede T, Rouzrokh P, Gamble CU, Trivedi HM, Wyles CC, Sellergren AB, Purkayastha S, Erickson BJ, Gichoya JW. Synthetically enhanced: unveiling synthetic data's potential in medical imaging research. EBioMedicine 2024; 104:105174. [PMID: 38821021 PMCID: PMC11177083 DOI: 10.1016/j.ebiom.2024.105174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2024] [Revised: 05/13/2024] [Accepted: 05/15/2024] [Indexed: 06/02/2024] Open
Abstract
BACKGROUND Chest X-rays (CXR) are essential for diagnosing a variety of conditions, but when used on new populations, model generalizability issues limit their efficacy. Generative AI, particularly denoising diffusion probabilistic models (DDPMs), offers a promising approach to generating synthetic images, enhancing dataset diversity. This study investigates the impact of synthetic data supplementation on the performance and generalizability of medical imaging research. METHODS The study employed DDPMs to create synthetic CXRs conditioned on demographic and pathological characteristics from the CheXpert dataset. These synthetic images were used to supplement training datasets for pathology classifiers, with the aim of improving their performance. The evaluation involved three datasets (CheXpert, MIMIC-CXR, and Emory Chest X-ray) and various experiments, including supplementing real data with synthetic data, training with purely synthetic data, and mixing synthetic data with external datasets. Performance was assessed using the area under the receiver operating curve (AUROC). FINDINGS Adding synthetic data to real datasets resulted in a notable increase in AUROC values (up to 0.02 in internal and external test sets with 1000% supplementation, p-value <0.01 in all instances). When classifiers were trained exclusively on synthetic data, they achieved performance levels comparable to those trained on real data with 200%-300% data supplementation. The combination of real and synthetic data from different sources demonstrated enhanced model generalizability, increasing model AUROC from 0.76 to 0.80 on the internal test set (p-value <0.01). INTERPRETATION Synthetic data supplementation significantly improves the performance and generalizability of pathology classifiers in medical imaging. FUNDING Dr. Gichoya is a 2022 Robert Wood Johnson Foundation Harold Amos Medical Faculty Development Program and declares support from RSNA Health Disparities grant (#EIHD2204), Lacuna Fund (#67), Gordon and Betty Moore Foundation, NIH (NIBIB) MIDRC grant under contracts 75N92020C00008 and 75N92020C00021, and NHLBI Award Number R01HL167811.
Collapse
Affiliation(s)
- Bardia Khosravi
- Department of Radiology, Mayo Clinic, Rochester, MN, USA; Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | - Frank Li
- Department of Radiology, Emory University, Atlanta, GA, USA
| | - Theo Dapamede
- Department of Radiology, Emory University, Atlanta, GA, USA
| | - Pouria Rouzrokh
- Department of Radiology, Mayo Clinic, Rochester, MN, USA; Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | | | - Hari M Trivedi
- Department of Radiology, Emory University, Atlanta, GA, USA
| | - Cody C Wyles
- Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA
| | | | - Saptarshi Purkayastha
- School of Informatics and Computing, Indiana University-Purdue University, Indianapolis, IN, USA
| | | | - Judy W Gichoya
- Department of Radiology, Emory University, Atlanta, GA, USA.
| |
Collapse
|
8
|
Eidex Z, Wang J, Safari M, Elder E, Wynne J, Wang T, Shu HK, Mao H, Yang X. High-resolution 3T to 7T ADC map synthesis with a hybrid CNN-transformer model. Med Phys 2024; 51:4380-4388. [PMID: 38630982 DOI: 10.1002/mp.17079] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Revised: 02/13/2024] [Accepted: 03/23/2024] [Indexed: 04/19/2024] Open
Abstract
BACKGROUND 7 Tesla (7T) apparent diffusion coefficient (ADC) maps derived from diffusion-weighted imaging (DWI) demonstrate improved image quality and spatial resolution over 3 Tesla (3T) ADC maps. However, 7T magnetic resonance imaging (MRI) currently suffers from limited clinical unavailability, higher cost, and increased susceptibility to artifacts. PURPOSE To address these issues, we propose a hybrid CNN-transformer model to synthesize high-resolution 7T ADC maps from multimodal 3T MRI. METHODS The Vision CNN-Transformer (VCT), composed of both Vision Transformer (ViT) blocks and convolutional layers, is proposed to produce high-resolution synthetic 7T ADC maps from 3T ADC maps and 3T T1-weighted (T1w) MRI. ViT blocks enabled global image context while convolutional layers efficiently captured fine detail. The VCT model was validated on the publicly available Human Connectome Project Young Adult dataset, comprising 3T T1w, 3T DWI, and 7T DWI brain scans. The Diffusion Imaging in Python library was used to compute ADC maps from the DWI scans. A total of 171 patient cases were randomly divided into 130 training cases, 20 validation cases, and 21 test cases. The synthetic ADC maps were evaluated by comparing their similarity to the ground truth volumes with the following metrics: peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and mean squared error (MSE). In addition, RESULTS: The results are as follows: PSNR: 27.0 ± 0.9 dB, SSIM: 0.945 ± 0.010, and MSE: 2.0E-3 ± 0.4E-3. Both qualitative and quantitative results demonstrate that VCT performs favorably against other state-of-the-art methods. We have introduced various efficiency improvements, including the implementation of flash attention and training on 176×208 resolution images. These enhancements have resulted in the reduction of parameters and training time per epoch by 50% in comparison to ResViT. Specifically, the training time per epoch has been shortened from 7.67 min to 3.86 min. CONCLUSION We propose a novel method to predict high-resolution 7T ADC maps from low-resolution 3T ADC maps and T1w MRI. Our predicted images demonstrate better spatial resolution and contrast compared to 3T MRI and prediction results made by ResViT and pix2pix. These high-quality synthetic 7T MR images could be beneficial for disease diagnosis and intervention, producing higher resolution and conformal contours, and as an intermediate step in generating synthetic CT for radiation therapy, especially when 7T MRI scanners are unavailable.
Collapse
Affiliation(s)
- Zach Eidex
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
- School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Jing Wang
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
| | - Mojtaba Safari
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
| | - Eric Elder
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
- Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Jacob Wynne
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Hui-Kuo Shu
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
- Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Hui Mao
- Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, Georgia, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology, Emory University, Atlanta, Georgia, USA
- School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA
- Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| |
Collapse
|
9
|
Kim W. Seeing the Unseen: Advancing Generative AI Research in Radiology. Radiology 2024; 311:e240935. [PMID: 38771182 DOI: 10.1148/radiol.240935] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/22/2024]
Affiliation(s)
- Woojin Kim
- From Rad AI, San Francisco, Calif; and Department of Radiology, Palo Alto VA Medical Center, 3801 Miranda Ave, Palo Alto, CA 94304
| |
Collapse
|
10
|
Safari M, Eidex Z, Chang CW, Qiu RL, Yang X. Fast MRI Reconstruction Using Deep Learning-based Compressed Sensing: A Systematic Review. ARXIV 2024:arXiv:2405.00241v1. [PMID: 38745700 PMCID: PMC11092677] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Magnetic resonance imaging (MRI) has revolutionized medical imaging, providing a non-invasive and highly detailed look into the human body. However, the long acquisition times of MRI present challenges, causing patient discomfort, motion artifacts, and limiting real-time applications. To address these challenges, researchers are exploring various techniques to reduce acquisition time and improve the overall efficiency of MRI. One such technique is compressed sensing (CS), which reduces data acquisition by leveraging image sparsity in transformed spaces. In recent years, deep learning (DL) has been integrated with CS-MRI, leading to a new framework that has seen remarkable growth. DL-based CS-MRI approaches are proving to be highly effective in accelerating MR imaging without compromising image quality. This review comprehensively examines DL-based CS-MRI techniques, focusing on their role in increasing MR imaging speed. We provide a detailed analysis of each category of DL-based CS-MRI including end-to-end, unroll optimization, self-supervised, and federated learning. Our systematic review highlights significant contributions and underscores the exciting potential of DL in CS-MRI. Additionally, our systematic review efficiently summarizes key results and trends in DL-based CS-MRI including quantitative metrics, the dataset used, acceleration factors, and the progress of and research interest in DL techniques over time. Finally, we discuss potential future directions and the importance of DL-based CS-MRI in the advancement of medical imaging. To facilitate further research in this area, we provide a GitHub repository that includes up-to-date DL-based CS-MRI publications and publicly available datasets - https://github.com/mosaf/Awesome-DL-based-CS-MRI.
Collapse
Affiliation(s)
- Mojtaba Safari
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, United States of America
| | - Zach Eidex
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, United States of America
| | - Chih-Wei Chang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, United States of America
| | - Richard L.J. Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, United States of America
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA 30322, United States of America
| |
Collapse
|
11
|
Pan S, Abouei E, Wynne J, Chang CW, Wang T, Qiu RLJ, Li Y, Peng J, Roper J, Patel P, Yu DS, Mao H, Yang X. Synthetic CT generation from MRI using 3D transformer-based denoising diffusion model. Med Phys 2024; 51:2538-2548. [PMID: 38011588 PMCID: PMC10994752 DOI: 10.1002/mp.16847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2023] [Revised: 11/02/2023] [Accepted: 11/03/2023] [Indexed: 11/29/2023] Open
Abstract
BACKGROUND AND PURPOSE Magnetic resonance imaging (MRI)-based synthetic computed tomography (sCT) simplifies radiation therapy treatment planning by eliminating the need for CT simulation and error-prone image registration, ultimately reducing patient radiation dose and setup uncertainty. In this work, we propose a MRI-to-CT transformer-based improved denoising diffusion probabilistic model (MC-IDDPM) to translate MRI into high-quality sCT to facilitate radiation treatment planning. METHODS MC-IDDPM implements diffusion processes with a shifted-window transformer network to generate sCT from MRI. The proposed model consists of two processes: a forward process, which involves adding Gaussian noise to real CT scans to create noisy images, and a reverse process, in which a shifted-window transformer V-net (Swin-Vnet) denoises the noisy CT scans conditioned on the MRI from the same patient to produce noise-free CT scans. With an optimally trained Swin-Vnet, the reverse diffusion process was used to generate noise-free sCT scans matching MRI anatomy. We evaluated the proposed method by generating sCT from MRI on an institutional brain dataset and an institutional prostate dataset. Quantitative evaluations were conducted using several metrics, including Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Multi-scale Structure Similarity Index (SSIM), and Normalized Cross Correlation (NCC). Dosimetry analyses were also performed, including comparisons of mean dose and target dose coverages for 95% and 99%. RESULTS MC-IDDPM generated brain sCTs with state-of-the-art quantitative results with MAE 48.825 ± 21.491 HU, PSNR 26.491 ± 2.814 dB, SSIM 0.947 ± 0.032, and NCC 0.976 ± 0.019. For the prostate dataset: MAE 55.124 ± 9.414 HU, PSNR 28.708 ± 2.112 dB, SSIM 0.878 ± 0.040, and NCC 0.940 ± 0.039. MC-IDDPM demonstrates a statistically significant improvement (with p < 0.05) in most metrics when compared to competing networks, for both brain and prostate synthetic CT. Dosimetry analyses indicated that the target dose coverage differences by using CT and sCT were within ± 0.34%. CONCLUSIONS We have developed and validated a novel approach for generating CT images from routine MRIs using a transformer-based improved DDPM. This model effectively captures the complex relationship between CT and MRI images, allowing for robust and high-quality synthetic CT images to be generated in a matter of minutes. This approach has the potential to greatly simplify the treatment planning process for radiation therapy by eliminating the need for additional CT scans, reducing the amount of time patients spend in treatment planning, and enhancing the accuracy of treatment delivery.
Collapse
Affiliation(s)
- Shaoyan Pan
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
- Department of Biomedical Informatics, Emory University, Atlanta, Georgia, USA
| | - Elham Abouei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Jacob Wynne
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Chih-Wei Chang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Tonghe Wang
- Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, New York, USA
| | - Richard L J Qiu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Yuheng Li
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Junbo Peng
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Pretesh Patel
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - David S Yu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
| | - Hui Mao
- Department of Radiology and Imaging Sciences, Winship Cancer Institute, Atlanta, Georgia, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, Georgia, USA
- Department of Biomedical Informatics, Emory University, Atlanta, Georgia, USA
| |
Collapse
|
12
|
Yu X, Yang Q, Tang Y, Gao R, Bao S, Cai LY, Lee HH, Huo Y, Moore AZ, Ferrucci L, Landman BA. Deep conditional generative model for longitudinal single-slice abdominal computed tomography harmonization. J Med Imaging (Bellingham) 2024; 11:024008. [PMID: 38571764 PMCID: PMC10987005 DOI: 10.1117/1.jmi.11.2.024008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Revised: 01/18/2024] [Accepted: 03/14/2024] [Indexed: 04/05/2024] Open
Abstract
Purpose Two-dimensional single-slice abdominal computed tomography (CT) provides a detailed tissue map with high resolution allowing quantitative characterization of relationships between health conditions and aging. However, longitudinal analysis of body composition changes using these scans is difficult due to positional variation between slices acquired in different years, which leads to different organs/tissues being captured. Approach To address this issue, we propose C-SliceGen, which takes an arbitrary axial slice in the abdominal region as a condition and generates a pre-defined vertebral level slice by estimating structural changes in the latent space. Results Our experiments on 2608 volumetric CT data from two in-house datasets and 50 subjects from the 2015 Multi-Atlas Abdomen Labeling Challenge Beyond the Cranial Vault (BTCV) dataset demonstrate that our model can generate high-quality images that are realistic and similar. We further evaluate our method's capability to harmonize longitudinal positional variation on 1033 subjects from the Baltimore longitudinal study of aging dataset, which contains longitudinal single abdominal slices, and confirmed that our method can harmonize the slice positional variance in terms of visceral fat area. Conclusion This approach provides a promising direction for mapping slices from different vertebral levels to a target slice and reducing positional variance for single-slice longitudinal analysis. The source code is available at: https://github.com/MASILab/C-SliceGen.
Collapse
Affiliation(s)
- Xin Yu
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
| | - Qi Yang
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
| | - Yucheng Tang
- Vanderbilt University, Department of Electrical and Computer Engineering, Nashville, Tennessee, United States
| | - Riqiang Gao
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
| | - Shunxing Bao
- Vanderbilt University, Department of Electrical and Computer Engineering, Nashville, Tennessee, United States
| | - Leon Y. Cai
- Vanderbilt University, Department of Biomedical Engineering, Nashville, Tennessee, United States
| | - Ho Hin Lee
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
| | - Yuankai Huo
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
- Vanderbilt University, Department of Electrical and Computer Engineering, Nashville, Tennessee, United States
| | | | - Luigi Ferrucci
- National Institute on Aging, Baltimore, Maryland, United States
| | - Bennett A. Landman
- Vanderbilt University, Department of Computer Science, Nashville, Tennessee, United States
- Vanderbilt University, Department of Electrical and Computer Engineering, Nashville, Tennessee, United States
- Vanderbilt University, Department of Biomedical Engineering, Nashville, Tennessee, United States
| |
Collapse
|
13
|
Choi JY, Ryu IH, Kim JK, Lee IS, Yoo TK. Development of a generative deep learning model to improve epiretinal membrane detection in fundus photography. BMC Med Inform Decis Mak 2024; 24:25. [PMID: 38273286 PMCID: PMC10811871 DOI: 10.1186/s12911-024-02431-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Accepted: 01/17/2024] [Indexed: 01/27/2024] Open
Abstract
BACKGROUND The epiretinal membrane (ERM) is a common retinal disorder characterized by abnormal fibrocellular tissue at the vitreomacular interface. Most patients with ERM are asymptomatic at early stages. Therefore, screening for ERM will become increasingly important. Despite the high prevalence of ERM, few deep learning studies have investigated ERM detection in the color fundus photography (CFP) domain. In this study, we built a generative model to enhance ERM detection performance in the CFP. METHODS This deep learning study retrospectively collected 302 ERM and 1,250 healthy CFP data points from a healthcare center. The generative model using StyleGAN2 was trained using single-center data. EfficientNetB0 with StyleGAN2-based augmentation was validated using independent internal single-center data and external datasets. We randomly assigned healthcare center data to the development (80%) and internal validation (20%) datasets. Data from two publicly accessible sources were used as external validation datasets. RESULTS StyleGAN2 facilitated realistic CFP synthesis with the characteristic cellophane reflex features of the ERM. The proposed method with StyleGAN2-based augmentation outperformed the typical transfer learning without a generative adversarial network. The proposed model achieved an area under the receiver operating characteristic (AUC) curve of 0.926 for internal validation. AUCs of 0.951 and 0.914 were obtained for the two external validation datasets. Compared with the deep learning model without augmentation, StyleGAN2-based augmentation improved the detection performance and contributed to the focus on the location of the ERM. CONCLUSIONS We proposed an ERM detection model by synthesizing realistic CFP images with the pathological features of ERM through generative deep learning. We believe that our deep learning framework will help achieve a more accurate detection of ERM in a limited data setting.
Collapse
Affiliation(s)
- Joon Yul Choi
- Department of Biomedical Engineering, Yonsei University, Wonju, South Korea
| | - Ik Hee Ryu
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
- Research and development department, VISUWORKS, Seoul, South Korea
| | - Jin Kuk Kim
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
- Research and development department, VISUWORKS, Seoul, South Korea
| | - In Sik Lee
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
| | - Tae Keun Yoo
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea.
- Research and development department, VISUWORKS, Seoul, South Korea.
| |
Collapse
|
14
|
Ren Y, Wang G, Wang P, Liu K, Liu Q, Sun H, Li X, Wei B. MM-SFENet: multi-scale multi-task localization and classification of bladder cancer in MRI with spatial feature encoder network. Phys Med Biol 2024; 69:025009. [PMID: 38091612 DOI: 10.1088/1361-6560/ad1548] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Accepted: 12/13/2023] [Indexed: 01/12/2024]
Abstract
Objective. Bladder cancer is a common malignant urinary carcinoma, with muscle-invasive and non-muscle-invasive as its two major subtypes. This paper aims to achieve automated bladder cancer invasiveness localization and classification based on MRI.Approach. Different from previous efforts that segment bladder wall and tumor, we propose a novel end-to-end multi-scale multi-task spatial feature encoder network (MM-SFENet) for locating and classifying bladder cancer, according to the classification criteria of the spatial relationship between the tumor and bladder wall. First, we built a backbone with residual blocks to distinguish bladder wall and tumor; then, a spatial feature encoder is designed to encode the multi-level features of the backbone to learn the criteria.Main Results. We substitute Smooth-L1 Loss with IoU Loss for multi-task learning, to improve the accuracy of the classification task. By learning two datasets collected from bladder cancer patients at the hospital, the mAP, IoU, Acc, Sen and Spec are used as the evaluation metrics. The experimental result could reach 93.34%, 83.16%, 85.65%, 81.51%, 89.23% on test set1 and 80.21%, 75.43%, 79.52%, 71.87%, 77.86% on test set2.Significance. The experimental result demonstrates the effectiveness of the proposed MM-SFENet on the localization and classification of bladder cancer. It may provide an effective supplementary diagnosis method for bladder cancer staging.
Collapse
Affiliation(s)
- Yu Ren
- College of Electronic Engineering and Intelligent Manufacturing, Anqing Normal University, Anqing 246133, People's Republic of China
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| | - Guoli Wang
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| | - Pingping Wang
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| | - Kunmeng Liu
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| | - Quanjin Liu
- College of Electronic Engineering and Intelligent Manufacturing, Anqing Normal University, Anqing 246133, People's Republic of China
| | - Hongfu Sun
- Urological department, Affiliated Hospital of Shandong University of Traditional Chinese Medicine, Jinan 250011, People's Republic of China
| | - Xiang Li
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| | - Bengzheng Wei
- Center for Medical Artificial Intelligence, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
- Qingdao Academy of Chinese Medical Sciences, Shandong University of Traditional Chinese Medicine, Qingdao 266112, People's Republic of China
| |
Collapse
|
15
|
Shao L, Chen B, Zhang Z, Zhang Z, Chen X. Artificial intelligence generated content (AIGC) in medicine: A narrative review. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2024; 21:1672-1711. [PMID: 38303483 DOI: 10.3934/mbe.2024073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/03/2024]
Abstract
Recently, artificial intelligence generated content (AIGC) has been receiving increased attention and is growing exponentially. AIGC is generated based on the intentional information extracted from human-provided instructions by generative artificial intelligence (AI) models. AIGC quickly and automatically generates large amounts of high-quality content. Currently, there is a shortage of medical resources and complex medical procedures in medicine. Due to its characteristics, AIGC can help alleviate these problems. As a result, the application of AIGC in medicine has gained increased attention in recent years. Therefore, this paper provides a comprehensive review on the recent state of studies involving AIGC in medicine. First, we present an overview of AIGC. Furthermore, based on recent studies, the application of AIGC in medicine is reviewed from two aspects: medical image processing and medical text generation. The basic generative AI models, tasks, target organs, datasets and contribution of studies are considered and summarized. Finally, we also discuss the limitations and challenges faced by AIGC and propose possible solutions with relevant studies. We hope this review can help readers understand the potential of AIGC in medicine and obtain some innovative ideas in this field.
Collapse
Affiliation(s)
- Liangjing Shao
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| | - Benshuang Chen
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| | - Ziqun Zhang
- Information office, Fudan University, Shanghai 200032, China
| | - Zhen Zhang
- Baoshan Branch of Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200444, China
| | - Xinrong Chen
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| |
Collapse
|
16
|
Liang P, Chen J, Yao L, Yu Y, Liang K, Chang Q. DAWTran: dynamic adaptive windowing transformer network for pneumothorax segmentation with implicit feature alignment. Phys Med Biol 2023; 68:175020. [PMID: 37541224 DOI: 10.1088/1361-6560/aced79] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Accepted: 08/04/2023] [Indexed: 08/06/2023]
Abstract
Objective. This study aims to address the significant challenges posed by pneumothorax segmentation in computed tomography images due to the resemblance between pneumothorax regions and gas-containing structures such as the trachea and bronchus.Approach. We introduce a novel dynamic adaptive windowing transformer (DAWTran) network incorporating implicit feature alignment for precise pneumothorax segmentation. The DAWTran network consists of an encoder module, which employs a DAWTran, and a decoder module. We have proposed a unique dynamic adaptive windowing strategy that enables multi-head self-attention to effectively capture multi-scale information. The decoder module incorporates an implicit feature alignment function to minimize information deviation. Moreover, we utilize a hybrid loss function to address the imbalance between positive and negative samples.Main results. Our experimental results demonstrate that the DAWTran network significantly improves the segmentation performance. Specifically, it achieves a higher dice similarity coefficient (DSC) of 91.35% (a larger DSC value implies better performance), showing an increase of 2.21% compared to the TransUNet method. Meanwhile, it significantly reduces the Hausdorff distance (HD) to 8.06 mm (a smaller HD value implies better performance), reflecting a reduction of 29.92% in comparison to the TransUNet method. Incorporating the dynamic adaptive windowing (DAW) mechanism has proven to enhance DAWTran's performance, leading to a 4.53% increase in DSC and a 15.85% reduction in HD as compared to SwinUnet. The application of the implicit feature alignment (IFA) further improves the segmentation accuracy, increasing the DSC by an additional 0.11% and reducing the HD by another 10.01% compared to the model only employing DAW.Significance. These results highlight the potential of the DAWTran network for accurate pneumothorax segmentation in clinical applications, suggesting that it could be an invaluable tool in improving the precision and effectiveness of diagnosis and treatment in related healthcare scenarios. The improved segmentation performance with the inclusion of DAW and IFA validates the effectiveness of our proposed model and its components.
Collapse
Affiliation(s)
- Pengchen Liang
- The Department of School of Microelectronics, Shanghai University, Shanghai, 201800, People's Republic of China
| | - Jianguo Chen
- The School of Software Engineering, Sun Yat-sen University, Zhuhai, Guangdong Province, 519000, People's Republic of China
| | - Lei Yao
- The Department of School of Microelectronics, Shanghai University, Shanghai, 201800, People's Republic of China
| | - Yanfang Yu
- The Department of Pulmonary and Critical Care Medicine, Jiading Central Hospital, Shanghai University of Medicine and Health Sciences, Shanghai, 201800, People's Republic of China
| | - Kaiyi Liang
- The Department of Radiology Jiading District Central Hospital Affiliated with the Shanghai University of Medicine and Health Sciences, Shanghai, 201800, People's Republic of China
| | - Qing Chang
- The Department Shanghai Key Laboratory of Gastric Neoplasms, Department of Surgery, Shanghai Institute of Digestive Surgery, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 201800, People's Republic of China
| |
Collapse
|