1
|
Buddenkotte T, Rundo L, Woitek R, Escudero Sanchez L, Beer L, Crispin-Ortuzar M, Etmann C, Mukherjee S, Bura V, McCague C, Sahin H, Pintican R, Zerunian M, Allajbeu I, Singh N, Sahdev A, Havrilesky L, Cohn DE, Bateman NW, Conrads TP, Darcy KM, Maxwell GL, Freymann JB, Öktem O, Brenton JD, Sala E, Schönlieb CB. Deep learning-based segmentation of multisite disease in ovarian cancer. Eur Radiol Exp 2023; 7:77. [PMID: 38057616 PMCID: PMC10700248 DOI: 10.1186/s41747-023-00388-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Accepted: 09/21/2023] [Indexed: 12/08/2023] Open
Abstract
PURPOSE To determine if pelvic/ovarian and omental lesions of ovarian cancer can be reliably segmented on computed tomography (CT) using fully automated deep learning-based methods. METHODS A deep learning model for the two most common disease sites of high-grade serous ovarian cancer lesions (pelvis/ovaries and omentum) was developed and compared against the well-established "no-new-Net" framework and unrevised trainee radiologist segmentations. A total of 451 CT scans collected from four different institutions were used for training (n = 276), evaluation (n = 104) and testing (n = 71) of the methods. The performance was evaluated using the Dice similarity coefficient (DSC) and compared using a Wilcoxon test. RESULTS Our model outperformed no-new-Net for the pelvic/ovarian lesions in cross-validation, on the evaluation and test set by a significant margin (p values being 4 × 10-7, 3 × 10-4, 4 × 10-2, respectively), and for the omental lesions on the evaluation set (p = 1 × 10-3). Our model did not perform significantly differently in segmenting pelvic/ovarian lesions (p = 0.371) compared to a trainee radiologist. On an independent test set, the model achieved a DSC performance of 71 ± 20 (mean ± standard deviation) for pelvic/ovarian and 61 ± 24 for omental lesions. CONCLUSION Automated ovarian cancer segmentation on CT scans using deep neural networks is feasible and achieves performance close to a trainee-level radiologist for pelvic/ovarian lesions. RELEVANCE STATEMENT Automated segmentation of ovarian cancer may be used by clinicians for CT-based volumetric assessments and researchers for building complex analysis pipelines. KEY POINTS • The first automated approach for pelvic/ovarian and omental ovarian cancer lesion segmentation on CT images has been presented. • Automated segmentation of ovarian cancer lesions can be comparable with manual segmentation of trainee radiologists. • Careful hyperparameter tuning can provide models significantly outperforming strong state-of-the-art baselines.
Collapse
Affiliation(s)
- Thomas Buddenkotte
- Department, of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Department for Diagnostic and Interventional Radiology and Nuclear Medicine, University Hospital Hamburg-Eppendorf, Hamburg, Germany
- jung diagnostics GmbH, Hamburg, Germany
| | - Leonardo Rundo
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Department of Information and Electrical Engineering and Applied Mathematics, University of Salerno, Fisciano, Italy
| | - Ramona Woitek
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Department of Medicine, Danube Private University, Krems, Austria
| | - Lorena Escudero Sanchez
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
| | - Lucian Beer
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Department of Biomedical Imaging and Image-Guided Therapy, Medical University Vienna, Vienna, Austria
| | - Mireia Crispin-Ortuzar
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK
- Department of Oncology, University of Cambridge, Cambridge, UK
| | - Christian Etmann
- Department, of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| | - Subhadip Mukherjee
- Department, of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| | - Vlad Bura
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Department of Radiology and Medical Imaging, County Clinical Emergency Hospital, Cluj-Napoca-Napoca, Romania
| | - Cathal McCague
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
| | - Hilal Sahin
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Department of Radiology, Tepecik Training and Research Hospital, Izmir, Turkey
| | - Roxana Pintican
- Department of Radiology and Medical Imaging, County Clinical Emergency Hospital, Cluj-Napoca-Napoca, Romania
- Department of Radiology, Iuliu Hațieganu University of Medicine and Pharmacy, Cluj-Napoca-Napoca, Romania
| | - Marta Zerunian
- Department of Medical-Surgical and Translational Medicine-Radiology Unit, Sapienza University of Rome, Sant'Andrea Hospital, Rome, Italy
| | - Iris Allajbeu
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK
| | - Naveena Singh
- Department of Clinical Pathology, Barts Health NHS Trust, London, UK
| | - Anju Sahdev
- Department of Radiology, Barts Health NHS Trust, London, UK
| | | | - David E Cohn
- Departmant of Obstetrics and Gynecology, Division of Gynecologic Oncology, Ohio State University Comprehensive Cancer Center, Ohio State University College of Medicine, Columbus, OH, USA
| | - Nicholas W Bateman
- Department of Obstetrics and Gynecology, Gynecologic Cancer Center of Excellence, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- The John P. Murtha Cancer Center, Walter Reed National Military Medical Center, Uniformed Services University, Bethesda, MD, USA
| | - Thomas P Conrads
- Department of Obstetrics and Gynecology, Gynecologic Cancer Center of Excellence, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- The John P. Murtha Cancer Center, Walter Reed National Military Medical Center, Uniformed Services University, Bethesda, MD, USA
- Department of Obstetrics and Gynecology, Inova Fairfax Medical Campus, Falls Church, VA, USA
- Inova Center for Personalized Health, Inova Schar Cancer Institute, Falls Church, VA, USA
| | - Kathleen M Darcy
- Department of Obstetrics and Gynecology, Gynecologic Cancer Center of Excellence, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- The John P. Murtha Cancer Center, Walter Reed National Military Medical Center, Uniformed Services University, Bethesda, MD, USA
| | - G Larry Maxwell
- Department of Obstetrics and Gynecology, Gynecologic Cancer Center of Excellence, Walter Reed National Military Medical Center, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
- The John P. Murtha Cancer Center, Walter Reed National Military Medical Center, Uniformed Services University, Bethesda, MD, USA
- Department of Obstetrics and Gynecology, Inova Fairfax Medical Campus, Falls Church, VA, USA
| | - John B Freymann
- Cancer Imaging Informatics Lab, Frederick National Laboratory for Cancer Research, Frederick, MD, USA
| | - Ozan Öktem
- Department of Mathematics, KTH Royal Institute of Technology, Stockholm, Sweden
| | - James D Brenton
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK
- Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, UK
| | - Evis Sala
- Department of Radiology, University of Cambridge, Box 218, Cambridge Biomedical Campus, Cambridge, CB2 0QQ, UK.
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, UK.
- Dipartimento Di Scienze Radiologiche Ed Ematologiche, Universita Cattolica del Sacro Cuore, Rome, Italy.
- Dipartimento Diagnostica Per Immagini, Radioterapia Oncologica Ed Ematologia, Policlinico Universitario A. Gemelli IRCCS, Rome, Italy.
| | - Carola-Bibiane Schönlieb
- Department, of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| |
Collapse
|
2
|
Buddenkotte T, Escudero Sanchez L, Crispin-Ortuzar M, Woitek R, McCague C, Brenton JD, Öktem O, Sala E, Rundo L. Calibrating ensembles for scalable uncertainty quantification in deep learning-based medical image segmentation. Comput Biol Med 2023; 163:107096. [PMID: 37302375 DOI: 10.1016/j.compbiomed.2023.107096] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 04/16/2023] [Accepted: 05/27/2023] [Indexed: 06/13/2023]
Abstract
Uncertainty quantification in automated image analysis is highly desired in many applications. Typically, machine learning models in classification or segmentation are only developed to provide binary answers; however, quantifying the uncertainty of the models can play a critical role for example in active learning or machine human interaction. Uncertainty quantification is especially difficult when using deep learning-based models, which are the state-of-the-art in many imaging applications. The current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. Scalable solutions often rely on classical techniques, such as dropout, during inference or training ensembles of identical models with different random seeds to obtain a posterior distribution. In this paper, we present the following contributions. First, we show that the classical approaches fail to approximate the classification probability. Second, we propose a scalable and intuitive framework for uncertainty quantification in medical image segmentation that yields measurements that approximate the classification probability. Third, we suggest the usage of k-fold cross-validation to overcome the need for held out calibration data. Lastly, we motivate the adoption of our method in active learning, creating pseudo-labels to learn from unlabeled images and human-machine collaboration.
Collapse
Affiliation(s)
- Thomas Buddenkotte
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom; Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Department for Diagnostic and Interventional Radiology and Nuclear Medicine, University Hospital Hamburg-Eppendorf, Hamburg, Germany; Jung diagnostics GmbH, Hamburg, Germany.
| | - Lorena Escudero Sanchez
- Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom
| | - Mireia Crispin-Ortuzar
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, United Kingdom; Department of Oncology, University of Cambridge, Cambridge, United Kingdom
| | - Ramona Woitek
- Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom; Medical Image Analysis & Artificial Intelligence (MIAAI), Department of Medicine, Danube Private University, Krems, Austria
| | - Cathal McCague
- Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom
| | - James D Brenton
- Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Institute, University of Cambridge, Cambridge, United Kingdom; Department of Oncology, University of Cambridge, Cambridge, United Kingdom
| | - Ozan Öktem
- Department of Mathematics, KTH Royal Institute of Technology, Stockholm, Sweden
| | - Evis Sala
- Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom; Dipartimento di Scienze Radiologiche ed Ematologiche, Università Cattolica del Sacro Cuore, Rome, Italy; Dipartimento Diagnostica per Immagini, Radioterapia Oncologica ed Ematologia, Policlinico Universitario A. Gemelli IRCCS, Rome, Italy
| | - Leonardo Rundo
- Department of Radiology, University of Cambridge, Cambridge, United Kingdom; Cancer Research UK Cambridge Centre, University of Cambridge, Cambridge, United Kingdom; Department of Information and Electrical Engineering and Applied Mathematics, University of Salerno, Fisciano (SA), Italy
| |
Collapse
|
3
|
Escudero Sanchez L, Buddenkotte T, Al Sa’d M, McCague C, Darcy J, Rundo L, Samoshkin A, Graves MJ, Hollamby V, Browne P, Crispin-Ortuzar M, Woitek R, Sala E, Schönlieb CB, Doran SJ, Öktem O. Integrating Artificial Intelligence Tools in the Clinical Research Setting: The Ovarian Cancer Use Case. Diagnostics (Basel) 2023; 13:2813. [PMID: 37685352 PMCID: PMC10486639 DOI: 10.3390/diagnostics13172813] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 07/31/2023] [Accepted: 08/22/2023] [Indexed: 09/10/2023] Open
Abstract
Artificial intelligence (AI) methods applied to healthcare problems have shown enormous potential to alleviate the burden of health services worldwide and to improve the accuracy and reproducibility of predictions. In particular, developments in computer vision are creating a paradigm shift in the analysis of radiological images, where AI tools are already capable of automatically detecting and precisely delineating tumours. However, such tools are generally developed in technical departments that continue to be siloed from where the real benefit would be achieved with their usage. Significant effort still needs to be made to make these advancements available, first in academic clinical research and ultimately in the clinical setting. In this paper, we demonstrate a prototype pipeline based entirely on open-source software and free of cost to bridge this gap, simplifying the integration of tools and models developed within the AI community into the clinical research setting, ensuring an accessible platform with visualisation applications that allow end-users such as radiologists to view and interact with the outcome of these AI tools.
Collapse
Affiliation(s)
- Lorena Escudero Sanchez
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- National Cancer Imaging Translational Accelerator (NCITA) Consortium, UK
| | - Thomas Buddenkotte
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, Cambridge CB3 0WA, UK
- Department for Diagnostic and Interventional Radiology and Nuclear Medicine, University Hospital Hamburg-Eppendorf, 20246 Hamburg, Germany
- Jung Diagnostics GmbH, 22335 Hamburg, Germany
| | - Mohammad Al Sa’d
- National Cancer Imaging Translational Accelerator (NCITA) Consortium, UK
- Cancer Imaging Centre, Department of Surgery & Cancer, Imperial College, London SW7 2AZ, UK
| | - Cathal McCague
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- Cambridge University Hospitals NHS Foundation Trust, Cambridge CB2 0QQ, UK
| | - James Darcy
- National Cancer Imaging Translational Accelerator (NCITA) Consortium, UK
- Division of Radiotherapy and Imaging, Institute of Cancer Research, London SW7 3RP, UK
| | - Leonardo Rundo
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- Department of Information and Electrical Engineering and Applied Mathematics (DIEM), University of Salerno, 84084 Fisciano, Italy
| | - Alex Samoshkin
- Office for Translational Research, School of Clinical Medicine, University of Cambridge, Cambridge CB2 0SP, UK
| | - Martin J. Graves
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cambridge University Hospitals NHS Foundation Trust, Cambridge CB2 0QQ, UK
| | - Victoria Hollamby
- Research and Information Governance, School of Clinical Medicine, University of Cambridge, Cambridge CB2 0SP, UK
| | - Paul Browne
- High Performance Computing Department, University of Cambridge, Cambridge CB3 0RB, UK
| | - Mireia Crispin-Ortuzar
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- Department of Oncology, University of Cambridge, Cambridge CB2 0XZ, UK
| | - Ramona Woitek
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- Research Centre for Medical Image Analysis and Artificial Intelligence (MIAAI), Department of Medicine, Faculty of Medicine and Dentistry, Danube Private University, 3500 Krems, Austria
| | - Evis Sala
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
- Cancer Research UK Cambridge Centre, Li Ka Shing Centre, Cambridge CB2 0RE, UK
- National Cancer Imaging Translational Accelerator (NCITA) Consortium, UK
- Cambridge University Hospitals NHS Foundation Trust, Cambridge CB2 0QQ, UK
- Dipartimento di Scienze Radiologiche ed Ematologiche, Universita Cattolica del Sacro Cuore, 00168 Rome, Italy
- Dipartimento Diagnostica per Immagini, Radioterapia Oncologica ed Ematologia, Policlinico Universitario A. Gemelli IRCCS, 00168 Rome, Italy
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, Cambridge CB3 0WA, UK
| | - Simon J. Doran
- National Cancer Imaging Translational Accelerator (NCITA) Consortium, UK
- Division of Radiotherapy and Imaging, Institute of Cancer Research, London SW7 3RP, UK
| | - Ozan Öktem
- Department of Mathematics, KTH-Royal Institute of Technology, SE-100 44 Stockholm, Sweden
| |
Collapse
|
4
|
Kimanius D, Zickert G, Nakane T, Adler J, Lunz S, Schönlieb CB, Öktem O, Scheres SHW. Exploiting prior knowledge about biological macromolecules in cryo-EM structure determination. IUCrJ 2021; 8:60-75. [PMID: 33520243 PMCID: PMC7793004 DOI: 10.1107/s2052252520014384] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/10/2020] [Accepted: 10/29/2020] [Indexed: 05/07/2023]
Abstract
Three-dimensional reconstruction of the electron-scattering potential of biological macromolecules from electron cryo-microscopy (cryo-EM) projection images is an ill-posed problem. The most popular cryo-EM software solutions to date rely on a regularization approach that is based on the prior assumption that the scattering potential varies smoothly over three-dimensional space. Although this approach has been hugely successful in recent years, the amount of prior knowledge that it exploits compares unfavorably with the knowledge about biological structures that has been accumulated over decades of research in structural biology. Here, a regularization framework for cryo-EM structure determination is presented that exploits prior knowledge about biological structures through a convolutional neural network that is trained on known macromolecular structures. This neural network is inserted into the iterative cryo-EM structure-determination process through an approach that is inspired by regularization by denoising. It is shown that the new regularization approach yields better reconstructions than the current state of the art for simulated data, and options to extend this work for application to experimental cryo-EM data are discussed.
Collapse
Affiliation(s)
- Dari Kimanius
- MRC Laboratory of Molecular Biology, Cambridge, United Kingdom
| | - Gustav Zickert
- Department of Mathematics, Royal Institute of Technology (KTH), Sweden
| | - Takanori Nakane
- MRC Laboratory of Molecular Biology, Cambridge, United Kingdom
| | | | - Sebastian Lunz
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom
| | - Carola-Bibiane Schönlieb
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, United Kingdom
| | - Ozan Öktem
- Department of Mathematics, Royal Institute of Technology (KTH), Sweden
| | | |
Collapse
|
5
|
Andrade-Loarca H, Kutyniok G, Öktem O. Shearlets as feature extractor for semantic edge detection: the model-based and data-driven realm. Proc Math Phys Eng Sci 2020; 476:20190841. [PMID: 33363436 PMCID: PMC7735309 DOI: 10.1098/rspa.2019.0841] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2019] [Accepted: 11/02/2020] [Indexed: 11/16/2022] Open
Abstract
Semantic edge detection has recently gained a lot of attention as an image-processing task, mainly because of its wide range of real-world applications. This is based on the fact that edges in images contain most of the semantic information. Semantic edge detection involves two tasks, namely pure edge detection and edge classification. Those are in fact fundamentally distinct in terms of the level of abstraction that each task requires. This fact is known as the distracted supervision paradox and limits the possible performance of a supervised model in semantic edge detection. In this work, we will present a novel hybrid method that is based on a combination of the model-based concept of shearlets, which provides probably optimally sparse approximations of a model class of images, and the data-driven method of a suitably designed convolutional neural network. We show that it avoids the distracted supervision paradox and achieves high performance in semantic edge detection. In addition, our approach requires significantly fewer parameters than a pure data-driven approach. Finally, we present several applications such as tomographic reconstruction and show that our approach significantly outperforms former methods, thereby also indicating the value of such hybrid methods for biomedical imaging.
Collapse
Affiliation(s)
| | - Gitta Kutyniok
- Institut für Mathematik, Technische Universität Berlin, 10623 Berlin, Germany
- Fakultät Elektrotechnik und Informatik, Technische Universität Berlin, 10587 Berlin, Germany
- Department of Physics and Technology, University of Tromsø, Tromsø, Norway
| | - Ozan Öktem
- Department of Mathematics, KTH - Royal Institute of Technology, SE-100 44 Stockholm, Sweden
| |
Collapse
|
6
|
Abstract
Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a multi-scale learned iterative reconstruction scheme that computes iterates on discretisations of increasing resolution. This procedure does not only reduce memory requirements, it also considerably speeds up reconstruction and training times, but most importantly is scalable to large scale inverse problems with non-trivial forward operators, such as those that arise in many 3D tomographic applications. In particular, we propose a hybrid network that combines the multiscale iterative approach with a particularly expressive network architecture which in combination exhibits excellent scalability in 3D. Applicability of the algorithm is demonstrated for 3D cone beam computed tomography from real measurement data of an organic phantom. Additionally, we examine scalability and reconstruction quality in comparison to established learned reconstruction methods in two dimensions for low dose computed tomography on human phantoms.
Collapse
Affiliation(s)
- Andreas Hauptmann
- Research Unit of Mathematical Sciences; University of Oulu, Oulu, Finland and with the Department of Computer Science; University College London, London, United Kingdom
| | - Jonas Adler
- Elekta, Stockholm, Sweden and KTH - Royal Institute of Technology, Stockolm, Sweden. He is currently with DeepMind, London, UK
| | - Simon Arridge
- Department of Computer Science; University College London, London, United Kingdom
| | - Ozan Öktem
- Department of Mathematics, KTH - Royal Institute of Technology, Stockholm, Sweden
| |
Collapse
|
7
|
Bergstrand J, Xu L, Miao X, Li N, Öktem O, Franzén B, Auer G, Lomnytska M, Widengren J. Super-resolution microscopy can identify specific protein distribution patterns in platelets incubated with cancer cells. Nanoscale 2019; 11:10023-10033. [PMID: 31086875 DOI: 10.1039/c9nr01967g] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Protein contents in platelets are frequently changed upon tumor development and metastasis. However, how cancer cells can influence protein-selective redistribution and release within platelets, thereby promoting tumor development, remains largely elusive. With fluorescence-based super-resolution stimulated emission depletion (STED) imaging we reveal how specific proteins, implicated in tumor progression and metastasis, re-distribute within platelets, when subject to soluble activators (thrombin, adenosine diphosphate and thromboxane A2), and when incubated with cancer (MCF-7, MDA-MB-231, EFO21) or non-cancer cells (184A1, MCF10A). Upon cancer cell incubation, the cell-adhesion protein P-selectin was found to re-distribute into circular nano-structures, consistent with accumulation into the membrane of protein-storing alpha-granules within the platelets. These changes were to a significantly lesser extent, if at all, found in platelets incubated with normal cells, or in platelets subject to soluble platelet activators. From these patterns, we developed a classification procedure, whereby platelets exposed to cancer cells, to non-cancer cells, soluble activators, as well as non-activated platelets all could be identified in an automatic, objective manner. We demonstrate that STED imaging, in contrast to electron and confocal microscopy, has the necessary spatial resolution and labelling efficiency to identify protein distribution patterns in platelets and can resolve how they specifically change upon different activations. Combined with image analyses of specific protein distribution patterns within the platelets, STED imaging can thus have a role in future platelet-based cancer diagnostics and therapeutic monitoring. The presented approach can also bring further clarity into fundamental mechanisms for cancer cell-platelet interactions, and into non-contact cell-to-cell interactions in general.
Collapse
Affiliation(s)
- Jan Bergstrand
- Royal Institute of Technology (KTH), Department of Applied Physics, Experimental Biomolecular Physics, Albanova Univ Center, SE-106 91 Stockholm, Sweden.
| | - Lei Xu
- Royal Institute of Technology (KTH), Department of Applied Physics, Experimental Biomolecular Physics, Albanova Univ Center, SE-106 91 Stockholm, Sweden.
| | - Xinyan Miao
- Royal Institute of Technology (KTH), Department of Applied Physics, Experimental Biomolecular Physics, Albanova Univ Center, SE-106 91 Stockholm, Sweden.
| | - Nailin Li
- Karolinska Institutet, Department of Medicine-Solna, Clinical Pharmacology, L7:03, Karolinska University Hospital-Solna, SE-171 76 Stockholm, Sweden
| | - Ozan Öktem
- Royal Institute of Technology (KTH), Department of Mathematics, Lindstedsvägen 25, SE-100 44 Stockholm, Sweden
| | - Bo Franzén
- Karolinska Institutet, Department of Oncology-Pathology, K7, Z1:00, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | - Gert Auer
- Karolinska Institutet, Department of Oncology-Pathology, K7, Z1:00, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | - Marta Lomnytska
- Karolinska Institutet, Department of Oncology-Pathology, K7, Z1:00, Karolinska University Hospital, 171 76 Stockholm, Sweden
| | - Jerker Widengren
- Royal Institute of Technology (KTH), Department of Applied Physics, Experimental Biomolecular Physics, Albanova Univ Center, SE-106 91 Stockholm, Sweden.
| |
Collapse
|
8
|
Tavabi AH, Beleggia M, Migunov V, Savenko A, Öktem O, Dunin-Borkowski RE, Pozzi G. Tunable Ampere phase plate for low dose imaging of biomolecular complexes. Sci Rep 2018; 8:5592. [PMID: 29618785 PMCID: PMC5884816 DOI: 10.1038/s41598-018-23100-3] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2017] [Accepted: 02/07/2018] [Indexed: 11/09/2022] Open
Abstract
A novel device that can be used as a tunable support-free phase plate for transmission electron microscopy of weakly scattering specimens is described. The device relies on the generation of a controlled phase shift by the magnetic field of a segment of current-carrying wire that is oriented parallel or antiparallel to the electron beam. The validity of the concept is established using both experimental electron holographic measurements and a theoretical model based on Ampere's law. Computer simulations are used to illustrate the resulting contrast enhancement for studies of biological cells and macromolecules.
Collapse
Affiliation(s)
- Amir H Tavabi
- Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons and Peter Grünberg Institute, Forschungszentrum Jülich, 52428, Jülich, Germany.
| | - Marco Beleggia
- Center for Electron Nanoscopy, Technical University of Denmark, 2800, Kgs Lyngby, Denmark
| | - Vadim Migunov
- Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons and Peter Grünberg Institute, Forschungszentrum Jülich, 52428, Jülich, Germany
| | - Alexey Savenko
- FEI Company, Achtseweg Noord 5, 5600 KA, Eindhoven, The Netherlands
| | - Ozan Öktem
- Centre for Industrial and Applied Mathematics, Department of Mathematics, KTH - Royal Institute of Technology, Stockholm, Sweden
| | - Rafal E Dunin-Borkowski
- Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons and Peter Grünberg Institute, Forschungszentrum Jülich, 52428, Jülich, Germany
| | - Giulio Pozzi
- Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons and Peter Grünberg Institute, Forschungszentrum Jülich, 52428, Jülich, Germany.,Department of Physics and Astronomy, University of Bologna, Viale B. Pichat 6/2, 40127, Bologna, Italy
| |
Collapse
|
9
|
Öktem O, Chen C, Domaniç NO, Ravikumar P, Bajaj C. SHAPE BASED IMAGE RECONSTRUCTION USING LINEARIZED DEFORMATIONS. Inverse Probl 2017; 33:035004. [PMID: 28855745 PMCID: PMC5573282 DOI: 10.1088/1361-6420/aa55af] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
We introduce a reconstruction framework that can account for shape related a priori information in ill-posed linear inverse problems in imaging. It is a variational scheme that uses a shape functional defined using deformable templates machinery from shape theory. As proof of concept, we apply the proposed shape based reconstruction to 2D tomography with very sparse measurements, and demonstrate strong empirical results.
Collapse
Affiliation(s)
- Ozan Öktem
- Department of Mathematics, KTH - Royal Institute of Technology, 100 44 Stockholm, Sweden
| | - Chong Chen
- Department of Mathematics, KTH - Royal Institute of Technology, 100 44 Stockholm, Sweden and LSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing 100190, China
| | - Nevzat Onur Domaniç
- Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA
| | - Pradeep Ravikumar
- Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA
| | - Chandrajit Bajaj
- Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA
| |
Collapse
|
10
|
Vulović M, Ravelli RBG, van Vliet LJ, Koster AJ, Lazić I, Lücken U, Rullgård H, Öktem O, Rieger B. Image formation modeling in cryo-electron microscopy. J Struct Biol 2013; 183:19-32. [PMID: 23711417 DOI: 10.1016/j.jsb.2013.05.008] [Citation(s) in RCA: 59] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2013] [Revised: 05/07/2013] [Accepted: 05/14/2013] [Indexed: 11/16/2022]
Abstract
Accurate modeling of image formation in cryo-electron microscopy is an important requirement for quantitative image interpretation and optimization of the data acquisition strategy. Here we present a forward model that accounts for the specimen's scattering properties, microscope optics, and detector response. The specimen interaction potential is calculated with the isolated atom superposition approximation (IASA) and extended with the influences of solvent's dielectric and ionic properties as well as the molecular electrostatic distribution. We account for an effective charge redistribution via the Poisson-Boltzmann approach and find that the IASA-based potential forms the dominant part of the interaction potential, as the contribution of the redistribution is less than 10%. The electron wave is propagated through the specimen by a multislice approach and the influence of the optics is included via the contrast transfer function. We incorporate the detective quantum efficiency of the camera due to the difference between signal and noise transfer characteristics, instead of using only the modulation transfer function. The full model was validated against experimental images of 20S proteasome, hemoglobin, and GroEL. The simulations adequately predict the effects of phase contrast, changes due to the integrated electron flux, thickness, inelastic scattering, detective quantum efficiency and acceleration voltage. We suggest that beam-induced specimen movements are relevant in the experiments whereas the influence of the solvent amorphousness can be neglected. All simulation parameters are based on physical principles and, when necessary, experimentally determined.
Collapse
Affiliation(s)
- Miloš Vulović
- Quantitative Imaging Group, Faculty of Applied Sciences, Delft University of Technology, Lorentzweg 1, 2628 CJ Delft, The Netherlands
| | | | | | | | | | | | | | | | | |
Collapse
|
11
|
Abstract
We introduce a tomographic reconstruction method implemented using a shape-based regularization technique. Spatial models of known features in the structure being reconstructed are integrated into the reconstruction process as regularizers. Our regularization scheme is driven locally through shape information obtained from segmentation and compared with a known spatial model. We demonstrated our method on tomography data from digital phantoms, simulated data, and experimental electron tomography (ET) data of virus complexes. Our reconstruction showed reduced blurring and an improvement in the resolution of the reconstructed volume was also measured. This method also produced improved demarcation of spike boundaries in viral membranes when compared with popular techniques like weighted back projection and the algebraic reconstruction technique. Improved ET reconstructions will provide better structure elucidation and improved feature visualization, which can aid in solving key biological issues. Our method can also be generalized to other tomographic modalities.
Collapse
Affiliation(s)
- Ajay Gopinath
- Department of Electrical and Computer Engineering, University of Texas at Austin, Austin, TX 78712 USA.
| | | | | | | | | | | |
Collapse
|