1
|
Franco-Barranco D, Lin Z, Jang WD, Wang X, Shen Q, Yin W, Fan Y, Li M, Chen C, Xiong Z, Xin R, Liu H, Chen H, Li Z, Zhao J, Chen X, Pape C, Conrad R, Nightingale L, de Folter J, Jones ML, Liu Y, Ziaei D, Huschauer S, Arganda-Carreras I, Pfister H, Wei D. Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3956-3971. [PMID: 37768797 PMCID: PMC10753957 DOI: 10.1109/tmi.2023.3320497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/30/2023]
Abstract
In this paper, we present the results of the MitoEM challenge on mitochondria 3D instance segmentation from electron microscopy images, organized in conjunction with the IEEE-ISBI 2021 conference. Our benchmark dataset consists of two large-scale 3D volumes, one from human and one from rat cortex tissue, which are 1,986 times larger than previously used datasets. At the time of paper submission, 257 participants had registered for the challenge, 14 teams had submitted their results, and six teams participated in the challenge workshop. Here, we present eight top-performing approaches from the challenge participants, along with our own baseline strategies. Posterior to the challenge, annotation errors in the ground truth were corrected without altering the final ranking. Additionally, we present a retrospective evaluation of the scoring system which revealed that: 1) challenge metric was permissive with the false positive predictions; and 2) size-based grouping of instances did not correctly categorize mitochondria of interest. Thus, we propose a new scoring system that better reflects the correctness of the segmentation results. Although several of the top methods are compared favorably to our own baselines, substantial errors remain unsolved for mitochondria with challenging morphologies. Thus, the challenge remains open for submission and automatic evaluation, with all volumes available for download.
Collapse
Affiliation(s)
- Daniel Franco-Barranco
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, and also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain
| | - Zudi Lin
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Won-Dong Jang
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Xueying Wang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Qijia Shen
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU Oxford, U.K
| | - Wenjie Yin
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Yutian Fan
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Mingxing Li
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Chang Chen
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Zhiwei Xiong
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Rui Xin
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Hao Liu
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Huai Chen
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Zhili Li
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Jie Zhao
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Xuejin Chen
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Constantin Pape
- European Molecular Biology Laboratory (EMBL), 69117 Heidelberg, Germany. He is now with the Institute for Computer Science, Georg-August-Universität Göttingen, Göttingen, Germany
| | - Ryan Conrad
- Center for Molecular Microscopy, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda, MD 20892 USA, and also with the Cancer Research Technology Program, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | | | | | - Yanling Liu
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | - Dorsa Ziaei
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | - Ignacio Arganda-Carreras
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain, also with the IKERBASQUE, Basque Foundation for Science, 48009 Bilbao, Spain, and also with the Biofisika Institute, 48940 Leioa, Spain
| | - Hanspeter Pfister
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Donglai Wei
- Computer Science Department, Boston College, Chestnut Hill, MA 02467 USA
| |
Collapse
|
2
|
Powell KA, Bohrer LR, Stone NE, Hittle B, Anfinson KR, Luangphakdy V, Muschler G, Mullins RF, Stone EM, Tucker BA. Automated human induced pluripotent stem cell colony segmentation for use in cell culture automation applications. SLAS Technol 2023; 28:416-422. [PMID: 37454765 PMCID: PMC10775697 DOI: 10.1016/j.slast.2023.07.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Revised: 06/28/2023] [Accepted: 07/13/2023] [Indexed: 07/18/2023]
Abstract
Human induced pluripotent stem cells (hiPSCs) have demonstrated great promise for a variety of applications that include cell therapy and regenerative medicine. Production of clinical grade hiPSCs requires reproducible manufacturing methods with stringent quality-controls such as those provided by image-controlled robotic processing systems. In this paper we present an automated image analysis method for identifying and picking hiPSC colonies for clonal expansion using the CellXTM robotic cell processing system. This method couples a light weight deep learning segmentation approach based on the U-Net architecture to automatically segment the hiPSC colonies in full field of view (FOV) high resolution phase contrast images with a standardized approach for suggesting pick locations. The utility of this method is demonstrated using images and data obtained from the CellXTM system where clinical grade hiPSCs were reprogrammed, clonally expanded, and differentiated into retinal organoids for use in treatment of patients with inherited retinal degenerative blindness.
Collapse
Affiliation(s)
- Kimerly A Powell
- Department of Biomedical Informatics, The Ohio State University, Columbus, OH 43210, USA.
| | - Laura R Bohrer
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| | - Nicholas E Stone
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| | - Bradley Hittle
- Department of Biomedical Informatics, The Ohio State University, Columbus, OH 43210, USA
| | - Kristin R Anfinson
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| | - Viviane Luangphakdy
- Department of Biomedical Engineering, Lerner Research Institute, Cleveland Clinic, Cleveland, OH, USA; Cell X Technologies Inc., Cleveland, OH, USA
| | - George Muschler
- Department of Biomedical Engineering, Lerner Research Institute, Cleveland Clinic, Cleveland, OH, USA; Department of Orthopedic Surgery, Cleveland Clinic, Cleveland, OH, USA
| | - Robert F Mullins
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| | - Edwin M Stone
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| | - Budd A Tucker
- Institute for Vision Research, Carver College of Medicine, University of Iowa, Iowa City, IA, USA; Department of Ophthalmology and Visual Sciences, Carver College of Medicine, University of Iowa, Iowa City, IA, USA
| |
Collapse
|
3
|
Andrés-San Román JA, Gordillo-Vázquez C, Franco-Barranco D, Morato L, Fernández-Espartero CH, Baonza G, Tagua A, Vicente-Munuera P, Palacios AM, Gavilán MP, Martín-Belmonte F, Annese V, Gómez-Gálvez P, Arganda-Carreras I, Escudero LM. CartoCell, a high-content pipeline for 3D image analysis, unveils cell morphology patterns in epithelia. CELL REPORTS METHODS 2023; 3:100597. [PMID: 37751739 PMCID: PMC10626192 DOI: 10.1016/j.crmeth.2023.100597] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 07/19/2023] [Accepted: 08/31/2023] [Indexed: 09/28/2023]
Abstract
Decades of research have not yet fully explained the mechanisms of epithelial self-organization and 3D packing. Single-cell analysis of large 3D epithelial libraries is crucial for understanding the assembly and function of whole tissues. Combining 3D epithelial imaging with advanced deep-learning segmentation methods is essential for enabling this high-content analysis. We introduce CartoCell, a deep-learning-based pipeline that uses small datasets to generate accurate labels for hundreds of whole 3D epithelial cysts. Our method detects the realistic morphology of epithelial cells and their contacts in the 3D structure of the tissue. CartoCell enables the quantification of geometric and packing features at the cellular level. Our single-cell cartography approach then maps the distribution of these features on 2D plots and 3D surface maps, revealing cell morphology patterns in epithelial cysts. Additionally, we show that CartoCell can be adapted to other types of epithelial tissues.
Collapse
Affiliation(s)
- Jesús A Andrés-San Román
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - Carmen Gordillo-Vázquez
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - Daniel Franco-Barranco
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastian, Spain; Donostia International Physics Center (DIPC), 20018 San Sebastian, Spain
| | - Laura Morato
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - Cecilia H Fernández-Espartero
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - Gabriel Baonza
- Program of Tissue and Organ Homeostasis, Centro de Biología Molecular Severo Ochoa, CSIC-UAM and Ramón & Cajal Health Research Institute (IRYCIS), Hospital Universitario Ramón y Cajal, 28034 Madrid, Spain
| | - Antonio Tagua
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | | | - Ana M Palacios
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - María P Gavilán
- Centro Andaluz de Biología Molecular y Medicina Regenerativa (CABIMER), JA/CSIC/Universidad de Sevilla/Universidad Pablo de Olavide and Departamento de Citología e Histología Normal y Patológica, Facultad de Medicina, Universidad de Sevilla, 41009 Seville, Spain
| | - Fernando Martín-Belmonte
- Program of Tissue and Organ Homeostasis, Centro de Biología Molecular Severo Ochoa, CSIC-UAM and Ramón & Cajal Health Research Institute (IRYCIS), Hospital Universitario Ramón y Cajal, 28034 Madrid, Spain
| | - Valentina Annese
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain
| | - Pedro Gómez-Gálvez
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain; MRC Laboratory of Molecular Biology, Cambridge Biomedical Campus, Francis Crick Avenue, Trumpington, Cambridge CB2 0QH, UK; Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge CB2 3EG, UK.
| | - Ignacio Arganda-Carreras
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastian, Spain; Donostia International Physics Center (DIPC), 20018 San Sebastian, Spain; Ikerbasque, Basque Foundation for Science, 48009 Bilbao, Spain; Biofisika Institute, 48940 Leioa, Spain.
| | - Luis M Escudero
- Instituto de Biomedicina de Sevilla (IBiS), Hospital Universitario Virgen del Rocío/CSIC/Universidad de Sevilla and Departamento de Biología Celular, Facultad de Biología, Universidad de Sevilla, 41013 Seville, Spain; Biomedical Network Research Centre on Neurodegenerative Diseases (CIBERNED), 28029 Madrid, Spain.
| |
Collapse
|
4
|
Aswath A, Alsahaf A, Giepmans BNG, Azzopardi G. Segmentation in large-scale cellular electron microscopy with deep learning: A literature survey. Med Image Anal 2023; 89:102920. [PMID: 37572414 DOI: 10.1016/j.media.2023.102920] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2022] [Revised: 07/05/2023] [Accepted: 07/31/2023] [Indexed: 08/14/2023]
Abstract
Electron microscopy (EM) enables high-resolution imaging of tissues and cells based on 2D and 3D imaging techniques. Due to the laborious and time-consuming nature of manual segmentation of large-scale EM datasets, automated segmentation approaches are crucial. This review focuses on the progress of deep learning-based segmentation techniques in large-scale cellular EM throughout the last six years, during which significant progress has been made in both semantic and instance segmentation. A detailed account is given for the key datasets that contributed to the proliferation of deep learning in 2D and 3D EM segmentation. The review covers supervised, unsupervised, and self-supervised learning methods and examines how these algorithms were adapted to the task of segmenting cellular and sub-cellular structures in EM images. The special challenges posed by such images, like heterogeneity and spatial complexity, and the network architectures that overcame some of them are described. Moreover, an overview of the evaluation measures used to benchmark EM datasets in various segmentation tasks is provided. Finally, an outlook of current trends and future prospects of EM segmentation is given, especially with large-scale models and unlabeled images to learn generic features across EM datasets.
Collapse
Affiliation(s)
- Anusha Aswath
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University Groningen, Groningen, The Netherlands; Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands.
| | - Ahmad Alsahaf
- Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands
| | - Ben N G Giepmans
- Department of Biomedical Sciences of Cells and Systems, University Groningen, University Medical Center Groningen, Groningen, The Netherlands
| | - George Azzopardi
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, University Groningen, Groningen, The Netherlands
| |
Collapse
|
5
|
Conrad R, Narayan K. Instance segmentation of mitochondria in electron microscopy images with a generalist deep learning model trained on a diverse dataset. Cell Syst 2023; 14:58-71.e5. [PMID: 36657391 PMCID: PMC9883049 DOI: 10.1016/j.cels.2022.12.006] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/06/2022] [Revised: 10/10/2022] [Accepted: 12/14/2022] [Indexed: 01/19/2023]
Abstract
Mitochondria are extremely pleomorphic organelles. Automatically annotating each one accurately and precisely in any 2D or volume electron microscopy (EM) image is an unsolved computational challenge. Current deep learning-based approaches train models on images that provide limited cellular contexts, precluding generality. To address this, we amassed a highly heterogeneous ∼1.5 × 106 image 2D unlabeled cellular EM dataset and segmented ∼135,000 mitochondrial instances therein. MitoNet, a model trained on these resources, performs well on challenging benchmarks and on previously unseen volume EM datasets containing tens of thousands of mitochondria. We release a Python package and napari plugin, empanada, to rapidly run inference, visualize, and proofread instance segmentations. A record of this paper's transparent peer review process is included in the supplemental information.
Collapse
Affiliation(s)
- Ryan Conrad
- Center for Molecular Microscopy, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda 20892, Maryland, USA.,Cancer Research Technology Program, Frederick National Laboratory for Cancer Research, Frederick 21702, Maryland, USA
| | - Kedar Narayan
- Center for Molecular Microscopy, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda 20892, Maryland, USA.,Cancer Research Technology Program, Frederick National Laboratory for Cancer Research, Frederick 21702, Maryland, USA
| |
Collapse
|
6
|
Franco-Barranco D, Pastor-Tronch J, González-Marfil A, Muñoz-Barrutia A, Arganda-Carreras I. Deep learning based domain adaptation for mitochondria segmentation on EM volumes. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 222:106949. [PMID: 35753105 DOI: 10.1016/j.cmpb.2022.106949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Revised: 06/05/2022] [Accepted: 06/09/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Accurate segmentation of electron microscopy (EM) volumes of the brain is essential to characterize neuronal structures at a cell or organelle level. While supervised deep learning methods have led to major breakthroughs in that direction during the past years, they usually require large amounts of annotated data to be trained, and perform poorly on other data acquired under similar experimental and imaging conditions. This is a problem known as domain adaptation, since models that learned from a sample distribution (or source domain) struggle to maintain their performance on samples extracted from a different distribution or target domain. In this work, we address the complex case of deep learning based domain adaptation for mitochondria segmentation across EM datasets from different tissues and species. METHODS We present three unsupervised domain adaptation strategies to improve mitochondria segmentation in the target domain based on (1) state-of-the-art style transfer between images of both domains; (2) self-supervised learning to pre-train a model using unlabeled source and target images, and then fine-tune it only with the source labels; and (3) multi-task neural network architectures trained end-to-end with both labeled and unlabeled images. Additionally, to ensure good generalization in our models, we propose a new training stopping criterion based on morphological priors obtained exclusively in the source domain. The code and its documentation are publicly available at https://github.com/danifranco/EM_domain_adaptation. RESULTS We carried out all possible cross-dataset experiments using three publicly available EM datasets. We evaluated our proposed strategies and those of others based on the mitochondria semantic labels predicted on the target datasets. CONCLUSIONS The methods introduced here outperform the baseline methods and compare favorably to the state of the art. In the absence of validation labels, monitoring our proposed morphology-based metric is an intuitive and effective way to stop the training process and select in average optimal models.
Collapse
Affiliation(s)
- Daniel Franco-Barranco
- Dept. of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain; Donostia International Physics Center (DIPC), Spain.
| | - Julio Pastor-Tronch
- Dept. of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain
| | - Aitor González-Marfil
- Dept. of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain
| | - Arrate Muñoz-Barrutia
- Universidad Carlos III de Madrid, Spain; Instituto de Investigación Sanitaria Gregorio Marañón, Spain
| | - Ignacio Arganda-Carreras
- Dept. of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), Spain; Donostia International Physics Center (DIPC), Spain; Ikerbasque, Basque Foundation for Science, Spain
| |
Collapse
|