1
|
Yang HY, Leahy BD, Jang WD, Wei D, Kalma Y, Rahav R, Carmon A, Kopel R, Azem F, Venturas M, Kelleher CP, Cam L, Pfister H, Needleman DJ, Ben-Yosef D. BlastAssist: a deep learning pipeline to measure interpretable features of human embryos. Hum Reprod 2024; 39:698-708. [PMID: 38396213 DOI: 10.1093/humrep/deae024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 01/05/2024] [Indexed: 02/25/2024] Open
Abstract
STUDY QUESTION Can the BlastAssist deep learning pipeline perform comparably to or outperform human experts and embryologists at measuring interpretable, clinically relevant features of human embryos in IVF? SUMMARY ANSWER The BlastAssist pipeline can measure a comprehensive set of interpretable features of human embryos and either outperform or perform comparably to embryologists and human experts in measuring these features. WHAT IS KNOWN ALREADY Some studies have applied deep learning and developed 'black-box' algorithms to predict embryo viability directly from microscope images and videos but these lack interpretability and generalizability. Other studies have developed deep learning networks to measure individual features of embryos but fail to conduct careful comparisons to embryologists' performance, which are fundamental to demonstrate the network's effectiveness. STUDY DESIGN, SIZE, DURATION We applied the BlastAssist pipeline to 67 043 973 images (32 939 embryos) recorded in the IVF lab from 2012 to 2017 in Tel Aviv Sourasky Medical Center. We first compared the pipeline measurements of individual images/embryos to manual measurements by human experts for sets of features, including: (i) fertilization status (n = 207 embryos), (ii) cell symmetry (n = 109 embryos), (iii) degree of fragmentation (n = 6664 images), and (iv) developmental timing (n = 21 036 images). We then conducted detailed comparisons between pipeline outputs and annotations made by embryologists during routine treatments for features, including: (i) fertilization status (n = 18 922 embryos), (ii) pronuclei (PN) fade time (n = 13 781 embryos), (iii) degree of fragmentation on Day 2 (n = 11 582 embryos), and (iv) time of blastulation (n = 3266 embryos). In addition, we compared the pipeline outputs to the implantation results of 723 single embryo transfer (SET) cycles, and to the live birth results of 3421 embryos transferred in 1801 cycles. PARTICIPANTS/MATERIALS, SETTING, METHODS In addition to EmbryoScope™ image data, manual embryo grading and annotations, and electronic health record (EHR) data on treatment outcomes were also included. We integrated the deep learning networks we developed for individual features to construct the BlastAssist pipeline. Pearson's χ2 test was used to evaluate the statistical independence of individual features and implantation success. Bayesian statistics was used to evaluate the association of the probability of an embryo resulting in live birth to BlastAssist inputs. MAIN RESULTS AND THE ROLE OF CHANCE The BlastAssist pipeline integrates five deep learning networks and measures comprehensive, interpretable, and quantitative features in clinical IVF. The pipeline performs similarly or better than manual measurements. For fertilization status, the network performs with very good parameters of specificity and sensitivity (area under the receiver operating characteristics (AUROC) 0.84-0.94). For symmetry score, the pipeline performs comparably to the human expert at both 2-cell (r = 0.71 ± 0.06) and 4-cell stages (r = 0.77 ± 0.07). For degree of fragmentation, the pipeline (acc = 69.4%) slightly under-performs compared to human experts (acc = 73.8%). For developmental timing, the pipeline (acc = 90.0%) performs similarly to human experts (acc = 91.4%). There is also strong agreement between pipeline outputs and annotations made by embryologists during routine treatments. For fertilization status, the pipeline and embryologists strongly agree (acc = 79.6%), and there is strong correlation between the two measurements (r = 0.683). For degree of fragmentation, the pipeline and embryologists mostly agree (acc = 55.4%), and there is also strong correlation between the two measurements (r = 0.648). For both PN fade time (r = 0.787) and time of blastulation (r = 0.887), there's strong correlation between the pipeline and embryologists. For SET cycles, 2-cell time (P < 0.01) and 2-cell symmetry (P < 0.03) are significantly correlated with implantation success rate, while other features showed correlations with implantation success without statistical significance. In addition, 2-cell time (P < 5 × 10-11), PN fade time (P < 5 × 10-10), degree of fragmentation on Day 3 (P < 5 × 10-4), and 2-cell symmetry (P < 5 × 10-3) showed statistically significant correlation with the probability of the transferred embryo resulting in live birth. LIMITATIONS, REASONS FOR CAUTION We have not tested the BlastAssist pipeline on data from other clinics or other time-lapse microscopy (TLM) systems. The association study we conducted with live birth results do not take into account confounding variables, which will be necessary to construct an embryo selection algorithm. Randomized controlled trials (RCT) will be necessary to determine whether the pipeline can improve success rates in clinical IVF. WIDER IMPLICATIONS OF THE FINDINGS BlastAssist provides a comprehensive and holistic means of evaluating human embryos. Instead of using a black-box algorithm, BlastAssist outputs meaningful measurements of embryos that can be interpreted and corroborated by embryologists, which is crucial in clinical decision making. Furthermore, the unprecedentedly large dataset generated by BlastAssist measurements can be used as a powerful resource for further research in human embryology and IVF. STUDY FUNDING/COMPETING INTEREST(S) This work was supported by Harvard Quantitative Biology Initiative, the NSF-Simons Center for Mathematical and Statistical Analysis of Biology at Harvard (award number 1764269), the National Institute of Heath (award number R01HD104969), the Perelson Fund, and the Sagol fund for embryos and stem cells as part of the Sagol Network. The authors declare no competing interests. TRIAL REGISTRATION NUMBER Not applicable.
Collapse
Affiliation(s)
- Helen Y Yang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Department of Biophysics, Harvard Graduate School of Arts and Sciences, Cambridge, MA, USA
| | - Brian D Leahy
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Department of Applied Physics, Harvard School of Engineering and Applied Sciences, Cambridge, MA, USA
| | - Won-Dong Jang
- Department of Computer Science, Harvard School of Engineering and Applied Sciences, Cambridge, MA, USA
| | - Donglai Wei
- Department of Computer Science, Harvard School of Engineering and Applied Sciences, Cambridge, MA, USA
| | - Yael Kalma
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Roni Rahav
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Ariella Carmon
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Rotem Kopel
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Foad Azem
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
| | - Marta Venturas
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Colm P Kelleher
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Liz Cam
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Hanspeter Pfister
- Department of Computer Science, Harvard School of Engineering and Applied Sciences, Cambridge, MA, USA
| | - Daniel J Needleman
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Department of Applied Physics, Harvard School of Engineering and Applied Sciences, Cambridge, MA, USA
| | - Dalit Ben-Yosef
- Department of Reproduction and IVF, Lis Maternity Hospital Tel-Aviv Sourasky Medical Center, Tel Aviv, Israel
- Department of Cell and Developmental Biology, Sackler Faculty of Medicine, Sagol School of Neuroscience, Tel-Aviv University, Tel-Aviv, Israel
| |
Collapse
|
2
|
Franco-Barranco D, Lin Z, Jang WD, Wang X, Shen Q, Yin W, Fan Y, Li M, Chen C, Xiong Z, Xin R, Liu H, Chen H, Li Z, Zhao J, Chen X, Pape C, Conrad R, Nightingale L, de Folter J, Jones ML, Liu Y, Ziaei D, Huschauer S, Arganda-Carreras I, Pfister H, Wei D. Current Progress and Challenges in Large-Scale 3D Mitochondria Instance Segmentation. IEEE Trans Med Imaging 2023; 42:3956-3971. [PMID: 37768797 PMCID: PMC10753957 DOI: 10.1109/tmi.2023.3320497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/30/2023]
Abstract
In this paper, we present the results of the MitoEM challenge on mitochondria 3D instance segmentation from electron microscopy images, organized in conjunction with the IEEE-ISBI 2021 conference. Our benchmark dataset consists of two large-scale 3D volumes, one from human and one from rat cortex tissue, which are 1,986 times larger than previously used datasets. At the time of paper submission, 257 participants had registered for the challenge, 14 teams had submitted their results, and six teams participated in the challenge workshop. Here, we present eight top-performing approaches from the challenge participants, along with our own baseline strategies. Posterior to the challenge, annotation errors in the ground truth were corrected without altering the final ranking. Additionally, we present a retrospective evaluation of the scoring system which revealed that: 1) challenge metric was permissive with the false positive predictions; and 2) size-based grouping of instances did not correctly categorize mitochondria of interest. Thus, we propose a new scoring system that better reflects the correctness of the segmentation results. Although several of the top methods are compared favorably to our own baselines, substantial errors remain unsolved for mitochondria with challenging morphologies. Thus, the challenge remains open for submission and automatic evaluation, with all volumes available for download.
Collapse
Affiliation(s)
- Daniel Franco-Barranco
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, and also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain
| | - Zudi Lin
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Won-Dong Jang
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Xueying Wang
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Qijia Shen
- Wellcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of Oxford, OX3 9DU Oxford, U.K
| | - Wenjie Yin
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Yutian Fan
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138 USA
| | - Mingxing Li
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Chang Chen
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Zhiwei Xiong
- Department of Electronic Engineering and Information Science (EEIS), University of Science and Technology of China, Anhui 230026, China
| | - Rui Xin
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Hao Liu
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Huai Chen
- Institute of Image Processing and Pattern Recognition, Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Zhili Li
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Jie Zhao
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Xuejin Chen
- National Engineering Laboratory for Brain-Inspired Intelligence Technology and Application, University of Science and Technology of China, Anhui 230026, China
| | - Constantin Pape
- European Molecular Biology Laboratory (EMBL), 69117 Heidelberg, Germany. He is now with the Institute for Computer Science, Georg-August-Universität Göttingen, Göttingen, Germany
| | - Ryan Conrad
- Center for Molecular Microscopy, Center for Cancer Research, National Cancer Institute, National Institutes of Health, Bethesda, MD 20892 USA, and also with the Cancer Research Technology Program, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | | | | | - Yanling Liu
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | - Dorsa Ziaei
- Advanced Biomedical Computational Science Group, Frederick National Laboratory for Cancer Research, Frederick, MD 21701 USA
| | | | - Ignacio Arganda-Carreras
- Department of Computer Science and Artificial Intelligence, University of the Basque Country (UPV/EHU), 20018 San Sebastián, Spain, also with the Donostia International Physics Center (DIPC), 20018 San Sebastián, Spain, also with the IKERBASQUE, Basque Foundation for Science, 48009 Bilbao, Spain, and also with the Biofisika Institute, 48940 Leioa, Spain
| | - Hanspeter Pfister
- Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), Harvard University, Allston, MA 02134 USA
| | - Donglai Wei
- Computer Science Department, Boston College, Chestnut Hill, MA 02467 USA
| |
Collapse
|
3
|
Magid SA, Zhang Y, Wei D, Jang WD, Lin Z, Fu Y, Pfister H. Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution. Proc IEEE Int Conf Comput Vis 2021; 2021:4268-4277. [PMID: 35368831 DOI: 10.1109/iccv48922.2021.00425] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Deep convolutional neural networks (CNNs) have pushed forward the frontier of super-resolution (SR) research. However, current CNN models exhibit a major flaw: they are biased towards learning low-frequency signals. This bias becomes more problematic for the image SR task which targets reconstructing all fine details and image textures. To tackle this challenge, we propose to improve the learning of high-frequency features both locally and globally and introduce two novel architectural units to existing SR models. Specifically, we propose a dynamic highpass filtering (HPF) module that locally applies adaptive filter weights for each spatial location and channel group to preserve high-frequency signals. We also propose a matrix multi-spectral channel attention (MMCA) module that predicts the attention map of features decomposed in the frequency domain. This module operates in a global context to adaptively recalibrate feature responses at different frequencies. Extensive qualitative and quantitative results demonstrate that our proposed modules achieve better accuracy and visual improvements against state-of-the-art methods on several benchmark datasets.
Collapse
|
4
|
Lukyanenko S, Jang WD, Wei D, Struyven R, Kim Y, Leahy B, Yang H, Rush A, Ben-Yosef D, Needleman D, Pfister H. Developmental Stage Classification of Embryos Using Two-Stream Neural Network with Linear-Chain Conditional Random Field. Med Image Comput Comput Assist Interv 2021; 12908:363-372. [PMID: 34671767 PMCID: PMC8526069 DOI: 10.1007/978-3-030-87237-3_35] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
Abstract
The developmental process of embryos follows a monotonic order. An embryo can progressively cleave from one cell to multiple cells and finally transform to morula and blastocyst. For time-lapse videos of embryos, most existing developmental stage classification methods conduct per-frame predictions using an image frame at each time step. However, classification using only images suffers from overlapping between cells and imbalance between stages. Temporal information can be valuable in addressing this problem by capturing movements between neighboring frames. In this work, we propose a two-stream model for developmental stage classification. Unlike previous methods, our two-stream model accepts both temporal and image information. We develop a linear-chain conditional random field (CRF) on top of neural network features extracted from the temporal and image streams to make use of both modalities. The linear-chain CRF formulation enables tractable training of global sequential models over multiple frames while also making it possible to inject monotonic development order constraints into the learning process explicitly. We demonstrate our algorithm on two time-lapse embryo video datasets: i) mouse and ii) human embryo datasets. Our method achieves 98.1% and 80.6% for mouse and human embryo stage classification, respectively. Our approach will enable more pro-found clinical and biological studies and suggests a new direction for developmental stage classification by utilizing temporal information.
Collapse
Affiliation(s)
| | - Won-Dong Jang
- School of Engineering and Applied Sciences, Harvard University, USA
| | - Donglai Wei
- School of Engineering and Applied Sciences, Harvard University, USA
| | - Robbert Struyven
- School of Engineering and Applied Sciences, Harvard University, USA
- University College London, UK
| | | | - Brian Leahy
- School of Engineering and Applied Sciences, Harvard University, USA
- Department of Molecular and Cellular Biology, Harvard University, USA
| | - Helen Yang
- Department of Molecular and Cellular Biology, Harvard University, USA
- Graduate Program in Biophysics, Harvard University, USA
| | | | - Dalit Ben-Yosef
- Lis Maternity Hospital, Tel-Aviv Sourasky Medical Center, Israel
- Cell and Developmental Biology, Tel-Aviv University, Israel
| | - Daniel Needleman
- School of Engineering and Applied Sciences, Harvard University, USA
- Department of Molecular and Cellular Biology, Harvard University, USA
- Center for Computational Biology, Flatiron Institute, USA
| | | |
Collapse
|
5
|
Wei D, Lin Z, Franco-Barranco D, Wendt N, Liu X, Yin W, Huang X, Gupta A, Jang WD, Wang X, Arganda-Carreras I, Lichtman JW, Pfister H. MitoEM Dataset: Large-scale 3D Mitochondria Instance Segmentation from EM Images. Med Image Comput Comput Assist Interv 2020; 12265:66-76. [PMID: 33283212 PMCID: PMC7713709 DOI: 10.1007/978-3-030-59722-1_7] [Citation(s) in RCA: 28] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
Abstract
Electron microscopy (EM) allows the identification of intracellular organelles such as mitochondria, providing insights for clinical and scientific studies. However, public mitochondria segmentation datasets only contain hundreds of instances with simple shapes. It is unclear if existing methods achieving human-level accuracy on these small datasets are robust in practice. To this end, we introduce the MitoEM dataset, a 3D mitochondria instance segmentation dataset with two (30μm)3 volumes from human and rat cortices respectively, 3, 600× larger than previous benchmarks. With around 40K instances, we find a great diversity of mitochondria in terms of shape and density. For evaluation, we tailor the implementation of the average precision (AP) metric for 3D data with a 45× speedup. On MitoEM, we find existing instance segmentation methods often fail to correctly segment mitochondria with complex shapes or close contacts with other instances. Thus, our MitoEM dataset poses new challenges to the field. We release our code and data: https://donglaiw.github.io/page/mitoEM/index.html.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | | | - Ignacio Arganda-Carreras
- Donostia International Physics Center
- University of the Basque Country
- Ikerbasque, Basque Foundation for Science
| | | | | |
Collapse
|
6
|
Lin Z, Wei D, Jang WD, Zhou S, Chen X, Wang X, Schalek R, Berger D, Matejek B, Kamentsky L, Peleg A, Haehn D, Jones T, Parag T, Lichtman J, Pfister H. Two Stream Active Query Suggestion for Active Learning in Connectomics. Comput Vis ECCV 2020; 12363:103-120. [PMID: 33345257 PMCID: PMC7746018 DOI: 10.1007/978-3-030-58523-5_7] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/03/2022]
Abstract
For large-scale vision tasks in biomedical images, the labeled data is often limited to train effective deep models. Active learning is a common solution, where a query suggestion method selects representative unlabeled samples for annotation, and the new labels are used to improve the base model. However, most query suggestion models optimize their learnable parameters only on the limited labeled data and consequently become less effective for the more challenging unlabeled data. To tackle this, we propose a two-stream active query suggestion approach. In addition to the supervised feature extractor, we introduce an unsupervised one optimized on all raw images to capture diverse image features, which can later be improved by fine-tuning on new labels. As a use case, we build an end-to-end active learning framework with our query suggestion method for 3D synapse detection and mitochondria segmentation in connectomics. With the framework, we curate, to our best knowledge, the largest connectomics dataset with dense synapses and mitochondria annotation. On this new dataset, our method outperforms previous state-of-the-art methods by 3.1% for synapse and 3.8% for mitochondria in terms of region-of-interest proposal accuracy. We also apply our method to image classification, where it outperforms previous approaches on CIFAR-10 under the same limited annotation budget. The project page is https://zudi-lin.github.io/projects/#two_stream_active.
Collapse
|
7
|
Krueger R, Beyer J, Jang WD, Kim NW, Sokolov A, Sorger PK, Pfister H. Facetto: Combining Unsupervised and Supervised Learning for Hierarchical Phenotype Analysis in Multi-Channel Image Data. IEEE Trans Vis Comput Graph 2020; 26:227-237. [PMID: 31514138 PMCID: PMC7045445 DOI: 10.1109/tvcg.2019.2934547] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/20/2023]
Abstract
Facetto is a scalable visual analytics application that is used to discover single-cell phenotypes in high-dimensional multi-channel microscopy images of human tumors and tissues. Such images represent the cutting edge of digital histology and promise to revolutionize how diseases such as cancer are studied, diagnosed, and treated. Highly multiplexed tissue images are complex, comprising 109 or more pixels, 60-plus channels, and millions of individual cells. This makes manual analysis challenging and error-prone. Existing automated approaches are also inadequate, in large part, because they are unable to effectively exploit the deep knowledge of human tissue biology available to anatomic pathologists. To overcome these challenges, Facetto enables a semi-automated analysis of cell types and states. It integrates unsupervised and supervised learning into the image and feature exploration process and offers tools for analytical provenance. Experts can cluster the data to discover new types of cancer and immune cells and use clustering results to train a convolutional neural network that classifies new cells accordingly. Likewise, the output of classifiers can be clustered to discover aggregate patterns and phenotype subsets. We also introduce a new hierarchical approach to keep track of analysis steps and data subsets created by users; this assists in the identification of cell types. Users can build phenotype trees and interact with the resulting hierarchical structures of both high-dimensional feature and image spaces. We report on use-cases in which domain scientists explore various large-scale fluorescence imaging datasets. We demonstrate how Facetto assists users in steering the clustering and classification process, inspecting analysis results, and gaining new scientific insights into cancer biology.
Collapse
|