Multi-Disease Segmentation of Gliomas and White Matter Hyperintensities in the BraTS Data Using a 3D Convolutional Neural Network

The University of California San Francisco Adult Longitudinal Post-Treatment Diffuse Glioma MRI Dataset

Supplemental material is available for this article.

Non-invasive prediction of overall survival time for glioblastoma multiforme patients based on multimodal MRI radiomics

Glioblastoma multiforme (GBM) is the most common and deadly primary malignant brain tumor. As GBM tumor is aggressive and shows high biological heterogeneity, the overall survival (OS) time is extremely low even with the most aggressive treatment. If the OS time can be predicted before surgery, developing personalized treatment plans for GBM patients will be beneficial. Magnetic resonance imaging (MRI) is a commonly used diagnostic tool for brain tumors with high-resolution and sound imaging effects. However, in clinical practice, doctors mainly rely on manually segmenting the tumor regions in MRI and predicting the OS time of GBM patients, which is time-consuming, subjective and repetitive, limiting the effectiveness of clinical diagnosis and treatment. Therefore, it is crucial to segment the brain tumor regions in MRI, and an accurate pre-operative prediction of OS time for personalized treatment is highly desired. In this study, we present a multimodal MRI radiomics-based automatic framework for non-invasive prediction of the OS time for GBM patients. A modified 3D-UNet model is built to segment tumor subregions in MRI of GBM patients; then, the radiomic features in the tumor subregions are extracted and combined with the clinical features input into the Support Vector Regression (SVR) model to predict the OS time. In the experiments, the BraTS2020, BraTS2019 and BraTS2018 datasets are used to evaluate our framework. Our model achieves competitive OS time prediction accuracy compared to most typical approaches.

External Validation and Retraining of DeepBleed: The First Open-Source 3D Deep Learning Network for the Segmentation of Spontaneous Intracerebral and Intraventricular Hemorrhage

BACKGROUND

The objective of this study was to assess the performance of the first publicly available automated 3D segmentation for spontaneous intracerebral hemorrhage (ICH) based on a 3D neural network before and after retraining.

METHODS

We performed an independent validation of this model using a multicenter retrospective cohort. Performance metrics were evaluated using the dice score (DSC), sensitivity, and positive predictive values (PPV). We retrained the original model (OM) and assessed the performance via an external validation design. A multivariate linear regression model was used to identify independent variables associated with the model's performance. Agreements in volumetric measurements and segmentation were evaluated using Pearson's correlation coefficients (r) and intraclass correlation coefficients (ICC), respectively. With 1040 patients, the OM had a median DSC, sensitivity, and PPV of 0.84, 0.79, and 0.93, compared to thoseo f 0.83, 0.80, and 0.91 in the retrained model (RM). However, the median DSC for infratentorial ICH was relatively low and improved significantly after retraining, at p < 0.001. ICH volume and location were significantly associated with the DSC, at p < 0.05. The agreement between volumetric measurements (r > 0.90, p > 0.05) and segmentations (ICC ≥ 0.9, p < 0.001) was excellent.

CONCLUSION

The model demonstrated good generalization in an external validation cohort. Location-specific variances improved significantly after retraining. External validation and retraining are important steps to consider before applying deep learning models in new clinical settings.

GaNDLF: the generally nuanced deep learning framework for scalable end-to-end clinical workflows

Deep Learning (DL) has the potential to optimize machine learning in both the scientific and clinical communities. However, greater expertise is required to develop DL algorithms, and the variability of implementations hinders their reproducibility, translation, and deployment. Here we present the community-driven Generally Nuanced Deep Learning Framework (GaNDLF), with the goal of lowering these barriers. GaNDLF makes the mechanism of DL development, training, and inference more stable, reproducible, interpretable, and scalable, without requiring an extensive technical background. GaNDLF aims to provide an end-to-end solution for all DL-related tasks in computational precision medicine. We demonstrate the ability of GaNDLF to analyze both radiology and histology images, with built-in support for k -fold cross-validation, data augmentation, multiple modalities and output classes. Our quantitative performance evaluation on numerous use cases, anatomies, and computational tasks supports GaNDLF as a robust application framework for deployment in clinical workflows.

The increasing complexity of the implementation and operation of deep learning techniques hinders their reproducibility and deployment at scale, especially in healthcare. Pati and colleagues introduce a deep learning framework to analyse healthcare data without requiring extensive computational experience, facilitating the integration of artificial intelligence in clinical workflows.

3D Capsule Networks for Brain Image Segmentation

BACKGROUND AND PURPOSE

Current autosegmentation models such as UNets and nnUNets have limitations, including the inability to segment images that are not represented during training and lack of computational efficiency. 3D capsule networks have the potential to address these limitations.

MATERIALS AND METHODS

We used 3430 brain MRIs, acquired in a multi-institutional study, to train and validate our models. We compared our capsule network with standard alternatives, UNets and nnUNets, on the basis of segmentation efficacy (Dice scores), segmentation performance when the image is not well-represented in the training data, performance when the training data are limited, and computational efficiency including required memory and computational speed.

RESULTS

The capsule network segmented the third ventricle, thalamus, and hippocampus with Dice scores of 95%, 94%, and 92%, respectively, which were within 1% of the Dice scores of UNets and nnUNets. The capsule network significantly outperformed UNets in segmenting images that were not well-represented in the training data, with Dice scores 30% higher. The computational memory required for the capsule network is less than one-tenth of the memory required for UNets or nnUNets. The capsule network is also >25% faster to train compared with UNet and nnUNet.

CONCLUSIONS

We developed and validated a capsule network that is effective in segmenting brain images, can segment images that are not well-represented in the training data, and is computationally efficient compared with alternatives.

Comparing 3D, 2.5D, and 2D Approaches to Brain Image Auto-Segmentation

Deep-learning methods for auto-segmenting brain images either segment one slice of the image (2D), five consecutive slices of the image (2.5D), or an entire volume of the image (3D). Whether one approach is superior for auto-segmenting brain images is not known. We compared these three approaches (3D, 2.5D, and 2D) across three auto-segmentation models (capsule networks, UNets, and nnUNets) to segment brain structures. We used 3430 brain MRIs, acquired in a multi-institutional study, to train and test our models. We used the following performance metrics: segmentation accuracy, performance with limited training data, required computational memory, and computational speed during training and deployment. The 3D, 2.5D, and 2D approaches respectively gave the highest to lowest Dice scores across all models. 3D models maintained higher Dice scores when the training set size was decreased from 3199 MRIs down to 60 MRIs. 3D models converged 20% to 40% faster during training and were 30% to 50% faster during deployment. However, 3D models require 20 times more computational memory compared to 2.5D or 2D models. This study showed that 3D models are more accurate, maintain better performance with limited training data, and are faster to train and deploy. However, 3D models require more computational memory compared to 2.5D or 2D models.

Federated learning enables big data for rare cancer boundary detection

Although machine learning (ML) has shown promise across disciplines, out-of-sample generalizability is concerning. This is currently addressed by sharing multi-site data, but such centralization is challenging/infeasible to scale due to various limitations. Federated ML (FL) provides an alternative paradigm for accurate and generalizable ML, by only sharing numerical model updates. Here we present the largest FL study to-date, involving data from 71 sites across 6 continents, to generate an automatic tumor boundary detector for the rare disease of glioblastoma, reporting the largest such dataset in the literature (n = 6, 314). We demonstrate a 33% delineation improvement for the surgically targetable tumor, and 23% for the complete tumor extent, over a publicly trained model. We anticipate our study to: 1) enable more healthcare studies informed by large diverse data, ensuring meaningful results for rare diseases and underrepresented populations, 2) facilitate further analyses for glioblastoma by releasing our consensus model, and 3) demonstrate the FL effectiveness at such scale and task-complexity as a paradigm shift for multi-site collaborations, alleviating the need for data-sharing.

Longitudinal Assessment of Posttreatment Diffuse Glioma Tissue Volumes with Three-dimensional Convolutional Neural Networks

Neural networks were trained for segmentation and longitudinal assessment of posttreatment diffuse glioma. A retrospective cohort (from January 2018 to December 2019) of 298 patients with diffuse glioma (mean age, 52 years ± 14 [SD]; 177 men; 152 patients with glioblastoma, 72 patients with astrocytoma, and 74 patients with oligodendroglioma) who underwent two consecutive multimodal MRI examinations were randomly selected into training (n = 198) and testing (n = 100) samples. A posttreatment tumor segmentation three-dimensional nnU-Net convolutional neural network with multichannel inputs (T1, T2, and T1 postcontrast and fluid-attenuated inversion recovery [FLAIR]) was trained to segment three multiclass tissue types (peritumoral edematous, infiltrated, or treatment-changed tissue [ED]; active tumor or enhancing tissue [AT]; and necrotic core). Separate longitudinal change nnU-Nets were trained on registered and subtracted FLAIR and T1 postlongitudinal images to localize and better quantify and classify changes in ED and AT. Segmentation Dice scores, volume similarities, and 95th percentile Hausdorff distances ranged from 0.72 to 0.89, 0.90 to 0.96, and 2.5 to 3.6 mm, respectively. Accuracy rates of the posttreatment tumor segmentation and longitudinal change networks being able to classify longitudinal changes in ED and AT as increased, decreased, or unchanged were 76%-79% and 90%-91%, respectively. The accuracy levels of the longitudinal change networks were not significantly different from those of three neuroradiologists (accuracy, 90%-92%; κ, 0.58-0.63; P > .05). The results of this study support the potential clinical value of artificial intelligence-based automated longitudinal assessment of posttreatment diffuse glioma. Keywords: MR Imaging, Neuro-Oncology, Neural Networks, CNS, Brain/Brain Stem, Segmentation, Quantification, Convolutional Neural Network (CNN) Supplemental material is available for this article. © RSNA, 2022.

Automated detection and quantification of brain metastases on clinical MRI data using artificial neural networks

Background

Reliable detection and precise volumetric quantification of brain metastases (BM) on MRI are essential for guiding treatment decisions. Here we evaluate the potential of artificial neural networks (ANN) for automated detection and quantification of BM.

Methods

A consecutive series of 308 patients with BM was used for developing an ANN (with a 4:1 split for training/testing) for automated volumetric assessment of contrast-enhancing tumors (CE) and non-enhancing FLAIR signal abnormality including edema (NEE). An independent consecutive series of 30 patients was used for external testing. Performance was assessed case-wise for CE and NEE and lesion-wise for CE using the case-wise/lesion-wise DICE-coefficient (C/L-DICE), positive predictive value (L-PPV) and sensitivity (C/L-Sensitivity).

Results

The performance of detecting CE lesions on the validation dataset was not significantly affected when evaluating different volumetric thresholds (0.001–0.2 cm3; P = .2028). The median L-DICE and median C-DICE for CE lesions were 0.78 (IQR = 0.6–0.91) and 0.90 (IQR = 0.85–0.94) in the institutional as well as 0.79 (IQR = 0.67–0.82) and 0.84 (IQR = 0.76–0.89) in the external test dataset. The corresponding median L-Sensitivity and median L-PPV were 0.81 (IQR = 0.63–0.92) and 0.79 (IQR = 0.63–0.93) in the institutional test dataset, as compared to 0.85 (IQR = 0.76–0.94) and 0.76 (IQR = 0.68–0.88) in the external test dataset. The median C-DICE for NEE was 0.96 (IQR = 0.92–0.97) in the institutional test dataset as compared to 0.85 (IQR = 0.72–0.91) in the external test dataset.

Conclusion

The developed ANN-based algorithm (publicly available at www.github.com/NeuroAI-HD/HD-BM) allows reliable detection and precise volumetric quantification of CE and NEE compartments in patients with BM.

Interinstitutional Portability of a Deep Learning Brain MRI Lesion Segmentation Algorithm

PURPOSE

To assess how well a brain MRI lesion segmentation algorithm trained at one institution performed at another institution, and to assess the effect of multi-institutional training datasets for mitigating performance loss.

MATERIALS AND METHODS

In this retrospective study, a three-dimensional U-Net for brain MRI abnormality segmentation was trained on data from 293 patients from one institution (IN1) (median age, 54 years; 165 women; patients treated between 2008 and 2018) and tested on data from 51 patients from a second institution (IN2) (median age, 46 years; 27 women; patients treated between 2003 and 2019). The model was then trained on additional data from various sources: (a) 285 multi-institution brain tumor segmentations, (b) 198 IN2 brain tumor segmentations, and (c) 34 IN2 lesion segmentations from various brain pathologic conditions. All trained models were tested on IN1 and external IN2 test datasets, assessing segmentation performance using Dice coefficients.

RESULTS

The U-Net accurately segmented brain MRI lesions across various pathologic conditions. Performance was lower when tested at an external institution (median Dice score, 0.70 [IN2] vs 0.76 [IN1]). Addition of 483 training cases of a single pathologic condition, including from IN2, did not raise performance (median Dice score, 0.72; P = .10). Addition of IN2 training data with heterogeneous pathologic features, representing only 10% (34 of 329) of total training data, increased performance to baseline (Dice score, 0.77; P < .001). This final model produced total lesion volumes with a high correlation to the reference standard (Spearman r = 0.98).

CONCLUSION

For brain MRI lesion segmentation, adding a modest amount of relevant training data from an external institution to a previously trained model supported successful application of the model to this external institution.Keywords: Neural Networks, Brain/Brain Stem, Segmentation Supplemental material is available for this article. © RSNA, 2021.

Trends in Development of Novel Machine Learning Methods for the Identification of Gliomas in Datasets That Include Non-Glioma Images: A Systematic Review

Purpose

Machine learning has been applied to the diagnostic imaging of gliomas to augment classification, prognostication, segmentation, and treatment planning. A systematic literature review was performed to identify how machine learning has been applied to identify gliomas in datasets which include non-glioma images thereby simulating normal clinical practice.

Materials and Methods

Four databases were searched by a medical librarian and confirmed by a second librarian for all articles published prior to February 1, 2021: Ovid Embase, Ovid MEDLINE, Cochrane trials (CENTRAL), and Web of Science-Core Collection. The search strategy included both keywords and controlled vocabulary combining the terms for: artificial intelligence, machine learning, deep learning, radiomics, magnetic resonance imaging, glioma, as well as related terms. The review was conducted in stepwise fashion with abstract screening, full text screening, and data extraction. Quality of reporting was assessed using TRIPOD criteria.

Results

A total of 11,727 candidate articles were identified, of which 12 articles were included in the final analysis. Studies investigated the differentiation of normal from abnormal images in datasets which include gliomas (7 articles) and the differentiation of glioma images from non-glioma or normal images (5 articles). Single institution datasets were most common (5 articles) followed by BRATS (3 articles). The median sample size was 280 patients. Algorithm testing strategies consisted of five-fold cross validation (5 articles), and the use of exclusive sets of images within the same dataset for training and for testing (7 articles). Neural networks were the most common type of algorithm (10 articles). The accuracy of algorithms ranged from 0.75 to 1.00 (median 0.96, 10 articles). Quality of reporting assessment utilizing TRIPOD criteria yielded a mean individual TRIPOD ratio of 0.50 (standard deviation 0.14, range 0.37 to 0.85).

Conclusion

Systematic review investigating the identification of gliomas in datasets which include non-glioma images demonstrated multiple limitations hindering the application of these algorithms to clinical practice. These included limited datasets, a lack of generalizable algorithm training and testing strategies, and poor quality of reporting. The development of more robust and heterogeneous datasets is needed for algorithm development. Future studies would benefit from using external datasets for algorithm testing as well as placing increased attention on quality of reporting standards.

Systematic Review Registration

www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020209938, International Prospective Register of Systematic Reviews (PROSPERO 2020 CRD42020209938).

Brain MRI Deep Learning and Bayesian Inference System Augments Radiology Resident Performance

Automated quantitative and probabilistic medical image analysis has the potential to improve the accuracy and efficiency of the radiology workflow. We sought to determine whether AI systems for brain MRI diagnosis could be used as a clinical decision support tool to augment radiologist performance. We utilized previously developed AI systems that combine convolutional neural networks and expert-derived Bayesian networks to distinguish among 50 diagnostic entities on multimodal brain MRIs. We tested whether these systems could augment radiologist performance through an interactive clinical decision support tool known as Adaptive Radiology Interpretation and Education System (ARIES) in 194 test cases. Four radiology residents and three academic neuroradiologists viewed half of the cases unassisted and half with the results of the AI system displayed on ARIES. Diagnostic accuracy of radiologists for top diagnosis (TDx) and top three differential diagnosis (T3DDx) was compared with and without ARIES. Radiology resident performance was significantly better with ARIES for both TDx (55% vs 30%; P < .001) and T3DDx (79% vs 52%; P = 0.002), with the largest improvement for rare diseases (39% increase for T3DDx; P < 0.001). There was no significant difference between attending performance with and without ARIES for TDx (72% vs 69%; P = 0.48) or T3DDx (86% vs 89%; P = 0.39). These findings suggest that a hybrid deep learning and Bayesian inference clinical decision support system has the potential to augment diagnostic accuracy of non-specialists to approach the level of subspecialists for a large array of diseases on brain MRI.

Three-dimensional U-Net Convolutional Neural Network for Detection and Segmentation of Intracranial Metastases

PURPOSE

To develop and validate a neural network for automated detection and segmentation of intracranial metastases on brain MRI studies obtained for stereotactic radiosurgery treatment planning.

MATERIALS AND METHODS

In this retrospective study, 413 patients (average age, 61 years ± 12 [standard deviation]; 238 women) with a total of 5202 intracranial metastases (median volume, 0.05 cm3; interquartile range, 0.02-0.18 cm3) undergoing stereotactic radiosurgery at one institution were included (January 2017 to February 2020). A total of 563 MRI examinations were performed among the patients, and studies were split into training (n = 413), validation (n = 50), and test (n = 100) datasets. A three-dimensional (3D) U-Net convolutional network was trained and validated on 413 T1 postcontrast or subtraction scans, and several loss functions were evaluated. After model validation, 100 discrete test patients, who underwent imaging after the training and validation patients, were used for final model evaluation. Performance for detection and segmentation of metastases was evaluated using Dice scores, false discovery rates, and false-negative rates, and a comparison with neuroradiologist interrater reliability was performed.

RESULTS

The median Dice score for segmenting enhancing metastases in the test set was 0.75 (interquartile range, 0.63-0.84). There were strong correlations between manually segmented and predicted metastasis volumes (r = 0.98, P < .001) and between the number of manually segmented and predicted metastases (R = 0.95, P < .001). Higher Dice scores were strongly correlated with larger metastasis volumes on a logarithmically transformed scale (r = 0.71). Sensitivity across the whole test sample was 70.0% overall and 96.4% for metastases larger than 6 mm. There was an average of 0.46 false-positive results per scan, with the positive predictive value being 91.5%. In comparison, the median Dice score between two neuroradiologists was 0.85 (interquartile range, 0.80-0.89), with sensitivity across the test sample being 87.9% overall and 98.4% for metastases larger than 6 mm.

CONCLUSION

A 3D U-Net-based convolutional neural network was able to segment brain metastases with high accuracy and perform detection at the level of human interrater reliability for metastases larger than 6 mm.Keywords: Adults, Brain/Brain Stem, CNS, Feature detection, MR-Imaging, Neural Networks, Neuro-Oncology, Quantification, Segmentation© RSNA, 2021.

Deep learning shows good reliability for automatic segmentation and volume measurement of brain hemorrhage, intraventricular extension, and peripheral edema

OBJECTIVES

To evaluate for the first time the performance of a deep learning method based on no-new-Net for fully automated segmentation and volumetric measurements of intracerebral hemorrhage (ICH), intraventricular extension of intracerebral hemorrhage (IVH), and perihematomal edema (PHE) in primary ICH on CT.

METHODS

Three hundred and eighty primary ICH patients who underwent CT at hospital arrival were divided into a training cohort (n = 300) and a validation cohort (n = 80). An independent cohort with 80 patients was used for testing. Ground truth (segmentation masks) was manually generated by radiologists. Model performance on lesion segmentation and volumetric measurement of ICH, IVH, and PHE were evaluated by comparing the model results with the segmentations performed by radiologists.

RESULTS

In the test cohort, the Dice scores of lesion segmentation were 0.92, 0.79, and 0.71 for ICH, IVH, and PHE, respectively. The sensitivities were 0.93 for ICH, 0.88 for IVH, and 0.81 for PHE. The positive predictive values were 0.92, 0.76, and 0.69 for ICH, IVH, and PHE, respectively. Excellent concordance (concordance correlation coefficients [CCCs] ≥ 0.98) of ICH and IVH and good concordance of PHE (CCCs ≥ 0.92) were demonstrated between manually and automatically measured volumes. The model took approximately 15 s to provide automatic segmentation and volume analysis for each patient.

CONCLUSION

Our model demonstrates good reliability for automatic segmentation and volume measurement of ICH, IVH, and PHE in primary ICH, which can be useful to reduce the effort and time of doctors to calculate volumes of ICH, IVH, and PHE.

KEY POINTS

• Deep learning algorithms can provide automatic and reliable assessment of intracerebral hemorrhage, intraventricular hemorrhage, and perihematomal edema on CT. • Non-contrast CT-based deep learning method can be helpful to provide efficient and accurate measurements of ICH, IVH, and PHE in primary ICH patients, thereby reducing the effort and time of doctors to segment and calculate volumes of ICH, IVH, and PHE in primary ICH patients.

Deep learning-enabled analysis reveals distinct neuronal phenotypes induced by aging and cold-shock

BACKGROUND

Access to quantitative information is crucial to obtain a deeper understanding of biological systems. In addition to being low-throughput, traditional image-based analysis is mostly limited to error-prone qualitative or semi-quantitative assessment of phenotypes, particularly for complex subcellular morphologies. The PVD neuron in Caenorhabditis elegans, which is responsible for harsh touch and thermosensation, undergoes structural degeneration as nematodes age characterized by the appearance of dendritic protrusions. Analysis of these neurodegenerative patterns is labor-intensive and limited to qualitative assessment.

RESULTS

In this work, we apply deep learning to perform quantitative image-based analysis of complex neurodegeneration patterns exhibited by the PVD neuron in C. elegans. We apply a convolutional neural network algorithm (Mask R-CNN) to identify neurodegenerative subcellular protrusions that appear after cold-shock or as a result of aging. A multiparametric phenotypic profile captures the unique morphological changes induced by each perturbation. We identify that acute cold-shock-induced neurodegeneration is reversible and depends on rearing temperature and, importantly, that aging and cold-shock induce distinct neuronal beading patterns.

CONCLUSION

The results of this work indicate that implementing deep learning for challenging image segmentation of PVD neurodegeneration enables quantitatively tracking subtle morphological changes in an unbiased manner. This analysis revealed that distinct patterns of morphological alteration are induced by aging and cold-shock, suggesting different mechanisms at play. This approach can be used to identify the molecular components involved in orchestrating neurodegeneration and to characterize the effect of other stressors on PVD degeneration.

Artificial Intelligence System Approaching Neuroradiologist-level Differential Diagnosis Accuracy at Brain MRI

Background Although artificial intelligence (AI) shows promise across many aspects of radiology, the use of AI to create differential diagnoses for rare and common diseases at brain MRI has not been demonstrated. Purpose To evaluate an AI system for generation of differential diagnoses at brain MRI compared with radiologists. Materials and Methods This retrospective study tested performance of an AI system for probabilistic diagnosis in patients with 19 common and rare diagnoses at brain MRI acquired between January 2008 and January 2018. The AI system combines data-driven and domain-expertise methodologies, including deep learning and Bayesian networks. First, lesions were detected by using deep learning. Then, 18 quantitative imaging features were extracted by using atlas-based coregistration and segmentation. Third, these image features were combined with five clinical features by using Bayesian inference to develop probability-ranked differential diagnoses. Quantitative feature extraction algorithms and conditional probabilities were fine-tuned on a training set of 86 patients (mean age, 49 years ± 16 [standard deviation]; 53 women). Accuracy was compared with radiology residents, general radiologists, neuroradiology fellows, and academic neuroradiologists by using accuracy of top one, top two, and top three differential diagnoses in 92 independent test set patients (mean age, 47 years ± 18; 52 women). Results For accuracy of top three differential diagnoses, the AI system (91% correct) performed similarly to academic neuroradiologists (86% correct; P = .20), and better than radiology residents (56%; P < .001), general radiologists (57%; P < .001), and neuroradiology fellows (77%; P = .003). The performance of the AI system was not affected by disease prevalence (93% accuracy for common vs 85% for rare diseases; P = .26). Radiologists were more accurate at diagnosing common versus rare diagnoses (78% vs 47% across all radiologists; P < .001). Conclusion An artificial intelligence system for brain MRI approached overall top one, top two, and top three differential diagnoses accuracy of neuroradiologists and exceeded that of less-specialized radiologists. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Zaharchuk in this issue.