Probabilistic Prediction of Laboratory Test Information Yield

Low-yield repetitive laboratory diagnostics burden patients and inflate cost of care. In this study, we assess whether stability in repeated laboratory diagnostic measurements is predictable with uncertainty estimates using electronic health record data available before the diagnostic is ordered. We use probabilistic regression to predict a distribution of plausible values, allowing use-time customization for various definitions of "stability" given dynamic ranges and clinical scenarios. After converting distributions into "stability" scores, the models achieve a sensitivity of 29% for white blood cells, 60% for hemoglobin, 100% for platelets, 54% for potassium, 99% for albumin and 35% for creatinine for predicting stability at 90% precision, suggesting those fractions of repetitive tests could be reduced with low risk of missing important changes. The findings demonstrate the feasibility of using electronic health record data to identify low-yield repetitive tests and offer personalized guidance for better usage of testing while ensuring high quality care.

VetLLM: Large Language Model for Predicting Diagnosis from Veterinary Notes

Lack of diagnosis coding is a barrier to leveraging veterinary notes for medical and public health research. Previous work is limited to develop specialized rule-based or customized supervised learning models to predict diagnosis coding, which is tedious and not easily transferable. In this work, we show that open-source large language models (LLMs) pretrained on general corpus can achieve reasonable performance in a zero-shot setting. Alpaca-7B can achieve a zero-shot F1 of 0.538 on CSU test data and 0.389 on PP test data, two standard benchmarks for coding from veterinary notes. Furthermore, with appropriate fine-tuning, the performance of LLMs can be substantially boosted, exceeding those of strong state-of-the-art supervised models. VetLLM, which is fine-tuned on Alpaca-7B using just 5000 veterinary notes, can achieve a F1 of 0.747 on CSU test data and 0.637 on PP test data. It is of note that our fine-tuning is data-efficient: using 200 notes can outperform supervised models trained with more than 100,000 notes. The findings demonstrate the great potential of leveraging LLMs for language processing tasks in medicine, and we advocate this new paradigm for processing clinical text.

Evaluating progress in automatic chest X-ray radiology report generation

Artificial intelligence (AI) models for automatic generation of narrative radiology reports from images have the potential to enhance efficiency and reduce the workload of radiologists. However, evaluating the correctness of these reports requires metrics that can capture clinically pertinent differences. In this study, we investigate the alignment between automated metrics and radiologists' scoring of errors in report generation. We address the limitations of existing metrics by proposing new metrics, RadGraph F1 and RadCliQ, which demonstrate stronger correlation with radiologists' evaluations. In addition, we analyze the failure modes of the metrics to understand their limitations and provide guidance for metric selection and interpretation. This study establishes RadGraph F1 and RadCliQ as meaningful metrics for guiding future research in radiology report generation.

Transfer learning enables prediction of myocardial injury from continuous single-lead electrocardiography

OBJECTIVE

Chest pain is common, and current risk-stratification methods, requiring 12-lead electrocardiograms (ECGs) and serial biomarker assays, are static and restricted to highly resourced settings. Our objective was to predict myocardial injury using continuous single-lead ECG waveforms similar to those obtained from wearable devices and to evaluate the potential of transfer learning from labeled 12-lead ECGs to improve these predictions.

METHODS

We studied 10 874 Emergency Department (ED) patients who received continuous ECG monitoring and troponin testing from 2020 to 2021. We defined myocardial injury as newly elevated troponin in patients with chest pain or shortness of breath. We developed deep learning models of myocardial injury using continuous lead II ECG from bedside monitors as well as conventional 12-lead ECGs from triage. We pretrained single-lead models on a pre-existing corpus of labeled 12-lead ECGs. We compared model predictions to those of ED physicians.

RESULTS

A transfer learning strategy, whereby models for continuous single-lead ECGs were first pretrained on 12-lead ECGs from a separate cohort, predicted myocardial injury as accurately as models using patients' own 12-lead ECGs: area under the receiver operating characteristic curve 0.760 (95% confidence interval [CI], 0.721-0.799) and area under the precision-recall curve 0.321 (95% CI, 0.251-0.397). Models demonstrated a high negative predictive value for myocardial injury among patients with chest pain or shortness of breath, exceeding the predictive performance of ED physicians, while attending to known stigmata of myocardial injury.

CONCLUSIONS

Deep learning models pretrained on labeled 12-lead ECGs can predict myocardial injury from noisy, continuous monitor data early in a patient's presentation. The utility of continuous single-lead ECG in the risk stratification of chest pain has implications for wearable devices and preclinical settings, where external validation of the approach is needed.

CheXED: Comparison of a Deep Learning Model to a Clinical Decision Support System for Pneumonia in the Emergency Department

PURPOSE

Patients with pneumonia often present to the emergency department (ED) and require prompt diagnosis and treatment. Clinical decision support systems for the diagnosis and management of pneumonia are commonly utilized in EDs to improve patient care. The purpose of this study is to investigate whether a deep learning model for detecting radiographic pneumonia and pleural effusions can improve functionality of a clinical decision support system (CDSS) for pneumonia management (ePNa) operating in 20 EDs.

MATERIALS AND METHODS

In this retrospective cohort study, a dataset of 7434 prior chest radiographic studies from 6551 ED patients was used to develop and validate a deep learning model to identify radiographic pneumonia, pleural effusions, and evidence of multilobar pneumonia. Model performance was evaluated against 3 radiologists' adjudicated interpretation and compared with performance of the natural language processing of radiology reports used by ePNa.

RESULTS

The deep learning model achieved an area under the receiver operating characteristic curve of 0.833 (95% confidence interval [CI]: 0.795, 0.868) for detecting radiographic pneumonia, 0.939 (95% CI: 0.911, 0.962) for detecting pleural effusions and 0.847 (95% CI: 0.800, 0.890) for identifying multilobar pneumonia. On all 3 tasks, the model achieved higher agreement with the adjudicated radiologist interpretation compared with ePNa.

CONCLUSIONS

A deep learning model demonstrated higher agreement with radiologists than the ePNa CDSS in detecting radiographic pneumonia and related findings. Incorporating deep learning models into pneumonia CDSS could enhance diagnostic performance and improve pneumonia management.

Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

In tasks involving the interpretation of medical images, suitably trained machine-learning models often exceed the performance of medical experts. Yet such a high-level of performance typically requires that the models be trained with relevant datasets that have been painstakingly annotated by experts. Here we show that a self-supervised model trained on chest X-ray images that lack explicit annotations performs pathology-classification tasks with accuracies comparable to those of radiologists. On an external validation dataset of chest X-rays, the self-supervised model outperformed a fully supervised model in the detection of three pathologies (out of eight), and the performance generalized to pathologies that were not explicitly annotated for model training, to multiple image-interpretation tasks and to datasets from multiple institutions.

Development and Validation of an Artificial Intelligence System to Optimize Clinician Review of Patient Records

IMPORTANCE

Physicians are required to work with rapidly growing amounts of medical data. Approximately 62% of time per patient is devoted to reviewing electronic health records (EHRs), with clinical data review being the most time-consuming portion.

OBJECTIVE

To determine whether an artificial intelligence (AI) system developed to organize and display new patient referral records would improve a clinician's ability to extract patient information compared with the current standard of care.

DESIGN, SETTING, AND PARTICIPANTS

In this prognostic study, an AI system was created to organize patient records and improve data retrieval. To evaluate the system on time and accuracy, a nonblinded, prospective study was conducted at a single academic medical center. Recruitment emails were sent to all physicians in the gastroenterology division, and 12 clinicians agreed to participate. Each of the clinicians participating in the study received 2 referral records: 1 AI-optimized patient record and 1 standard (non-AI-optimized) patient record. For each record, clinicians were asked 22 questions requiring them to search the assigned record for clinically relevant information. Clinicians reviewed records from June 1 to August 30, 2020.

MAIN OUTCOMES AND MEASURES

The time required to answer each question, along with accuracy, was measured for both records, with and without AI optimization. Participants were asked to assess overall satisfaction with the AI system, their preferred review method (AI-optimized vs standard), and other topics to assess clinical utility.

RESULTS

Twelve gastroenterology physicians/fellows completed the study. Compared with standard (non-AI-optimized) patient record review, the AI system saved first-time physician users 18% of the time used to answer the clinical questions (10.5 [95% CI, 8.5-12.6] vs 12.8 [95% CI, 9.4-16.2] minutes; P = .02). There was no significant decrease in accuracy when physicians retrieved important patient information (83.7% [95% CI, 79.3%-88.2%] with the AI-optimized vs 86.0% [95% CI, 81.8%-90.2%] without the AI-optimized record; P = .81). Survey responses from physicians were generally positive across all questions. Eleven of 12 physicians (92%) preferred the AI-optimized record review to standard review. Despite a learning curve pointed out by respondents, 11 of 12 physicians believed that the technology would save them time to assess new patient records and were interested in using this technology in their clinic.

CONCLUSIONS AND RELEVANCE

In this prognostic study, an AI system helped physicians extract relevant patient information in a shorter time while maintaining high accuracy. This finding is particularly germane to the ever-increasing amounts of medical data and increased stressors on clinicians. Increased user familiarity with the AI system, along with further enhancements in the system itself, hold promise to further improve physician data extraction from large quantities of patient health records.

Data augmentation with Mobius transformations

Data augmentation has led to substantial improvements in the performance and generalization of deep models, and remains a highly adaptable method to evolving model architectures and varying amounts of data—in particular, extremely scarce amounts of available training data. In this paper, we present a novel method of applying Möbius transformations to augment input images during training. Möbius transformations are bijective conformal maps that generalize image translation to operate over complex inversion in pixel space. As a result, Möbius transformations can operate on the sample level and preserve data labels. We show that the inclusion of Möbius transformations during training enables improved generalization over prior sample-level data augmentation techniques such as cutout and standard crop-and-flip transformations, most notably in low data regimes.

Automated coronary calcium scoring using deep learning with multicenter external validation

Coronary artery disease (CAD), the most common manifestation of cardiovascular disease, remains the most common cause of mortality in the United States. Risk assessment is key for primary prevention of coronary events and coronary artery calcium (CAC) scoring using computed tomography (CT) is one such non-invasive tool. Despite the proven clinical value of CAC, the current clinical practice implementation for CAC has limitations such as the lack of insurance coverage for the test, need for capital-intensive CT machines, specialized imaging protocols, and accredited 3D imaging labs for analysis (including personnel and software). Perhaps the greatest gap is the millions of patients who undergo routine chest CT exams and demonstrate coronary artery calcification, but their presence is not often reported or quantitation is not feasible. We present two deep learning models that automate CAC scoring demonstrating advantages in automated scoring for both dedicated gated coronary CT exams and routine non-gated chest CTs performed for other reasons to allow opportunistic screening. First, we trained a gated coronary CT model for CAC scoring that showed near perfect agreement (mean difference in scores = -2.86; Cohen's Kappa = 0.89, P < 0.0001) with current conventional manual scoring on a retrospective dataset of 79 patients and was found to perform the task faster (average time for automated CAC scoring using a graphics processing unit (GPU) was 3.5 ± 2.1 s vs. 261 s for manual scoring) in a prospective trial of 55 patients with little difference in scores compared to three technologists (mean difference in scores = 3.24, 5.12, and 5.48, respectively). Then using CAC scores from paired gated coronary CT as a reference standard, we trained a deep learning model on our internal data and a cohort from the Multi-Ethnic Study of Atherosclerosis (MESA) study (total training n = 341, Stanford test n = 42, MESA test n = 46) to perform CAC scoring on routine non-gated chest CT exams with validation on external datasets (total n = 303) obtained from four geographically disparate health systems. On identifying patients with any CAC (i.e., CAC ≥ 1), sensitivity and PPV was high across all datasets (ranges: 80-100% and 87-100%, respectively). For CAC ≥ 100 on routine non-gated chest CTs, which is the latest recommended threshold to initiate statin therapy, our model showed sensitivities of 71-94% and positive predictive values in the range of 88-100% across all the sites. Adoption of this model could allow more patients to be screened with CAC scoring, potentially allowing opportunistic early preventive interventions.

DLBCL-Morph: Morphological features computed using deep learning for an annotated digital DLBCL image set

Diffuse Large B-Cell Lymphoma (DLBCL) is the most common non-Hodgkin lymphoma. Though histologically DLBCL shows varying morphologies, no morphologic features have been consistently demonstrated to correlate with prognosis. We present a morphologic analysis of histology sections from 209 DLBCL cases with associated clinical and cytogenetic data. Duplicate tissue core sections were arranged in tissue microarrays (TMAs), and replicate sections were stained with H&E and immunohistochemical stains for CD10, BCL6, MUM1, BCL2, and MYC. The TMAs are accompanied by pathologist-annotated regions-of-interest (ROIs) that identify areas of tissue representative of DLBCL. We used a deep learning model to segment all tumor nuclei in the ROIs, and computed several geometric features for each segmented nucleus. We fit a Cox proportional hazards model to demonstrate the utility of these geometric features in predicting survival outcome, and found that it achieved a C-index (95% CI) of 0.635 (0.574,0.691). Our finding suggests that geometric features computed from tumor nuclei are of prognostic importance, and should be validated in prospective studies.

Measurement(s)	B-cell lymphoma • histology
Technology Type(s)	machine learning
Factor Type(s)	patient
Sample Characteristic - Organism	Homo sapiens

Machine-accessible metadata file describing the reported data: 10.6084/m9.figshare.14465178

CheXaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with HIV

Tuberculosis (TB) is the leading cause of preventable death in HIV-positive patients, and yet often remains undiagnosed and untreated. Chest x-ray is often used to assist in diagnosis, yet this presents additional challenges due to atypical radiographic presentation and radiologist shortages in regions where co-infection is most common. We developed a deep learning algorithm to diagnose TB using clinical information and chest x-ray images from 677 HIV-positive patients with suspected TB from two hospitals in South Africa. We then sought to determine whether the algorithm could assist clinicians in the diagnosis of TB in HIV-positive patients as a web-based diagnostic assistant. Use of the algorithm resulted in a modest but statistically significant improvement in clinician accuracy (p = 0.002), increasing the mean clinician accuracy from 0.60 (95% CI 0.57, 0.63) without assistance to 0.65 (95% CI 0.60, 0.70) with assistance. However, the accuracy of assisted clinicians was significantly lower (p < 0.001) than that of the stand-alone algorithm, which had an accuracy of 0.79 (95% CI 0.77, 0.82) on the same unseen test cases. These results suggest that deep learning assistance may improve clinician accuracy in TB diagnosis using chest x-rays, which would be valuable in settings with a high burden of HIV/TB co-infection. Moreover, the high accuracy of the stand-alone algorithm suggests a potential value particularly in settings with a scarcity of radiological expertise.

Incorporating machine learning and social determinants of health indicators into prospective risk adjustment for health plan payments

BACKGROUND

Risk adjustment models are employed to prevent adverse selection, anticipate budgetary reserve needs, and offer care management services to high-risk individuals. We aimed to address two unknowns about risk adjustment: whether machine learning (ML) and inclusion of social determinants of health (SDH) indicators improve prospective risk adjustment for health plan payments.

METHODS

We employed a 2-by-2 factorial design comparing: (i) linear regression versus ML (gradient boosting) and (ii) demographics and diagnostic codes alone, versus additional ZIP code-level SDH indicators. Healthcare claims from privately-insured US adults (2016-2017), and Census data were used for analysis. Data from 1.02 million adults were used for derivation, and data from 0.26 million to assess performance. Model performance was measured using coefficient of determination (R²), discrimination (C-statistic), and mean absolute error (MAE) for the overall population, and predictive ratio and net compensation for vulnerable subgroups. We provide 95% confidence intervals (CI) around each performance measure.

RESULTS

Linear regression without SDH indicators achieved moderate determination (R² 0.327, 95% CI: 0.300, 0.353), error ($6992; 95% CI: $6889, $7094), and discrimination (C-statistic 0.703; 95% CI: 0.701, 0.705). ML without SDH indicators improved all metrics (R² 0.388; 95% CI: 0.357, 0.420; error $6637; 95% CI: $6539, $6735; C-statistic 0.717; 95% CI: 0.715, 0.718), reducing misestimation of cost by $3.5 M per 10,000 members. Among people living in areas with high poverty, high wealth inequality, or high prevalence of uninsured, SDH indicators reduced underestimation of cost, improving the predictive ratio by 3% (~$200/person/year).

CONCLUSIONS

ML improved risk adjustment models and the incorporation of SDH indicators reduced underpayment in several vulnerable populations.

PENet-a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging

Pulmonary embolism (PE) is a life-threatening clinical problem and computed tomography pulmonary angiography (CTPA) is the gold standard for diagnosis. Prompt diagnosis and immediate treatment are critical to avoid high morbidity and mortality rates, yet PE remains among the diagnoses most frequently missed or delayed. In this study, we developed a deep learning model-PENet, to automatically detect PE on volumetric CTPA scans as an end-to-end solution for this purpose. The PENet is a 77-layer 3D convolutional neural network (CNN) pretrained on the Kinetics-600 dataset and fine-tuned on a retrospective CTPA dataset collected from a single academic institution. The PENet model performance was evaluated in detecting PE on data from two different institutions: one as a hold-out dataset from the same institution as the training data and a second collected from an external institution to evaluate model generalizability to an unrelated population dataset. PENet achieved an AUROC of 0.84 [0.82-0.87] on detecting PE on the hold out internal test set and 0.85 [0.81-0.88] on external dataset. PENet also outperformed current state-of-the-art 3D CNN models. The results represent successful application of an end-to-end 3D CNN model for the complex task of PE diagnosis without requiring computationally intensive and time consuming preprocessing and demonstrates sustained performance on data from an external institution. Our model could be applied as a triage tool to automatically identify clinically important PEs allowing for prioritization for diagnostic radiology interpretation and improved care pathways via more efficient diagnosis.

AppendiXNet: Deep Learning for Diagnosis of Appendicitis from A Small Dataset of CT Exams Using Video Pretraining

The development of deep learning algorithms for complex tasks in digital medicine has relied on the availability of large labeled training datasets, usually containing hundreds of thousands of examples. The purpose of this study was to develop a 3D deep learning model, AppendiXNet, to detect appendicitis, one of the most common life-threatening abdominal emergencies, using a small training dataset of less than 500 training CT exams. We explored whether pretraining the model on a large collection of natural videos would improve the performance of the model over training the model from scratch. AppendiXNet was pretrained on a large collection of YouTube videos called Kinetics, consisting of approximately 500,000 video clips and annotated for one of 600 human action classes, and then fine-tuned on a small dataset of 438 CT scans annotated for appendicitis. We found that pretraining the 3D model on natural videos significantly improved the performance of the model from an AUC of 0.724 (95% CI 0.625, 0.823) to 0.810 (95% CI 0.725, 0.895). The application of deep learning to detect abnormalities on CT examinations using video pretraining could generalize effectively to other challenging cross-sectional medical imaging tasks when training data is limited.

Impact of a deep learning assistant on the histopathologic classification of liver cancer

Artificial intelligence (AI) algorithms continue to rival human performance on a variety of clinical tasks, while their actual impact on human diagnosticians, when incorporated into clinical workflows, remains relatively unexplored. In this study, we developed a deep learning-based assistant to help pathologists differentiate between two subtypes of primary liver cancer, hepatocellular carcinoma and cholangiocarcinoma, on hematoxylin and eosin-stained whole-slide images (WSI), and evaluated its effect on the diagnostic performance of 11 pathologists with varying levels of expertise. Our model achieved accuracies of 0.885 on a validation set of 26 WSI, and 0.842 on an independent test set of 80 WSI. Although use of the assistant did not change the mean accuracy of the 11 pathologists (p = 0.184, OR = 1.281), it significantly improved the accuracy (p = 0.045, OR = 1.499) of a subset of nine pathologists who fell within well-defined experience levels (GI subspecialists, non-GI subspecialists, and trainees). In the assisted state, model accuracy significantly impacted the diagnostic decisions of all 11 pathologists. As expected, when the model's prediction was correct, assistance significantly improved accuracy (p = 0.000, OR = 4.289), whereas when the model's prediction was incorrect, assistance significantly decreased accuracy (p = 0.000, OR = 0.253), with both effects holding across all pathologist experience levels and case difficulty levels. Our results highlight the challenges of translating AI models into the clinical setting, and emphasize the importance of taking into account potential unintended negative consequences of model assistance when designing and testing medical AI-assistance tools.

Deep Learning-Assisted Diagnosis of Cerebral Aneurysms Using the HeadXNet Model

IMPORTANCE

Deep learning has the potential to augment clinician performance in medical imaging interpretation and reduce time to diagnosis through automated segmentation. Few studies to date have explored this topic.

OBJECTIVE

To develop and apply a neural network segmentation model (the HeadXNet model) capable of generating precise voxel-by-voxel predictions of intracranial aneurysms on head computed tomographic angiography (CTA) imaging to augment clinicians' intracranial aneurysm diagnostic performance.

DESIGN, SETTING, AND PARTICIPANTS

In this diagnostic study, a 3-dimensional convolutional neural network architecture was developed using a training set of 611 head CTA examinations to generate aneurysm segmentations. Segmentation outputs from this support model on a test set of 115 examinations were provided to clinicians. Between August 13, 2018, and October 4, 2018, 8 clinicians diagnosed the presence of aneurysm on the test set, both with and without model augmentation, in a crossover design using randomized order and a 14-day washout period. Head and neck examinations performed between January 3, 2003, and May 31, 2017, at a single academic medical center were used to train, validate, and test the model. Examinations positive for aneurysm had at least 1 clinically significant, nonruptured intracranial aneurysm. Examinations with hemorrhage, ruptured aneurysm, posttraumatic or infectious pseudoaneurysm, arteriovenous malformation, surgical clips, coils, catheters, or other surgical hardware were excluded. All other CTA examinations were considered controls.

MAIN OUTCOMES AND MEASURES

Sensitivity, specificity, accuracy, time, and interrater agreement were measured. Metrics for clinician performance with and without model augmentation were compared.

RESULTS

The data set contained 818 examinations from 662 unique patients with 328 CTA examinations (40.1%) containing at least 1 intracranial aneurysm and 490 examinations (59.9%) without intracranial aneurysms. The 8 clinicians reading the test set ranged in experience from 2 to 12 years. Augmenting clinicians with artificial intelligence-produced segmentation predictions resulted in clinicians achieving statistically significant improvements in sensitivity, accuracy, and interrater agreement when compared with no augmentation. The clinicians' mean sensitivity increased by 0.059 (95% CI, 0.028-0.091; adjusted P = .01), mean accuracy increased by 0.038 (95% CI, 0.014-0.062; adjusted P = .02), and mean interrater agreement (Fleiss κ) increased by 0.060, from 0.799 to 0.859 (adjusted P = .05). There was no statistically significant change in mean specificity (0.016; 95% CI, -0.010 to 0.041; adjusted P = .16) and time to diagnosis (5.71 seconds; 95% CI, 7.22-18.63 seconds; adjusted P = .19).

CONCLUSIONS AND RELEVANCE

The deep learning model developed successfully detected clinically significant intracranial aneurysms on CTA. This suggests that integration of an artificial intelligence-assisted diagnostic model may augment clinician performance with dependable and accurate predictions and thereby optimize patient care.

Clinical Value of Predicting Individual Treatment Effects for Intensive Blood Pressure Therapy

BACKGROUND

The absolute risk reduction (ARR) in cardiovascular events from therapy is generally assumed to be proportional to baseline risk-such that high-risk patients benefit most. Yet newer analyses have proposed using randomized trial data to develop models that estimate individual treatment effects. We tested 2 hypotheses: first, that models of individual treatment effects would reveal that benefit from intensive blood pressure therapy is proportional to baseline risk; and second, that a machine learning approach designed to predict heterogeneous treatment effects-the X-learner meta-algorithm-is equivalent to a conventional logistic regression approach.

METHODS AND RESULTS

We compared conventional logistic regression to the X-learner approach for prediction of 3-year cardiovascular disease event risk reduction from intensive (target systolic blood pressure <120 mm Hg) versus standard (target <140 mm Hg) blood pressure treatment, using individual participant data from the SPRINT (Systolic Blood Pressure Intervention Trial; N=9361) and ACCORD BP (Action to Control Cardiovascular Risk in Diabetes Blood Pressure; N=4733) trials. Each model incorporated 17 covariates, an indicator for treatment arm, and interaction terms between covariates and treatment. Logistic regression had lower C statistic for benefit than the X-learner (0.51 [95% CI, 0.49-0.53] versus 0.60 [95% CI, 0.58-0.63], respectively). Following the logistic regression's recommendation for individualized therapy produced restricted mean time until cardiovascular disease event of 1065.47 days (95% CI, 1061.04-1069.35), while following the X-learner's recommendation improved mean time until cardiovascular disease event to 1068.71 days (95% CI, 1065.42-1072.08). Calibration was worse for logistic regression; it over-estimated ARR attributable to intensive treatment (slope between predicted and observed ARR of 0.73 [95% CI, 0.30-1.14] versus 1.06 [95% CI, 0.74-1.32] for the X-learner, compared with the ideal of 1). Predicted ARRs using logistic regression were generally proportional to baseline pretreatment cardiovascular risk, whereas the X-learner observed-correctly-that individual treatment effects were often not proportional to baseline risk.

CONCLUSIONS

Predictions for individual treatment effects from trial data reveal that patients may experience ARRs not simply proportional to baseline cardiovascular disease risk. Machine learning methods may improve discrimination and calibration of individualized treatment effect estimates from clinical trial data.

CLINICAL TRIAL REGISTRATION

URL: https://www.clinicaltrials.gov . Unique identifiers: NCT01206062; NCT00000620.

Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists

BACKGROUND

Chest radiograph interpretation is critical for the detection of thoracic diseases, including tuberculosis and lung cancer, which affect millions of people worldwide each year. This time-consuming task typically requires expert radiologists to read the images, leading to fatigue-based diagnostic error and lack of diagnostic expertise in areas of the world where radiologists are not available. Recently, deep learning approaches have been able to achieve expert-level performance in medical image interpretation tasks, powered by large network architectures and fueled by the emergence of large labeled datasets. The purpose of this study is to investigate the performance of a deep learning algorithm on the detection of pathologies in chest radiographs compared with practicing radiologists.

METHODS AND FINDINGS

We developed CheXNeXt, a convolutional neural network to concurrently detect the presence of 14 different pathologies, including pneumonia, pleural effusion, pulmonary masses, and nodules in frontal-view chest radiographs. CheXNeXt was trained and internally validated on the ChestX-ray8 dataset, with a held-out validation set consisting of 420 images, sampled to contain at least 50 cases of each of the original pathology labels. On this validation set, the majority vote of a panel of 3 board-certified cardiothoracic specialist radiologists served as reference standard. We compared CheXNeXt's discriminative performance on the validation set to the performance of 9 radiologists using the area under the receiver operating characteristic curve (AUC). The radiologists included 6 board-certified radiologists (average experience 12 years, range 4-28 years) and 3 senior radiology residents, from 3 academic institutions. We found that CheXNeXt achieved radiologist-level performance on 11 pathologies and did not achieve radiologist-level performance on 3 pathologies. The radiologists achieved statistically significantly higher AUC performance on cardiomegaly, emphysema, and hiatal hernia, with AUCs of 0.888 (95% confidence interval [CI] 0.863-0.910), 0.911 (95% CI 0.866-0.947), and 0.985 (95% CI 0.974-0.991), respectively, whereas CheXNeXt's AUCs were 0.831 (95% CI 0.790-0.870), 0.704 (95% CI 0.567-0.833), and 0.851 (95% CI 0.785-0.909), respectively. CheXNeXt performed better than radiologists in detecting atelectasis, with an AUC of 0.862 (95% CI 0.825-0.895), statistically significantly higher than radiologists' AUC of 0.808 (95% CI 0.777-0.838); there were no statistically significant differences in AUCs for the other 10 pathologies. The average time to interpret the 420 images in the validation set was substantially longer for the radiologists (240 minutes) than for CheXNeXt (1.5 minutes). The main limitations of our study are that neither CheXNeXt nor the radiologists were permitted to use patient history or review prior examinations and that evaluation was limited to a dataset from a single institution.

CONCLUSIONS

In this study, we developed and validated a deep learning algorithm that classified clinically important abnormalities in chest radiographs at a performance level comparable to practicing radiologists. Once tested prospectively in clinical settings, the algorithm could have the potential to expand patient access to chest radiograph diagnostics.

Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet

BACKGROUND

Magnetic resonance imaging (MRI) of the knee is the preferred method for diagnosing knee injuries. However, interpretation of knee MRI is time-intensive and subject to diagnostic error and variability. An automated system for interpreting knee MRI could prioritize high-risk patients and assist clinicians in making diagnoses. Deep learning methods, in being able to automatically learn layers of features, are well suited for modeling the complex relationships between medical images and their interpretations. In this study we developed a deep learning model for detecting general abnormalities and specific diagnoses (anterior cruciate ligament [ACL] tears and meniscal tears) on knee MRI exams. We then measured the effect of providing the model's predictions to clinical experts during interpretation.

METHODS AND FINDINGS

Our dataset consisted of 1,370 knee MRI exams performed at Stanford University Medical Center between January 1, 2001, and December 31, 2012 (mean age 38.0 years; 569 [41.5%] female patients). The majority vote of 3 musculoskeletal radiologists established reference standard labels on an internal validation set of 120 exams. We developed MRNet, a convolutional neural network for classifying MRI series and combined predictions from 3 series per exam using logistic regression. In detecting abnormalities, ACL tears, and meniscal tears, this model achieved area under the receiver operating characteristic curve (AUC) values of 0.937 (95% CI 0.895, 0.980), 0.965 (95% CI 0.938, 0.993), and 0.847 (95% CI 0.780, 0.914), respectively, on the internal validation set. We also obtained a public dataset of 917 exams with sagittal T1-weighted series and labels for ACL injury from Clinical Hospital Centre Rijeka, Croatia. On the external validation set of 183 exams, the MRNet trained on Stanford sagittal T2-weighted series achieved an AUC of 0.824 (95% CI 0.757, 0.892) in the detection of ACL injuries with no additional training, while an MRNet trained on the rest of the external data achieved an AUC of 0.911 (95% CI 0.864, 0.958). We additionally measured the specificity, sensitivity, and accuracy of 9 clinical experts (7 board-certified general radiologists and 2 orthopedic surgeons) on the internal validation set both with and without model assistance. Using a 2-sided Pearson's chi-squared test with adjustment for multiple comparisons, we found no significant differences between the performance of the model and that of unassisted general radiologists in detecting abnormalities. General radiologists achieved significantly higher sensitivity in detecting ACL tears (p-value = 0.002; q-value = 0.019) and significantly higher specificity in detecting meniscal tears (p-value = 0.003; q-value = 0.019). Using a 1-tailed t test on the change in performance metrics, we found that providing model predictions significantly increased clinical experts' specificity in identifying ACL tears (p-value < 0.001; q-value = 0.006). The primary limitations of our study include lack of surgical ground truth and the small size of the panel of clinical experts.

CONCLUSIONS

Our deep learning model can rapidly generate accurate clinical pathology classifications of knee MRI exams from both internal and external datasets. Moreover, our results support the assertion that deep learning models can improve the performance of clinical experts during medical imaging interpretation. Further research is needed to validate the model prospectively and to determine its utility in the clinical setting.

Autonomous Helicopter Tracking and Localization Using a Self-surveying Camera Array

A Self-surveying Camera Array (SSCA) is a vision-based local-area positioning system consisting of multiple ground-deployed cameras that are capable of self-surveying their extrinsic parameters while tracking and localizing a moving target. This paper presents the self-surveying algorithm being used to track a target helicopter in each camera frame and to localize the helicopter in an array-fixed frame. Three cameras are deployed independently in an arbitrary arrangement that allows each camera to view the helicopter's flight volume. The helicopter then flies an unplanned path that allows the cameras to calibrate the relative locations and orientations by utilizing a self-surveying algorithm that is extended from the well-known structure from motion algorithm and the bundle adjustment technique. This yields the cameras'extrinsic parameters enabling real-time helicopter positioning via triangulation. This paper also presents results from field trials, which verify the feasibility of the SSCA as a readily-deployable system applicable to helicopter tracking and localization. The results demonstrate that, compared to the differential GPS solution as true reference, the SSCA alone is capable of positioning the helicopter with meter-level accuracy. The SSCA has been integrated with onboard inertial sensors providing a reliable positioning system to enable successful autonomous hovering.

Simultaneous Localization and Mapping with Sparse Extended Information Filters

In this paper we describe a scalable algorithm for the simultaneous mapping and localization (SLAM) problem. SLAM is the problem of acquiring a map of a static environment with a mobile robot. The vast majority of SLAM algorithms are based on the extended Kalman filter (EKF). In this paper we advocate an algorithm that relies on the dual of the EKF, the extended information filter (EIF). We show that when represented in the information form, map posteriors are dominated by a small number of links that tie together nearby features in the map. This insight is developed into a sparse variant of the EIF, called the sparse extended information filter (SEIF). SEIFs represent maps by graphical networks of features that are locally interconnected, where links represent relative information between pairs of nearby features, as well as information about the robot’s pose relative to the map. We show that all essential update equations in SEIFs can be executed in constant time, irrespective of the size of the map. We also provide empirical results obtained for a benchmark data set collected in an outdoor environment, and using a multi-robot mapping simulation.

Unsupervised Deep Learning Applied to Breast Density Segmentation and Mammographic Risk Scoring

Mammographic risk scoring has commonly been automated by extracting a set of handcrafted features from mammograms, and relating the responses directly or indirectly to breast cancer risk. We present a method that learns a feature hierarchy from unlabeled data. When the learned features are used as the input to a simple classifier, two different tasks can be addressed: i) breast density segmentation, and ii) scoring of mammographic texture. The proposed model learns features at multiple scales. To control the models capacity a novel sparsity regularizer is introduced that incorporates both lifetime and population sparsity. We evaluated our method on three different clinical datasets. Our state-of-the-art results show that the learned breast density scores have a very strong positive relationship with manual ones, and that the learned texture scores are predictive of breast cancer. The model is easy to apply and generalizes to many other segmentation and scoring problems.

The Stanford LittleDog: A learning and rapid replanning approach to quadruped locomotion

Legged robots have the potential to navigate a wide variety of terrain that is inaccessible to wheeled vehicles. In this paper we consider the planning and control tasks of navigating a quadruped robot over challenging terrain, including terrain that it has not seen until run-time. We present a software architecture that makes use of both static and dynamic gaits, as well as specialized dynamic maneuvers, to accomplish this task. Throughout the paper we highlight two themes that have been central to our approach: (1) the prevalent use of learning algorithms, and (2) a focus on rapid recovery and replanning techniques; we present several novel methods and algorithms that we developed for the quadruped and that illustrate these two themes. We evaluate the performance of these different methods, and also present and discuss the performance of our system on the official Learning Locomotion tests.

Autonomous Helicopter Aerobatics through Apprenticeship Learning

Autonomous helicopter flight is widely regarded to be a highly challenging control problem. Despite this fact, human experts can reliably fly helicopters through a wide range of maneuvers, including aerobatic maneuvers at the edge of the helicopter’s capabilities. We present apprenticeship learning algorithms, which leverage expert demonstrations to efficiently learn good controllers for tasks being demonstrated by an expert. These apprenticeship learning algorithms have enabled us to significantly extend the state of the art in autonomous helicopter aerobatics. Our experimental results include the first autonomous execution of a wide range of maneuvers, including but not limited to in-place flips, in-place rolls, loops and hurricanes, and even auto-rotation landings, chaos and tic-tocs, which only exceptional human pilots can perform. Our results also include complete airshows, which require autonomous transitions between many of these maneuvers. Our controllers perform as well as, and often even better than, our expert pilot.

Make3D: learning 3D scene structure from a single still image

We consider the problem of estimating detailed 3D structure from a single still image of an unstructured environment. Our goal is to create 3D models that are both quantitatively accurate as well as visually pleasing. For each small homogeneous patch in the image, we use a Markov Random Field (MRF) to infer a set of "plane parameters" that capture both the 3D location and 3D orientation of the patch. The MRF, trained via supervised learning, models both image depth cues as well as the relationships between different parts of the image. Other than assuming that the environment is made up of a number of small planes, our model makes no explicit assumptions about the structure of the scene; this enables the algorithm to capture much more detailed 3D structure than does prior art and also give a much richer experience in the 3D flythroughs created using image-based rendering, even for scenes with significant nonvertical structure. Using this approach, we have created qualitatively correct 3D models for 64.9 percent of 588 images downloaded from the Internet. We have also extended our model to produce large-scale 3D models from a few images.

Robotic Grasping of Novel Objects using Vision

We consider the problem of grasping novel objects, specifically objects that are being seen for the first time through vision. Grasping a previously unknown object, one for which a 3-d model is not available, is a challenging problem. Furthermore, even if given a model, one still has to decide where to grasp the object. We present a learning algorithm that neither requires nor tries to build a 3-d model of the object. Given two (or more) images of an object, our algorithm attempts to identify a few points in each image corresponding to good locations at which to grasp the object. This sparse set of points is then triangulated to obtain a 3-d location at which to attempt a grasp. This is in contrast to standard dense stereo, which tries to triangulate every single point in an image (and often fails to return a good 3-d model). Our algorithm for identifying grasp locations from an image is trained by means of supervised learning, using synthetic images for the training set. We demonstrate this approach on two robotic manipulation platforms. Our algorithm successfully grasps a wide variety of objects, such as plates, tape rolls, jugs, cellphones, keys, screwdrivers, staplers, a thick coil of wire, a strangely shaped power horn and others, none of which were seen in the training set. We also apply our method to the task of unloading items from dishwashers.

Cloning and characterization of UROC28, a novel gene overexpressed in prostate, breast, and bladder cancers

A novel gene, designated UROC28, was identified by an agarose gel-based differential display technique, and it was found to be up-regulated in prostate, breast, and bladder cancer. Expression of UROC28 was also up-regulated in prostate cancer cells in the presence of androgens as demonstrated by relative quantitative reverse transcription-PCR. The elevated expression of this gene was observed to increase in surgically removed tissues concomitantly with rising Gleason grade and was most elevated in metastatic tissue. UROC28 protein was detected in serum by Western slot blot analyses, and a significant higher UROC28 protein level was found in sera of prostate cancer individuals compared with normal individuals and individuals with nonmalignant prostatic hyperplasia. Northern analyses in normal tissues showed that the UROC28 cDNA hybridizes to two mRNAs at about 2.1 and 2.5 kb. Nucleic acid sequence analyses indicated that these two alternatively spliced mRNA variants differ only at the 3' untranslated region. These two mRNAs encode the same protein with 135 amino acids. Bioinformation analyses suggest that there is a possible transmembrane domain from amino acid aa34 to aa50, three protein kinase-C phosphorylation sites at aa62 (SQK), aa89 (TMK), and aa94 (SMK), and one myristylation site at aa118 (GLECCL). Genomic Southern hybridization and chromosomal mapping demonstrated that UROC28 is encoded by a single copy of gene at chromosome 6q23-24. In situ hybridization and immunohistochemistry experiments further confirmed up-regulation of this gene in prostate and breast cancers with the expression localizing to the glandular epithelium. This gene did not demonstrate increased expression in lung and colon cancer tissues.

Phenylbutyrate-induced apoptosis and differential expression of Bcl-2, Bax, p53 and Fas in human prostate cancer cell lines

OBJECTIVE

To assess the mechanisms of action of phenylbutyrate (PB), an investigational chemotherapeutic agent for prostate cancer (PCa), in apoptosis induction in PCa cell lines in vitro.

STUDY DESIGN

We analyzed the differential expression of different apoptosis modulators, Bcl-2, Bax, p53 and Fas, for their potential role in PB-induced apoptosis using relative quantitative flow cytometry (FCM). Both androgen-dependent (LNCaP) and androgen-independent (C-4-2, PC-3-PF and DU145) human PCa cell lines were examined.

RESULTS

PB induced apoptosis in PCa cell lines in a dose-dependent manner. Fifty percent apoptosis could be induced by 5-10 mM PB. Bcl-2 was down-regulated 30-75% and the Bax:Bcl-2 ratio elevated in apoptotic PCa cell lines regardless of their androgen dependency or p53 status. FCM revealed a heterogeneous stimulatory effect on the expression of Bax and Bcl-2 in PC3-PF cells at 0.5-2.5 mM PB. In a p53-positive cell line (DU145), p53 was repressed by 70% and Fas elevated sixfold with 10 mM PB.

CONCLUSION

Our data show that PB-induced PCa apoptosis is associated with the relative repression of Bcl-2 and with up-regulation of Bax and Fas proteins and that this PB-induced apoptosis is independent of p53 and androgen-dependency status of PCa cell lines.

A novel transcription factor, ELF5, belongs to the ELF subfamily of ETS genes and maps to human chromosome 11p13-15, a region subject to LOH and rearrangement in human carcinoma cell lines

The ETS transcription factors are a large family implicated in the control of cellular proliferation and tumorigenesis. In addition, chromosomal translocations involving ETS family members are associated with a range of different human cancers. Given the extensive involvement of ETS factors in tumorigenesis, it becomes important to identify any additional ETS genes that may also play oncogenic roles. We identify a novel gene, ELF5, that appears to belong to the ELF (E74-like-factor) subfamily of the ETS transcription factor family, based upon similarity within the 'ETS domain'. ELF5 displays a similar, but more restricted, expression pattern to that of the newly isolated epithelium-specific ETS gene, ELF3. Unlike most other ETS family members, ELF5 is not expressed in hematopoietic compartments, but is restricted to organs such as lung, stomach, kidney, prostate, bladder and mammary gland. ELF5 is localized to human chromosome 11p13-15, a region that frequently undergoes loss of heterozygosity (LOH) in several types of carcinoma, including those of breast, kidney and prostate. We find that ELF5 expression is not detectable in a number of carcinoma cell lines, some of which display loss or rearrangement of an ELF5 allele. Similar to other ETS family members, ELF5 displays specific binding to DNA sequences containing a GGAA-core. In addition, ELF5 is able to transactivate through these ETS sequences, present upstream from a minimal promoter. Our data suggest that ELF5 may play roles in mammary, lung, prostate and/or kidney function, and possibly also in tumorigenesis.

Interstitial Dup(1p) with findings of Kabuki make-up syndrome

We describe a male patient with interstitial duplication of the short arm of chromosome 1 with breakpoints involving 1p13.1 and 1p22.1. The patient presented with some clinical findings of Kabuki make-up syndrome (KMS), including mental retardation, small head, eversion of the lateral part of lower eyelids, epicanthic folds, lateral flare of the eyebrows, short columella, and persistent fetal finger pads. This cytogenetic finding may provide clues for gene mapping of the syndrome.

A novel epithelial-expressed ETS gene, ELF3: human and murine cDNA sequences, murine genomic organization, human mapping to 1q32.2 and expression in tissues and cancer

The ETS family of genes are implicated in cancers such as Ewings sarcoma, acute myeloid leukemia and chronic myelomonocytic leukemia. Further, they have important functions in embryonic development. Hence, identification and characterization of members of this family are important. We identify a novel ETS family member, ELF3, and report its human and murine cDNA sequences. The mouse cDNA has an alternatively spliced transcript with an extra 60 bp inserted. Hence we present the organization of the murine Elf3 gene together with its exon/intron structure. This gene consists of 9 exons and 8 introns spanning 4.8 kb. ELF3 binds and transactivates ETS sequences and interestingly also shows the ability to bind a GGAT-like purine core, a preferential ETS1/ETS2 type binding site. The expression of ELF3, unlike most other ETS family members, is absent in hematopoietic cells and hematopoietic organs in humans and mice. Intriguingly, the gene is specifically expressed in cell lines of epithelial origin and in organs such as lung, stomach, intestine, kidney that have specialized epithelial cells. We localize the human gene to 1q32.2, a region that is amplified in epithelial tumors of the breast, lung and prostate. Finally, we show that ELF3 expression is increased in a lung carcinoma and adenocarcinoma, as compared to normal tissue. ELF3 is also expressed in cell lines derived from lung cancers. These results suggest that this novel ETS gene may be involved in lung tumorigenesis.

Umbilical cord transforming growth factor-beta 3: isolation, comparison with recombinant TGF-beta 3 and cellular localization

The transforming growth factor beta (TGF-beta) family of growth modulators play critical roles in tissue development and maintenance. Recent data suggest that individual TGF-beta isoforms (TGF-beta 1, -beta 2 and -beta 3) have overlapping yet distinct biological actions and target cell specificities, both in developing and adult tissues. The TGF-beta 3 isoform was purified to homogeneity from both natural and recombinant sources and characterized by laser desorption mass spectrometry, by protein sequencing, by amino acid analysis and by biological activity. TGF-beta 3 was the major TGF-beta isoform in umbilical cord (230 ng/g), and was physically and biologically indistinguishable from recombinant TGF-beta 3 and from the tumor growth inhibitory (TGI) protein found in umbilical cord. Immunohistochemistry using antipeptide TGF-beta 3 specific antibody showed TGF-beta 3 localization in perivascular smooth muscle.

Loss of heterozygosity on the short arm of chromosome 3 in nasopharyngeal carcinoma

A consistent loss of constitutional heterozygosity within a specific chromosome locus in a tumor type is suggestive of a tumor suppressor gene important in the genesis of that tumor. We studied whether such genetic alterations are involved, in the development of nasopharyngeal carcinoma (NPC). Tumor and matched blood leukocytes DNA from eleven Hong Kong Chinese patients with primary NPC stages I to IV were subjected to restriction fragment length polymorphism (RFLP) analysis using chromosome 3-specific polymorphic probes. Such probes are assigned to chromosomal region 3p25 (RAF-1), 3p24-22.1 (ERBA beta), 3p21 (DNF15S2), 3p14 (D3S3), and 3q12 (D3S1). The breakpoint varied among tumors, ranging in extent from 3p21-14. However, 100% frequency of complete loss of heterozygosity was observed at two chromosomal loci: RAF-1 locus (ten of ten cases at 3p25) and D3S3 locus (nine of nine cases at 3p14), in all evaluable NPC patients, suggesting the presence of putative tumor suppressor gene(s) within or close to these defined regions. The observed consistent deletion of alleles on the short arm of chromosome 3 in the NPC cases, which is in line with our previously reported and present cytogenetic findings, may represent a critical event in the multistep genesis of NPC. The present report also identifies defined loci for linkage studies on NPC families.

Integration of HPV-16 DNA in cervical carcinoma cell line CC3/CUHK3 and its xenografts

A recently established cell line, designated CC3/CUHK3, derived from a squamous cell carcinoma of the uterine cervix of a Hong Kong Chinese patient was investigated for its association with human papillomavirus (HPV) by Southern blot hybridization studies. A "grafted tumour" or xenograft of CC3/CUHK3 has also been successfully established in athymic mice, which showed similar histology to that of the original biopsy. The epithelial nature of the xenografts, like the original biopsy. The epithelial nature of the xenografts, like the original tumour specimen, was also confirmed by transmission electron microscopy which demonstrated the presence of desmosomes and tonofilaments. Moreover transmission electron microscopic examination revealed no HPV particle in either the tumour biopsy or the xenograft tissues. Analysis of the DNA samples extracted from the cervical cancer cell line and the xenograft tissues derived from it showed the presence of HPV type 16 DNA. No DNA sequence related to HPV type 6, 11, or 18 was demonstrated. The viral DNA was found to be integrated into the cellular genome at multiple sites as single copy or head-to-tail tandem repeats. Deletion or rearrangement of the HPV DNA had probably occurred on or subsequent to integration. Viral sequence deletion has also been observed in the grafted tumours derived from CC3/CUHK3.

Nontraumatic prehospital cardiac arrest ages 1 to 39 years

Clinical and autopsy records were retrospectively reviewed for 105 patients between the ages of 1 and 39 years who came in to the emergency department with nontraumatic cardiac arrest. There were 65 male (62%) and 40 female patients (38%). Forty-eight percent of the patients were resuscitated. Long-term survival rate was 23%. The most common presenting rhythm was ventricular fibrillation (45%). Cardiac diseases constituted the most common cause of arrest (38%). Atherosclerotic coronary artery disease represented 50% of all cardiac causes. The second most common etiology was overdose or toxic exposure (21%). Witnessed arrest and an etiology of primary cardiac dysrhythmia for arrest were statistically significant factors related to favorable outcome. Asystole as the initial cardiac rhythm was a negative prognostic indicator. Age, sex, race, bystander cardiopulmonary resuscitation, and paramedic response time were not significant prognostic factors for long-term survival.

The optic nerve of the cat: appearance and loss of axons during normal development

The number of axons in the optic nerve has been estimated in cats ranging in age from mid-gestation to adulthood. At mid-gestation the number of axons present in the nerve (218,000) already exceeded adult levels. The number of axons, nevertheless, more than doubled over the next 10-20 days, reaching a maximum of 450,000-483,000. In the last prenatal week and the first postnatal week, the number of axons declined rapidly, stabilizing at adult levels 2-3 weeks after birth. Myelination began just before birth and reached adult levels (over 95% of axons myelinated) 6-8 weeks after birth. Several mechanisms which may underlie the loss of axons are discussed.