Inter operator variability of machine learning researchers predicting all-cause mortality in patients admitted to intensive care unit

Background

The number of publications using machine learning (ML) to predict cardiovascular outcomes and identify clusters of patients at greater risk has risen dramatically in recent years. However, research papers which use ML often fail to provide sufficient information about their algorithms to enable results to be replicated by others in the same or different datasets.

Aim

To test the reproducibility of results from ML algorithms given three different levels of information commonly found in publications: model type alone, a description of the model, and complete algorithm.

Methods

MIMIC-III is a healthcare dataset comprising detailed information from over 60,000 intensive care unit (ICU) admissions from the Beth Israel Deaconess Medical Centre between 2001 and 2012. Access is available to everyone pending approval and completion of a short training course.

Using this dataset, three models for predicting all-cause in-hospital mortality were created, two from a PhD student working in ML, and one from an existing research paper which used the same dataset and provided complete model information. A second researcher (a PhD student in ML and cardiology) was given the same dataset and was tasked with reproducing their results. Initially, this second researcher was told what type of model was created in each case, followed by a brief description of the algorithms. Finally, the complete algorithms from each participant were provided. In all three scenarios, recreated models were compared to original models using Area Under the Receiver Operating Characteristic Curve (AUC).

Results

After excluding those younger than 18 years and events with missing or invalid entries, 21,139 ICU admissions remained from 18,094 patients between 2001 and 2012, including 2,797 in-hospital deaths. Three models were produced: two Recurrent Neural Networks (RNNs) which differed significantly in internal weights and variables, and a Boosted Tree Classifier (BTC). The AUC of the first reproduced RNN matched that of the original RNN (Figure 1), however the second RNN and the BTC could not be reproduced given model type alone. As more information was provided about these algorithms, the results from the reproduced models matched the original results more closely.

Conclusions

In order to create clinically useful ML tools with results that are reproducible and consistent, it is vital that researchers share enough detail about their models. Model type alone is not enough to guarantee reproducibility. Although some models can be recreated with limited information, this is not always the case, and the best results are found when the complete algorithm is shared. These findings have huge relevance when trying to apply ML in clinical practice.

Funding Acknowledgement

Type of funding sources: None.

A comparison of machine learning models for predicting rehospitalisation and death after a first hospitalisation with heart failure

Background

Many machine learning models exist, including Multilayer Perceptron (MLP), Random Forest algorithm (RF), Support Vector Machine (SVM), and Gradient Boosted Machine (GBM), but their value for predicting outcome in patients with heart failure has not been compared.

Aim

To predict rehospitalisation (all-cause) and death (all-cause) at 1-, 3- and 12 months after discharge from a first hospitalisation for heart failure using four machine learning models.

Methods

The National Health Service Greater Glasgow and Clyde Health Board serves a population of ∼1.1 million. We obtained de-identified administrative data, including investigations, diagnosis and prescriptions, linked to hospital admissions and deaths for anyone with a diagnosis of vascular disease or heart failure or prescribed loop diuretics, statins or neuro-endocrine antagonists at any time between 1st January 2010 and 1st June 2018. Patients who were under 18 or had no prior hospitalisation for heart failure were excluded. Four ML algorithms using 46 variables were applied.

Results

Of 360,000 people who met the above criteria between 2010–2018, 6,372 had a hospitalisation for heart failure prior to 1st January 2010 and 8,304 had a first hospitalisation for heart failure thereafter. Between 2010 and 2018 there were 3,086 re-hospitalisations over 24 hours and 3,706 patients died, with 5,070 patients experiencing the composite outcome.

GBM and RF consistently outperformed MLP and SVM when comparing AUC, sensitivity and specificity combined, with GBM performing best in all scenarios. Since GBM and RF are both tree-based models, and with SVM and MLP regularly reporting very poor sensitivity or specificity despite a similar AUC to the others, this suggests that SVM and MLP may be suffering from overfitting and might perform better in larger data-sets.

Both GBM and RF work by ordering variables, so the final model can be used to determine the most important prediction variables. Age, number of times a blood sample was taken out of hospital, length of stay, social deprivation index and haemoglobin concentration consistently ranked amongst the most important variables. Models predicted all 1-month events better than later events.

Conclusions

Some, but not all, ML models applied to this data-set predicted rehospitalisation and death with great accuracy for up to 3 months after a first hospitalisation for heart failure. The models identified several important prognostic variables that are currently seldom collected in clinical research registries but perhaps should be.

Funding Acknowledgement

Type of funding source: Public grant(s) – National budget only. Main funding source(s): Medical Research Council

Population prevalence of atrial fibrillation and QRS prolongation and mortality in people with suspected heart failure

Background

The 12-lead electrocardiogram (ECG) is an essential tool for the diagnosis and management of heart failure (HF). There are few population-based studies on the prevalence and prognostic implications of ECG abnormalities in patients with HF. There are also no robust diagnostic criteria for HF. We explored these issues in a large administrative database.

Methods

The National Health Service Greater Glasgow and Clyde Health Board serves a population of ∼1.1 million. We obtained de-identified administrative data, including investigations, diagnosis and prescriptions, linked to hospital admissions and deaths, for anyone with a diagnosis of vascular disease or HF or prescribed loop diuretics (LD) or neuro-endocrine antagonists between 1st January 2012 and 1st April 2018. People were classified into 5 exclusive groups: a) prevalent HF; b) incident (or latent) HF with onset during follow-up; c) people taking LDs but with no diagnosis of HF at any time; d) new prescription of LDs during follow-up but with no diagnosis of HF at any time and d) people to whom none of the above applied. ECGs were classified according to heart rhythm (sinus, AF or flutter or pacemaker/CRT/ICD) and QRS duration <100ms, 100–130ms or >130ms. Follow-up for each group started on 1st of January 2012 (prior to the onset of the classifying event for incident groups).

Results

During the observation period, of 316,350 people included, 158,421 had a recorded ECG (mean of 3.2 per person with an ECG), including 8,768 prevalent and 13,195 incident cases of HF. Of those who never got a diagnosis of heart failure, 11,508 were receiving and a further 14,633 were newly prescribed LD during follow-up. There were 110,317 people who did not fall into the above groups, of whom 51,089 were aged ≥60 years. A higher proportion of those who were prescribed loop diuretics without a diagnosis of heart failure were women. A similar proportion of those with heart failure and those prescribed diuretics alone had, lung disease and renal dysfunction but patients with heart failure had more ischaemic heart disease, more often had a heart rhythm other than sinus and had longer QRS duration. By three years, 8,816 people (11%) had died, of whom 2,919 (33%) had a diagnosis of heart failure and 2,694 (31%) had been prescribed LD without a diagnosis of HF, together accounting for 64% of all deaths. Patients with a rhythm other than sinus had a worse prognosis in all 5 groups of patients. QRS duration >130ms was associated with a worse prognosis in patients with HF or taking LD.

Conclusions

Most people with cardiovascular disease who die will first develop HF or be prescribed a LD (indicating possible undiagnosed HF). Patient characteristics of those prescribed LD suggest that many might have HF with preserved left ventricular ejection fraction (HFpEF).

Baseline characteristics and HR

Funding Acknowledgement

Type of funding source: None

Cognitive Dysfunction Among Primi gravidae Attending an Ante Natal Clinic in Kano, Northwest Nigeria

Cognitive deficits among pregnant women have been reported, though the evidence is equivocal. Cognitive dysfunction during pregnancy may have negative consequences on maternal and child health.  Yet, very little is known about cognitive function of pregnant women in general and primi gravidae in particular in the area under study. This study aimed to evaluate cognitive function among primi gravidae. About 120 primi gravidae were studied in a large urban hospital in Kano. Mini-Mental State Examination (MMSE) was used to evaluate cognitive function; socio-demographic and clinical data were obtained. Data were processed using IBM SPSS statistics version 20.0. Significant values of P were those < 0.05. The median age of the primi gravidae was 20.0 (3) years; 80 % of them had secondary school education, 88.3 % were in their third trimester of pregnancy and 42.5 % did not have any medical complaints at the time of presentation. The median MMSE score of the primi gravidae was 22.00 (19.0), indicating mild cognitive impairment. Majority of the women (88.3%) had either mild (58.3%) or severe (30%) cognitive dysfunction. The cognitive dysfunction was influenced by level of education (X2 = 11.961, P = 0.003) and type of presenting complaints (X2 = 13.514, P = 0.036). There was significant association between the mild cognitive impairment and level of education (X2 = 11.426, P = 0.022).   This study concluded that the primi gravidae had mild cognitive impairment; with majority (88.3%) of them having cognitive dysfunction, which was significantly associated with level of educational attainment and was influenced by the level of education and presenting complaints.

Retropharyngeal abscess after blunt trauma in an adolescent

Retropharyngeal abscess is an unusual infection in teenagers, and it is rarely associated with blunt trauma. We present the case of 17-year-old male who developed signs and symptoms of retropharyngeal abscess soon after being struck by a car. The case illustrates that retropharyngeal abscess should be considered, even in an adolescent, if the patient has fever, neck stiffness, and sore throat with symptoms out of proportion to the oropharyngeal findings.

Method for assessing food intakes in terms of servings based on food guidance

We describe a method for comparing food intakes reported in surveys with recommendations given in the US Department of Agriculture's food guide pyramid. Such comparison was previously difficult because many foods are eaten as mixtures that cannot be categorized into pyramid food groups and because food intake data and food recommendations are expressed in different units of measurement. We developed a recipe file showing ingredients at multiple levels of breakdown and used it to disaggregate food mixtures into their ingredients so that they could be placed into pyramid food groups. Food-specific weight consistent with the pyramid definitions for serving sizes were derived from the food code book of the 1989-1991 Continuing Survey of Food Intakes by Individuals and used to develop a serving-weight file. Data collected from 8181 adults aged > or = 20 y in the survey were used to estimate the mean number of servings from pyramid food groups and percentages of the population meeting pyramid recommendations. The mean numbers of servings were close to the minimum recommendations for the grain, vegetable, and meat groups and well below the minimum recommendations for the fruit and dairy groups. large proportions of adults (43-72%) failed to meet the dietary recommendations. Our method has several nutrition monitoring and nutrition education applications.

The students for peace project: a comprehensive violence-prevention program for middle school students

Students for Peace is a three-year project (October 1993- September 1996) designed to evaluate a comprehensive, school-based intervention that seeks to prevent violence among sixth-, seventh-, and eighth-grade students in a large urban school district in Texas. This study examines the hypothesis that students exposed to a two-year multiple-component intervention will reduce aggressive behavior compared to students who receive the district's "usual care" of violence prevention activities. Students for Peace is based largely on Social Learning. Theory (SLT), which addresses both the psychosocial dynamics underlying health behavior and the methods of promoting behavior change, while emphasizing cognitive processes and their effect on behavior. SLT explains human behavior in terms of a model in which three factors-behavior, social-environmental influences, and personal factors (such as personality, perceptions and expectations, and affect)-all interact. Theoretically, an individual's behavior is uniquely determined by a combination of these factors; thus, these factors become the elements for intervention strategies. The intervention program includes four main components: (1) modification of the school environment, (2) a violence-prevention curriculum, (3) peer leadership, and (4) parent education. Students for Peace is using a nested cross-sectional and cohort design in which school is the unit of design, allocation, and analysis. Eight schools, four intervention and four control, are participating. In May 1994, a questionnaire was administered to all students in school the day of the survey. A posttest evaluation was taken in the spring of 1995 and will be followed by a final posttest in spring 1996. A total of 8,865 students responded to the baseline survey. Nearly all variables indicated comparability between treatment and control conditions. As a population, Students for Peace participants are largely Hispanic (65%) or African American (19%). Violence-related variables indicated 30-day fighting prevalence, 23%; 12-month prevalence of injuries due to fighting, 14%; 30-day hand-gun carrying prevalence, 11%; 30-day prevalence of taunts and threats at school, 27%, and threats going to and from school, 26%. Overall, the data from Year 1 activities indicate a population in need of violence-prevention intervention. The challenge is to mold existing district resources into a theoretically sound program of interventions. If that program is found effective, the district will already have the necessary documentation, personnel, and skills for broader dissemination.

Prevention of youth violence: rationale and characteristics of 15 evaluation projects

Interpersonal violence is a major cause of injury, disability, and death, especially among youth. Evaluations of 15 youth violence-prevention projects are under way. Public health is concerned about health problems that need to be addressed via collective action. Public health involvement in addressing interpersonal violence among youths brings an emphasis on primary prevention, a systematic and scientific process, and integrative leadership. Few quantitative evaluations of violence-prevention projects have been done. The interventions are scientifically based and use a spectrum of strategies. Individually oriented strategies are more common than those directed toward peers, families, schools, or communities. Each project has a rigorous evaluation design. Twelve are randomized. Sample sizes range from 180 to 10,000. Participants range in age from 5 to 18 years, although most are in the middle-school years (11-14 years). At baseline, intervention and comparison groups are similar. Baseline data demonstrate high frequency of violent behavior, weapon carrying, and exposure to violence among the youthful participants. Field intervention and evaluation research is difficult and expensive. Difficulties encompass organizational programatic, and scientific issues; these difficulties reduce scientific interest and financial support for projects such as these. Public health has an important role to play in reducing violence. These projects will make important contributions to that task.

Attributional bias and reactive aggression

This article looks at a cognitive behavioral intervention designed to reduce minority youths' (Latino and African-American boys) levels of reactive peer-directed aggression. The BrainPower Program trains aggressive boys to recognize accidental causation in ambiguous interactions with peers. The objective of this research is to evaluate the effectiveness of this attribution retraining program in reducing levels of reactive, peer-directed aggression. This research hypothesizes that aggressive young boys' tendency to attribute hostile intentions to others in ambiguous social interactions causes display of inappropriate, peer-directed aggression. A reduction in attributional bias should produce a decrease in reactive physical and verbal aggression directed toward peers. A 12-session, attributional intervention has been designed to reduce aggressive students' tendency to infer hostile intentions in peers following ambiguous peer provocations. The program trains boys to (1) accurately perceive and categorize the available social cues in interactions with peers, (2) attribute negative outcomes of ambiguous causality to accidental or uncontrollable causes, and (3) generate behaviors appropriate to these retrained attributions. African-American and Latino male elementary-school students (N = 384), in grades four-six, served as subjects in one of three groups: experimental attribution retraining program, attention training, and no-attention control group. Three broad categories of outcome data were collected: teacher and administrator reports of behavior, independent observations of behavior, and self-reports from participating students. Process measures to assess implementation fidelity include videotaped training sessions, observations of intervention sessions, student attendance records, and weekly team meetings. The baseline data indicated that students who were evenly distributed across the four sites were not significantly different on the baseline indicators: student cognitions, teacher perceptions of behavior, and student suspension rates. Substantial evidence has shown that aggressive boys tend to attribute hostile intentions to peers, often resulting in inappropriate retaliatory aggression. The BrainPower Program was designed to determine whether psychoeducational strategies in a school context are effective in reducing attributional bias and whether such reductions significantly reduce aggressive behavior.

Fruit and vegetable intakes of children and adolescents in the United States

OBJECTIVES

To identify the ways in which fruits and vegetables are consumed by children, to provide estimates of their intakes compared with recommendations, and to estimate the percentage of children meeting those recommendations.

DESIGN

We examined 3 days of dietary data from respondents in the US Department of Agriculture's 1989-1991 Continuing Survey of Food Intakes by Individuals. All foods reported in the survey were disaggregated into their component ingredients; all fruit and vegetable ingredients were assigned specific weights to correspond with a serving as defined by current dietary guidance materials; and the number of servings of each fruit and vegetable was tallied.

PARTICIPANTS

A total of 3148 children and adolescents aged 2 to 18 years in the 48 conterminous United States.

MAIN OUTCOME MEASURES

Percentages of fruit and vegetable servings consumed in various forms, mean number of servings consumed per day, and percentage of persons meeting various recommendations by sex/age, race/ethnicity, and household income.

RESULTS

Nearly one quarter of all vegetables consumed by children and adolescents were french fries. Their intakes of all fruits and of dark green and/or deep yellow vegetables were very low compared with recommendations. Only one in five children consumed five or more servings of fruits and vegetables per day.

CONCLUSION

Pediatricians should encourage the consumption of fruits and vegetables, especially dark green and deep yellow vegetables, by children.

US adults' fruit and vegetable intakes, 1989 to 1991: a revised baseline for the Healthy People 2000 objective

OBJECTIVES

This study provides revised baseline data for the Healthy People 2000 objective related to fruit and vegetable intakes, accounting for fruits and vegetable intakes, accounting for fruits and vegetables from all sources and measuring servings in a manner consistent with current dietary guidance.

METHODS

Dietary data from 8181 adults in the US Department of Agriculture's 1989-1991 Continuing Surveys of Food Intakes by Individuals were examined. All foods were disaggregated into their component ingredients; all fruit and vegetable ingredients were assigned specific weights to correspond to a serving as defined by current dietary guidance materials; and the number of servings was tallied.

RESULTS

While mean intakes of fruits and vegetables--4.3 servings per day--were not far from the Year 2000 objective, only 32% of American adults' intakes met the objective. When more stringent standards were set either to compensate for higher calorie levels or to achieve the balance between fruits and vegetables suggested in current guidance, only 24% and 12%, respectively, met the recommendations.

CONCLUSIONS

These results suggest a need to develop strategies for overcoming barriers to eating fruits and vegetables.

A microscopic incident in a monumental struggle: Huxley and antibiosis in 1875

In 1875 T. H. Huxley discovered that a secretion from the mouldpenicillium glaucumhad an ability, unconnected with oxygen deprivation, to inhibit bacterial growth. He recorded his observations in his notebooks and in a single letter to John Tyndall, who at that time was a friend of Lister and a correspondent of Pasteur. Neither Huxley nor Tyndall looked for an explanation of this phenomenon, and neither told anyone else about it.