Performance metrics for models designed to predict treatment effect

BACKGROUND

Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs).

METHODS

Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest.

RESULTS

As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics".

CONCLUSION

The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Does blood flow limit acute hypoxia performance in larval zebrafish (Danio rerio)?

Oxygen uptake (ṀO₂) in larval zebrafish prior to maturation of the gill relies on cutaneous O₂ transfer. Under normoxic conditions, rates of cutaneous O₂ transfer are unaffected by haemoglobin availability but are diminished in fish lacking a functional circulatory system, suggesting that internal convection is critically involved in setting the resting ṀO₂ in zebrafish larvae, even when relying on cutaneous O₂ transfer. The reliance of ṀO₂ on blood circulation led to the first objective of the current study, to determine whether loss of internal convection would reduce acute hypoxia performance (as determined by measuring critical PO₂; P_crit) in larval zebrafish under conditions of moderate hypoxia (PO₂ = 55 mmHg) at 28.5 and 34 °C. Internal convection was eliminated by preventing development of blood vessels using morpholino knockdown of vascular endothelial growth factor (VEGF); these fish are termed VEGF morphants. Breathing frequency (f_V) and heart rate (f_H) also were measured (at 28.5 °C) to determine whether any detriment in performance might be linked to cardiorespiratory dysfunction. Although ṀO₂ was reduced in the VEGF morphants, there was no significant effect on P_crit at 28.5 °C. Raising temperature to 34 °C resulted in the VEGF morphants exhibiting a higher P_crit than the shams, suggesting an impairment of hypoxia tolerance in the morphants at the higher temperature. The usual robust increase in f_V during hypoxia was absent or attenuated in VEGF morphants at 4 and 5 days post fertilization (dpf), respectively. Resting f_H was reduced in the VEGF morphants and unlike the sham fish, the morphants did not exhibit hypoxic tachycardia at 4 or 5 dpf. The number of cutaneous neuroepithelial cells (presumptive O₂ chemoreceptors) was significantly higher in the VEGF morphants and thus the cardiorespiratory impairment in the morphants during hypoxia was unlikely related to inadequate peripheral O₂ sensing.

Folate and related micronutrients, folate-metabolising genes and risk of ovarian cancer

BACKGROUND/OBJECTIVE

Folates are essential for DNA synthesis and methylation, and thus may have a role in carcinogenesis. Limited evidence suggests folate-containing foods might protect against some cancers and may partially mitigate the increased risk of breast cancer associated with alcohol intake, but there is little information regarding ovarian cancer. Our aim was to evaluate the role of folate and related micronutrients, polymorphisms in key folate-metabolising genes and environmental factors in ovarian carcinogenesis.

SUBJECTS/METHODS

Participants in the Australian Ovarian Cancer Study (1363 cases, 1414 controls) self-completed risk factor and food-frequency questionnaires. DNA samples (1638 cases, 1278 controls) were genotyped for 49 tag single-nucleotide polymorphisms (SNPs) in the methylene tetrahydrofolate reductase (MTHFR), methionine synthase (MTR) and MTR reductase (MTRR) genes. Logistic regression models were used to generate adjusted odds ratios and 95% confidence intervals.

RESULTS

We saw no overall association between the intake of folate, B vitamins or other methyl donors and ovarian cancer risk, although increasing folate from foods was associated with reduced risk among current smokers (P(trend)=0.03) and folic acid intake was associated with borderline significant increased risks among women who consumed ≥1 standard alcoholic drinks/day (odds ratio (OR)=1.64; 95% confidence interval (CI) 1.05-2.54, P(trend)=0.05). Two SNPs (rs7365052, rs7526063) showed borderline significant inverse associations with ovarian cancer risk; both had very low minor allele frequencies. There was little evidence for interaction between genotype and micronutrient intake or for variation between different histological subtypes of ovarian cancer.

CONCLUSIONS

Our data provide little evidence to support a protective role for folate in ovarian carcinogenesis but suggest further evaluation of the joint effects of folic acid and alcohol is warranted.

Validation of a FFQ to estimate the intake of PUFA using plasma phospholipid fatty acids and weighed foods records

Due to the growing knowledge about the role of specific fatty acids in health and disease, dietary intake measurements of individual fatty acids or classes of fatty acids are becoming increasingly important. The objective of this study was to evaluate the ability of the Nambour FFQ to estimate intakes of specific fatty acids, particularly PUFA. The study population was a sub-sample of adult participants in a randomised controlled trial of beta-carotene and sunscreen in the prevention of skin cancer (n 43). Dietary intake was assessed by a self-administered FFQ and a weighed food record (WFR). Non-fasting blood samples were collected and analysed for plasma phospholipid fatty acids. Median intakes on the FFQ were generally higher than the WFR except for the n-3 PUFA groups, where the FFQ estimated higher intakes. Correlations between the FFQ and WFR were moderate (r 0 x 32-0 x 59) except for trans fatty acids (r 0 x 03). Correlations between each of the dietary assessment methods and the plasma phospholipids were poor for all fatty acids other than the PUFA. Using the methods of triads approach, the FFQ validity coefficients for total n-3 fatty acids, total long chain n-3 fatty acids, EPA, arachidonic acid, docosapentaenoic acid and DHA were 0 x 50, 0 x 63, 0 x 45 and 0 x 62 and 0 x 62, respectively. For most fatty acids, the FFQ adequately estimates group mean fatty acid intakes and can adequately rank individuals; however, the ability of this FFQ to estimate trans fatty acids was poor.

Alcohol intake and risk of skin cancer: a prospective study

OBJECTIVE

To investigate the association between total alcohol intake and intake of different types of alcoholic beverages in relation to the risk of basal cell (BCC) and squamous cell (SCC) carcinoma of the skin.

DESIGN

Prospective cohort study.

SETTING

Follow-up data from a community-based skin cancer study in Australia.

SUBJECTS

Randomly selected sample of 1360 adult residents of the township of Nambour who completed a food frequency questionnaire in 1992 and were monitored for BCC and SCC until 31 December 2002.

RESULTS

No significant association was found between overall BCC or SCC risk and total alcohol intake, or intake of beer, white wine, red wine or sherry and port. However, among those with a prior skin cancer history, there was a significant doubling of risk of SCC for above-median consumption of sherry and port (multivariable adjusted relative risk 2.46, 95% confidence interval 1.06-5.72) compared with abstainers.

CONCLUSIONS

There are no associations between first occurrence of skin cancers and alcoholic beverage consumption. People with a history of skin cancer who consume above-average quantities of sherry or port may be at a raised risk of SCC, although replication of these findings in different study populations is needed to confirm this possible role of specific alcoholic beverages in secondary keratinocytic skin cancer risk.

Environmental influences on helminthiasis and nutritional status among Pacific schoolchildren

This paper describes a study undertaken to: (1) determine the prevalence of Ascaris lumbricoides, Trichuris trichiura and hookworm infections and nutritional status among Pacific Island school children; (2) identify factors influencing helminthiasis; (3) identify interventions to improve school health. A total of 3,683 children aged 5-12 years attending 27 primary schools in 13 Pacific Island countries were surveyed along with school environmental data. Stool samples were collected from 1996 children (54.2%) and analysed for ova and helminths. Total prevalence of helminthiasis was 32.8%. Anaemia prevalence was 12.4%. Children with helminthiasis and anaemia were found to be 8.7 times more likely to be stunted and 4.3 times more likely to be underweight than non-anaemic and non-infected children. Four significant environmental influences on helminthiasis were identified: (1) an inadequate water supply; (2); availability of a school canteen; (3) regular water/sanitation maintenance regimes; and (4) overcrowded classrooms. Helminthiasis was found to be strongly associated with anaemia, stunting and underweight and environmental influences identified. Although mass anti-helminthic drug administrations (MDA) have been taking place, reinfection is common as drug therapy alone is not enough. Programme effectiveness depends upon upgrading school environments to include an adequate water supply, controlled food preparation/provision, well-maintained water/sanitation facilities and class sizes of 30 students or less.

Critical care nursing for the patient with a spinal cord injury

Knowledge of the dynamics of spinal cord injury to anticipate the myriad of physiologic and psychological complications is essential for effective interventions to achieve optimal patient outcomes. Continuous expert nursing care must be provided to ensure that the patient survives life-threatening events and to facilitate optimal adaptation of the patient and family during this enormous emotional upheaval of their lives. Given the devastating effects of a spinal cord injury, the critical care nurse faces one of the most complex and potentially rewarding challenges of his or her career.

Hyperactivity and the attention deficit disorder

The attention deficit disorder is a clinical syndrome manifested by inattention, impulsivity and hyperactivity beginning before age seven and persisting for at least six months. The family physician plays a key role in consolidating the observations of parents and teachers with the findings on medical evaluation. The physician also initiates the comprehensive treatment program with counseling and stimulant medication, and coordinates the use of special education, mental health and other resources as needed.

Children with psychogenic abdominal pain and their families. Management during hospitalization

Children hospitalized with psychogenic abdominal pain and the children's families are discussed from the perspective of a child psychiatrist working within the pediatric hospital. Psychosocial and emotional difficulties for the child and parents are noted as families use somatic symptoms to deal with psychic distress. Principles for management are emphasized, utilizing a comprehensive team approach by physicians, nurses, and other health professionals. Through constructive use of hospitalization and informed follow-up, these children and their parents can be helped toward symptomatic improvement and a more meaningful understanding and resolution of their underlying emotional difficulties.