Simple imputation methods were inadequate for missing not at random (MNAR) quality of life data

A Conference (Missingness in Action) to Address Missingness in Data and AI in Health Care: Qualitative Thematic Analysis

BACKGROUND

Missingness in health care data poses significant challenges in the development and implementation of artificial intelligence (AI) and machine learning solutions. Identifying and addressing these challenges is critical to ensuring the continued growth and accuracy of these models as well as their equitable and effective use in health care settings.

OBJECTIVE

This study aims to explore the challenges, opportunities, and potential solutions related to missingness in health care data for AI applications through the conduct of a digital conference and thematic analysis of conference proceedings.

METHODS

A digital conference was held in September 2022, attracting 861 registered participants, with 164 (19%) attending the live event. The conference featured presentations and panel discussions by experts in AI, machine learning, and health care. Transcripts of the event were analyzed using the stepwise framework of Braun and Clark to identify key themes related to missingness in health care data.

RESULTS

Three principal themes-data quality and bias, human input in model development, and trust and privacy-emerged from the analysis. Topics included the accuracy of predictive models, lack of inclusion of underrepresented communities, partnership with physicians and other populations, challenges with sensitive health care data, and fostering trust with patients and the health care community.

CONCLUSIONS

Addressing the challenges of data quality, human input, and trust is vital when devising and using machine learning algorithms in health care. Recommendations include expanding data collection efforts to reduce gaps and biases, involving medical professionals in the development and implementation of AI models, and developing clear ethical guidelines to safeguard patient privacy. Further research and ongoing discussions are needed to ensure these conclusions remain relevant as health care and AI continue to evolve.

Making the Improbable Possible: Generalizing Models Designed for a Syndrome-Based, Heterogeneous Patient Landscape

Syndromic conditions, such as sepsis, are commonly encountered in the intensive care unit. Although these conditions are easy for clinicians to grasp, these conditions may limit the performance of machine-learning algorithms. Individual hospital practice patterns may limit external generalizability. Data missingness is another barrier to optimal algorithm performance and various strategies exist to mitigate this. Recent advances in data science, such as transfer learning, conformal prediction, and continual learning, may improve generalizability of machine-learning algorithms in critically ill patients. Randomized trials with these approaches are indicated to demonstrate improvements in patient-centered outcomes at this point.

A Bayesian Approach for Estimating the Survivor Average Causal Effect When Outcomes Are Truncated by Death in Cluster-Randomized Trials

Many studies encounter clustering due to multicenter enrollment and nonmortality outcomes, such as quality of life, that are truncated due to death-that is, missing not at random and nonignorable. Traditional missing-data methods and target causal estimands are suboptimal for statistical inference in the presence of these combined issues, which are especially common in multicenter studies and cluster-randomized trials (CRTs) carried out among the elderly or seriously ill. Using principal stratification, we developed a Bayesian estimator that jointly identifies the always-survivor principal stratum in a clustered/hierarchical data setting and estimates the average treatment effect among them (i.e., the survivor average causal effect (SACE)). In simulations, we observed low bias and good coverage with our method. In a motivating CRT, the SACE and the estimate from complete-case analysis differed in magnitude, but both were small, and neither was incompatible with a null effect. However, the SACE estimate has a clear causal interpretation. The option to assess the rigorously defined SACE estimand in studies with informative truncation and clustering can provide additional insight into an important subset of study participants. Based on the simulation study and CRT reanalysis, we provide practical recommendations for using the SACE in CRTs and software code to support future research.

Tweedie Model for Predicting Factors Associated with Distance Traveled to Access Inpatient Services in Kenya

Aim. This study aims to examine which factors influence the distance traveled by patients for inpatient care in Kenya. Methods. We used data from the fourth round of the Kenya Household Health Expenditure and Utilization survey. Our dependent variable was the self-reported distance traveled by patients to access inpatient care at public health facilities. As the clustered data were correlated, we used the generalized estimating equations approach with an exchangeable correlation under a Tweedie distribution. To select the best-fit covariates for predicting distance, we adopted a variable selection technique using the $QI{C}_u$ and $R^2$ criteria, wherein the lowest (highest) value for the former (latter) is preferred. Results. Using data for 451 participants from 47 counties, we found that three-fifths were admitted between 1 and 5 days, two-thirds resided in rural areas, and 90% were satisfied with the facilities’ service. Wealth quintiles were evenly distributed across respondents. Most admissions (81%) comprised $<$ 15, $>$ 65, and 25–54 years. Many households were of medium size (4–6 members) and had low education level (48%), and nine-tenths had no access to insurance. While two-thirds reported employment-based income, the same number reported not having cash to pay for inpatient services; 6 out of 10 paid over 3000 KES. Thus, differences in employment, ability to pay, and household size influence the distance traveled to access government healthcare facilities in Kenya. Interpretation. Low-income individuals more likely have large households and live in rural areas and, thus, are forced to travel farther to access inpatient care. Unlike the unemployed, the employed may have better socioeconomic status and possibly live near inpatient healthcare facilities, thereby explaining their short distances to access these services. Policymakers must support equal access to inpatient services, prioritize rural areas, open job opportunities, and encourage smaller families.

Retrospectively Collected EQ-5D-5L Data as Valid Proxies for Imputing Missing Information in Longitudinal Studies

OBJECTIVES

Studies face challenges with missing 5-level EQ-5D (EQ-5D-5L) data, often because of the need for longitudinal EQ-5D-5L data collection. There is a dearth of validated methodologies for dealing with missing EQ-5D-5L data in the literature. This study, for the first time, examined the possibility of using retrospectively collected EQ-5D-5L data as proxies for the missing data.

METHODS

Participants who had prospectively completed a 3rd month postdischarge EQ-5D-5L instrument (in-the-moment collection) were randomly interviewed to respond to a 2nd "retrospective collection" of their 3rd month EQ-5D-5L at 6th, 9th, or 12th month after hospital discharge. A longitudinal single imputation was also used to assess the relative performance of retrospective collection compared with the longitudinal single imputation. Concordances between the in-the-moment, retrospective, and imputed measures were assessed using intraclass correlation coefficients and weighted kappa statistics.

RESULTS

Considerable agreement was observed on the basis of weighted kappa (range 0.72-0.95) between the mobility, self-care, and usual activities dimensions of EQ-5D-5L collected in-the-moment and retrospectively. Concordance based on intraclass correlation coefficients was good to excellent (range 0.79-0.81) for utility indices computed, and excellent (range 0.93-0.96) for quality-adjusted life-years computed using in-the-moment compared with retrospective EQ-5D-5L. The longitudinal single imputation did not perform as well as the retrospective collection method.

CONCLUSIONS

This study demonstrates that retrospective collection of EQ-5D-5L has high concordance with "in-the-moment" EQ-5D-5L and could be a valid and attractive alternative for data imputation when longitudinally collected EQ-5D-5L data are missing. Future studies examining this method for other disease areas and populations are required to provide more generalizable evidence.

Handling missing data in a rheumatoid arthritis registry using random forest approach

Missing data in clinical epidemiological research violate the intention-to-treat principle, reduce the power of statistical analysis, and can introduce bias if the cause of missing data is related to a patient's response to treatment. Multiple imputation provides a solution to predict the values of missing data. The main objective of this study is to estimate and impute missing values in patient records. The data from the Kuwait Registry for Rheumatic Diseases was used to deal with missing values among patient records. A number of methods were implemented to deal with missing data; however, choosing the best imputation method was judged by the lowest root mean square error (RMSE). Among 1735 rheumatoid arthritis patients, we found missing values vary from 5% to 65.5% of the total observations. The results show that sequential random forest method can estimate these missing values with a high level of accuracy. The RMSE varied between 2.5 and 5.0. missForest had the lowest imputation error for both continuous and categorical variables under each missing data rate (10%, 20%, and 30%) and had the smallest prediction error difference when the models used the imputed laboratory values.

Predictive Modeling for Metabolomics Data

In recent years, mass spectrometry (MS)-based metabolomics has been extensively applied to characterize biochemical mechanisms, and study physiological processes and phenotypic changes associated with disease. Metabolomics has also been important for identifying biomarkers of interest suitable for clinical diagnosis. For the purpose of predictive modeling, in this chapter, we will review various supervised learning algorithms such as random forest (RF), support vector machine (SVM), and partial least squares-discriminant analysis (PLS-DA). In addition, we will also review feature selection methods for identifying the best combination of metabolites for an accurate predictive model. We conclude with best practices for reproducibility by including internal and external replication, reporting metrics to assess performance, and providing guidelines to avoid overfitting and to deal with imbalanced classes. An analysis of an example data will illustrate the use of different machine learning methods and performance metrics.

Missing not at random in end of life care studies: multiple imputation and sensitivity analysis on data from the ACTION study

Background

Missing data are common in end-of-life care studies, but there is still relatively little exploration of which is the best method to deal with them, and, in particular, if the missing at random (MAR) assumption is valid or missing not at random (MNAR) mechanisms should be assumed. In this paper we investigated this issue through a sensitivity analysis within the ACTION study, a multicenter cluster randomized controlled trial testing advance care planning in patients with advanced lung or colorectal cancer.

Methods

Multiple imputation procedures under MAR and MNAR assumptions were implemented. Possible violation of the MAR assumption was addressed with reference to variables measuring quality of life and symptoms. The MNAR model assumed that patients with worse health were more likely to have missing questionnaires, making a distinction between single missing items, which were assumed to satisfy the MAR assumption, and missing values due to completely missing questionnaire for which a MNAR mechanism was hypothesized. We explored the sensitivity to possible departures from MAR on gender differences between key indicators and on simple correlations.

Results

Up to 39% of follow-up data were missing. Results under MAR reflected that missingness was related to poorer health status. Correlations between variables, although very small, changed according to the imputation method, as well as the differences in scores by gender, indicating a certain sensitivity of the results to the violation of the MAR assumption.

Conclusions

The findings confirmed the importance of undertaking this kind of analysis in end-of-life care studies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-020-01180-y.

Full Law Identification in Graphical Models of Missing Data: Completeness Results

Missing data has the potential to affect analyses conducted in all fields of scientific study including healthcare, economics, and the social sciences. Several approaches to unbiased inference in the presence of non-ignorable missingness rely on the specification of the target distribution and its missingness process as a probability distribution that factorizes with respect to a directed acyclic graph. In this paper, we address the longstanding question of the characterization of models that are identifiable within this class of missing data distributions. We provide the first completeness result in this field of study - necessary and sufficient graphical conditions under which, the full data distribution can be recovered from the observed data distribution. We then simultaneously address issues that may arise due to the presence of both missing data and unmeasured confounding, by extending these graphical conditions and proofs of completeness, to settings where some variables are not just missing, but completely unobserved.

The ability of different imputation methods for missing values in mental measurement questionnaires

Background

Incomplete data are of particular important influence in mental measurement questionnaires. Most experts, however, mostly focus on clinical trials and cohort studies and generally pay less attention to this deficiency. We aim is to compare the accuracy of four common methods for handling items missing from different psychology questionnaires according to the items non-response rates.

Method

All data were drawn from the previous studies including the self-acceptance scale (SAQ), the activities of daily living scale (ADL) and self-esteem scale (RSES). SAQ and ADL dataset, simulation group, were used to compare and assess the ability of four imputation methods which are direct deletion, mode imputation, Hot-deck (HD) imputation and multiple imputation (MI) by absolute deviation, the root mean square error and average relative error in missing proportions of 5, 10, 15 and 20%. RSES dataset, validation group, was used to test the application of imputation methods. All analyses were finished by SAS 9.4.

Results

The biases obtained by MI are the smallest under various missing proportions. HD imputation approach performed the lowest absolute deviation of standard deviation values. But they got the similar results and the performances of them are obviously better than direct deletion and mode imputation. In a real world situation, the respondents’ average score in complete data set was 28.22 ± 4.63, which are not much different from imputed datasets. The direction of the influence of the five factors on self-esteem was consistent, although there were some differences in the size and range of OR values in logistic regression model.

Conclusion

MI shows the best performance while it demands slightly more data analytic capacity and skills of programming. And HD could be considered to impute missing values in psychological investigation when MI cannot be performed due to limited circumstances.

Performance of a Bayesian Approach for Imputing Missing Data on the SF-12 Health-Related Quality-of-Life Measure

BACKGROUND

Missing data in health-related quality-of-life outcomes are an ongoing problem. The 12-item short form health survey (SF-12) scores are no exception. Data imputation is complicated, because missingness may be partially predicted by the missing data themselves.

OBJECTIVES

To compare the performance of a Bayesian method for imputing SF-12 data with previously described frequentist imputation methods.

METHODS

SF-12 data were extracted from a trial assessing continence promotion on health-related quality of life in older women (n = 1052); the data set was split into a model development cohort for creating predictive models and a validation cohort to validate these models. Algorithms were constructed using data from the model development cohort to compute SF-12-related scores (physical health composite scale, the mental health composite scale, and the six-dimensional health state short form utilities). The Bayesian models used missing at random and missing not at random algorithms to impute missing SF-12 answers as categorical data. Comparative models replaced missing data with 0, used the mean weight of the sample, and regressed parameters from sociodemographic predictors. Data randomly deleted from the validation cohort were imputed with each algorithm, and the mean absolute error was used to gauge goodness of fit.

RESULTS

Each cohort included 526 persons; mean age was 78.1 ± 7.8 years. In the model development cohort, 15.6% of the participants had missing data. For the physical health composite scale, the mental health composite scale, and the six-dimensional health state short form utilities, the Bayesian model with missing at random data significantly outperformed all five comparison models, including the Bayesian models with missing not at random data.

CONCLUSIONS

Bayesian imputation was superior to other previously described methods for computing missing SF-12 data.

Apathy and health-related quality of life in nursing home residents

PURPOSE

To explore the association between apathy and health-related quality of life (HRQoL) from resident and proxy perspectives and whether cognition and depression moderate this relationship.

METHODS

Secondary analyses with baseline data from a cluster randomized trial on the effects of a care program for depression in Nursing Homes (NHs) were conducted. For HRQoL, the Visual Analogue Scale (VAS) and the Dutch version of the European Quality of Life (EQ-5D) were administered to 521 NH residents, and to professional caregivers reporting from the perspective of the NH resident (Resident-Proxy) and from their own perspective (Proxy-Proxy). Utility scores (U) were calculated for the three perspectives. Apathy, depression, and cognition were measured using the 10-item Apathy Evaluation Scale, the Cornell Scale for Depression in Dementia, and the standardized Mini-Mental State Examination, respectively.

RESULTS

Mixed models adjusted for clustering within NH units revealed that apathy was negatively associated with HRQoL both from the Resident-Proxy perspective (EQ-5D VAS: estimated effect, - 0.31, P < 0.001; EQ-5D Utility: - 0.30, P < 0.001) and from the Proxy-Proxy perspective (VAS: - 0.29, P < 0.001; U: - 0.03, P < 0.001), but not from the Resident-Resident perspective (VAS: - 0.05, P = 0.423; Utility: - 0.08, P = 0.161). Controlling for depression and cognition and their interaction terms with apathy did not change the results.

CONCLUSION

Apathy is negatively associated with NH resident HRQoL as reported by proxies. Depression and cognitive functioning do not moderate this association. NH residents do not self-report a relationship between apathy and HRQoL. More research is needed to understand caregiver and NH resident attitudes and underlying assumptions regarding apathy and HRQoL.

A Bayesian framework for health economic evaluation in studies with missing data

Health economics studies with missing data are increasingly using approaches such as multiple imputation that assume that the data are "missing at random." This assumption is often questionable, as-even given the observed data-the probability that data are missing may reflect the true, unobserved outcomes, such as the patients' true health status. In these cases, methodological guidelines recommend sensitivity analyses to recognise data may be "missing not at random" (MNAR), and call for the development of practical, accessible approaches for exploring the robustness of conclusions to MNAR assumptions. Little attention has been paid to the problem that data may be MNAR in health economics in general and in cost-effectiveness analyses (CEA) in particular. In this paper, we propose a Bayesian framework for CEA where outcome or cost data are missing. Our framework includes a practical, accessible approach to sensitivity analysis that allows the analyst to draw on expert opinion. We illustrate the framework in a CEA comparing an endovascular strategy with open repair for patients with ruptured abdominal aortic aneurysm, and provide software tools to implement this approach.

Development of a practical approach to expert elicitation for randomised controlled trials with missing health outcomes: Application to the IMPROVE trial

Background/aims:

The analyses of randomised controlled trials with missing data typically assume that, after conditioning on the observed data, the probability of missing data does not depend on the patient’s outcome, and so the data are ‘missing at random’ . This assumption is usually implausible, for example, because patients in relatively poor health may be more likely to drop out. Methodological guidelines recommend that trials require sensitivity analysis, which is best informed by elicited expert opinion, to assess whether conclusions are robust to alternative assumptions about the missing data. A major barrier to implementing these methods in practice is the lack of relevant practical tools for eliciting expert opinion. We develop a new practical tool for eliciting expert opinion and demonstrate its use for randomised controlled trials with missing data.

Methods:

We develop and illustrate our approach for eliciting expert opinion with the IMPROVE trial (ISRCTN 48334791), an ongoing multi-centre randomised controlled trial which compares an emergency endovascular strategy versus open repair for patients with ruptured abdominal aortic aneurysm. In the IMPROVE trial at 3 months post-randomisation, 21% of surviving patients did not complete health-related quality of life questionnaires (assessed by EQ-5D-3L). We address this problem by developing a web-based tool that provides a practical approach for eliciting expert opinion about quality of life differences between patients with missing versus complete data. We show how this expert opinion can define informative priors within a fully Bayesian framework to perform sensitivity analyses that allow the missing data to depend upon unobserved patient characteristics.

Results:

A total of 26 experts, of 46 asked to participate, completed the elicitation exercise. The elicited quality of life scores were lower on average for the patients with missing versus complete data, but there was considerable uncertainty in these elicited values. The missing at random analysis found that patients randomised to the emergency endovascular strategy versus open repair had higher average (95% credible interval) quality of life scores of 0.062 (−0.005 to 0.130). Our sensitivity analysis that used the elicited expert information as pooled priors found that the gain in average quality of life for the emergency endovascular strategy versus open repair was 0.076 (−0.054 to 0.198).

Conclusion:

We provide and exemplify a practical tool for eliciting the expert opinion required by recommended approaches to the sensitivity analyses of randomised controlled trials. We show how this approach allows the trial analysis to fully recognise the uncertainty that arises from making alternative, plausible assumptions about the reasons for missing data. This tool can be widely used in the design, analysis and interpretation of future trials, and to facilitate this, materials are available for download.

An Overview and Evaluation of Recent Machine Learning Imputation Methods Using Cardiac Imaging Data

Many clinical research datasets have a large percentage of missing values that directly impacts their usefulness in yielding high accuracy classifiers when used for training in supervised machine learning. While missing value imputation methods have been shown to work well with smaller percentages of missing values, their ability to impute sparse clinical research data can be problem specific. We previously attempted to learn quantitative guidelines for ordering cardiac magnetic resonance imaging during the evaluation for pediatric cardiomyopathy, but missing data significantly reduced our usable sample size. In this work, we sought to determine if increasing the usable sample size through imputation would allow us to learn better guidelines. We first review several machine learning methods for estimating missing data. Then, we apply four popular methods (mean imputation, decision tree, k-nearest neighbors, and self-organizing maps) to a clinical research dataset of pediatric patients undergoing evaluation for cardiomyopathy. Using Bayesian Rule Learning (BRL) to learn ruleset models, we compared the performance of imputation-augmented models versus unaugmented models. We found that all four imputation-augmented models performed similarly to unaugmented models. While imputation did not improve performance, it did provide evidence for the robustness of our learned models.

The current practice of handling and reporting missing outcome data in eight widely used PROMs in RCT publications: a review of the current literature

Purpose

Patient-reported outcome measures (PROMs) are designed to assess patients’ perceived health states or health-related quality of life. However, PROMs are susceptible to missing data, which can affect the validity of conclusions from randomised controlled trials (RCTs). This review aims to assess current practice in the handling, analysis and reporting of missing PROMs outcome data in RCTs compared to contemporary methodology and guidance.

Methods

This structured review of the literature includes RCTs with a minimum of 50 participants per arm. Studies using the EQ-5D-3L, EORTC QLQ-C30, SF-12 and SF-36 were included if published in 2013; those using the less commonly implemented HUI, OHS, OKS and PDQ were included if published between 2009 and 2013.

Results

The review included 237 records (4–76 per relevant PROM). Complete case analysis and single imputation were commonly used in 33 and 15 % of publications, respectively. Multiple imputation was reported for 9 % of the PROMs reviewed. The majority of publications (93 %) failed to describe the assumed missing data mechanism, while low numbers of papers reported methods to minimise missing data (23 %), performed sensitivity analyses (22 %) or discussed the potential influence of missing data on results (16 %).

Conclusions

Considerable discrepancy exists between approved methodology and current practice in handling, analysis and reporting of missing PROMs outcome data in RCTs. Greater awareness is needed for the potential biases introduced by inappropriate handling of missing data, as well as the importance of sensitivity analysis and clear reporting to enable appropriate assessments of treatment effects and conclusions from RCTs.

Electronic supplementary material

The online version of this article (doi:10.1007/s11136-015-1206-1) contains supplementary material, which is available to authorized users.

Variations in Health-Related Quality of Life (HRQoL) and survival 1 year after stroke: five European population-based registers

OBJECTIVE

There were two main objectives: to describe and compare clinical outcomes and Patient-Reported Outcome Measures (PROMs) collected using standardised procedures across the European Registers of Stroke (EROS) at 3 and 12 months after stroke; and to examine the relationship between patients' Health-Related Quality of Life (HRQoL) at 3 months after stroke and survival up to 1 year across the 5 populations.

DESIGN

Analysis of data from population-based stroke registers.

SETTING

European populations in Dijon (France); Kaunas (Lithuania); London (UK); Warsaw (Poland) and Sesto Fiorentino (Italy).

PARTICIPANTS

Patients with ischaemic or intracerebral haemorrhage (ICH) stroke, registered between 2004 and 2006.

OUTCOME MEASURES

(1) HRQoL, assessed by the physical component summary (PCS) and mental component summary (MCS) of the Short-Form Health Survey (SF-12), mapped into the EQ-5D to estimate responses on 5 dimensions (mobility, activity, pain, anxiety and depression, and self-care) and utility scores. (2) Mortality within 3 months and within 1 year of stroke.

RESULTS

Of 1848 patients, 325 were lost to follow-up and 500 died within a year of stroke. Significant differences in mortality, HRQoL and utility scores were found, and remained after adjustments. Kaunas had an increased risk of death; OR 2.34, 95% CI (1.32 to 4.14) at 3 months after stroke in Kaunas, compared with London. Sesto Fiorentino had the highest adjusted PCS: 43.54 (SD=0.96), and Dijon had the lowest adjusted

MCS

38.67 (SD=0.67). There are strong associations between levels of the EQ-5D at 3 months and survival within the year. The trend across levels suggests a dose-response relationship.

CONCLUSIONS

The study demonstrated significant variations in survival, HRQoL and utilities across populations that could not be explained by stroke severity and sociodemographic factors. Strong associations between HRQoL at 3 months and survival to 1 year after stroke were identified.

Implementing stratified primary care management for low back pain: cost-utility analysis alongside a prospective, population-based, sequential comparison study

STUDY DESIGN

Within-study cost-utility analysis.

OBJECTIVE

To explore the cost-utility of implementing stratified care for low back pain (LBP) in general practice, compared with usual care, within risk-defined patient subgroups (that is, patients at low, medium, and high risk of persistent disabling pain).

SUMMARY OF BACKGROUND DATA

Individual-level data collected alongside a prospective, sequential comparison of separate patient cohorts with 6-month follow-up.

METHODS

Adopting a cost-utility framework, the base case analysis estimated the incremental LBP-related health care cost per additional quality-adjusted life year (QALY) by risk subgroup. QALYs were constructed from responses to the 3-level EQ-5D, a preference-based health-related quality of life instrument. Uncertainty was explored with cost-utility planes and acceptability curves. Sensitivity analyses examined alternative methodological approaches, including a complete case analysis, the incorporation of non-back pain-related health care use and estimation of societal costs relating to work absence.

RESULTS

Stratified care was a dominant treatment strategy compared with usual care for patients at high risk, with mean health care cost savings of £124 and an incremental QALY estimate of 0.023. The likelihood that stratified care provides a cost-effective use of resources for patients at low and medium risk is no greater than 60% irrespective of a decision makers' willingness-to-pay for additional QALYs. Patients at medium and high risk of persistent disability in paid employment at 6-month follow-up reported, on average, 6 fewer days of LBP-related work absence in the stratified care cohort compared with usual care (associated societal cost savings per employed patient of £736 and £652, respectively).

CONCLUSION

At the observed level of adherence to screening tool recommendations for matched treatments, stratified care for LBP is cost-effective for patients at high risk of persistent disabling LBP only.

LEVEL OF EVIDENCE

2.

A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials

Missing data are a frequent problem in cost-effectiveness analysis (CEA) within a randomised controlled trial. Inappropriate methods to handle missing data can lead to misleading results and ultimately can affect the decision of whether an intervention is good value for money. This article provides practical guidance on how to handle missing data in within-trial CEAs following a principled approach: (i) the analysis should be based on a plausible assumption for the missing data mechanism, i.e. whether the probability that data are missing is independent of or dependent on the observed and/or unobserved values; (ii) the method chosen for the base-case should fit with the assumed mechanism; and (iii) sensitivity analysis should be conducted to explore to what extent the results change with the assumption made. This approach is implemented in three stages, which are described in detail: (1) descriptive analysis to inform the assumption on the missing data mechanism; (2) how to choose between alternative methods given their underlying assumptions; and (3) methods for sensitivity analysis. The case study illustrates how to apply this approach in practice, including software code. The article concludes with recommendations for practice and suggestions for future research.

Analysis of the progression of systolic blood pressure using imputation of missing phenotype values

We present a genome-wide association study of a quantitative trait, "progression of systolic blood pressure in time," in which 142 unrelated individuals of the Genetic Analysis Workshop 18 real genotype data were analyzed. Information on systolic blood pressure and other phenotypic covariates was missing at certain time points for a considerable part of the sample. We observed that the dropout process causing missingness is not independent of the initial systolic blood pressure; that is, the data is not missing completely at random. However, after the adjustment for age, the impact of systolic blood pressure on dropouts was no longer significant. Therefore, we decided to impute missing phenotype values by using information from individuals with complete phenotypic data. Progression of systolic blood pressure (∆SBP/∆t) was defined based on the imputed phenotypes and analyzed in a genome-wide fashion. We also conducted an exhaustive genome-wide search for interaction between single-nucleotide polymorphisms (7.14 × 10(10) tests) under an allelic model. The suggested data imputation and the association analysis strategy proved to be valid in the sense that there was no evidence of genome-wide inflation or increased type I error in general. Furthermore, we detected 2 single-nucleotide polymorphisms (SNPs) that met the criterion for genome-wide significance (p≤5 × 10(-8)), which was also confirmed via Monte-Carlo simulation. In view of the rather small sample size, however, the results have to be followed-up in larger studies.

Dietary, physical activity, and sedentary behaviors associated with percent body fat in rural Hispanic youth

INTRODUCTION

The objective of the present study was to assess dietary, physical activity, and sedentary behaviors associated with percent body fat in rural Hispanic youth.

METHOD

A total of 189 Hispanic children and adolescents ages 8 to 19 years completed the School Physical Activity and Nutrition questionnaire. Body composition (percent body fat) was determined by anthropometric skinfold methods. Logistic regression analysis was performed with percent body fat as the primary outcome dichotomized into excess body fat/normal body fat.

RESULTS

Gender was significantly associated with percent body fat in that girls were more likely to be in the excess percent body fat group. A significant interaction effect was noted between gender and sugar-sweetened beverages in that the effect of consuming sugar-sweetened drinks on excess adiposity was 6.28 times greater for boys than for girls.

DISCUSSION

Our data suggest that being a girl and consumption of sugar-sweetened beverages for boys may be risk factors for excess adiposity in rural Hispanic youth. Development of tailored, culturally sensitive interventions for this population may benefit from targeting these areas.

Tensor-based methods for handling missing data in quality-of-life questionnaires

A common problem with self-report quality-of-life questionnaires is missing data. Despite enormous care and effort to prevent it, some level of missing data is common and unavoidable. Missing data can have a detrimental impact on the data analysis. In this paper, a novel approach to imputing missing data in quality-of-life questionnaires is proposed, based on matrix and tensor decompositions. In order to illustrate and assess those methods, two datasets are considered: The first dataset contains the responses of 100 patients to a systemic lupus erythematosus-specific quality-of-life questionnaire; the other contains the responses of 43 patients to a rhino-conjunctivitis quality-of-life questionnaire. The two datasets contain almost no missing data, and for testing purposes, data entries are removed at random to have missing completely at random data. Several proportions of missing values are considered, and for each, the imputation error is assessed through k-fold cross validation. We also evaluate different imputation methods for missing at random and missing not at randomdata. The numerical results demonstrate that the proposed tensor factorization-based methods outperform standard methods in terms of root mean square error with at least 4% improvement, while the bias and variance are similar.

Multiple imputation methods for handling missing data in cost-effectiveness analyses that use data from hierarchical studies: an application to cluster randomized trials

PURPOSE

Multiple imputation (MI) has been proposed for handling missing data in cost-effectiveness analyses (CEAs). In CEAs that use cluster randomized trials (CRTs), the imputation model, like the analysis model, should recognize the hierarchical structure of the data. This paper contrasts a multilevel MI approach that recognizes clustering, with single-level MI and complete case analysis (CCA) in CEAs that use CRTs.

METHODS

We consider a multilevel MI approach compatible with multilevel analytical models for CEAs that use CRTs. We took fully observed data from a CEA that evaluated an intervention to improve diagnosis of active labor in primiparous women using a CRT (2078 patients, 14 clusters). We generated scenarios with missing costs and outcomes that differed, for example, according to the proportion with missing data (10%-50%), the covariates that predicted missing data (individual, cluster-level), and the missingness mechanism: missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). We estimated incremental net benefits (INBs) for each approach and compared them with the estimates from the fully observed data, the "true" INBs.

RESULTS

When costs and outcomes were assumed to be MCAR, the INBs for each approach were similar to the true estimates. When data were MAR, the point estimates from the CCA differed from the true estimates. Multilevel MI provided point estimates and standard errors closer to the true values than did single-level MI across all settings, including those in which a high proportion of observations had cost and outcome data MAR and when data were MNAR.

CONCLUSIONS

Multilevel MI accommodates the multilevel structure of the data in CEAs that use cluster trials and provides accurate cost-effectiveness estimates across the range of circumstances considered.

Methods in public health services and systems research: a systematic review

CONTEXT

Public Health Services and Systems Research (PHSSR) is concerned with evaluating the organization, financing, and delivery of public health services and their impact on public health. The strength of the current PHSSR evidence is somewhat dependent on the methods used to examine the field. Methods used in PHSSR articles, reports, and other documents were reviewed to assess their methodologic strengths and challenges in light of PHSSR goals.

EVIDENCE ACQUISITION

A total of 364 documents from the PHSSR library met the inclusion criteria as empirical and based in the U.S. After additional exclusions, 327 of these were analyzed.

EVIDENCE SYNTHESIS

A detailed codebook was used to classify articles in terms of (1) study design; (2) sampling; (3) instrumentation; (4) data collection; (5) data analysis; and (6) study validity. Inter-coder reliability was assessed for the codebook; once it was found reliable, the available empirical documents were coded.

CONCLUSIONS

Although there has been a dramatic increase in the amount of published PHSSR recently, methods used remain primarily cross-sectional and descriptive. Moreover, although appropriate for exploratory and foundational work in a new field, these approaches are limiting progress toward some PHSSR goals. Recommendations are given to advance and strengthen the methods used in PHSSR to better meet the goals and challenges facing the field.

Additional support for simple imputation of missing quality of life data in nursing research

Background. Missing data are a significant problem in health-related quality of life (HRQOL) research. We evaluated two imputation approaches: missing data estimation (MDE) and assignment of mean score (AMS). Methods. HRQOL data were collected using the Medical Outcomes Trust SF-12. Missing data were estimated using both approaches, summary statistics were produced for both, and results were compared using intraclass correlations (ICC). Results. Missing data were imputed for 21 participants. Mean values were similar, with ICC >.99 within both the Physical Component Summary and the Mental Component Summary when comparing the two methodologies. When imputed data were added into the full study sample, mean scores were identical regardless of methodology. Conclusion. Results support the use of a practical and simple imputation strategy of replacing missing values with the mean of the sample in cross-sectional studies when less than half of the required items of the SF-12 components are missing.

Measuring the Impact of Arthritis on Worker Productivity: Perspectives, Methodologic Issues, and Contextual Factors

Leading up to the Outcome Measures in Rheumatology (OMERACT) 10 meeting, the goal of the Worker Productivity Special Interest Group (WP-SIG) was to make progress on 3 key issues that relate to the application and interpretation of worker productivity outcomes in arthritis: (1) to review existing conceptual frameworks to help consolidate our intended target and scope of measurement; (2) to examine the methodologic issues associated with our goal of combining multiple indicators of worker productivity loss (e.g., absenteeism <—> presenteeism) into a single comprehensive outcome; and (3) to examine the relevant contextual factors of work and potential implications for the interpretation of scores derived from existing outcome measures. Progress was made on all 3 issues at OMERACT 10. We identified 3 theoretical frameworks that offered unique but converging perspectives on worker productivity loss and/or work disability to provide guidance with classification, selection, and future recommendation of outcomes. Several measurement and analytic approaches to combine absenteeism and presenteeism outcomes were proposed, and the need for further validation of such approaches was also recognized. Finally, participants at the WP-SIG were engaged to brainstorm and provide preliminary endorsements to support key contextual factors of worker productivity through an anonymous “dot voting” exercise. A total of 24 specific factors were identified, with 16 receiving ≥ 1 vote among members, reflecting highly diverse views on specific factors that were considered most important. Moving forward, further progress on these issues remains a priority to help inform the best application of worker productivity outcomes in arthritis research.

Imputation by the mean score should be avoided when validating a Patient Reported Outcomes questionnaire by a Rasch model in presence of informative missing data

Background

Nowadays, more and more clinical scales consisting in responses given by the patients to some items (Patient Reported Outcomes - PRO), are validated with models based on Item Response Theory, and more specifically, with a Rasch model. In the validation sample, presence of missing data is frequent. The aim of this paper is to compare sixteen methods for handling the missing data (mainly based on simple imputation) in the context of psychometric validation of PRO by a Rasch model. The main indexes used for validation by a Rasch model are compared.

Methods

A simulation study was performed allowing to consider several cases, notably the possibility for the missing values to be informative or not and the rate of missing data.

Results

Several imputations methods produce bias on psychometrical indexes (generally, the imputation methods artificially improve the psychometric qualities of the scale). In particular, this is the case with the method based on the Personal Mean Score (PMS) which is the most commonly used imputation method in practice.

Conclusions

Several imputation methods should be avoided, in particular PMS imputation. From a general point of view, it is important to use an imputation method that considers both the ability of the patient (measured for example by his/her score), and the difficulty of the item (measured for example by its rate of favourable responses). Another recommendation is to always consider the addition of a random process in the imputation method, because such a process allows reducing the bias. Last, the analysis realized without imputation of the missing data (available case analyses) is an interesting alternative to the simple imputation in this context.

Cautionary tales in the clinical interpretation of trials assessing therapy-induced changes in health status

Trials assessing the effects of therapies on symptoms, functional capacity, health-related quality of life and other aspects of health status are becoming more common in an era of chronic disease management. Such trials involve instruments for measuring health status whose reliability, validity and responsiveness need to be understood by clinicians and policy-makers in interpreting trial results. Deciding whether a treatment is clinically efficacious requires prior determination, based on empirical evidence, of what constitutes a minimal important difference (MID) between active treatment and control groups in the change in health status between study start and end. This MID should be used to calculate the sample size that will confer adequate power to detect a treatment effect if it truly exists. Many trials assessing health status have major methodological flaws: use of inappropriate or psychometrically unsound measurement instruments, lack of specification of MID, assumption that statistically significant results represent clinically significant treatment effects, and statement of conclusions inconsistent with observed results. This article provides guidance to clinicians in interpreting results of such trials in regard to clinical decision-making.

Predicting missing quality of life data that were later recovered: an empirical comparison of approaches

Background and Purpose The aim was to compare simple imputation, multiple imputation, and modeling approaches to deal with ‘missing’ quality of life data. Data were obtained from five clinical trials, which employed a reminder system for follow-up questionnaires. Previous studies have compared imputation strategies by artificially removing data according to prespecified mechanisms. Our approach differs from previous study as actual collected data are utilized.

Methods Data obtained by reminder were initially treated as missing. These missing values were imputed using a variety of simple and multiple imputation strategies. The trials were analyzed using the imputed datasets, and the resulting treatment effects compared to analyses using the full dataset including responses following reminders. A repeated measures model was also carried out on the available data and the pattern mixture models were employed. The accuracy of the different strategies was assessed by calculating the bias seen in the calculated treatment difference compared to the actual observed treatment difference.

Results Baseline carried forward or last value carried forward were shown to be the best simple imputation methods in this setting. Multiple imputation using a regression model or predictive mean match model tended to provide treatment difference estimates with the least bias when compared to the actual observed data. Pattern mixture models did not perform well. Overall, the multiple imputation procedures were generally the least biased approaches.

Limitations A number of imputation and modeling procedures have been investigated but this list is not exhaustive. All the example datasets come from the same data source and perhaps studies from additional disease areas would have been useful. However, we feel the results are generalizable to other quality of life outcomes and clinical areas.

Conclusions Multiple imputation is recommended for missing quality of life data as it makes the assumption of missing at random which in the quality of life setting is more plausible than the assumption of missing completely at random for which most simple imputation methods are based. Pattern mixture models can be complex and did not perform well in this setting. Clinical Trials 2010; 7: 333—342. http://ctj.sagepub.com

Investigating the missing data mechanism in quality of life outcomes: a comparison of approaches

BACKGROUND

Missing data is classified as missing completely at random (MCAR), missing at random (MAR) or missing not at random (MNAR). Knowing the mechanism is useful in identifying the most appropriate analysis. The first aim was to compare different methods for identifying this missing data mechanism to determine if they gave consistent conclusions. Secondly, to investigate whether the reminder-response data can be utilised to help identify the missing data mechanism.

METHODS

Five clinical trial datasets that employed a reminder system at follow-up were used. Some quality of life questionnaires were initially missing, but later recovered through reminders. Four methods of determining the missing data mechanism were applied. Two response data scenarios were considered. Firstly, immediate data only; secondly, all observed responses (including reminder-response).

RESULTS

In three of five trials the hypothesis tests found evidence against the MCAR assumption. Logistic regression suggested MAR, but was able to use the reminder-collected data to highlight potential MNAR data in two trials.

CONCLUSION

The four methods were consistent in determining the missingness mechanism. One hypothesis test was preferred as it is applicable with intermittent missingness. Some inconsistencies between the two data scenarios were found. Ignoring the reminder data could potentially give a distorted view of the missingness mechanism. Utilising reminder data allowed the possibility of MNAR to be considered.