Measurement of patient-reported outcomes. 2: Are current measures failing us?

Psychometric properties of patient-reported outcome measures, measuring fatigue in patients with multiple sclerosis, a systematic review

BACKGROUND

Multiple sclerosis (MS) is a chronic, inflammatory, degenerative disease of the central nervous system. One of the most common and disabling symptoms is fatigue. More than 80% of people with MS experience fatigue, which has a negative impact on their quality of life and level of independence in daily activities. The multidimensional nature of fatigue makes it essential to understand its impact from the patient's perspective. Patient-reported outcomes (PROs), defined by the FDA as "any report of a patient's health status that comes directly from the patient, without interpretation of the patient's response by the physician or other health care professional," were created to address this need.

OBJECTIVES

To identify and describe patient-reported outcomes (PROs) that measure the level of fatigue specific to patients with multiple sclerosis. To evaluate and analyze the quality of psychometric properties, methodological quality, and risk of bias of patient-reported outcomes that measure the level of fatigue specific to patients with multiple sclerosis.

METHODS

A systematic psychometric review was conducted and framed according to the standards of the Consensus for the Selection of Measurement Instruments (COSMIN) (Mokkink et al., 2010).

RESULTS

A total of 34 studies were included, from which a total of 40 references were extracted, as some studies reported two or more PROMs simultaneously. The evaluation and analysis of the risk of bias shows that the studies present a heterogeneous classification depending on the psychometric property evaluated, i.e. while the frequency of studies classified with low risk of bias is higher in measurement properties such as; structural validity; 25 studies (64.10%), internal consistency 25 studies (64.10%), criterion validity: 29 studies (74.36%). There is also a high frequency of studies rated as high or unclear risk of bias, mainly in psychometric properties such as reliability 19 studies (48.71%), cross-cultural validity measurement invariance 13 studies (33.33%).

CONCLUSIONS

PRO instruments are the best way to know the patients' perception of their symptomatology in this case of fatigue, which will undoubtedly contribute to a better approach and better intervention strategies in a personalized way, another component in the improvement of the quality of care and in line with the new paradigm of patient-centered care, which requires an assessment of fatigue by means of a PRO instrument. Therefore, it is of utmost importance to consider the current standards in the development of these instruments for a correct use and interpretability of their results.

Patient-reported outcome measures for dysphagia in head and neck cancer: A systematic review and appraisal of content validity and internal structure

Dysphagia is a major head and neck cancer (HNC) issue. Dysphagia-related patient-reported outcome measures (PROMs) are critical for patient-centred assessment and intervention tailoring. This systematic review aimed to derive a comprehensive inventory of HNC dysphagia PROMs and appraise their content validity and internal structure. Six electronic databases were searched to February 2023 for studies detailing PROM content validity or internal structure. Eligible PROMs were those developed or validated for HNC, with ≥20% of items related to swallowing. Two independent raters screened citations and full-text articles. Critical appraisal followed COSMIN guidelines. Overall, 114 studies were included, yielding 39 PROMs (17 dysphagia-specific and 22 generic). Of included studies, 33 addressed PROM content validity and 78 internal structure. Of all PROMs, only the SOAL met COSMIN standards for both sufficient content validity and internal structure. Notably, the development of 18 PROMs predated the publication of COSMIN standards. In conclusion, this review identified 39 PROMs addressing dysphagia in HNC, of which only one met COSMIN quality criteria. Given that half of PROMs were developed prior to COSMIN guidelines, future application of current standards is needed to establish their psychometric quality.

Structural equation modeling for identifying the drivers of health-related quality of life improvement experienced by patients with migraine receiving eptinezumab

BACKGROUND

As new migraine therapies emerge, it is crucial for measures to capture the complexities of health-related quality of life (HRQoL) improvement beyond improvements in monthly migraine day (MMD) reduction. Investigations into the correlations between MMD reduction, symptom management, and HRQoL are lacking, particularly those that focus on improvements in canonical symptoms and improvement in patient-identified most-bothersome symptoms (PI-MBS), in patients treated with eptinezumab. This exploratory analysis identified efficacy measures mediating the effect of eptinezumab on HRQoL improvements in patients with migraine.

METHODS

Data from the DELIVER study of patients with 2-4 prior preventive migraine treatment failures (NCT04418765) were inputted to two structural equation models describing sources of HRQoL improvement via Migraine-Specific Quality-of-Life Questionnaire (MSQ) scores. A single latent variable was defined to represent HRQoL and describe the sources of HRQoL in DELIVER. One model included all migraine symptoms while the second model included the PI-MBS as the only migraine symptom. Mediating variables capturing different aspects of efficacy included MMDs, other canonical symptoms, and PI-MBS.

RESULTS

In the first model, reductions in MMDs and other canonical symptoms accounted for 35% (standardized effect size [SES] - 0.11) and 25% (SES - 0.08) of HRQoL improvement, respectively, with 41% (SES - 0.13) of improvement comprising "direct treatment effect," i.e., unexplained by mediators. In the second model, substantial HRQoL improvement with eptinezumab (86%; SES - 0.26) is due to MMD reduction (17%; SES - 0.05) and change in PI-MBS (69%; SES - 0.21).

CONCLUSIONS

Improvements in HRQoL experienced by patients treated with eptinezumab can be substantially explained by its effect on migraine frequency and PI-MBS. Therefore, in addition to MMD reduction, healthcare providers should discuss PI-MBS improvements, since this may impact HRQoL. Health technology policymakers should consider implications of these findings in economic evaluation, as they point to alternative measurement of quality-adjusted life years to capture fully treatment benefits in cost-utility analyses.

TRIAL REGISTRATION

ClinicalTrials.gov (Identifier: NCT04418765 ; EudraCT (Identifier: 2019-004497-25; URL: https://www.clinicaltrialsregister.eu/ctr-search/search?query=2019-004497-25 ).

Exploring a New Application of Construct Specification Equations (CSEs) and Entropy: A Pilot Study with Balance Measurements

Both construct specification equations (CSEs) and entropy can be used to provide a specific, causal, and rigorously mathematical conceptualization of item attributes in order to provide fit-for-purpose measurements of person abilities. This has been previously demonstrated for memory measurements. It can also be reasonably expected to be applicable to other kinds of measures of human abilities and task difficulty in health care, but further exploration is needed about how to incorporate qualitative explanatory variables in the CSE formulation. In this paper we report two case studies exploring the possibilities of advancing CSE and entropy to include human functional balance measurements. In case study I, physiotherapists have formulated a CSE for balance task difficulty by principal component regression of empirical balance task difficulty values from Berg's Balance Scale transformed using the Rasch model. In case study II, four balance tasks of increasing difficulty due to diminishing bases of support and vision were briefly investigated in relation to entropy as a measure of the amount of information and order as well as physical thermodynamics. The pilot study has explored both methodological and conceptual possibilities and concerns to be considered in further work. The results should not be considered as fully comprehensive or absolute, but rather open up for further discussion and investigations to advance measurements of person balance ability in clinical practice, research, and trials.

Forward and Backward Recalling Sequences in Spatial and Verbal Memory Tasks: What Do We Measure?

There are different views in the literature about the number and inter-relationships of cognitive domains (such as memory and executive function) and a lack of understanding of the cognitive processes underlying these domains. In previous publications, we demonstrated a methodology for formulating and testing cognitive constructs for visuo-spatial and verbal recall tasks, particularly for working memory task difficulty where entropy is found to play a major role. In the present paper, we applied those insights to a new set of such memory tasks, namely, backward recalling block tapping and digit sequences. Once again, we saw clear and strong entropy-based construct specification equations (CSEs) for task difficulty. In fact, the entropy contributions in the CSEs for the different tasks were of similar magnitudes (within the measurement uncertainties), which may indicate a shared factor in what is being measured with both forward and backward sequences, as well as visuo-spatial and verbal memory recalling tasks more generally. On the other hand, the analyses of dimensionality and the larger measurement uncertainties in the CSEs for the backward sequences suggest that caution is needed when attempting to unify a single unidimensional construct based on forward and backward sequences with visuo-spatial and verbal memory tasks.

Traceability and comparability through crosswalks with the NeuroMET Memory Metric

Accurate assessment of memory ability for persons on the continuum of Alzheimer’s disease (AD) is vital for early diagnosis, monitoring of disease progression and evaluation of new therapies. However, currently available neuropsychological tests suffer from a lack of standardization and metrological quality assurance. Improved metrics of memory can be created by carefully combining selected items from legacy short-term memory tests, whilst at the same time retaining validity, and reducing patient burden. In psychometrics, this is known as “crosswalks” to link items empirically. The aim of this paper is to link items from different types of memory tests. Memory test data were collected from the European EMPIR NeuroMET and the SmartAge studies recruited at Charité Hospital (Healthy controls n = 92; Subjective cognitive decline n = 160; Mild cognitive impairment n = 50; and AD n = 58; age range 55–87). A bank of items (n = 57) was developed based on legacy short-term memory items (i.e., Corsi Block Test, Digit Span Test, Rey’s Auditory Verbal Learning Test, Word Learning Lists from the CERAD test battery and Mini Mental State Examination; MMSE). The NeuroMET Memory Metric (NMM) is a composite metric that comprises 57 dichotomous items (right/wrong). We previously reported on a preliminary item bank to assess memory based on immediate recall, and have now demonstrated direct comparability of measurements generated from the different legacy tests. We created crosswalks between the NMM and the legacy tests and between the NMM and the full MMSE using Rasch analysis (RUMM2030) and produced two conversion tables. Measurement uncertainties for estimates of person memory ability with the NMM across the full span were smaller than all individual legacy tests, which demonstrates the added value of the NMM. Comparisons with one (MMSE) of the legacy tests showed however higher measurement uncertainties of the NMM for people with a very low memory ability (raw score ≤ 19). The conversion tables developed through crosswalks in this paper provide clinicians and researchers with a practical tool to: (i) compensate for ordinality in raw scores, (ii) ensure traceability to make reliable and valid comparisons when measuring person ability, and (iii) enable comparability between test results from different legacy tests.

Systematic Review of the Content Validity of Patient Reported Outcome Measures of Transition to Parenthood

This review aims to identify self-report instruments examining aspects of transition to parenthood for use in practice and research. After performing a literature search in Embase, Medline, Web of Science, Cochrane, PsycINFO and Google Scholar, the Patient Reported Outcome Measures (PROMs) measuring (aspects of) transition to parenthood during pregnancy or up to 1-year postpartum were identified. Following COSMIN guidelines for systematic reviews on PROMs, the quality of the PROM development and PROM content validity was evaluated. From the 129 included studies, 39 PROMs assessed aspects of transition to parenthood. A total of 32 PROMs were included in the evaluation. The development quality of 30/32 PROMS was mostly rated as inadequate and the quality of 15 content validity studies was mostly rated as doubtful. All PROMs received inadequate or doubtful ratings on content validity. Most of the PROMs measuring aspects of the transition to parenthood didn't include parents' points of view when developing them. Many PROMs are being used for a long time without reassessing relevance, comprehensiveness, and comprehensibility among parents and/or practitioners. It is recommended that researchers and healthcare professionals assess content validity of the PROM before use with the target population.

Evaluation of the Unidimensional Fatigue Impact Scale (U-FIS) in Crohn's Disease: The Importance of Local Item Dependency

BACKGROUND AND PURPOSE

The Unidimensional Fatigue Impact Scale (U-FIS) was developed for use in a multiple sclerosis population. The aim was to determine whether the U-FIS is a valid tool for measuring the impact of fatigue in Crohn's disease (CD).

METHOD

CD patients completed the U-FIS as part of a validation study of the Crohn's Life Impact Questionnaire (CLIQ). Data were analyzed according to Rasch measurement theory (RMT).

RESULTS

Two hundred sixty-one completed U-FIS questionnaires were available for analysis. After rescoring the items to resolve disordered thresholds, all 22 items showed acceptable RMT fit. However, there was considerable local item dependency (LID).

CONCLUSION

The U-FIS did not provide unidimensional measurement in a sample of CD patients due to high levels of LID. Combining the three FIS outcomes into a single measure was not justified.

Fundamental Measurement: The Need Fulfilment Quality of Life (N-QOL) Measure

The quality adjusted life year (QALY) has serious problems related to its failure to adhere to measurement theory. If a QALY is to be meaningful, the utility score that translates time spent to an equivalent time spent in so-called perfect health must have ratio properties (i.e., it must support multiplication). Multiattribute utility scores (e.g. those generated by the EQ-5D-5L) fail to meet this standard. The multiattribute instruments produce ordinal scores that lack a true zero and they generate negative values. The manifest deficiencies of multiattribute utility instruments render them unfit, not only as a measure of therapy response but also in generating QALY claims. After 30 years of belief in their use, utilities and QALYs are clearly analytical dead ends.

The purpose of this commentary is to demonstrate a coherent way forward in health technology assessment by focusing, not on clinical attributes as surrogates for quality of life, but on measures that are based on a conceptual model describing patient value in terms of need-fulfilment. Building on an extensive, yet often overlooked literature, need-based measures that fit Rasch Measurement Theory criteria are converted from ordinal scores to interval scores to evaluate response to therapy. These measures meet the requirements of single attribute fundamental measurement which is the standard in the physical sciences. It is proposed that a translation from a Rasch interval scale (defined by logits) can be transformed to a bounded ratio scale. Need based Quality of Life (N-QOL) scales bounded by 0 (where no needs are fulfilled) to 1 (where all needs are fulfilled) form such scales. The N-QOL supports the full range of arithmetic operations. Multiattribute utilities and mathematically invalid QALYs can be put to one side as unfortunate historical curiosities in favor of a disease or target population specific N-QOL scale. Such a scale has the required properties to evaluate disease specific response to therapy This can also support N-QOL adjusted life years with a need- fulfillment life year (NALY) metric with ratio properties.

Medicaid Formulary Decisions and the Institute for Clinical and Economic Review: Abandoning Pseudoscience in Imaginary Pharmaceutical Pricing Claims

Medicaid formulary committees and other gatekeepers face a difficult task. On the one hand they can utilize technical expertise in evaluating the real world evidence for clinical, quality of life and resource utilization claims for competing products while on the other hand they may be asked to assess claims built by simulation models for pricing and product access. A common option has been to take modeled claims from third parties such as the Institute for Clinical and Economic Review (ICER) at face value without challenging the model structure, its assumptions and its incremental cost-per-QALY claims set against competing products or the existing standard of care. Unfortunately, from the available evidence, it seems clear that many formulary assessment groups, last but not least those for whom the ICER modeling claims are targeted, have little if any appreciation of the limitations of ICER modeling. There are two substantive issues: (i) a failure to appreciate the limitations imposed by the standards of normal science for credible, empirically evaluable and replicable product claims and (ii) an understanding of limitations imposed by the axioms of fundamental measurement. In the latter case, a failure to recognize that the quality adjusted life year (QALY) is an impossible mathematical construct (hence the I-QALY). To these limitations should be added the potential for constructing competing imaginary claims. Surprisingly, ICER has provided the ideal opportunity to construct competing claims with the launch in late 2020 of the ICER Analytics cloud platform. Formulary committees and other health decision makers should be aware that claims based on the ICER Analytics platform together with competing lifetime modelled claims all fail the standards of normal science. Factoring these into formulary decisions is not only misguided but may have unintended consequences for pricing and access that may disadvantage significantly patients and caregivers. We have spent too much time debating the merits or otherwise of the I-QALY for targeted patient groups with the parties failing to recognize that the focus on simulated cost-per-I-QALY value assessments is a mathematical folly; I-QALY claims are a chimera. The I-QALY, at long last, should be abandoned together with modelled lifetime simulations. Medicaid formulary decision makes should rethink the required evidence base for formulary decisions and negotiations. Care should be taken to revisit previous negotiations where ICER recommendations have been utilized to support pricing and access.

Setting and maintaining standards for patient-reported outcome measures: can we rely on the COSMIN checklists?

As test-developers we have often been troubled by published reviews of patient-reported outcome measures (PROMs). Too often minor issues are judged important while other reviews exclude the best measures available. Perhaps this led several groups to make recommendations for evaluating the quality of PROMs. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist is the latest set of recommendations. While reviewing the COSMIN literature and reviews conducted using their recommendations several concerns became apparent. The checklist is not evidence-based, relying on the opinion of researchers experienced in health-related quality of life. PROMs measuring other types of outcomes are inadequately covered by the checklist. COSMIN choose to focus on Classical Test Theory and the checklists are not appropriate for use with PROMs developed using modern measurement. Such an approach only obstructs progress in the field of outcome measurement. The retrospective nature of the evaluations also penalizes new PROMs. While the checklists imply that composite, ordinal level measurement is acceptable, crucial aspects of instrument development and quality are excluded. Reviews based on the COSMIN checklist produce contradictory conclusions and fail to provide evidence to support the recommendations. These problems suggest that the checklists themselves lack reliability and validity. It is also clear that several reviewers lack the expertise to apply the checklists. Researchers require a good grounding in instrument development and psychometrics to produce quality reviews. The science of modern PROM development is still in an early phase. Few available PROMs have sufficient quality, limiting the need for complex reviews. Standards need to be agreed for high quality outcome measurement. The issue is who should set these standards? Most published reviews merely scratch the surface and lack essential detail. All reviews of PROMs should be treated with caution, irrespective of whether the COSMIN checklist was employed.

To Dream the Impossible Dream: The Commitment by the Institute for Clinical and Economic Review to Rewrite the Axioms of Fundamental Measurement for Hemophilia A and Bladder Cancer Value Claims

Understandably, after 30 years of ignoring the axioms of fundamental measurement, advocates of creating approximate information through the construction of lifetime cost-per-QALY worlds are somewhat unnerved by the realization that their methodology is incompatible with those axioms. This is made all the more unnerving when it is pointed out that this incompatibility was pointed out over 30 years ago, following the formalization of those axioms almost 80 years ago. Why this was overlooked is a mystery. The result was a commitment to the application of ordinal utility and other patient reported outcome measures to support claims for response to competing therapies; most egregiously, the advocacy of cost-per-QALY lifetime models and willingness to pay thresholds to support recommendations for pricing and access to pharmaceutical products and devices. Although this incompatibility has been pointed out in respect of simulation modeling, to groups such as the Institute for Clinical and Economic Review (ICER) they press on, producing evidence reports and recommendations for emerging products that fail the standards of normal science. While these are an analytical dead end, ICER has nowhere else to go. This is their business model; to admit otherwise would mean withdrawing their many evidence reports and admit they were wrong. ICER has rejected this; rather it has decided, together with its academic consultants, to challenge the axioms of fundamental measurement, to produce a parallel measurement universe that can sustain QALYs and the imaginary simulation lifetime models. The purpose here is to make clear that ICER is manifestly wrong and that there is no way it can maintain its credibility in pursuing this path. This is achieved by a deconstruction of the arguments put forward by ICER to defend its new vision of the axioms of fundamental measurement, a vision which provides a case study in the distinction between justified belief and opinion. Fortunately, we have the framework for a new paradigm in value assessment; a paradigm that recognizes the standards of normal science and rejects belief in an alternative reality consistent with fundamental measurement axioms.

Value Assessment, Real World Evidence and Fundamental Measurement: Version 3.0 of the Minnesota Formulary Submission Guidelines

This latest version of the Minnesota guidelines is intended to reassert the application of the standards of normal science in formulary submissions for new and existing pharmaceutical products and devices. This represents a paradigm shift from the existing value assessment standards which are focused on imaginary or I-QALY modeling of lifetime claims. The proposed new paradigm rejects this as pseudoscience; a failure to recognize the standards of normal science, in particular a failure to recognize the constraints of fundamental measurement. As a result, current health technology assessment is dominated by value assessments that create claims that are neither credible, nor empirically evaluable or replicable. The fatal flaw is the failure to recognize that QALYS are an impossible mathematical construct (hence the term I-QALY). The proposed paradigm recognizes that if there are claims for product value then, regardless of whether the claim is for clinical impact, quality of life or resource utilization, all claims must be empirically evaluable. If not, then they should be rejected. The Minnesota guidelines propose a new evidence based approach to formulary assessment, together with ongoing disease area and therapeutic class reviews. The focus is on claims that are specific to target patient populations that are claims for specific attributes and are consistent with the axioms of fundamental measurement. Manufacturers are asked to support claims assessment through protocols detailing the evidence base for claims assessment, the timelines for those assessments and the process by which claims assessments are reported back to formulary committees. Value assessment leads naturally to value contracting, revisiting provisional prices as new information is discovered and delivered to the formulary committee.

Development of an Item Bank to Measure Medication Adherence: Systematic Review

Background

Medication adherence is important in managing the progression of chronic diseases. A promising approach to reduce cognitive burden when measuring medication adherence lies in the use of computer‐adaptive tests (CATs) or in the development of shorter patient-reported outcome measures (PROMs). However, the lack of an item bank currently hampers this progress.

Objective

We aim to develop an item bank to measure general medication adherence.

Methods

Using the preferred reporting items for systematic review and meta-analysis (PRISMA), articles published before October 2019 were retrieved from PubMed, Embase, CINAHL, the Cochrane Library, and Web of Science. Items from existing PROMs were classified and selected (“binned” and “winnowed”) according to standards published by the Patient-Reported Outcomes Measurement Information System (PROMIS) Cooperative Group.

Results

A total of 126 unique PROMs were identified from 213 studies in 48 countries. Items from the literature review (47 PROMs with 579 items for which permission has been obtained) underwent binning and winnowing. This resulted in 421 candidate items (77 extent of adherence and 344 reasons for adherence).

Conclusions

We developed an item bank for measuring general medication adherence using items from validated PROMs. This will allow researchers to create new PROMs from selected items and provide the foundation to develop CATs.

Composite outcome measurement in clinical research: the triumph of illusion over reality?

Composite measures that combine different types of indicators are widely used in medical research; to evaluate health systems, as outcomes in clinical trials and patient-reported outcome measurement. The potential advantages of such indices are clear. They are used to summarise complex data and to overcome the problem of evaluating new interventions when the most important outcome is rare or likely to occur far in the future. However, many scientists question the value of composite measures, primarily due to inadequate development methodology, lack of transparency or the likelihood of producing misleading results. It is argued that the real problems with composite measurement are related to their failure to take account of measurement theory and the absence of coherent theoretical models that justify the addition of the individual indicators that are combined into the composite index. All outcome measures must be unidimensional if they are to provide meaningful data. They should also have dimensional homogeneity. Ideally, a specification equation should be developed that can predict accurately how organisations or individuals will score on an index, based on their scores on the individual indicators that make up the measure. The article concludes that composite measures should not be used as they fail to apply measurement theory and, consequently, produce invalid and misleading scores.

Collection and use of EQ-5D for follow-up, decision-making, and quality improvement in health care - the case of the Swedish National Quality Registries

BACKGROUND

The Swedish National Quality Registries (NQRs) contain individual-level health care data for specific patient populations, or patients receiving specific interventions. Approximately 90% of the 105 Swedish NQRs include any patient-reported outcome measure, with EQ-5D being the most common. As there has been no general overview of EQ-5D data within the NQRs, this study fills a knowledge gap by reporting how the data are collected, presented, and used at different levels of the Swedish health care system.

METHODS

All 46 NQRs with a license for the use of EQ-5D were included. Information was retrieved from the registries' annual reports or from websites, using a template that was subsequently sent to each registry for completion and confirmation. If considered necessary, the contact was followed-up with an interview, either in-person or over the telephone. The uses of EQ-5D were categorised as denoting usage for follow-up, decision-making, or quality improvement in Swedish health care.

RESULTS

In total, 41 of the 46 licensed registries reported collection of EQ-5D data. EQ-5D is most commonly collected within registries related to the musculoskeletal system, but it has a wide application also in other disease areas. Thirty-six registries provide EQ-5D results to patients, clinicians, or other decision-makers. Twenty-two of the registries reported that EQ-5D data are being used for follow-up, decision-making or quality improvement. The registries most commonly reported use of data for assessing interventions, and in quality indicators to follow-up the quality of care at a national level.

CONCLUSION

Collection and use of EQ-5D data vary across the Swedish NQRs, which may partly be accounted for by the different purposes of the registries. The provided examples of use illustrate how EQ-5D data can inform decisions at different levels of the health care system. However, there is potential for improving the use of EQ-5D data.

Measurement, modeling and QALYs

Over the past 30 years, a mainstay of health technology assessment has been the creation of modeled incremental cost-per-quality adjusted life year (QALY) claims. These are intended to inform resource allocation decisions. Unfortunately, the reliance on the construction of QALYs from generic utility scales is misplaced. Those advocating QALY-based lifetime modeled claims fail to appreciate the limitations placed on these constructs by the axioms of fundamental measurement. Utility scales, such as those created by the EQ-5D-3L instrument, are nothing more than multidimensional, ordinal scales. Such scales cannot support basic arithmetic operations. Interval scales can support addition and subtraction; ratio scales the further operations of multiplication and division. Those who advocate the construction of QALYs fail to appreciate that such an operation is only possible if the utility scale is unidimensional and has ratio properties with a true zero. The utility measures available do not meet these requirements. As we cannot produce meaningful utility values, the QALY is an invalid construct. Consequently, cost-per-incremental QALY claims are impossible to sustain and the application of cost-per QALY thresholds meaningless. As utility is a latent, unidimensional variable, the best a measure of utility could achieve would be unidimensionality and interval scaling properties. Where such measures are available, they could support claims for response to therapy. Consequently, there would be no need to continue constructing imaginary lifetime value assessment frameworks. Admitting that the QALY is a fatally flawed construct means rejecting 30 years of cost-per-QALY models.

The Impossible QALY and the Denial of Fundamental Measurement: Rejecting the University of Washington Value Assessment of Targeted Immune Modulators (TIMS) in Ulcerative Colitis for the Institute for Clinical and Economic Review (ICER)

All too often organizations embrace standards for health technology assessment that fail to meet the standards of normal science. A continuing puzzle is why the axioms of fundamental measurement are ignored by researchers such as the University of Washington Model Group in constructing lifetime cost-per-QALY claims. The University of Washington Model Group is not alone; it is an accepted article of faith that multiattribute utility scales can be manipulated as if they had ratio scale properties, which they do not. This commitment to pseudoscientific claims, embracing intelligent design rather than natural selection, is endorsed by professional groups such as ISPOR as well as by self-appointed arbiters of value assessment such as ICER. Perhaps the answer is peer pressure rather than ignorance of the axioms of fundamental measurement. More to the point, if you have been an advocate of imaginary simulations a Damascene epiphany creates both psychological and professional challenges. After all, if cost-per-QALY constructs are rejected, then it is difficult to see what options there are for those attempting to model cost-effectiveness claims. If it is just ignorance of the axioms of fundamental measurement then a reasonable question is why these axioms, readily available on any number of internet sites, are ignored in health technology assessment programs. The purpose of this commentary is to review the ICER September 11th 2020 evidence report in ulcerative colitis, with particular reference to ICER's responses to questions raised in the public comment period on the measurement properties (or their absence) for utility scales; in this context the EQ-5D instruments. The critique pointed out that the utility scores had ordinal properties. ICER, without proof, disputed this statement asserting that health economists believed (or assumed) they were ratio scales. This is nonsensical. ICER has two options: first, to continue to believe that the EQ-5D instruments had ratio properties or second, to acknowledge that they indeed only had ordinal properties, rejecting their many modeled claims for pricing and access. Not surprisingly, the possibility of a Damascene epiphany was rejected. ICER maintained its assertion that health economists, presumably all of them, believe or possibly just assume for analytical convenience that the EQ-5D-3L and similar measures are in fact on a ratio scale. This introduces a new concept in fundamental measurement: a ratio scale without a true zero but with negative values. ICER is quite prepared to admit that negative I-QALYs are possible and their lifetime cost-per-incremental I-QALY modelling can yield negative I-QALYs.

The Great I-QALY Disaster

The QALY is an impossible construct; it defies common sense. It fails completely once we consider the axioms of fundamental measurement. Utilities as ordinal scales cannot be used to create QALYS. The QALY should never have been introduced to support the value assessment of pharmaceutical products and devices. The result is 30 years of QALY based assessments of pharmaceutical products and devices which are conceptually and technically wrong. They are a charade and will have contributed mistakenly to thousands of formulary decisions. In the search for a common metric to evaluate cost-effectiveness the impossibility of a QALY was overlooked. The result is a disaster, unfolding over decades. Our next steps must be to abandon the QALY paradigm and look ahead to a new value assessment framework.

The National Pharmaceutical Council: Endorsing the Construction of Imaginary Worlds in Health Technology Assessment

All too often, organizations embrace standards for health technology assessment that fail to meet those of normal science. A value assessment framework has been endorsed that is patently in the realm of pseudoscience. If a value assessment framework is to be accepted, then claims for the value of competing products must be credible, evaluable and replicable. If not, for example, when the assessment relies on the construction of an imaginary lifetime incremental cost-per-quality-adjusted-life-year (QALY) world, then that assessment should be rejected. Such an assessment would fail one of the central roles of normal science: the discovery of new facts through an ongoing process of conjecture and refutation where provisional claims can be continually challenged. It is no good defending an endorsement of a value framework that fails expected standards on the grounds that it has been endorsed by professional groups and reflects decades of development. This is intellectually lazy. If this is the case, then the scientific revolution of the 17th century need not have happened. The purpose of this commentary is to consider the recommended standards for health technology assessment of the National Pharmaceutical Council (NPC), with particular reference to proposed methodological standards in value assessment and the commitment to mathematically impossible QALYs.

Value Assessment in Cystic Fibrosis: ICER's Rejection of the Axioms of Fundamental Measurement

One of the features of the ICER stakeholder involvement in the development of ICER evidence reports is the ability for public comment. Unfortunately, and this may just a miscommunication, the replies from ICER to public comments frequently miss the point or fail to provide backup for their claims. The purpose of this commentary is to review ICER's responses to public comments by the author on the just released final evidence report on cystic fibrosis. The message is quite simple: the ICER value assessment framework lacks credibility. It fails to meet the standards of normal science. This is seen in ICERs apparent ignorance or rejection of the axioms of fundamental measurement which point quite clearly to the mathematical impossibility of creating QALYs from generic multiattribute utility scores. The ICER report also fails standards by creating a model from prior assumptions; there is no logical basis for constructing a value assessment claim. Either ICER should withdraw its value claims or admit the dubious basis on which the model is built, as a duty to its readership.

More Unnecessary Imaginary Worlds - Part 4: The ICER Evidence Report for Crizanlizumab, Voxelotor and L-Glutamine for Sickle Cell Disease

A number of commentaries have been published over the past 4 years by the present author on the manifest flaws in the reference case value assessment framework of the Institute for Economic and Clinical Review. The recent release of the evidence report on sickle cell disease continues ICER's commitment to what has been described as the creation of imaginary worlds to support value assessment. The purpose of the present commentary is to continue the critiques that have been presented for earlier evidence reports. This is important because of the apparent willingness to take ICER's recommendations at face value rather than a critical review of the value assessment framework. The case presented here points to a number of weaknesses in the ICER framework: (i) the fabrication of imaginary constructs with a lifetime cost-per-incremental QALY framework; (ii) the consequent failure to meet the standards of normal science; (iii) the illogical reliance of assumptions drawn from the literature to create future scenarios; (iv) the rejection of hypothesis testing in favor of 'approximate information' and (v) a belief that in the construction of QALYS that the EQ-5D-3L utility scale has ratio properties. This last point is demonstrably false which means that the ICER value assessment framework collapses. It is impossible mathematically, a failure to meet the axioms of fundamental measurement, for an ordinal utility scale to be combined with time spent in a disease state. The result is that the pricing and access recommendations for Crizanlizumab, Voxelotor and L-glutamine in sickle cell disease (SCD) are complete nonsense and should be rejected.

Nonsense on Stilts - Part 1: The ICER 2020-2023 Value Assessment Framework for Constructing Imaginary Worlds

Previous commentaries in the Formulary Evaluation section of INNOVATIONS in Pharmacy have pointed to the lack of credibility in modeled claims for cost-effectiveness and associated recommendations for pricing and access by the Institute for Clinical and Economic Review (ICER). The principal objection to ICER reports has been that their modeled claims fail the standards of normal science: they are best seen as pseudoscience. The purpose of this latest commentary is to provide a critique of the recently released ICER 2020 Value Assessment Framework (VAF). Although ICER has taken upon itself the pole position in health technology assessments and recommendations for product pricing in the US health care system, the incremental, lifetime cost-per-QALY modeling methodology should not be taken seriously. The creation of imaginary modeled worlds, built entirely from assumption, fails the demarcation test between science and pseudoscience. The ICER evidence reports are best seen as the health technology assessment equivalent of 'intelligent design' in counterpoint to 'natural selection'. It is surprising, therefore, that health care decision makers should take ICER's recommendations seriously as providing 'approximate information' for formulary decision making. What is not appreciated is that the claims made by ICER lack credibility, are impossible to evaluate and lack the ability to be replicated across treatment settings. Indeed, the models presented under the guise of a 'state of the art' value assessment were never intended to support evaluable claims. We have no idea and will never know if they are right or if they are wrong. ICER's position becomes even more untenable once the models presented are assessed in detail. Without in any way supporting the ICER methodology, it is worth noting that all too often ICER's claims for incremental QALYs in specific models are based upon what appears to be, from the limited evidence presented, a casual and ad hoc assemblage of utility scores from diverse constructs. This is a critical weakness given the role attributed by ICER to the modeled cost-per-QALY claims as central to ICERs imaginary value assessment. ICER also overlooks the fact that the utility scores it captures from the literature to populate its imaginary reference case world lack objectivity. They are ordinal rather than interval measures. To apply these manifest scores to time spent in a disease stage and then aggregate these over different disease stages is nonsensical. The critical issue is one of instrument development. The case made here is for the application of Rasch Measurement Theory (RMT) to construct a unidimensional instrument with interval properties, in this case from the needs fulfillment construct of quality of life (QoL). Unless an instrument meets RMT standards in its development, the logic of Rasch modeling to achieve fundamental measurement standards means that other scales are, by definition, ordinal. It is absurd to 'assume' they are interval. RMT is designed to create instruments to evaluate change and test hypotheses. In the absence of instruments that have RMT properties, the cost-per-QALY reference case modelling meme collapses. It is an analytical dead end. If we are to support a meaningful scientific program to discover new facts to support health care delivery and improve the lives of patients, caregivers and their families, then ICER should be put to one side.

More Unnecessary Imaginary Worlds - Part 1: The Institute for Clinical and Economic Review's Evidence Report on Janus Kinase (JAK) Inhibitors in Rheumatoid Arthritis

Previous commentaries in the Formulary Evaluation section of INNOVATIONS in Pharmacy have pointed to the lack of credibility in modeled claims for cost-effectiveness and associated recommendations for pricing by the Institute for Clinical and Economic Review (ICER). The principal objection to ICER reports has been that their modeled claims fail the standards of normal science: they are best seen as pseudoscience. The purpose of this latest commentary is to consider the recently released ICER evidence report for Janus Kinase (JAK) Inhibitors. As ICER continues, in the case of JAK Inhibitors, to apply its modeled cost utility framework with consequent recommendations for pricing adjustments, these recommendations also lack credibility. In contrast with previous ICER evidence reports, the present report adopts only a 12-month timeframe, one due, in large part, to ICER being unable to justify assumptions to drive its construction of imaginary worlds beyond 12 months. This commentary emphasizesagain, why the ICER methodology fails to meet the standards of normal science. Claims made by ICER for the competing JAK Inhibitor therapies lack credibility, are impossible to evaluate, let alone replicate across treatment settings. Even so, it is important to examine a number of key elements in the ICER invention of the 12-month JAK Inhibitor imaginary world. While this does not imply any degree of acceptance of the ICER methodology, one element that merits particular attention is thefailure of the ICER modeling to meet logically defensible measurement standards in its application of generic health related quality of life (HRQoL) ordinal metrics to create its QALY claims. The failure to meet the required standards of fundamental measurement means that the cost-per-QALY claims are invalid. This raises the issue of the application of Rasch Measurement Theory (RMT) in instrument development and the potential role of patient centric outcome (PCO) instruments that represent the patient voice in value claims. The case made here is that the ICER approach should be abandoned as an unnecessary distraction. If we are to meet standards for the discovery of new facts in therapy response then our focus must be on proposing credible, evaluable and replicable claims within disease states. Instruments, such as the Rheumatoid Arthritis Quality of Life (RAQoL)questionnaire that build on the common construct that QoL is the extent to which human needs are fulfilled should be the basis for value claims. HRQoL Instruments that are clinically focused and reflect the value calculus of providers and not patients in measuring response by symptoms and activity limitations are irrelevant. This puts to one side the belief that incremental cost-per-QALY models, the construction of imaginary worlds are, in any sense, a 'gold standard'; a meme embraced by the health technology assessment profession. Claims for incremental cost per QALY outcomes and recommendations for pricing and access driven by willingness to pay thresholds are irrelevant to formulary decisions.

Yet another Ersatz World: The ICER Final Evidence Report for Additive Cardiovascular Therapies

Previous commentaries in the Formulary Evaluation section of INNOVATIONS in Pharmacy have pointed to the lack of credibility in modeled claims for cost-effectiveness and associated recommendations for pricing by the Institute for Clinical and Economic Review (ICER). The principal objection to ICER reports has been that their modeled claims fail the standards of normal science: they are best seen as pseudoscience. The purpose of this latest commentary is to consider the recently released ICER report for Additive Cardiovascular Disease therapies. This report should not be taken seriously in its claims for cost-effectiveness and pricing in cardiovascular disease (CVD). The analytical framework applied by ICER fails to meet the standards of normal science in demarcating science from pseudoscience. Irrespective of the value judgements and recommendations of an ICER report, these lack credibility. They were never intended to be evaluable and replicable across treatment settings. The claims made are constructed, driven by assumption, and should be put to one side by health system decision makers. In this review the focus is on to the ICER modeled estimates of utility scores in CVD, the insistence on utilizing a generic utility algorithm (the EQ-5D-3L) and the consequent quality adjusted life year (QALY) estimates. Two issues are raised that will be the subject of future commentaries: the lack of appreciation of fundamental measurement and (ii) the importance of the patient voice in benefit claims. Given the importance in the ICER methodology of QALYS, the ad hoc nature of the ordinal utilities introduced to the cardiovascular model must raise concerns over the role the ICER evidence report may play in health care decision-making. These concerns extend to the claim by ICER that, on ICER's own affordability threshold for individual new molecular entities, the anticipated uptake of these therapies may raise questions of overall affordability. Again, we are dealing with an arbitraryconstruct that may adversely impact patient access.