Surgical trial design for incorporating the effects of learning: what is the current methodological guidance, and is it sufficient?

BACKGROUND

Surgical interventions are complex. Key elements of this complexity are the surgeon and their learning curve. They pose methodological challenges in the design, analysis and interpretation of surgical RCTs. We identify, summarise, and critically examine current guidance about how to incorporate learning curves in the design and analysis of RCTs in surgery.

EXAMINING CURRENT GUIDANCE

Current guidance presumes that randomisation must be between levels of just one treatment component, and that the evaluation of comparative effectiveness will be made via the average treatment effect (ATE). It considers how learning effects affect the ATE, and suggests solutions which seek to define the target population such that the ATE is a meaningful quantity to guide practice. We argue that these are solutions to a flawed formulation of the problem, and are inadequate for policymaking in this setting.

REFORMULATING THE PROBLEM

The premise that surgical RCTs are limited to single-component comparisons, evaluated via the ATE, has skewed the methodological discussion. Forcing a multi-component intervention, such as surgery, into the framework of the conventional RCT design ignores its factorial nature. We briefly discuss the multiphase optimisation strategy (MOST), which for a Stage 3 trial would endorse a factorial design. This would provide a wealth of information to inform nuanced policy but would likely be infeasible in this setting. We discuss in more depth the benefits of targeting the ATE conditional on operating surgeon experience (CATE). The value of estimating the CATE for exploring learning effects has been previously recognised, but with discussion limited to analysis methods only. The robustness and precision of such analyses can be ensured via the trial design, and we argue that trial designs targeting CATE represent a clear gap in current guidance.

CONCLUSION

Trial designs that facilitate robust, precise estimation of the CATE would allow for more nuanced policymaking, leading to patient benefit. No such designs are currently forthcoming. Further research in trial design to facilitate the estimation of the CATE is needed.

A systematic review and narrative synthesis of footwear and orthotic devices used in the management of ankle haemarthrosis and haemarthropathy in haemophilia

Introduction

Haemarthrosis is a clinical feature of haemophilia leading to haemarthropathy. The ankle joint is most commonly affected, resulting in significant pain, disability and a reduction in health‐related quality of life. Footwear and orthotic devices are effective in other diseases that affect the foot and ankle, such as rheumatoid arthritis, but little is known about their effect in haemophilia.

Aims

To review the efficacy and effectiveness of footwear and orthotic devices in the management of ankle joint haemarthrosis and haemarthropathy in haemophilia.

Methods

A systematic literature review was conducted. Two review authors independently screened studies for inclusion and appraised methodological quality using Joanna Briggs Institute Critical Appraisal checklists. A narrative analysis was undertaken.

Results

Ten studies involving 271 male participants were eligible for inclusion. All studies were quasi‐experimental; three employed a within‐subject design. Two studies included an independent comparison or control group. A range of footwear and orthotic devices were investigated. Limited evidence from non‐randomised studies suggested that footwear and orthotic devices improve the number of ankle joint bleeding episodes, gait parameters and patient‐reported pain.

Conclusion

This review demonstrates a lack of robust evidence regarding the efficacy and effectiveness of footwear and orthotic devices in the management of ankle joint haemarthrosis and haemarthropathy in haemophilia. Methodological heterogeneities and limitations with the study designs, small sample sizes and limited follow‐up of participants exist. Future studies utilising randomised designs, larger sample sizes, long‐term follow‐up and validated patient‐reported outcome measures are needed to inform the clinical management of ankle joint haemarthrosis and haemarthropathy.

Effectiveness of Dementia Care Mapping™ to reduce agitation in care home residents with dementia: an open-cohort cluster randomised controlled trial

OBJECTIVES

Agitation is common and problematic in care home residents with dementia. This study investigated the (cost)effectiveness of Dementia Care Mapping™ (DCM) for reducing agitation in this population.

METHOD

Pragmatic, cluster randomised controlled trial with cost-effectiveness analysis in 50 care homes, follow-up at 6 and 16 months and stratified randomisation to intervention (n = 31) and control (n = 19). Residents with dementia were recruited at baseline (n = 726) and 16 months (n = 261). Clusters were not blinded to allocation. Three DCM cycles were scheduled, delivered by two trained staff per home. Cycle one was supported by an external DCM expert. Agitation (Cohen-Mansfield Agitation Inventory (CMAI)) at 16 months was the primary outcome.

RESULTS

DCM was not superior to control on any outcomes (cross-sectional sample n = 675: 287 control, 388 intervention). The adjusted mean CMAI score difference was -2.11 points (95% CI -4.66 to 0.44, p = 0.104, adjusted ICC control = 0, intervention 0.001). Sensitivity analyses supported the primary analysis. Incremental cost per unit improvement in CMAI and QALYs (intervention vs control) on closed-cohort baseline recruited sample (n = 726, 418 intervention, 308 control) was £289 and £60,627 respectively. Loss to follow-up at 16 months in the original cohort was 312/726 (43·0%) mainly (87·2%) due to deaths. Intervention dose was low with only a quarter of homes completing more than one DCM cycle.

CONCLUSION

No benefits of DCM were evidenced. Low intervention dose indicates standard care homes may be insufficiently resourced to implement DCM. Alternative models of implementation, or other approaches to reducing agitation should be considered.

A hypothesis test of feasibility for external pilot trials assessing recruitment, follow-up, and adherence rates

The power of a large clinical trial can be adversely affected by low recruitment, follow-up, and adherence rates. External pilot trials estimate these rates and use them, via prespecified decision rules, to determine if the definitive trial is feasible and should go ahead. There is little methodological research underpinning how these decision rules, or the sample size of the pilot, should be chosen. In this article we propose a hypothesis test of the feasibility of a definitive trial, to be applied to the external pilot data and used to make progression decisions. We quantify feasibility by the power of the planned trial, as a function of recruitment, follow-up, and adherence rates. We use this measure to define hypotheses to test in the pilot, propose a test statistic, and show how the error rates of this test can be calculated for the common scenario of a two-arm parallel group definitive trial with a single normally distributed primary endpoint. We use our method to redesign TIGA-CUB, an external pilot trial comparing a psychotherapy with treatment as usual for children with conduct disorders. We then extend our formulation to include using the pilot data to estimate the standard deviation of the primary endpoint and incorporate this into the progression decision.

Bayesian design and analysis of external pilot trials for complex interventions

External pilot trials of complex interventions are used to help determine if and how a confirmatory trial should be undertaken, providing estimates of parameters such as recruitment, retention, and adherence rates. The decision to progress to the confirmatory trial is typically made by comparing these estimates to pre-specified thresholds known as progression criteria, although the statistical properties of such decision rules are rarely assessed. Such assessment is complicated by several methodological challenges, including the simultaneous evaluation of multiple endpoints, complex multi-level models, small sample sizes, and uncertainty in nuisance parameters. In response to these challenges, we describe a Bayesian approach to the design and analysis of external pilot trials. We show how progression decisions can be made by minimizing the expected value of a loss function, defined over the whole parameter space to allow for preferences and trade-offs between multiple parameters to be articulated and used in the decision-making process. The assessment of preferences is kept feasible by using a piecewise constant parametrization of the loss function, the parameters of which are chosen at the design stage to lead to desirable operating characteristics. We describe a flexible, yet computationally intensive, nested Monte Carlo algorithm for estimating operating characteristics. The method is used to revisit the design of an external pilot trial of a complex intervention designed to increase the physical activity of care home residents.

The Implementation of Dementia Care Mapping in a Randomized Controlled Trial in Long-Term Care: Results of a Process Evaluation

This study explored intervention implementation within a pragmatic, cluster randomized controlled trial of Dementia Care Mapping™ (DCM) in UK care homes. DCM is a practice development tool comprised of a 5 component cycle (staff briefing, mapping observations, data analysis and reporting, staff feedback, and action planning) that supports delivery of person-centered care. Two staff from the 31 intervention care homes were trained in DCM and asked to deliver 3 cycles over a 15-month period, supported by a DCM expert during cycle 1. Implementation data were collected after each mapping cycle. There was considerable variability in DCM implementation fidelity, dose, and reach. Not all homes trained 2 mappers on schedule, and some found it difficult to retain mappers. Only 26% of homes completed more than 1 cycle. Future DCM trials in care home settings should consider additional methods to support intervention completion including intervention delivery being conducted with ongoing external support.

The feasibility of using the EQ-5D-3L with adults with mild to moderate learning disabilities within a randomized control trial: a qualitative evaluation

Background

In trials incorporating a health economic evaluation component, reliable validated measures for health-related quality of life (HRQOL) are essential. The EQ-5D is the preferred measure for cost-effectiveness analysis in UK trials. This paper presents a qualitative evaluation of the use of the EQ-5D-3L in a feasibility randomised control trial with participants who had a mild- to  moderate learning disability and type 2 diabetes.

Methods

Researchers administered the EQ-5D-3L to 82 participants at baseline and 77 at follow-up. After each interview, researchers rated the ease of administering the EQ-5D-3L and made free-text entries on the administration experience. For a subset of 16 interviews, researchers audio-recorded more detailed journal entries. Ease of administration data were analysed using descriptive statistics. Free-text responses were subject to a basic content analysis. The EQ-5D-3L-related journal entries were transcribed, coded and analysed thematically.

Results

Over half of participants were perceived to experience difficulty answering some or all of the items in the EQ-5D-3L (60% at baseline; 54% at follow-up). Analysis of the free-text entries and audio journals identified four themes that question the use of the EQ-5D-3L in this population. The first theme is related to observations of participant intellectual ability and difficulties, for example, in understanding the wording of the measure. Theme 2 is related to the normalisation of adjustments for impairments, which rendered the measure less sensitive in this population. Theme 3 is related to researcher adaptation and non-standard administration. An overarching fourth theme was identified in that people with learning disabilities were viewed as ‘unreliable witnesses’ by both researchers and supporters.

Conclusions

It is recommended that the EQ-5D-3L should not be used in isolation to assess health-related quality of life outcomes in trials research in adults with a learning disability. Further research is required to develop and evaluate a version of the EQ-5D appropriate for this population in trials research. It is unrealistic to expect that adjustments to the wording alone will deliver an appropriate measure: supporter or researcher involvement will almost always be required. This requirement needs to be factored into the development and administration guidelines of any new version of the EQ-5D for adults with a learning disability.

Trial registration

Current Controlled Trials ISRCTN41897033 [registered 21 January 2013].

Electronic supplementary material

The online version of this article (10.1186/s40814-018-0357-6) contains supplementary material, which is available to authorized users.

Developing and feasibility testing of data collection methods for an economic evaluation of a supported selfmanagement programme for adults with a learning disability and type 2 diabetes

Background

The challenges of conducting research with hard to reach vulnerable groups are particularly pertinent for people with learning disabilities. Data collection methods for previous cost and cost-effectiveness analyses of health and social care interventions targeting people with learning disabilities have relied on health care/health insurance records or data collection forms completed by the service provider rather than by people with learning disabilities themselves. This paper reports on the development and testing of data collection methods for an economic evaluation within a randomised controlled trial (RCT) for a supported self-management programme for people with mild/moderate learning disabilities and type 2 diabetes.

Methods

A case finding study was conducted to identify types of health and social care use and data collection methods employed in previous studies with this population. Based on this evidence, resource use questionnaires for completion by GP staff and interviewer-administered participant questionnaires (covering a wider cost perspective and health-related quality of life) were tested within a feasibility RCT. Interviewer-administered questionnaires included the EQ-5D-3L (the NICE recommended measure for use in economic evaluation). Participants were adults > 18 years with a mild or moderate learning disability and type 2 diabetes, with mental capacity to give consent to research participation.

Results

Data collection for questionnaires completed by GP staff requesting data for the last 12 months proved time intensive and difficult. Whilst 82.3% (121/147) of questionnaires were returned, up to 17% of service use items were recorded as unknown. Subsequently, a shorter recall period (4 months) led to a higher return rate but with a higher rate of missing data. Missing data for interviewer-administered participant questionnaires was > 8% but the interviewers reported difficulty with participant recall. Almost 60% (48/80) of participants had difficulty completing the EQ-5D-3L.

Conclusions

Further investigation as to how service use can be recorded is recommended. Concerns about the reliability of identifying service use data directly from participants with a learning disability due to challenges in completion, specifically around recall, remain. The degree of difficulty to complete EQ-5D-3L indicates concerns regarding the appropriateness of using this measure in its current form in research with this population.

Trial registration

Current Controlled Trials ISRCTN41897033 (registered 21 January 2013).

The evaluation of enhanced feedback interventions to reduce unnecessary blood transfusions (AFFINITIE): protocol for two linked cluster randomised factorial controlled trials

Background

Blood for transfusion is a frequently used clinical intervention, and is also a costly and limited resource with risks. Many transfusions are given to stable and non-bleeding patients despite no clear evidence of benefit from clinical studies. Audit and feedback (A&F) is widely used to improve the quality of healthcare, including appropriate use of blood. However, its effects are often inconsistent, indicating the need for coordinated research including more head-to-head trials comparing different ways of delivering feedback. A programmatic series of research projects, termed the ‘Audit and Feedback INterventions to Increase evidence-based Transfusion practIcE’ (AFFINITIE) programme, aims to test different ways of developing and delivering feedback within an existing national audit structure.

Methods

The evaluation will comprise two linked 2×2 factorial, cross-sectional cluster-randomised controlled trials. Each trial will estimate the effects of two feedback interventions, ‘enhanced content’ and ‘enhanced follow-on support’, designed in earlier stages of the AFFINITIE programme, compared to current practice. The interventions will be embedded within two rounds of the UK National Comparative Audit of Blood Transfusion (NCABT) focusing on patient blood management in surgery and use of blood transfusions in patients with haematological malignancies. The unit of randomisation will be National Health Service (NHS) trust or health board. Clusters providing care relevant to the audit topics will be randomised following each baseline audit (separately for each trial), with stratification for size (volume of blood transfusions) and region (Regional Transfusion Committee). The primary outcome for each topic will be the proportion of patients receiving a transfusion coded as unnecessary. For each audit topic a linked, mixed-method fidelity assessment and cost-effectiveness analysis will be conducted in parallel to the trial.

Discussion

AFFINITIE involves a series of studies to explore how A&F may be refined to change practice including two cluster randomised trials linked to national audits of transfusion practice. The methodology represents a step-wise increment in study design to more fully evaluate the effects of two enhanced feedback interventions on patient- and trust-level clinical, cost, safety and process outcomes.

Trial registration

http://www.isrctn.com/ISRCTN15490813

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-017-0614-8) contains supplementary material, which is available to authorized users.

Evaluating the effectiveness and cost-effectiveness of Dementia Care Mapping™ to enable person-centred care for people with dementia and their carers (DCM-EPIC) in care homes: study protocol for a randomised controlled trial

BACKGROUND

Up to 90 % of people living with dementia in care homes experience one or more behaviours that staff may describe as challenging to support (BSC). Of these agitation is the most common and difficult to manage. The presence of agitation is associated with fewer visits from relatives, poorer quality of life and social isolation. It is recommended that agitation is treated through psychosocial interventions. Dementia Care Mapping™ (DCM™) is an established, widely used observational tool and practice development cycle, for ensuring a systematic approach to providing person-centred care. There is a body of practice-based literature and experience to suggests that DCM™ is potentially effective but limited robust evidence for its effectiveness, and no examination of its cost-effectiveness, as a UK health care intervention. Therefore, a definitive randomised controlled trial (RCT) of DCM™ in the UK is urgently needed.

METHODS/DESIGN

A pragmatic, multi-centre, cluster-randomised controlled trial of Dementia Care Mapping (DCM™) plus Usual Care (UC) versus UC alone, where UC is the normal care delivered within the care home following a minimum level of dementia awareness training. The trial will take place in residential, nursing and dementia-specialist care homes across West Yorkshire, Oxfordshire and London, with residents with dementia. A random sample of 50 care homes will be selected within which a minimum of 750 residents will be registered. Care homes will be randomised in an allocation ratio of 3:2 to receive either intervention or control. Outcome measures will be obtained at 6 and 16 months following randomisation. The primary outcome is agitation as measured by the Cohen-Mansfield Agitation Inventory, at 16 months post randomisation. Key secondary outcomes are other BSC and quality of life. There will be an integral cost-effectiveness analysis and a process evaluation.

DISCUSSION

The protocol was refined following a pilot of trial procedures. Changes include replacement of a questionnaire, whose wording caused some residents distress, to an adapted version specifically designed for use in care homes, a change to the randomisation stratification factors, adaption in how the staff measures are collected to encourage greater compliance, and additional reminders to intervention homes of when mapping cycles are due, via text message.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN82288852 . Registered on 16 January 2014. Full protocol version and date: v7.1: 18 December 2015.

Supported self-management for adults with type 2 diabetes and a learning disability (OK-Diabetes): study protocol for a randomised controlled feasibility trial

BACKGROUND

Individuals with a learning disability (LD) are at higher risk of developing type 2 diabetes, but LD is not straightforward to define or identify, especially at the milder end of the spectrum, which makes case finding difficult. While supported self-management of health problems is now established, current material is largely educational and didactic with little that facilitates behavioural change. The interaction between the person with diabetes and others supporting their care is also largely unknown. For these reasons, there is considerable work needed to prepare for a definitive trial. The aim of this paper is to publish the abridged protocol of this preparatory work.

METHODS/DESIGN

Phase I is a prospective case-finding study (target n = 120 to 350) to identify and characterise potential participants, while developing a standardised supported self-management intervention. Phase II is a randomised feasibility trial (target n = 80) with blinded outcome assessment. Patients identified in Phase I will be interviewed and consented prior to being randomised to (1) standard treatment, or (2) supported self-management. Both arms will also be provided with an 'easy read' accessible information resource on managing type 2 diabetes. The intervention will be standardised but delivered flexibly depending on patient need, including components for the participant, a supporter, and shared activities. Outcomes will be (i) robust estimates of eligibility, consent and recruitment rates with refined recruitment procedures; (ii) characterisation of the eligible population; (iii) a standardised intervention with associated written materials, (iv) adherence and negative outcomes measures; (v) preliminary estimates of adherence, acceptability, follow-up and missing data rates, along with refined procedures; and (vi) description of standard treatment.

DISCUSSION

Our study will provide important information on the nature of type 2 diabetes in adults with LD living in the community, on the challenges of identifying those with milder LD, and on the possibilities of evaluating a standardised intervention to improve self-management in this population.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN41897033 (registered 21 January 2013).

Lifestyle referral assessment in an acute cardiology setting: study protocol for a randomized controlled feasibility trial

Background

Lifestyle and behaviour change are important factors in the prevention of cardiovascular disease and reduction of premature mortality. Public health initiatives have focused on opportunities for healthcare staff to deliver lifestyle advice routinely in primary and secondary care but there is no consistent approach to onward referrals and the rate of uptake of advice remains low. We do not know if advice is more effective in supporting behaviour change when a systematic approach is taken that includes identification of barriers to change, directing patients toward services, referral to services, and feedback on outcome.

Methods and design

This is a single-centre, randomized, unblinded feasibility trial in an acute hospital setting which aims to assess the feasibility of a definitive trial and provide proof of concept for the systematic delivery of individualized lifestyle advice in patients managed through an acute cardiology in-patient service.

Patients will be recruited before discharge and randomized to two groups. A control group will receive the usual lifestyle assessment and referral, while an intervention group will receive the usual assessment plus the new individualized lifestyle assessment and referral. The new assessment will inform assignment of each patient to one of three categories based on personal barriers to change. Patients may be referred to a formal lifestyle-change programme, through the ‘Leeds Let’s Change’ website, or they may be guided in self-management, using goal setting, or they may be assigned to a ‘deferment’ category, for reassessment at follow-up. These latter patients will be given a contact card for the ‘Leeds Let’s Change’ service.

Discussion

Lifestyle change is an important mechanism for improving health and wellbeing across the population but there are widely acknowledged difficulties in addressing lifestyle factors with patients and supporting behaviour change. A systematic approach to assessment would facilitate audit and provide an indicator of the quality of care. The new assessment template has been designed to be quick and easy to use in practice and could, for example, be added to a primary care consultation or form part of a nursing discharge assessment in an acute setting.

Trial registration

Current Controlled Trials ISRCTN41781196.