A thematic analysis of the strengths and weaknesses of manufacturers' submissions to the NICE Single Technology Assessment (STA) process

Quantifying the Limitations of Clinical and Technology-based Flap Monitoring Strategies using a Systematic Thematic Analysis

Background

Multiple techniques exist to monitor free flap viability postoperatively, varying with practical and personal preference, yet the limitations of each technique remain unquantified. This systematic review aims to identify the most commonly reported limitations of these techniques in clinical practice.

Methods

A systematic review was conducted according to PRISMA guidelines using MEDLINE, EMBASE, and Web of Science with search criteria for postoperative free flap monitoring techniques. Search results were independently screened using defined criteria by two authors and a senior clinician. Limitations of the techniques found in the discussion section of eligible articles were recorded and categorized using thematic analysis.

Results

A total of 4699 records were identified. In total, 2210 articles met the eligibility criteria and were subsequently reviewed, with 195 papers included in the final analysis. The most frequently reported limitations of clinical monitoring were interpretation requiring expertise (25% of related papers), unsuitability for buried flaps (21%), and lack of quantitative/objective values (19%). For noninvasive technologies, the limitations were lack of quantitative/objective values (21%), cost (16%), and interpretation requiring expertise (13%). For invasive technologies, the limitations were application requiring expertise (25%), equipment design and malfunction (13%), and cost (13%).

Conclusions

This is the first systematic review to quantify the limitations of different flap monitoring techniques, as reported in the literature. This information may enhance the choice in monitoring strategy for a reconstructive service, and inform the development and refinement of new flap monitoring technologies.

A systematic recurrent theme analysis of the reported limitations of facial electromyography

Background

Advances in digital technology hold promise in expanding the clinical and consumer applications of facial electromyography (EMG) through thedevelopment of wireless pervasive systems capable of operating in a nonclinical environment. This systematic review aims to appraise the most commonly reported limitations of the technology in clinical research and practice.

Methods

A systematic search for clinical facial EMG literature was performed using MEDLINE, EMBASE, PsychINFO and CINAHL. No language limits were applied. Search results were screened using defined criteria by two authors with disagreements resolved by a third. Practical limitations in the technology, as reported by the authors, were recorded and characterised using recurrent theme analysis.

Results

A total of 4,983 records were identified. Of those, 1,061 articles met eligibility criteria and were subsequently reviewed. In the medical domain, the most common area of application was in psychosocial studies (28% of medical studies); in the surgical domain monitoring of facial nerve integrity was the most common application of facial electromyography (27% of surgical studies). Collectively, the three most commonly reported limitations were motion artefact (13.7%), inter-subject variability in response and anatomy (13.1%), and muscle crosstalk (12.0%).

Conclusions

This is the first study to evaluate the limitations of facial EMG using a systematic analysis of author reports. Highlighting technology limitations in this non-biased manner raises awareness to users key issues and reliably informs the development of future systems.

•

The limitations of facial electromyography have been systematically reviewed.

•

The respective limitations of surface and needle facial electromyography have been examined across both medical and surgical domains.

•

Quantification of such limitations informs continued development of future systems.

Issues Related to the Frequency of Exploratory Analyses by Evidence Review Groups in the NICE Single Technology Appraisal Process

BACKGROUND

Evidence Review Groups (ERGs) critically appraise company submissions as part of the National Institute for Health and Care Excellence (NICE) Single Technology Appraisal (STA) process. As part of their critique of the evidence submitted by companies, the ERGs undertake exploratory analyses to explore uncertainties in the company's model. The aim of this study was to explore pre-defined factors that might influence or predict the extent of ERG exploratory analyses.

OBJECTIVE

The aim of this study was to explore predefined factors that might influence or predict the extent of ERG exploratory analyses.

METHODS

We undertook content analysis of over 400 documents, including ERG reports and related documentation for the 100 most recent STAs (2009-2014) for which guidance has been published. Relevant data were extracted from the documents and narrative synthesis was used to summarise the extracted data. All data were extracted and checked by two researchers.

RESULTS

Forty different companies submitted documents as part of the NICE STA process. The most common disease area covered by the STAs was cancer (44%), and most ERG reports (n = 93) contained at least one exploratory analysis. The incidence and frequency of ERG exploratory analyses does not appear to be related to any developments in the appraisal process, the disease area covered by the STA, or the company's base-case incremental cost-effectiveness ratio (ICER). However, there does appear to be a pattern in the mean number of analyses conducted by particular ERGs, but the reasons for this are unclear and potentially complex.

CONCLUSIONS

No clear patterns were identified regarding the presence or frequency of exploratory analyses, apart from the mean number conducted by individual ERGs. More research is needed to understand this relationship.

The use of exploratory analyses within the National Institute for Health and Care Excellence single technology appraisal process: an evaluation and qualitative analysis

BACKGROUND

As part of the National Institute for Health and Care Excellence (NICE) single technology appraisal (STA) process, independent Evidence Review Groups (ERGs) critically appraise the company submission. During the critical appraisal process the ERG may undertake analyses to explore uncertainties around the company's model and their implications for decision-making. The ERG reports are a central component of the evidence considered by the NICE Technology Appraisal Committees (ACs) in their deliberations.

OBJECTIVE

The aim of this research was to develop an understanding of the number and type of exploratory analyses undertaken by the ERGs within the STA process and to understand how these analyses are used by the NICE ACs in their decision-making.

METHODS

The 100 most recently completed STAs with published guidance were selected for inclusion in the analysis. The documents considered were ERG reports, clarification letters, the first appraisal consultation document and the final appraisal determination. Over 400 documents were assessed in this study. The categories of types of exploratory analyses included fixing errors, fixing violations, addressing matters of judgement and the ERG-preferred base case. A content analysis of documents (documentary analysis) was undertaken to identify and extract relevant data, and narrative synthesis was then used to rationalise and present these data.

RESULTS

The level and type of detail in ERG reports and clarification letters varied considerably. The vast majority (93%) of ERG reports reported one or more exploratory analyses. The most frequently reported type of analysis in these 93 ERG reports related to the category 'matters of judgement', which was reported in 83 (89%) reports. The category 'ERG base-case/preferred analysis' was reported in 45 (48%) reports, the category 'fixing errors' was reported in 33 (35%) reports and the category 'fixing violations' was reported in 17 (18%) reports. The exploratory analyses performed were the result of issues raised by an ERG in its critique of the submitted economic evidence. These analyses had more influence on recommendations earlier in the STA process than later on in the process.

LIMITATIONS

The descriptions of analyses undertaken were often highly specific to a particular STA and could be inconsistent across ERG reports and thus difficult to interpret.

CONCLUSIONS

Evidence Review Groups frequently conduct exploratory analyses to test or improve the economic evaluations submitted by companies as part of the STA process. ERG exploratory analyses often have an influence on the recommendations produced by the ACs.

FUTURE WORK

More in-depth analysis is needed to understand how ERGs make decisions regarding which exploratory analyses should be undertaken. More research is also needed to fully understand which types of exploratory analyses are most useful to ACs in their decision-making.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Developing an evidence-based methodological framework to systematically compare HTA coverage decisions: A mixed methods study

Health Technology Assessment (HTA) often results in different coverage recommendations across countries for a same medicine despite similar methodological approaches. This paper develops and pilots a methodological framework that systematically identifies the reasons for these differences using an exploratory sequential mixed methods research design. The study countries were England, Scotland, Sweden and France. The methodological framework was built around three stages of the HTA process: (a) evidence, (b) its interpretation, and (c) its influence on the final recommendation; and was applied to two orphan medicinal products. The criteria accounted for at each stage were qualitatively analyzed through thematic analysis. Piloting the framework for two medicines, eight trials, 43 clinical endpoints and seven economic models were coded 155 times. Eighteen different uncertainties about this evidence were coded 28 times, 56% of which pertained to evidence commonly appraised and 44% to evidence considered by only some agencies. The poor agreement in interpreting this evidence (κ=0.183) was partly explained by stakeholder input (ns=48 times), or by agency-specific risk (nu=28 uncertainties) and value preferences (noc=62 "other considerations"), derived through correspondence analysis. Accounting for variability at each stage of the process can be achieved by codifying its existence and quantifying its impact through the application of this framework. The transferability of this framework to other disease areas, medicines and countries is ensured by its iterative and flexible nature, and detailed description.

ASSESSING SEARCHES IN NICE SINGLE TECHNOLOGY APPRAISALS: PRACTICE AND CHECKLIST

Objectives: No guidelines exist in the approach that Evidence Review Groups (ERGs) should take to appraise search methodologies in the manufacturer's submission (MS) in Single Technology Appraisals (STA). As a result, ERGs are left to appraise searches using their own approach. This study investigates the limitations of manufacturers' search methodologies as critiqued by ERGs in published STA reports and to provide a recommended checklist.

Methods: Limitations from search critiques in 83 ERG reports published in the NIHR Web site between 2006 and May 2011 were extracted. The limitations were grouped into themes. Comparisons were made between limitations reported in the clinical effectiveness versus cost-effectiveness searches.

Results: Twelve themes were identified, six relating to the search strategy, source, limits, filters, translation, reporting, and missing studies. The search strategy theme contained the most limitations. Missing studies were frequently found by the ERG group in the clinical effectiveness searches. The omission of searches by manufacturers for unpublished and ongoing trials was frequently reported by the ERG. By contrast, failure of the manufacturer to report strategies was the most common limitation in the cost-effectiveness searches. Themes with the most frequent limitations in both types of searches are search strategy, reporting and source.

Conclusions: It is recommended that a checklist that has reporting, source and search strategy elements be used in the appraisal of manufacturer's searches during the STA process.