Identifying women with undetected ovarian cancer: independent and external validation of QCancer(®) (Ovarian) prediction model

Early diagnosis of symptomatic ovarian cancer in primary care in the UK: opportunities and challenges

Background:

Ovarian cancer is the sixth most common cause of cancer-related death in the UK amongst women. Ovarian cancer presents particular challenges for general practitioners (GPs) to diagnose due to its rarity and presentation with non-specific symptoms.

Methods:

A narrative overview of the literature was conducted by searching PubMed and Researchgate for relevant articles, using keywords such as “ovarian cancer,” “primary care” and “diagnosis.”

Results and Discussion:

Studies have shown that in the UK, GPs have a lower readiness to refer and investigate potential cancer symptoms compared with their international counterparts; and this has been correlated with reduced survival. Early diagnosis can be facilitated through a people-focussed and system-based approach which involves both educating GPs and using risk algorithms, rapid diagnostic centres/multi-disciplinary centres and being data-driven through the identification of best practice from national audits. Further research is required into the best evidence-based early investigations for ovarian cancer and more effective biomarkers.

Artificial intelligence predictive system of individual survival rate for lung adenocarcinoma

Background

The current research aimed to develop an artificial intelligence predictive system for individual survival rate of lung adenocarcinoma (LUAD).

Methods

Independent risk variables were identified by multivariate Cox regression. Artificial intelligence predictive system was constructed using three different data mining algorithms.

Results

Stage, PM, chemotherapy, PN, age, PT, sex, and radiation_surgery were determined as risk factors for LUAD patients. For 12-month survival rate in model cohort, concordance indexes of RFS, MTLR, and Cox models were 0.852, 0.821, and 0.835, respectively. For 36-month survival rate in model cohort, concordance indexes of RFS, MTLR, and Cox models were 0.901, 0.864, and 0.862, respectively. For 60-month survival rate in model cohort, concordance indexes of RFS, MTLR, and Cox models were 0.899, 0.874, and 0.866, respectively. The concordance indexes in validation dataset were similar to those in model dataset.

Conclusions

The current study designed an individualized survival predictive system, which could provide individual survival curves using three different artificial intelligence algorithms. This artificial intelligence predictive system could directly convey treatment benefits by comparing individual mortality risk curves under different treatments. This artificial intelligence predictive tool is available at https://zhangzhiqiao11.shinyapps.io/Artificial_Intelligence_Survival_Prediction_System_AI_E1001/.

Cancer diagnostic tools to aid decision-making in primary care: mixed-methods systematic reviews and cost-effectiveness analysis

BACKGROUND

Tools based on diagnostic prediction models are available to help general practitioners diagnose cancer. It is unclear whether or not tools expedite diagnosis or affect patient quality of life and/or survival.

OBJECTIVES

The objectives were to evaluate the evidence on the validation, clinical effectiveness, cost-effectiveness, and availability and use of cancer diagnostic tools in primary care.

METHODS

Two systematic reviews were conducted to examine the clinical effectiveness (review 1) and the development, validation and accuracy (review 2) of diagnostic prediction models for aiding general practitioners in cancer diagnosis. Bibliographic searches were conducted on MEDLINE, MEDLINE In-Process, EMBASE, Cochrane Library and Web of Science) in May 2017, with updated searches conducted in November 2018. A decision-analytic model explored the tools' clinical effectiveness and cost-effectiveness in colorectal cancer. The model compared patient outcomes and costs between strategies that included the use of the tools and those that did not, using the NHS perspective. We surveyed 4600 general practitioners in randomly selected UK practices to determine the proportions of general practices and general practitioners with access to, and using, cancer decision support tools. Association between access to these tools and practice-level cancer diagnostic indicators was explored.

RESULTS

Systematic review 1 - five studies, of different design and quality, reporting on three diagnostic tools, were included. We found no evidence that using the tools was associated with better outcomes. Systematic review 2 - 43 studies were included, reporting on prediction models, in various stages of development, for 14 cancer sites (including multiple cancers). Most studies relate to QCancer^® (ClinRisk Ltd, Leeds, UK) and risk assessment tools.

DECISION MODEL

In the absence of studies reporting their clinical outcomes, QCancer and risk assessment tools were evaluated against faecal immunochemical testing. A linked data approach was used, which translates diagnostic accuracy into time to diagnosis and treatment, and stage at diagnosis. Given the current lack of evidence, the model showed that the cost-effectiveness of diagnostic tools in colorectal cancer relies on demonstrating patient survival benefits. Sensitivity of faecal immunochemical testing and specificity of QCancer and risk assessment tools in a low-risk population were the key uncertain parameters.

SURVEY

Practitioner- and practice-level response rates were 10.3% (476/4600) and 23.3% (227/975), respectively. Cancer decision support tools were available in 83 out of 227 practices (36.6%, 95% confidence interval 30.3% to 43.1%), and were likely to be used in 38 out of 227 practices (16.7%, 95% confidence interval 12.1% to 22.2%). The mean 2-week-wait referral rate did not differ between practices that do and practices that do not have access to QCancer or risk assessment tools (mean difference of 1.8 referrals per 100,000 referrals, 95% confidence interval -6.7 to 10.3 referrals per 100,000 referrals).

LIMITATIONS

There is little good-quality evidence on the clinical effectiveness and cost-effectiveness of diagnostic tools. Many diagnostic prediction models are limited by a lack of external validation. There are limited data on current UK practice and clinical outcomes of diagnostic strategies, and there is no evidence on the quality-of-life outcomes of diagnostic results. The survey was limited by low response rates.

CONCLUSION

The evidence base on the tools is limited. Research on how general practitioners interact with the tools may help to identify barriers to implementation and uptake, and the potential for clinical effectiveness.

FUTURE WORK

Continued model validation is recommended, especially for risk assessment tools. Assessment of the tools' impact on time to diagnosis and treatment, stage at diagnosis, and health outcomes is also recommended, as is further work to understand how tools are used in general practitioner consultations.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42017068373 and CRD42017068375.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology programme and will be published in full in Health Technology Assessment; Vol. 24, No. 66. See the NIHR Journals Library website for further project information.

Identifying Ovarian Cancer in Symptomatic Women: A Systematic Review of Clinical Tools

Simple Summary

Most women with ovarian cancer are diagnosed after they develop symptoms—identifying symptomatic women earlier has the potential to improve outcomes. Tools, ranging from simple symptom checklists to diagnostic prediction models that incorporate tests and risk factors, have been developed to help identify women at increased risk of undiagnosed ovarian cancer. In this review, we systematically identified studies evaluating these tools and then compared the reported diagnostic performance of tools. All included studies had some quality concerns and most tools had only been evaluated in a single study. However, four tools were evaluated in multiple studies and showed moderate diagnostic performance, with relatively little difference in performance between tools. While encouraging, further large and well-conducted studies are needed to ensure these tools are acceptable to patients and clinicians, are cost-effective and facilitate the early diagnosis of ovarian cancer.

Abstract

In the absence of effective ovarian cancer screening programs, most women are diagnosed following the onset of symptoms. Symptom-based tools, including symptom checklists and risk prediction models, have been developed to aid detection. The aim of this systematic review was to identify and compare the diagnostic performance of these tools. We searched MEDLINE, EMBASE and Cochrane CENTRAL, without language restriction, for relevant studies published between 1 January 2000 and 3 March 2020. We identified 1625 unique records and included 16 studies, evaluating 21 distinct tools in a range of settings. Fourteen tools included only symptoms; seven also included risk factors or blood tests. Four tools were externally validated—the Goff Symptom Index (sensitivity: 56.9–83.3%; specificity: 48.3–98.9%), a modified Goff Symptom Index (sensitivity: 71.6%; specificity: 88.5%), the Society of Gynaecologic Oncologists consensus criteria (sensitivity: 65.3–71.5%; specificity: 82.9–93.9%) and the QCancer Ovarian model (10% risk threshold—sensitivity: 64.1%; specificity: 90.1%). Study heterogeneity precluded meta-analysis. Given the moderate accuracy of several tools on external validation, they could be of use in helping to select women for ovarian cancer investigations. However, further research is needed to assess the impact of these tools on the timely detection of ovarian cancer and on patient survival.

Stratifying Ovarian Cancer Risk Using Personal Health Data

Purpose: Screening the general population for ovarian cancer is not recommended by every major medical or public health organization because the harms from screening outweigh the benefit it provides. To improve ovarian cancer detection and survival many are looking at high-risk populations who would benefit from screening.

Methods: We train a neural network on readily available personal health data to predict and stratify ovarian cancer risk. We use two different datasets to train our network: The National Health Interview Survey and Prostate, Lung, Colorectal, and Ovarian Cancer Screening Trial.

Results: Our model has an area under the receiver operating characteristic curve of 0.71. We further demonstrate how the model could be used to stratify patients into different risk categories. A simple 3-tier scheme classifies 23.8% of those with cancer and 1.0% of those without as high-risk similar to genetic testing, and 1.1% of those with cancer and 24.4% of those without as low risk.

Conclusion: The developed neural network offers a cost-effective and non-invasive way to identify those who could benefit from targeted screening.

Availability and use of cancer decision-support tools: a cross-sectional survey of UK primary care

Background

Decision-support tools quantify the risk of undiagnosed cancer in symptomatic patients, and may help GPs when making referrals.

Aim

To quantify the availability and use of cancer decision-support tools (QCancer^® and risk assessment tools) and to explore the association between tool availability and 2-week-wait (2WW) referrals for suspected cancer.

Design and setting

A cross-sectional postal survey in UK primary care.

Methods

Out of 975 UK randomly selected general practices, 4600 GPs and registrars were invited to participate. Outcome measures included the proportions of UK general practices where cancer decision-support tools are available and at least one GP uses the tool. Weighted least-squares linear regression with robust errors tested the association between tool availability and number of 2WW referrals, adjusting for practice size, sex, age, and Index of Multiple Deprivation.

Results

In total, 476 GPs in 227 practices responded (response rates: practitioner, 10.3%; practice, 23.3%). At the practice level, 83/227 (36.6%, 95% confidence interval [CI] = 30.3 to 43.1) practices had at least one GP or registrar with access to cancer decision-support tools. Tools were available and likely to be used in 38/227 (16.7%, 95% CI = 12.1 to 22.2) practices. In subgroup analyses of 172 English practices, there was no difference in mean 2WW referral rate between practices with tools and those without (mean adjusted difference in referrals per 100 000: 3.1, 95% CI = −5.5 to 11.7).

Conclusion

This is the first survey of cancer decision-support tool availability and use. It suggests that the tools are an underused resource in the UK. Given the cost of cancer investigation, a randomised controlled trial of such clinical decision-support aids would be appropriate.

Development and validation of prediction models to estimate risk of primary total hip and knee replacements using data from the UK: two prospective open cohorts using the UK Clinical Practice Research Datalink

Objectives

The ability to efficiently and accurately predict future risk of primary total hip and knee replacement (THR/TKR) in earlier stages of osteoarthritis (OA) has potentially important applications. We aimed to develop and validate two models to estimate an individual’s risk of primary THR and TKR in patients newly presenting to primary care.

Methods

We identified two cohorts of patients aged ≥40 years newly consulting hip pain/OA and knee pain/OA in the Clinical Practice Research Datalink. Candidate predictors were identified by systematic review, novel hypothesis-free ‘Record-Wide Association Study’ with replication, and panel consensus. Cox proportional hazards models accounting for competing risk of death were applied to derive risk algorithms for THR and TKR. Internal–external cross-validation (IECV) was then applied over geographical regions to validate two models.

Results

45 predictors for THR and 53 for TKR were identified, reviewed and selected by the panel. 301 052 and 416 030 patients newly consulting between 1992 and 2015 were identified in the hip and knee cohorts, respectively (median follow-up 6 years). The resultant model C-statistics is 0.73 (0.72, 0.73) and 0.79 (0.78, 0.79) for THR (with 20 predictors) and TKR model (with 24 predictors), respectively. The IECV C-statistics ranged between 0.70–0.74 (THR model) and 0.76–0.82 (TKR model); the IECV calibration slope ranged between 0.93–1.07 (THR model) and 0.92–1.12 (TKR model).

Conclusions

Two prediction models with good discrimination and calibration that estimate individuals’ risk of THR and TKR have been developed and validated in large-scale, nationally representative data, and are readily automated in electronic patient records.

Weight loss as a predictor of cancer in primary care: a systematic review and meta-analysis

BACKGROUND

Weight loss is a non-specific cancer symptom for which there are no clinical guidelines about investigation in primary care.

AIM

To summarise the available evidence on weight loss as a clinical feature of cancer in patients presenting to primary care.

DESIGN AND SETTING

A diagnostic test accuracy review and meta-analysis.

METHOD

Studies reporting 2 × 2 diagnostic accuracy data for weight loss (index test) in adults presenting to primary care and a subsequent diagnosis of cancer (reference standard) were included. QUADAS-2 was used to assess study quality. Sensitivity, specificity, positive likelihood ratios, and positive predictive values were calculated, and a bivariate meta-analysis performed.

RESULTS

A total of 25 studies were included, with 23 (92%) using primary care records. Of these, 20 (80%) defined weight loss as a physician's coding of the symptom; the remainder collected data directly. One defined unexplained weight loss using objective measurements. Positive associations between weight loss and cancer were found for 10 cancer sites: prostate, colorectal, lung, gastro-oesophageal, pancreatic, non-Hodgkin's lymphoma, ovarian, myeloma, renal tract, and biliary tree. Sensitivity ranged from 2% to 47%, and specificity from 92% to 99%, across cancer sites. The positive predictive value for cancer in male and female patients with weight loss for all age groups ≥60 years exceeded the 3% risk threshold that current UK guidance proposes for further investigation.

CONCLUSION

A primary care clinician's decision to code for weight loss is highly predictive of cancer. For such patients, urgent referral pathways are justified to investigate for cancer across multiple sites.

Symptom Signatures and Diagnostic Timeliness in Cancer Patients: A Review of Current Evidence

Early diagnosis is an important aspect of contemporary cancer prevention and control strategies, as the majority of patients are diagnosed following symptomatic presentation. The nature of presenting symptoms can critically influence the length of the diagnostic intervals from symptom onset to presentation (the patient interval), and from first presentation to specialist referral (the primary care interval). Understanding which symptoms are associated with longer diagnostic intervals to help the targeting of early diagnosis initiatives is an area of emerging research. In this Review, we consider the methodological challenges in studying the presenting symptoms and intervals to diagnosis of cancer patients, and summarize current evidence on presenting symptoms associated with a range of common and rarer cancer sites. We propose a taxonomy of cancer sites considering their symptom signature and the predictive value of common presenting symptoms. Finally, we consider evidence on associations between symptomatic presentations and intervals to diagnosis before discussing implications for the design, implementation, and evaluation of public health or health system interventions to achieve the earlier detection of cancer.

Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model

Continuous predictors are routinely encountered when developing a prognostic model. Investigators, who are often non-statisticians, must decide how to handle continuous predictors in their models. Categorising continuous measurements into two or more categories has been widely discredited, yet is still frequently done because of its simplicity, investigator ignorance of the potential impact and of suitable alternatives, or to facilitate model uptake. We examine three broad approaches for handling continuous predictors on the performance of a prognostic model, including various methods of categorising predictors, modelling a linear relationship between the predictor and outcome and modelling a nonlinear relationship using fractional polynomials or restricted cubic splines. We compare the performance (measured by the c-index, calibration and net benefit) of prognostic models built using each approach, evaluating them using separate data from that used to build them. We show that categorising continuous predictors produces models with poor predictive performance and poor clinical usefulness. Categorising continuous predictors is unnecessary, biologically implausible and inefficient and should not be used in prognostic model development. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

A GPS-Based Methodology to Analyze Environment-Health Associations at the Trip Level: Case-Crossover Analyses of Built Environments and Walking

Environmental health studies have examined associations between context and health with individuals as statistical units. However, investigators have been unable to investigate momentary exposures, and such studies are often vulnerable to confounding from, for example, individual-level preferences. We present a Global Positioning System (GPS)-based methodology for segmenting individuals' observation periods into visits to places and trips, enabling novel life-segment investigations and case-crossover analysis for improved inferences. We analyzed relationships between built environments and walking in trips. Participants were tracked for 7 days with GPS receivers and accelerometers and surveyed with a Web-based mapping application about their transport modes during each trip (Residential Environment and Coronary Heart Disease (RECORD) GPS Study, France, 2012-2013; 6,313 trips made by 227 participants). Contextual factors were assessed around residences and the trips' origins and destinations. Conditional logistic regression modeling was used to estimate associations between environmental factors and walking or accelerometry-assessed steps taken in trips. In case-crossover analysis, the probability of walking during a trip was 1.37 (95% confidence interval: 1.23, 1.61) times higher when trip origin was in the fourth (vs. first) quartile of service density and 1.47 (95% confidence interval: 1.23, 1.68) times higher when trip destination was in the fourth (vs. first) quartile of service density. Green spaces at the origin and destination of trips were also associated with within-individual, trip-to-trip variations in walking. Our proposed approach using GPS and Web-based surveys enables novel life-segment epidemiologic investigations.

Risk Prediction for Epithelial Ovarian Cancer in 11 United States-Based Case-Control Studies: Incorporation of Epidemiologic Risk Factors and 17 Confirmed Genetic Loci

Previously developed models for predicting absolute risk of invasive epithelial ovarian cancer have included a limited number of risk factors and have had low discriminatory power (area under the receiver operating characteristic curve (AUC) < 0.60). Because of this, we developed and internally validated a relative risk prediction model that incorporates 17 established epidemiologic risk factors and 17 genome-wide significant single nucleotide polymorphisms (SNPs) using data from 11 case-control studies in the United States (5,793 cases; 9,512 controls) from the Ovarian Cancer Association Consortium (data accrued from 1992 to 2010). We developed a hierarchical logistic regression model for predicting case-control status that included imputation of missing data. We randomly divided the data into an 80% training sample and used the remaining 20% for model evaluation. The AUC for the full model was 0.664. A reduced model without SNPs performed similarly (AUC = 0.649). Both models performed better than a baseline model that included age and study site only (AUC = 0.563). The best predictive power was obtained in the full model among women younger than 50 years of age (AUC = 0.714); however, the addition of SNPs increased the AUC the most for women older than 50 years of age (AUC = 0.638 vs. 0.616). Adapting this improved model to estimate absolute risk and evaluating it in prospective data sets is warranted.

A Systematic Review of Symptoms for the Diagnosis of Ovarian Cancer

CONTEXT

Ovarian cancer is common and has significant morbidity and mortality, partly because it is often diagnosed at a late stage. This study sought to determine the accuracy of individual symptoms and combinations of symptoms for the diagnosis of ovarian cancer.

EVIDENCE ACQUISITION

MEDLINE was searched, identifying 2,492 abstracts, reviewing 71 articles in full, and ultimately identifying 17 studies published between 2001 and 2014 that met the inclusion criteria. Data were abstracted by two researchers, and quality was assessed using the QUADAS-2 criteria adapted to the study question. Bivariate random effects meta-analysis was used where possible, and heterogeneity and threshold effects were explored using receiver operating characteristic curves. Data were analyzed in 2015.

EVIDENCE SYNTHESIS

Most studies were at high risk of bias, primarily because of case-control design or differential verification bias. The highest positive likelihood ratios (LRs+) were found for presence of abdominal mass (LR+, 30.0); abdominal distension or increased girth (LR+, 16.0); abdominal or pelvic pain (LR+, 10.4); abdominal or pelvic bloating (LR+, 9.3); loss of appetite (LR+, 9.2); and a family history of ovarian cancer (LR+, 7.5). No symptoms were helpful at ruling out ovarian cancer when absent. The Ovarian Cancer Symptom Index was validated in five studies and (after excluding one outlier with different inclusion criteria) was 63% sensitive and 95% specific (LR+, 12.6; LR-, 0.39). Two other symptom scores had not been validated prospectively.

CONCLUSIONS

Several individual signs and symptoms significantly increase the likelihood of ovarian cancer when present. More work is needed to validate decision rules and develop new decision support tools integrating risk factors, symptoms, and possibly biomarkers to identify women at increased ovarian cancer risk.

Sample size considerations for the external validation of a multivariable prognostic model: a resampling study

After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c-index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events.

Identifying patients with undetected pancreatic cancer in primary care: an independent and external validation of QCancer(®) (Pancreas)

BACKGROUND

Despite its rarity, the prognosis of pancreatic cancer is very poor and it is a major cause of cancer mortality; being ranked fourth in the world, it has one of the worst survival rates of any cancer.

AIM

To evaluate the performance of QCancer(®) (Pancreas) for predicting the absolute risk of pancreatic cancer in an independent UK cohort of patients, from general practice records.

DESIGN AND SETTING

Prospective cohort study to evaluate the performance QCancer (Pancreas) prediction models in 364 practices from the UK, contributing to The Health Improvement Network (THIN) database.

METHOD

Records were extracted from the THIN database for 2.15 million patients registered with a general practice surgery between 1 January 2000 and 30 June 2008, aged 30-84 years (3.74 million person-years), with 618 pancreatic cancer cases. Pancreatic cancer was defined as incident diagnosis of pancreatic cancer during the 2 years after study entry.

RESULTS

The results from this independent and external validation of QCancer (Pancreas) demonstrated good performance data on a large cohort of general practice patients. QCancer (Pancreas) had very good discrimination properties, with areas under the receiver operating characteristic curve of 0.89 and 0.92 for females and males respectively. QCancer (Pancreas) explained 60% and 67% of the variation in females and males respectively. QCancer (Pancreas) over-predicted risk in both females and males, notably in older patients.

CONCLUSION

QCancer (Pancreas) is potentially useful for identifying undetected cases of pancreatic cancer in primary care in the UK.

Identifying patients with undetected renal tract cancer in primary care: an independent and external validation of QCancer® (Renal) prediction model

INTRODUCTION

To evaluate the performance of QCancer® (Renal) for predicting the absolute risk of renal tract cancer in a large independent UK cohort of patients from general practice records.

MATERIALS AND METHODS

Open cohort study to validate QCancer® (Renal) prediction model. Record from 365 practices from United Kingdom contributing to The Health Improvement Network (THIN) database. 2.1 million patients registered with a general practice surgery between 01 January 2000 and 30 June 2008, aged 30-84 years (3.7 million person years) with 2283 renal tract cancer cases. Renal tract cancer was defined as incident diagnosis of renal tract cancer during the 2 years after study entry. Model discrimination was measured using the receiver operating characteristics derived area under the curve. Calibration plots examined the relationship between predicted and observed probabilities of undetected renal tract cancer.

RESULTS

The results from this independent and external validation of QCancer® (Renal) demonstrated good performance data on a large cohort of general practice patients. QCancer® (Renal) had very good discrimination with areas under the ROC curve of 0.92 and 0.95 for women and men respectively. QCancer® (Renal) was well calibrated across all tenths of risk and over all age ranges with predicted risks closely matching observed risks. QCancer® (Renal) explained 74.4% and 74.2% of the variation in men and women respectively. A limitation of our study is the recording of symptoms might be less complete, as patients with mild symptoms may not visit their general practitioner or not report mild symptoms.

CONCLUSIONS

QCancer® (Renal) are useful tools to help in identifying undetected cases of undiagnosed renal tract cancer in primary care in the UK.