Developing and Validating a Primary Care EMR-based Frailty Definition using Machine Learning

Development and validation of machine learning models to predict frailty risk for elderly

AIMS

Early identification and intervention of the frailty of the elderly will help lighten the burden of social medical care and improve the quality of life of the elderly. Therefore, we used machine learning (ML) algorithm to develop models to predict frailty risk in the elderly.

DESIGN

A prospective cohort study.

METHODS

We collected data on 6997 elderly people from Chinese Longitudinal Healthy Longevity Study wave 6-7 surveys (2011-2012, 2014). After the baseline survey in 1998 (wave 1), the project conducted follow-up surveys (wave 2-8) in 2000-2018. The osteoporotic fractures index was used to assess frailty. Four ML algorithms (random forest [RF], support vector machine, XGBoost and logistic regression [LR]) were used to develop models to identify the risk factors of frailty and predict the risk of frailty. Different ML models were used for the prediction of frailty risk in the elderly and frailty risk was trained on a cohort of 4385 elderly people with frailty (split into a training cohort [75%] and internal validation cohort [25%]). The best-performing model for each study outcome was tested in an external validation cohort of 6997 elderly people with frailty pooled from the surveys (wave 6-7). Model performance was assessed by receiver operating curve and F2-score.

RESULTS

Among the four ML models, the F2-score values were similar (0.91 vs. 0.91 vs. 0.88 vs. 0.90), and the area under the curve (AUC) values of RF model was the highest (0.75), followed by LR model (0.74). In the final two models, the AUC values of RF and LR model were similar (0.77 vs. 0.76) and their accuracy was identical (87.4% vs. 87.4%).

CONCLUSION

Our study developed a preliminary prediction model based on two different ML approaches to help predict frailty risk in the elderly.

IMPACT

The presented models from this study can be used to inform healthcare providers to predict the frailty probability among older adults and maybe help guide the development of effective frailty risk management interventions.

IMPLICATIONS FOR THE PROFESSION AND/OR PATIENT CARE

Detecting frailty at an early stage and implementing timely targeted interventions may help to improve the allocation of health care resources and to reduce frailty-related burden. Identifying risk factors for frailty could be beneficial to provide tailored and personalized care intervention for older adults to more accurately prevent or improve their frail conditions so as to improve their quality of life.

REPORTING METHOD

The study has adhered to STROBE guidelines.

PATIENT OR PUBLIC CONTRIBUTION

No patient or public contribution.

Perspectives on the representation of frailty in the electronic frailty index

BACKGROUND

Frailty is a state of increased vulnerability from physical, social, and cognitive factors resulting in greater risk of negative health-related outcomes and increased healthcare expenditure. A 36-factor electronic frailty index (eFI) developed in the United Kingdom calculates frailty scores using electronic medical record data. There is currently no standardization of frailty screening in Canadian primary care. In order to implement the eFI in a Canadian context, adaptation of the tool is necessary because frailty is represented by different clinical terminologies in the UK and Canada. In considering the promise of implementing an eFI in British Columbia, Canada, we first looked at the content validation of the 36-factor eFI. Our research question was: Does the eFI represent frailty from the perspectives of primary care clinicians and older adults in British Columbia?

METHODS

A modified Delphi using three rounds of questionnaires with a panel of 23 experts (five family physicians, five nurse practitioners, five nurses, four allied health professionals, four older adults) reviewed and provided feedback on the 36-factor eFI. These professional groups were chosen because they closely work as interprofessional teams within primary care settings with older adults. Older adults provide real life context and experiences. Questionnaires involved rating the importance of each frailty factor on a 0-10 scale and providing rationale for ratings. Panelists were also given the opportunity to suggest additional factors that ought to be included in the screening tool. Suggested factors were similarly rated in two Delphi rounds.

RESULTS

Thirty-three of the 36 eFI factors achieved consensus (> 80% of panelists provided a rating of ≥ 8). Factors that did not achieve consensus were hypertension, thyroid disorder and peptic ulcer. These factors were perceived as easily treatable or manageable and/or not considered reflective of frailty on their own. Additional factors suggested by panelists that achieved consensus included: cancer, challenges to healthcare access, chronic pain, communication challenges, fecal incontinence, food insecurity, liver failure/cirrhosis, mental health challenges, medication noncompliance, poverty/financial difficulties, race/ethnic disparity, sedentary/low activity levels, and substance use/misuse. There was a 100% retention rate in each of the three Delphi rounds.

CONCLUSIONS AND NEXT STEPS

Three key findings emerged from this study: the conceptualization of frailty varied across participants, identification of frailty in community/primary care remains challenging, and social determinants of health affect clinicians' assessments and perceptions of frailty status. This study will inform the next phase of a broader mixed-method sequential study to build a frailty screening tool that could ultimately become a standard of practice for frailty screening in Canadian primary care. Early detection of frailty can help tailor decision making, frame discussions about goals of care, prevent advancement on the frailty trajectory, and ultimately decrease health expenditures, leading to improved patient and system level outcomes.

Digital health technology combining wearable gait sensors and machine learning improve the accuracy in prediction of frailty

Background

Frailty is a dynamic and complex geriatric condition characterized by multi-domain declines in physiological, gait and cognitive function. This study examined whether digital health technology can facilitate frailty identification and improve the efficiency of diagnosis by optimizing analytical and machine learning approaches using select factors from comprehensive geriatric assessment and gait characteristics.

Methods

As part of an ongoing study on observational study of Aging, we prospectively recruited 214 individuals living independently in the community of Southern China. Clinical information and fragility were assessed using comprehensive geriatric assessment (CGA). Digital tool box consisted of wearable sensor-enabled 6-min walk test (6MWT) and five machine learning algorithms allowing feature selections and frailty classifications.

Results

It was found that a model combining CGA and gait parameters was successful in predicting frailty. The combination of these features in a machine learning model performed better than using either CGA or gait parameters alone, with an area under the curve of 0.93. The performance of the machine learning models improved by 4.3-11.4% after further feature selection using a smaller subset of 16 variables. SHapley Additive exPlanation (SHAP) dependence plot analysis revealed that the most important features for predicting frailty were large-step walking speed, average step size, age, total step walking distance, and Mini Mental State Examination score.

Conclusion

This study provides evidence that digital health technology can be used for predicting frailty and identifying the key gait parameters in targeted health assessments.

Using electronic consultation (eConsult) to identify frailty in provider-to-provider communication: a feasibility and validation study

BACKGROUND

Frailty is a complex age-related clinical condition that increases vulnerability to stressors. Early recognition of frailty is challenging. While primary care providers (PCPs) serve as the first point of contact for most older adults, convenient tools for identifying frailty in primary care are lacking. Electronic consultation (eConsult), a platform connecting PCPs to specialists, is a rich source of provider-to-provider communication data. Text-based patient descriptions on eConsult may provide opportunities for earlier identification of frailty. We sought to explore the feasibility and validity of identifying frailty status using eConsult data.

METHODS

eConsult cases closed in 2019 and submitted on behalf of long-term care (LTC) residents or community-dwelling older adults were sampled. A list of frailty-related terms was compiled through a review of the literature and consultation with experts. To identify frailty, eConsult text was parsed to measure the frequency of frailty-related terms. Feasibility of this approach was assessed by examining the availability of frailty-related terms in eConsult communication logs, and by asking clinicians to indicate whether they can assess likelihood of frailty by reviewing the cases. Construct validity was assessed by comparing the number of frailty-related terms in cases about LTC residents with those about community-dwelling older adults. Criterion validity was assessed by comparing clinicians' ratings of frailty to the frequency of frailty-related terms.

RESULTS

One hundred thirteen LTC and 112 community cases were included. Frailty-related terms identified per case averaged 4.55 ± 3.95 in LTC and 1.96 ± 2.68 in the community (p < .001). Clinicians consistently rated cases with ≥ 5 frailty-related terms as highly likely of living with frailty.

CONCLUSIONS

The availability of frailty-related terms establishes the feasibility of using provider-to-provider communication on eConsult to identify patients with high likelihood of living with this condition. The higher average of frailty-related terms in LTC (versus community) cases, and agreement between clinician-provided frailty ratings and the frequency of frailty-related terms, support the validity of an eConsult-based approach to identifying frailty. There is potential for eConsult to be used as a case-finding tool in primary care for early recognition and proactive initiation of care processes for older patients living with frailty.

Identifying Frail Patients by Using Electronic Health Records in Primary Care: Current Status and Future Directions

With the rapidly aging population, frailty, characterized by an increased risk of adverse outcomes, has become a major public health problem globally. Several frailty guidelines or consensuses recommend screening for frailty, especially in primary care settings. However, most of the frailty assessment tools are based on questionnaires or physical examinations, adding to the clinical workload, which is the major obstacle to converting frailty research into clinical practice. Medical data naturally generated by routine clinical work containing frailty indicators are stored in electronic health records (EHRs) (also called electronic health record (EHR) data), which provide resources and possibilities for frailty assessment. We reviewed several frailty assessment tools based on primary care EHRs and summarized the features and novel usage of these tools, as well as challenges and trends. Further research is needed to develop and validate frailty assessment tools based on EHRs in primary care in other parts of the world.

Strategies for working across Canadian practice-based research and learning networks (PBRLNs) in primary care: focus on frailty

Background

Practice based research and learning networks (PBRLNs) are groups of learning communities that focus on improving delivery and quality of care. Accurate data from primary care electronic medical records (EMRs) is crucial in forming the backbone for PBRLNs. The purpose of this work is to: (1) report on descriptive findings from recent frailty work, (2) describe strategies for working across PBRLNs in primary care, and (3) provide lessons learned for engaging PBRLNs.

Methods

We carried out a participatory based descriptive study that engaged five different PBRLNs. We collected Clinical Frailty Scale scores from a sample of participating physicians within each PBRLN. Descriptive statistics were used to analyze frailty scores and patients’ associated risk factors and demographics. We used the Consolidated Framework for Implementation Research to inform thematic analysis of qualitative data (meeting minutes, notes, and conversations with co-investigators of each network) in recognizing challenges of working across networks.

Results

One hundred nine physicians participated in collecting CFS scores across the five provinces (n = 5466). Percentages of frail (11-17%) and not frail (82-91%) patients were similar in all networks, except Ontario who had a higher percentage of frail patients (25%). The majority of frail patients were female (65%) and had a significantly higher prevalence of hypertension, dementia, and depression. Frail patients had more prescribed medications and numbers of healthcare encounters. There were several noteworthy challenges experienced throughout the research process related to differences across provinces in the areas of: numbers of stakeholders/staff involved and thus levels of burden, recruitment strategies, data collection strategies, enhancing engagement, and timelines.

Discussion

Lessons learned throughout this multi-jurisdictional work included: the need for continuity in ethics, regular team meetings, enhancing levels of engagement with stakeholders, the need for structural support and recognizing differences in data sharing across provinces.

Conclusion

The differences noted across CPCSSN networks in our frailty study highlight the challenges of multi-jurisdictional work across provinces and the need for consistent and collaborative healthcare planning efforts.

Machine learning for identification of frailty in Canadian primary care practices

Introduction

Frailty is a medical syndrome, commonly affecting people aged 65 years and over and is characterized by a greater risk of adverse outcomes following illness or injury. Electronic medical records contain a large amount of longitudinal data that can be used for primary care research. Machine learning can fully utilize this wide breadth of data for the detection of diseases and syndromes. The creation of a frailty case definition using machine learning may facilitate early intervention, inform advanced screening tests, and allow for surveillance.

Objectives

The objective of this study was to develop a validated case definition of frailty for the primary care context, using machine learning.

Methods

Physicians participating in the Canadian Primary Care Sentinel Surveillance Network across Canada were asked to retrospectively identify the level of frailty present in a sample of their own patients (total n = 5,466), collected from 2015–2019. Frailty levels were dichotomized using a cut-off of 5. Extracted features included previously prescribed medications, billing codes, and other routinely collected primary care data. We used eight supervised machine learning algorithms, with performance assessed using a hold-out test set. A balanced training dataset was also created by oversampling. Sensitivity analyses considered two alternative dichotomization cut-offs. Model performance was evaluated using area under the receiver-operating characteristic curve, F1, accuracy, sensitivity, specificity, negative predictive value and positive predictive value.

Results

The prevalence of frailty within our sample was 18.4%. Of the eight models developed to identify frail patients, an XGBoost model achieved the highest sensitivity (78.14%) and specificity (74.41%). The balanced training dataset did not improve classification performance. Sensitivity analyses did not show improved performance for cut-offs other than 5.

Conclusion

Supervised machine learning was able to create well performing classification models for frailty. Future research is needed to assess frailty inter-rater reliability, and link multiple data sources for frailty identification.

Can Linked Electronic Medical Record and Administrative Data Help Us Identify Those Living with Frailty?

Introduction

Frailty is a complex condition that affects many aspects of patients’ wellbeing and health outcomes.

Objectives

We used available Electronic Medical Record (EMR) and administrative data to determine definitions of frailty. We also examined whether there were differences in demographics or health conditions among those identified as frail in either the EMR or administrative data.

Methods

EMR and administrative data were linked in British Columbia (BC) and Manitoba (MB) to identify those aged 65 years and older who were frail. The EMR data were obtained from the Canadian Primary Care Sentinel Surveillance Network (CPCSSN) and the administrative data (e.g. billing, hospitalizations) was obtained from Population Data BC and the Manitoba Population Research Data Repository. Sociodemographic characteristics, risk factors, prescribed medications, use and costs of healthcare are described for those identified as frail.

Results

Sociodemographic and utilization differences were found among those identified as frail from the EMR compared to those in the administrative data. Among those who were >65 years, who had a record in both EMR and administrative data, 5%-8% (n=191 of 3,553, BC; n=2,396 of 29,382, MB) were identified as frail. There was a higher likelihood of being frail with increasing age and being a woman. In BC and MB, those identified as frail in both data sources have approximately twice the number of contacts with primary care (n=20 vs. n=10) and more days in hospital (n=7.2 vs. n=1.9 in BC; n=9.8 vs. n=2.8 in MB) compared to those who are not frail; 27% (BC) and 14% (MB) of those identified as frail in 2014 died in 2015.

Conclusions

Identifying frailty using EMR data is particularly challenging because many functional deficits are not routinely recorded in structured data fields. Our results suggest frailty can be captured along a continuum using both EMR and administrative data.