Validation studies of claims data in the Asia‐Pacific region: A comprehensive review

Bayesian Latent Class Models for Evaluating the Validity of Claim-based Definitions of Disease Outcomes

BACKGROUND

Large electronic databases have been widely used in recent years; however, they can be susceptible to bias due to incomplete information. To address this, validation studies have been conducted to assess the accuracy of disease diagnoses defined in databases. However, such studies may be constrained by potential misclassification in references and the interdependence between diagnoses from the same data source.

METHODS

This study employs latent class modeling with Bayesian inference to estimate the sensitivity, specificity, and positive/negative predictive values of different diagnostic definitions. Four models are defined with/without assumptions of the gold standard and conditional independence, and then compared with breast cancer study data as a motivating example. Additionally, simulations that generated data under various true values are used to compare the performance of each model with bias, Pearson-type goodness-of-fit statistics, and widely applicable information criterion.

RESULTS

The model assuming conditional dependence and non-gold standard references exhibited the best predictive performance among the four models in the motivating example data analysis. The disease prevalence was slightly higher than that in previous findings, and the sensitivities were significantly lower than those of the other models. Additionally, bias evaluation showed that the Bayesian models with more assumptions and the frequentist model performed better under the true value conditions. The Bayesian model with fewer assumptions performed well in terms of goodness of fit and widely applicable information criteria.

CONCLUSIONS

The current assessments of outcome validation can introduce bias. The proposed approach can be adopted broadly as a valuable method for validation studies.

Validation of claims-based algorithms for rheumatoid arthritis in Japan: Results from the VALIDATE-J study

AIM

Validity of Algorithms in Large Databases: Infectious Diseases, Rheumatoid Arthritis, and Tumor Evaluation in Japan (VALIDATE-J) study examined algorithms for identifying rheumatoid arthritis (RA) in Japanese claims data.

METHODS

VALIDATE-J was a multicenter, cross-sectional retrospective study. Disease-identifying algorithms were used to detect RA diagnosed between January 2012 and December 2016 using claims data from two Japanese hospitals. An RA diagnosis was confirmed using one of four gold standard definitions. Positive predictive values (PPVs) were calculated for prevalent (regardless of baseline RA-free period) and incident (preceded by a 12-month RA-free period) cases.

RESULTS

Of patients identified using claims-based algorithms, a random sample of 389 prevalent and 134 incident cases of RA were included. Cases identified by an RA diagnosis, no diagnosis of psoriasis, and treatment with any disease-modifying antirheumatic drugs (DMARDs) resulted in the highest PPVs versus other claims-based treatment categories (29.0%-88.3% [prevalent] and 41.0%-78.2% [incident]); cases identified by an RA diagnosis, no diagnosis of psoriasis, and glucocorticoid-only treatment had the lowest PPVs. Across claims-based algorithms, PPVs were highest when a physician diagnosis or decision by adjudicators (confirmed and probable cases) was used as the gold standard and were lowest when American College of Rheumatology/European Alliance of Associations for Rheumatology 2010 criteria were applied. PPVs of claims-based algorithms for RA in patients aged ≥66 years were slightly higher versus a USA Medicare population (maximum PPVs of 95.0% and 88.9%, respectively).

CONCLUSION

VALIDATE-J demonstrated high PPVs for most claims-based algorithms for diagnosis of prevalent and incident RA using Japanese claims data. These findings will help inform appropriate RA definitions for future claims database research in Japan.

Identification of complications requiring interventions after gastrointestinal cancer surgery from real-world data: An external validation study

Background

Recently, real-world data have been recognized to have a significant role for research and quality improvement worldwide. The decision on the existence or nonexistence of postoperative complications is complex in clinical practice. This multicenter validation study aimed to evaluate the accuracy of identification of patients who underwent gastrointestinal (GI) cancer surgery and extraction of postoperative complications from Japanese administrative claims data.

Methods

We compared data extracted from both the Diagnosis Procedure Combination (DPC) and chart review of patients who underwent GI cancer surgery from April 2016 to March 2019. Using data of 658 patients at Kyoto University Hospital, we developed algorithms for the extraction of patients and postoperative complications requiring interventions, which included an invasive procedure, reoperation, mechanical ventilation, hemodialysis, intensive care unit management, and in-hospital mortality. The accuracy of the algorithms was externally validated using the data of 1708 patients at two other hospitals.

Results

In the overall validation set, 1694 of 1708 eligible patients were correctly extracted by DPC (sensitivity 0.992 and positive predictive value 0.992). All postoperative complications requiring interventions had a sensitivity of >0.798 and a specificity of almost 1.000. The overall sensitivity and specificity of Clavien-Dindo ≥grade IIIb complications was 1.000 and 0.995, respectively.

Conclusion

Patients undergoing GI cancer surgery and postoperative complications requiring interventions can be accurately identified using the real-world data. This multicenter external validation study may contribute to future research on hospital quality improvement or to a large-scale comparison study among nationwide hospitals using real-world data.

Validation study of case-identifying algorithms for severe hypoglycemia using hospital administrative data in Japan

OBJECTIVE

The purpose of this study was to evaluate the performance of algorithms for identifying cases of severe hypoglycemia in Japanese hospital administrative data.

METHODS

This was a multicenter, retrospective, observational study conducted at 3 acute-care hospitals in Japan. The study population included patients aged ≥18 years with diabetes who had an outpatient visit or hospital admission for possible hypoglycemia. Possible cases of severe hypoglycemia were identified using health insurance claims data and Diagnosis Procedure Combination data. Sixty-one algorithms using combinations of diagnostic codes and prescription of high concentration (≥20% mass/volume) injectable glucose were used to define severe hypoglycemia. Independent manual chart reviews by 2 physicians at each hospital were used as the reference standard. Algorithm validity was evaluated using standard performance metrics.

RESULTS

In total, 336 possible cases of severe hypoglycemia were identified, and 260 were consecutively sampled for validation. The best performing algorithms included 6 algorithms that had sensitivity ≥0.75, and 6 algorithms that had positive predictive values ≥0.75 with sensitivity ≥0.30. The best-performing algorithm with sensitivity ≥0.75 included any diagnoses for possible hypoglycemia or prescription of high-concentration glucose but excluded suspected diagnoses (sensitivity: 0.986 [95% confidence interval 0.959-1.013]; positive predictive value: 0.345 [0.280-0.410]). Restricting the algorithm definition to those with both a diagnosis of possible hypoglycemia and a prescription of high-concentration glucose improved the performance of the algorithm to correctly classify cases as severe hypoglycemia but lowered sensitivity (sensitivity: 0.375 [0.263-0.487]; positive predictive value: 0.771 [0.632-0.911]).

CONCLUSION

The case-identifying algorithms in this study showed moderate positive predictive value and sensitivity for identification of severe hypoglycemia in Japanese healthcare data and can be employed by future pharmacoepidemiological studies using Japanese hospital administrative databases.

Validation of diagnosis codes in healthcare databases in Taiwan, a literature review

PURPOSE

To compile validation findings of diagnosis codes and related algorithms for health outcomes of interest from National Health Insurance (NHI) or electronic medical records in Taiwan.

METHODS

We carried out a literature review of English articles in PubMed® and Embase from 2000 through July 2022 with appropriate search terms. Potentially relevant articles were identified through review of article titles and abstracts, full text search of methodology terms "validation", "positive predictive value", and "algorithm" in Subjects & Methods (or Methods) and Results sections of articles, followed by full text review of potentially eligible articles.

RESULTS

We identified 50 published reports with validation findings of diagnosis codes and related algorithms for a wide range of health outcomes of interest in Taiwan, including cardiovascular diseases, stroke, renal impairment, malignancy, diabetes, mental health diseases, respiratory diseases, viral (B and C) hepatitis, and tuberculosis. Most of the reported PPVs were in the 80% ~ 99% range. Assessment of algorithms based on ICD-10 systems were reported in 8 articles, all published in 2020 or later.

CONCLUSIONS

Investigators have published validation reports that may serve as empirical evidence to evaluate the utility of secondary health data environment in Taiwan for research and regulatory purpose.

Validation studies of Japanese administrative health care data: A scoping review

PURPOSE

Large-scale administrative health care databases are increasingly being utilized for research. However, there has not been much literature that validated administrative data in Japan; a previous review identified six validation studies published between 2011 and 2017. We conducted a literature review of studies that assessed the validity of Japanese administrative health care data.

METHODS

We searched for studies published by March 2022 that compared individual-level administrative data with a reference standard from another data source, as well as studies that validated administrative data using other data within the same database. The eligible studies were also summarized based on characteristics which included data types, settings, reference standard used, numbers of patients, and conditions validated.

RESULTS

There were 36 eligible studies, including 29 that used external reference standard and seven that validated administrative data using other data within the same database. Chart review was the reference standard in 21 studies (range of the numbers of patients, 72-1674; 11 studies conducted in single institutions and nine studies in 2-5 institutions). Five studies used a disease registry as the reference standard. Diagnoses of cardiovascular diseases, cancer, and diabetes were frequently evaluated.

CONCLUSIONS

Validation studies are being conducted at an increasing rate in Japan, although most of them are small scale. Further large-scale comprehensive validation studies are necessary to effectively utilize the databases for research.

Validation of algorithms to identify colorectal cancer patients from administrative claims data of a Japanese hospital

BACKGROUND

Administrative claims data are a valuable source for clinical studies; however, the use of validated algorithms to identify patients is essential to minimize bias. We evaluated the validity of diagnostic coding algorithms for identifying patients with colorectal cancer from a hospital's administrative claims data.

METHODS

This validation study used administrative claims data from a Japanese university hospital between April 2017 and March 2019. We developed diagnostic coding algorithms, basically based on the International Classification of Disease (ICD) 10th codes of C18-20 and Japanese disease codes, to identify patients with colorectal cancer. For random samples of patients identified using our algorithms, case ascertainment was performed using chart review as the gold standard. The positive predictive value (PPV) was calculated to evaluate the accuracy of the algorithms.

RESULTS

Of 249 random samples of patients identified as having colorectal cancer by our coding algorithms, 215 were confirmed cases, yielding a PPV of 86.3% (95% confidence interval [CI], 81.5-90.1%). When the diagnostic codes were restricted to site-specific (right colon, left colon, transverse colon, or rectum) cancer codes, 94 of the 100 random samples were true cases of colorectal cancer. Consequently, the PPV increased to 94.0% (95% CI, 87.2-97.4%).

CONCLUSION

Our diagnostic coding algorithms based on ICD-10 codes and Japanese disease codes were highly accurate in detecting patients with colorectal cancer from this hospital's claims data. The exclusive use of site-specific cancer codes further improved the PPV from 86.3 to 94.0%, suggesting their desirability in identifying these patients more precisely.

Performance of algorithms for identifying patients with chronic hepatitis B or C infection in the french health insurance claims databases using the ANRS CO22 HEPATHER cohort

The validity of algorithms for identifying patients with chronic hepatitis B or C virus (HBV or HCV) infection in claims databases has been little explored. The performance of 15 algorithms was evaluated. Data from HBV- or HCV-infected patients enrolled between August 2012 and December 2015 in French hepatology centres (ANRS CO22 HEPATHER cohort) were individually linked to the French national health insurance system (SNDS). The SNDS covers 99% of the French population and contains healthcare reimbursement data. Performance metrics were calculated by comparing the viral status established by clinicians with those obtained with the algorithms identifying chronic HBV- and HCV-infected patients. A total of 14 751 patients (29% with chronic HBV and 63% with chronic HCV infection) followed-up until December 2018 were selected. Despite good specificity, the algorithms relying on ICD-10 codes performed poorly. By contrast, the multi-criteria algorithms combining ICD-10 codes, antiviral dispensing, laboratory diagnostic tests (HBV DNA or HCV RNA detection and quantification, HCV genotyping), examinations for the assessment of liver fibrosis and long-term disease registrations were the most effective (sensitivity 0.92, 95% CI, 0.91-0.93 and specificity 0.96, 95% CI, 0.95-0.96 for identifying chronic HBV-infected patients; sensitivity 0.94, 95% CI, 0.94-0.94 and specificity 0.85, 95% CI, 0.84-0.86 for identifying chronic HCV-infected patients). In conclusion, the multi-criteria algorithms perform well in identifying patients with chronic hepatitis B or C infection and can be used to estimate the magnitude of the public health burden associated with hepatitis B and C in France.

A data-informed approach using individualised dispensing patterns to estimate medicine exposure periods and dose from pharmaceutical claims data

Pharmaceutical claims data are often used as the primary information source to define medicine exposure periods in pharmacoepidemiological studies. However, often critical information on directions for use and the intended duration of medicine supply are not available. In the absence of this information, alternative approaches are needed to support the assignment of exposure periods. This study summarises the key methods commonly used to estimate medicine exposure periods and dose from pharmaceutical claims data; and describes a method using individualised dispensing patterns to define time-dependent estimates of medicine exposure and dose. This method extends on important features of existing methods and also accounts for recent changes in an individual's medicine use. Specifically, this method constructs medicine exposure periods and estimates the dose used by considering characteristics from an individual's prior dispensings, accounting for the time between prior dispensings and the amount supplied at prior dispensings. Guidance on the practical applications of this method is also provided. Although developed primarily for application to databases, which do not contain duration of supply or dose information, use of this method may also facilitate investigations when such information is available and there is a need to consider individualised and/or changing dosing regimens. By shifting the reliance on prescribed duration and dose to determine exposure and dose estimates, individualised dispensing information is used to estimate patterns of exposure and dose for an individual. Reflecting real-world individualised use of medicines with complex and variable dosing regimens, this method offers a pragmatic approach that can be applied to all medicine classes.

Decline in oral antimicrobial prescription in the outpatient setting after nationwide implementation of financial incentives and provider education: An interrupted time-series analysis

OBJECTIVES

To assess the impact of nationwide outpatient antimicrobial stewardship interventions in the form of financial incentives for providers and provider education when antimicrobials are deemed unnecessary for uncomplicated respiratory infections and acute diarrhea.

METHODS

We collected data from a large claims database from April 2013 through March 2020 and performed a quasi-experimental, interrupted time-series analysis. The outcome of interest was oral antimicrobial prescription rate defined as the number of monthly antimicrobial prescriptions divided by the number of outpatient visits each month. We examined the effects of financial incentive to providers (ie, targeted prescriptions for those aged ≤2 years) and provider education (ie, targeted prescriptions for those aged ≥6 years) on the overall antimicrobial prescription rates and how these interventions affected different age groups before and after their implementation.

RESULTS

In total, 21,647,080 oral antimicrobials were prescribed to 2,920,381 unique outpatients during the study period. At baseline, prescription rates for all age groups followed a downward trend throughout the study period. Immediately after the financial incentive implementation, substantial reductions in prescription rates were observed among only those aged 0-2 years (-47.5 prescriptions per 1,000 clinic visits each month; 95% confidence interval, -77.3 to -17.6; P = .003), whereas provider education immediately reduced prescription rates in all age groups uniformly. These interventions did not affect the long-term trend for any age group.

CONCLUSION

These results suggest that the nationwide implementation of financial incentives and provider education had an immediate effect on the antimicrobial prescription but no long-term effect.

Validation of Algorithms to Identify Bone Metastases Using Administrative Claims Data in a Japanese Hospital

BACKGROUND

Validated coding algorithms are essential to generate high-quality, real-world evidence from claims data studies.

OBJECTIVE

We aimed to evaluate the validity of the algorithms to identify patients with bone metastases using claims data from a Japanese hospital.

PATIENTS AND METHODS

This study used administrative claims data and electronic medical records at Juntendo University Hospital from April 2017 to March 2019. We developed two candidate claims-based algorithms to detect bone metastases, one based on diagnosis codes alone (Algorithm 1) and the other based on the combination of diagnosis and imaging test codes (Algorithm 2). Of the patients identified by Algorithm 1, 100 patients were randomly sampled. Among these 100 patients, 88 patients met the conditions of Algorithm 2; further, 12 additional patients were randomly sampled from those identified by Algorithm 2, thus obtaining a total of 100 patients for Algorithm 2. They were evaluated for their true diagnosis using the patient chart review as the gold standard. The positive predictive value (PPV) was calculated to assess the accuracy of each algorithm.

RESULTS

For Algorithm 1, 82 patients were analyzed after excluding 18 patients without diagnostic imaging reports. Of these, 69 patients were true positive by chart review, resulting in a PPV of 84.1% (95% confidence interval (CI) 74.5-90.6). For Algorithm 2, 92 patients were analyzed after excluding eight patients whose diagnoses were not judged by chart review. Of these, 76 patients were confirmed positive by chart review, yielding a PPV of 82.6% (95% CI 73.4-89.1).

CONCLUSION

Both claims-based algorithms yielded high PPVs of approximately 85%, with no improvement in PPV by adding imaging test conditions. The diagnosis code-based algorithm is sufficient and valid for identifying bone metastases in this Japanese hospital.

Validation study on definition of cause of death in Japanese claims data

Identifying the cause of death is important for the study of end-of-life patients using claims data in Japan. However, the validity of how cause of death is identified using claims data remains unknown. Therefore, this study aimed to verify the validity of the method used to identify the cause of death based on Japanese claims data. Our study population included patients who died at two institutions between January 1, 2018 and December 31, 2019. Claims data consisted of medical data and Diagnosis Procedure Combination (DPC) data, and five definitions developed from disease classification in each dataset were compared with death certificates. Nine causes of death, including cancer, were included in the study. The definition with the highest positive predictive values (PPVs) and sensitivities in this study was the combination of "main disease" in both medical and DPC data. For cancer, these definitions had PPVs and sensitivities of > 90%. For heart disease, these definitions had PPVs of > 50% and sensitivities of > 70%. For cerebrovascular disease, these definitions had PPVs of > 80% and sensitivities of> 70%. For other causes of death, PPVs and sensitivities were < 50% for most definitions. Based on these results, we recommend definitions with a combination of "main disease" in both medical and DPC data for cancer and cerebrovascular disease. However, a clear argument cannot be made for other causes of death because of the small sample size. Therefore, the results of this study can be used with confidence for cancer and cerebrovascular disease but should be used with caution for other causes of death.

Development and validation of claims-based algorithms to identify pregnancy based on data from a university hospital in Japan

OBJECTIVE

When using administrative data, validation is essential since these data are not collected for research purposes and misclassification can occur. Thus, this study aimed to develop algorithms identifying pregnancy and to evaluate the validity of administrative claims data in Japan.

METHODS

All females who visited the Tohoku University Hospital Department of Obstetrics in 2018 were included. The diagnosis, medical procedure, medication, and medical service addition fee data were utilized to identify pregnancy, with the electronic medical records set as the gold standard. Combination algorithms were developed using predefined pregnancy-related claims data with a positive predictive value (PPV) ≥80%. Sensitivity (SE), specificity (SP), PPV, and negative predictive value (NPV) with their corresponding 95% confidence intervals (CIs) were calculated for these combination algorithms.

RESULTS

This study included 1757 females with a mean age of 32.8 (standard deviation: 5.9) years. In general, the individual claims data were able to identify pregnancy with a PPV ≥80%; however, the number of pregnancies identified using a single claims data was limited. Based on the combination algorithm with all of the categories, including diagnosis, medical procedure, medication, and medical service addition, the calculated SE, SP, PPV, and NPV were 73.4% (95% CI: 71.2%-75.4%), 96.9% (95% CI: 89.3%-99.6%), 99.8%,(95% CI: 99.4%-100.0%), and 12.3% (95% CI: 9.6%-15.4%), respectively.

CONCLUSIONS

The combination algorithm to identify pregnancy demonstrated a high PPV and moderate SE. The algorithm validated in this study is expected to accelerate future studies that aim to identify pregnancies and evaluate pregnancy outcome.

Validity of Administrative Data for Identifying Birth-Related Outcomes with the End Date of Pregnancy in a Japanese University Hospital

This study aimed to develop and validate claims-based algorithms for identifying live birth, fetal death, and cesarean section by utilizing administrative data from a university hospital in Japan. We included women who visited the Department of Obstetrics at a university hospital in 2018. The diagnosis, medical procedures, and medication data were used to identify potential cases of live birth, fetal death, and cesarean section. By reviewing electronic medical records, we evaluated the positive predictive values (PPVs) and the accuracy of the end date of pregnancy for each claims datum. “Selected algorithm 1” based on PPVs and “selected algorithm 2” based on both the PPVs and the accuracy of the end date of pregnancy were developed. A total of 1757 women were included, and the mean age was 32.8 years. The PPVs of “selected algorithm 1” and “selected algorithm 2” were both 98.1% for live birth, 99.0% and 98.9% for fetal death, and 99.7% and 100.0% for cesarean section, respectively. These findings suggest that the developed algorithms are useful for future studies for evaluating live birth, fetal death, and cesarean section with an accurate end date of pregnancy.

Context and Considerations for Use of Two Japanese Real-World Databases in Japan: Medical Data Vision and Japanese Medical Data Center

In Japan, an increasing interest in real-world evidence for hypothesis generation and decision-making has emerged in order to overcome limitations and restrictions of clinical trials. We sought to characterize the context and concrete considerations of when to use Medical Data Vision (MDV) and JMDC databases, the main Japanese real-world data (RWD) sources accessible by pharmaceutical companies. Use cases for these databases, and related issues and considerations, were identified and summarized based on a literature search and experience-based knowledge. Studies conducted using MDV or JMDC were mostly descriptive in nature, or explored potential risk factors by evaluating associations with a target outcome. Considerations such as variable ascertainment at different time points, including issues relating to treatment identification and missing data, were highlighted for these two databases. Although several issues were commonly shared (e.g., only month of event occurrence reported), some database-specific issues were also identified and need to be accounted for. In conclusion, MDV and JMDC present limitations that are relatively typical of RWD sources, though some of them are unique to Japan, such as the identification of event occurrence and the inability to track patients visiting different healthcare settings. Addressing study design and careful result interpretation with respect to the specificities and uniqueness of the Japanese healthcare system is of particular importance. This aspect is especially relevant with respect to the growing global interest of conducting RWD studies in Japan.

Validation of a claims-based algorithm to identify cases of ulcerative colitis in Japan

BACKGROUND AND AIM

The prevalence of ulcerative colitis (UC) is increasing in Japan. Validated claims-based definitions are required to investigate the epidemiology of UC and its treatment and disease course in clinical practice. This study aimed to develop a claims-based algorithm for UC in Japan.

METHODS

A committee of epidemiologists, gastroenterologists, and internal medicine physicians developed a claims-based definition for UC, based on diagnostic codes and claims for UC treatments, procedures (cytapheresis), or surgery (postoperative claims). Claims data and medical records for a random sample of 200 cases per site at two large tertiary care academic centers in Japan were used to calculate the positive predictive value (PPV) of the algorithm for three gold standards of diagnosis, defined as physician diagnosis in the medical records, adjudicated cases, or registration in the Japanese Intractable Disease Registry (IDR).

RESULTS

Overall, 1139 claims-defined UC cases were identified. Among 393 randomly sampled cases (mean age 44; 48% female), 94% had received ≥ 1 systemic treatment (immunosuppressants, tumor necrosis factor inhibitors, corticosteroids, or antidiarrheals), 7% had cytapheresis, and 7% had postoperative claims. When physician diagnosis was used as a gold standard, PPV was 90.6% (95% confidence interval [CI]: 87.7-93.5). PPV with expert adjudication was also 90.6% (95% CI: 87.7-93.5). PPVs with enrollment in the IDR as gold standard were lower at 41.5% (95% CI: 36.6-46.3) due to incomplete case registration.

CONCLUSIONS

The claims-based algorithm developed for use in Japan is likely to identify UC cases with high PPV for clinical studies using administrative claims databases.

Use of diagnostic likelihood ratio of outcome to evaluate misclassification bias in the planning of database studies

Background

The diagnostic likelihood ratio (DLR) and its utility are well-known in the field of medical diagnostic testing. However, its use has been limited in the context of an outcome validation study. We considered that wider recognition of the utility of DLR would enhance the practices surrounding database studies. This is particularly timely and important since the use of healthcare-related databases for pharmacoepidemiology research has greatly expanded in recent years. In this paper, we aimed to advance the use of DLR, focusing on the planning of a new database study.

Methods

Theoretical frameworks were developed for an outcome validation study and a comparative cohort database study; these two were combined to form the overall relationship. Graphical presentations based on these relationships were used to examine the implications of validation study results on the planning of a database study. Additionally, novel uses of graphical presentations were explored using some examples.

Results

Positive DLR was identified as a pivotal parameter that connects the expected positive-predictive value (PPV) with the disease prevalence in the planned database study, where the positive DLR is equal to sensitivity/(1-specificity). Moreover, positive DLR emerged as a pivotal parameter that links the expected risk ratio with the disease risk of the control group in the planned database study. In one example, graphical presentations based on these relationships provided a transparent and informative summary of multiple validation study results. In another example, the potential use of a graphical presentation was demonstrated in selecting a range of positive DLR values that best represented the relevant validation studies.

Conclusions

Inclusion of the DLR in the results section of a validation study would benefit potential users of the study results. Furthermore, investigators planning a database study can utilize the DLR to their benefit. Wider recognition of the full utility of the DLR in the context of a validation study would contribute meaningfully to the promotion of good practice in planning, conducting, analyzing, and interpreting database studies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12911-022-01757-1.

Validation Study of Algorithms to Identify Malignant Tumors and Serious Infections in a Japanese Administrative Healthcare Database

BACKGROUND

This retrospective observational study validated case-finding algorithms for malignant tumors and serious infections in a Japanese administrative healthcare database.

METHODS

Random samples of possible cases of each disease (January 2015-January 2018) from two hospitals participating in the Medical Data Vision Co., Ltd. (MDV) database were identified using combinations of ICD-10 diagnostic codes and other procedural/billing codes. For each disease, two physicians identified true cases among the random samples of possible cases by medical record review; a third physician made the final decision in cases where the two physicians disagreed. The accuracy of case-finding algorithms was assessed using positive predictive value (PPV) and sensitivity.

RESULTS

There were 2,940 possible cases of malignant tumor; 180 were randomly selected and 108 were identified as true cases after medical record review. One case-finding algorithm gave a high PPV (64.1%) without substantial loss in sensitivity (90.7%) and included ICD-10 codes for malignancy and photographing/imaging. There were 3,559 possible cases of serious infection; 200 were randomly selected and 167 were identified as true cases after medical record review. Two case-finding algorithms gave a high PPV (85.6%) with no loss in sensitivity (100%). Both case-finding algorithms included the relevant diagnostic code and immunological infection test/other related test and, of these, one also included pathological diagnosis within 1 month of hospitalization.

CONCLUSIONS

The case-finding algorithms in this study showed good PPV and sensitivity for identification of cases of malignant tumors and serious infections from an administrative healthcare database in Japan.

Validity of diagnoses and procedures in Japanese dental claims data

BACKGROUND

Dental claims data have been used for epidemiological studies without establishing the validity of the recorded diagnoses or procedures. The present study aimed to examine the accuracy of diagnoses, procedures, operation time, and the number of teeth recorded in dental claims data.

METHODS

We reviewed the charts of 200 patients who visited and 100 patients who were hospitalized in the Department of General Dentistry, Orthodontics, and Oral and Maxillofacial Surgery in an academic hospital between August 2012 and December 2017. The sensitivity and specificity of the dental claims data for five diseases and 15 procedures were evaluated. We assessed the difference in the number of teeth and duration of general anesthesia between claims data and chart reviews.

RESULTS

Sensitivity was more than 86% for six out of seven diagnoses except for pericoronitis (67%). Specificity ranged from 72% (periodontal disease) to 100% (oral cancer for inpatient). The sensitivity of procedures ranged from 10% (scaling for inpatient) to 100%, and the specificity ranged from 6% (food intake on the day of the surgery) to 100%. The mean (standard deviation [SD]) number of teeth in the chart review was 22.6 (6.8), and in the dental claims was 21.6 (8.6). The mean (SD) operation time was 171.2 (120.3) minutes, while the duration of general anesthesia was 270.9 (171.3) minutes.

CONCLUSIONS

The present study is the first study to validate dental claims data, and indicates the extent of usefulness of each diagnosis and procedure for future dental research using administrative data.

Claims-based algorithms for common chronic conditions were efficiently constructed using machine learning methods

Identification of medical conditions using claims data is generally conducted with algorithms based on subject-matter knowledge. However, these claims-based algorithms (CBAs) are highly dependent on the knowledge level and not necessarily optimized for target conditions. We investigated whether machine learning methods can supplement researchers' knowledge of target conditions in building CBAs. Retrospective cohort study using a claims database combined with annual health check-up results of employees' health insurance programs for fiscal year 2016-17 in Japan (study population for hypertension, N = 631,289; diabetes, N = 152,368; dyslipidemia, N = 614,434). We constructed CBAs with logistic regression, k-nearest neighbor, support vector machine, penalized logistic regression, tree-based model, and neural network for identifying patients with three common chronic conditions: hypertension, diabetes, and dyslipidemia. We then compared their association measures using a completely hold-out test set (25% of the study population). Among the test cohorts of 157,822, 38,092, and 153,608 enrollees for hypertension, diabetes, and dyslipidemia, 25.4%, 8.4%, and 38.7% of them had a diagnosis of the corresponding condition. The areas under the receiver operating characteristic curve (AUCs) of the logistic regression with/without subject-matter knowledge about the target condition were .923/.921 for hypertension, .957/.938 for diabetes, and .739/.747 for dyslipidemia. The logistic lasso, logistic elastic-net, and tree-based methods yielded AUCs comparable to those of the logistic regression with subject-matter knowledge: .923-.931 for hypertension; .958-.966 for diabetes; .747-.773 for dyslipidemia. We found that machine learning methods can attain AUCs comparable to the conventional knowledge-based method in building CBAs.

Validity of claims-based algorithms for selected cancers in Japan: Results from the VALIDATE-J study

PURPOSE

Real-world data from large administrative claims databases in Japan have recently become available, but limited evidence exists to support their validity. VALIDATE-J validated claims-based algorithms for selected cancers in Japan.

METHODS

VALIDATE-J was a multicenter, cross-sectional, retrospective study. Disease-identifying algorithms were used to identify cancers diagnosed between January or March 2012 and December 2016 using claims data from two hospitals in Japan. Positive predictive values (PPVs), specificity, and sensitivity were calculated for prevalent (regardless of baseline cancer-free period) and incident (12-month cancer-free period; with claims and registry periods in the same month) cases, using hospital cancer registry data as gold standard.

RESULTS

22 108 cancers were identified in the hospital claims databases. PPVs (number of registry cases) for prevalent/incident cases were: any malignancy 79.0% (25 934)/73.1% (18 119); colorectal 84.4% (3519)/65.6% (2340); gastric 87.4% (3534)/76.8% (2279); lung 88.1% (2066)/79.9% (1636); breast 86.4% (4959)/59.9% (3185); pancreatic 87.1% (582)/80.4% (508); melanoma 48.7% (46)/42.9% (36); and lymphoma 83.6% (1457)/77.8% (1035). Specificity ranged from 98.3% to 100% (prevalent)/99.5% to 100% (incident); sensitivity ranged from 39.1% to 67.6% (prevalent)/12.5% to 31.4% (incident). PPVs of claims-based algorithms for several cancers in patients ≥66 years of age were slightly higher than those in a US Medicare population.

CONCLUSIONS

VALIDATE-J demonstrated high specificity and modest-to-moderate sensitivity for claims-based algorithms of most malignancies using Japanese claims data. Use of claims-based algorithms will enable identification of patient populations from claims databases, while avoiding direct patient identification. Further research is needed to confirm the generalizability of our results and applicability to specific subgroups of patient populations.

Identification and Validation of Hemophilia-Related Outcomes on Japanese Electronic Medical Record Database (Hemophilia-REAL V Study)

Purpose

Routinely collected data are useful for epidemiological study in hemophilia, but few studies validated the algorithm accuracy. We aimed to develop and validate algorithms to identify patients with hemophilia A and hemophilia A-related events.

Patients and Methods

This validation study compared data from medical chart reviews to a database of routinely collected health data, including claims data and discharge abstracts, and especially electronic medical records (EMR), at a single Japanese hospital (Kurashiki Central Hospital) using a stratified sampling method. Two physicians reviewed the charts for all patients at high risk for hemophilia A, and randomly sampled patients with moderate risk. Diagnostic accuracy was determined based on sensitivity, specificity, positive predictive value (PPV), and negative predictive value.

Results

There were 1,033,845 eligible patients, of whom 31 had a diagnosis of hemophilia A. ICD-10 diagnosis code D66 in the EMR identified hemophilia A with a sensitivity of 93.5% (95% confidence interval: 78.6-99) and PPV of 61.7% (95% confidence interval: 46.4-75.5). The administration of ≥10,000 units/month of factor VIII products, as documented in the EMR, identified 81.3% of patients with prophylactic factor replacement therapy. The ICD-10 diagnosis code for intracranial bleeding in the EMR identified 75.0% of patients with intracranial bleeding, but those of gastrointestinal bleeding and major joint bleeding identified only 11.1% and 1.7%, respectively.

Conclusion

We developed and validated algorithms to identify congenital hemophilia A and hemophilia A-related events. Hemophilia A could be identified with high sensitivity and PPV, but it was still challenging to identify hemophilia A-related events.

Accuracy of Japanese claims data in identifying diabetes-related complications

PURPOSE

To evaluate the accuracy of various claims-based definitions of diabetes-related complications (coronary artery disease [CAD], heart failure, cerebrovascular disease and dialysis).

METHODS

We evaluated data on 1379 inpatients who received care at the Niigata University Medical & Dental Hospital in September 2018. Manual electronic medical chart reviews were conducted for all patients with regard to diabetes-related complications and were used as the gold standard. Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of each claims-based definition associated with diabetes-related complications based on Diagnosis Procedure Combination (DPC), International Classification of Diseases, Tenth Revision (ICD-10) codes, procedure codes and medication codes were calculated.

RESULTS

DPC-based definitions had higher sensitivity, specificity, and PPV than ICD-10 code definitions for CAD and cerebrovascular disease, with sensitivity of 0.963-1.000 and 0.905-0.952, specificity of 1.000 and 1.000, and PPV of 1.000 and 1.000, respectively. Sensitivity, specificity, and PPV were high using procedure codes for CAD and dialysis, with sensitivity of 0.963 and 1.000, specificity of 1.000 and 1.000, and PPV of 1.000 and 1.000, respectively. DPC and/or ICD-10 codes + medication were better for heart failure than the ICD-10 code definition, with sensitivity of 0.933, specificity of 1.000, and PPV of 1.000. The PPVs were lower than 60% for all diabetes-related complications using ICD-10 codes only.

CONCLUSION

The DPC-based definitions for CAD and cerebrovascular disease, procedure codes for CAD and dialysis, and DPC or ICD-10 codes with medication codes for heart failure could accurately identify these diabetes-related complications from claims databases.

Narrative Review: The Process of Expanding the Manual of Antimicrobial Stewardship by the Government of Japan

The Ministry of Health, Labour and Welfare has published the Manual of Antimicrobial Stewardship (1st edition) in June 2017 to improve the prescribing practice of antimicrobials for immunocompetent adult and pediatric (both school-aged and older children) patients. Due to the increasing demand for further promoting outpatient antimicrobial stewardship, we conducted a literature and national guideline review to identify the area of need. The results of our review revealed a high antimicrobial prescription rate in the Japanese pediatric population. Furthermore, although the Japanese clinical guidelines/guidance covered the fields of almost all infectious diseases, no system exists to estimate the incidence and treatment patterns of important infectious diseases such as asymptomatic bacteriuria, skin and soft tissue infections, and dental practices in Japan. Therefore, addressing the issues of both establishing surveillance systems and the implementation of guidelines/guidance can be the next step to promote further outpatient antimicrobial stewardship.

Validity of Claims Diagnosis Codes for Cardiovascular Diseases in Diabetes Patients in Japanese Administrative Database

Background

Observational studies using large claims databases for diabetes patients have been increasingly conducted. While validation of outcomes is important in such studies, validation studies from Japan are still scarce and small in scale with questions remaining on the representativeness of their findings. We examined the positive predictive value (PPV) of outcomes that often develop in type 2 diabetes patients: cardiovascular outcomes including congestive heart failure (CHF), myocardial infarction (MI), stroke-related diseases, and renal outcomes including end stage renal disease (ESRD), and death using a large Japanese database containing administrative claims and electronic medical record (EMR) data.

Patients and Methods

We used patient-level administrative claims data from 2003 and EMR data from 1985 to the most recent data up to December 2018 provided by Real World Data Co., Ltd. The database consisted of data from over 200 hospitals including ≥12 million uniquely identifiable patients. Among patients who had ≥1 type 2 diabetes diagnosis in the EMR, those who had administrative claims for each outcome were identified, and then the PPV was calculated for each outcome using the EMR as the gold standard.

Results

The numbers of patients identified for each outcome were 1,700 for MI, 2,027 for hemorrhagic stroke, 3,722 for ESRD, 4,723 for ischemic stroke, 5,404 for CHF, 6,678 for any type of stroke, and 10,815 for death. PPVs ranged from 67.4% for ESRD, 78.7% for MI, 80.3% for death, 85.7% for ischemic stroke, 88.9% for any type of stroke, 89.9% for hemorrhagic stroke, and 95.7% for CHF. A post hoc analysis showed PPV for ESRD as 83.8%.

Conclusion

This large-scale validation study on diagnosis in administrative claims showed reasonable PPVs for the outcomes. We believe that the definitions of outcomes can be considered to be appropriate for future studies using Japanese administrative claims data.

Validity of ICD-10-CM Codes Used to Identify Patients with Chronic Hepatitis B and C Virus Infection in Administrative Claims Data from the Taiwan National Health Insurance Outpatient Claims Dataset

PURPOSE

To validate the use of International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes to identify patients with chronic hepatitis B virus (HBV) and hepatitis C virus (HCV) infection in the Taiwan National Health Insurance (NHI) Outpatient Claims Dataset.

METHODS

We conducted a retrospective study using results of HBV surface antigen (HBsAg), HBV e antigen (HBeAg), and anti-HCV antibody tests in the NHI Lab & Exam Dataset from January 1 to March 31, 2018, as the reference standard to confirm HBV and HCV infection cases. We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) to assess the performance of HBV infection-specific ICD-10-CM codes (B180, B181, and B191) and HCV infection-specific ICD-10-CM codes (B182 and B192) recorded in the NHI Outpatient Claims Dataset to identify patients with HBV or HCV infection.

RESULTS

In total, 196,635 and 120,628 patients had analyzable results for HBsAg/HBeAg tests and anti-HCV tests, respectively. Moreover, 44,574 and 14,443 were confirmed to have HBV and HCV infection, respectively. The sensitivity, specificity, PPV, and NPV were, respectively, 46%, 83%, 45%, and 84% for HBV infection-specific ICD-10-CM codes and 47%, 99%, 81%, and 93% for HCV infection-specific ICD-10-CM codes. The sensitivity demonstrated great variation by region, clinical setting, and physician specialty.

CONCLUSION

The HBV and HCV infection-specific ICD-10-CM codes recorded by physicians in Taiwan NHI outpatient claims data in 2018 had moderate sensitivity and high specificity for both HBV and HCV infection. The PPV was high for HCV ICD-10-CM codes, yet moderate for HBV ICD-10-CM codes.

Identifying incident cancer cases in dispensing claims: A validation study using Australia's Repatriation Pharmaceutical Benefits Scheme (PBS) data

Introduction

Dispensing claims are used commonly as proxy measures in pharmacoepidemiological studies; however, their validity is often untested.

Objectives

To assess the performance of a proxy for identifying cancer cases based on the dispensing of anticancer medicines and estimate the misclassification of cancer status and potential for bias researchers may encounter when using this proxy.

Methods

We conducted our validation study using Department of Veterans’ Affairs (DVA) client data linked with the New South Wales (NSW) Cancer Registry and Repatriation Pharmaceutical Benefits Scheme data. We included DVA clients aged ≥65 years residing in NSW between July 2004 and December 2012. We matched clients with a cancer diagnosis to clients without a diagnosis based on demographic characteristics and available observation time. We used dispensing claims for anticancer medicines dispensed between July 2004 and December 2013 as a proxy to identify clients with cancer and calculated sensitivity, specificity, positive predictive values and negative predictive values compared with cancer registrations (gold standard), overall and by cancer site. We illustrated misclassification by the proxy in a cohort of people initiating opioid therapy. Using the proxy, we excluded people with cancer from the cohort, in an attempt to delineate people potentially using opioids for cancer rather than chronic non-cancer pain.

Results

We identified 15,679 new cancer diagnoses in 14,112 DVA clients from the cancer registry and 62,663 clients without a diagnosis. Sensitivity of the proxy based on dispensing claims was 30% for all cancers and around 20% for specific cancers (range: 10-67%). Specificity was above 90% for all cancers. The dispensing proxy correctly identified 26% of people with a cancer diagnosis who initiated opioid therapy and failed to identify 74% those with a cancer diagnosis; the proxy was most robust for clients with breast cancer where 61% were correctly identified by proxy.

Conclusions

Using dispensing of anticancer medicines to identify people with a cancer diagnosis performed poorly. Excluding patients with evidence of anticancer medicine use from cohort studies may result removal of a disproportionate number of women with breast cancer. Researchers excluding or otherwise using anticancer medicine dispensing to identify people with cancer in pharmacoepidemiological studies should acknowledge the potential biases introduced to their findings.

Keywords

cancer, diagnosis, proxy, dispensing records, validation study