Validation studies of verbal autopsy methods: a systematic review

BACKGROUND

Verbal autopsy (VA) has emerged as an increasingly popular technique to assign cause of death in parts of the world where the majority of deaths occur without proper medical certification. The purpose of this study was to examine the key characteristics of studies that have attempted to validate VA cause of death against an established cause of death.

METHODS

A systematic review was conducted by searching the MEDLINE, EMBASE, Cochrane-library, and Scopus electronic databases. Included studies contained 1) a VA component, 2) a validation component, and 3) original analysis or re-analysis. Characteristics of VA studies were extracted. A total of 527 studies were assessed, and 481 studies screened to give 66 studies selected for data extraction.

RESULTS

Sixty-six studies were included from multiple countries. Ten studies used an existing database. Sixteen studies used the World Health Organization VA questionnaire and 5 studies used the Population Health Metrics Research Consortium VA questionnaire. Physician certification was used in 36 studies and computer coded methods were used in 14 studies. Thirty-seven studies used high level comparator data with detailed laboratory investigations.

CONCLUSION

Most studies found VA to be an effective cause of death assignment method and compared VA cause of death to a high-quality established cause of death. Nonetheless, there were inconsistencies in the methodologies of the validation studies, and many used poor quality comparison cause of death data. Future VA validation studies should adhere to consistent methodological criteria so that policymakers can easily interpret the findings to select the most appropriate VA method.

PROSPERO REGISTRATION

CRD42020186886.

Validation of physician certified verbal autopsy using conventional autopsy: a large study of adult non-external causes of death in a metropolitan area in Brazil

Background

Reliable mortality data are essential for the development of public health policies. In Brazil, although there is a well-consolidated universal system for mortality data, the quality of information on causes of death (CoD) is not even among Brazilian regions, with a high proportion of ill-defined CoD. Verbal autopsy (VA) is an alternative to improve mortality data. This study aimed to evaluate the performance of an adapted and reduced version of VA in identifying the underlying causes of non-forensic deaths, in São Paulo, Brazil. This is the first time that a version of the questionnaire has been validated considering the autopsy as the gold standard.

Methods

The performance of a physician-certified verbal autopsy (PCVA) was evaluated considering conventional autopsy (macroscopy plus microscopy) as gold standard, based on a sample of 2060 decedents that were sent to the Post-Mortem Verification Service (SVOC-USP). All CoD, from the underlying to the immediate, were listed by both parties, and ICD-10 attributed by a senior coder. For each cause, sensitivity and chance corrected concordance (CCC) were computed considering first the underlying causes attributed by the pathologist and PCVA, and then any CoD listed in the death certificate given by PCVA. Cause specific mortality fraction accuracy (CSMF-accuracy) and chance corrected CSMF-accuracy were computed to evaluate the PCVA performance at the populational level.

Results

There was substantial variability of the sensitivities and CCC across the causes. Well-known chronic diseases with accurate diagnoses that had been informed by physicians to family members, such as various cancers, had sensitivities above 40% or 50%. However, PCVA was not effective in attributing Pneumonia, Cardiomyopathy and Leukemia/Lymphoma as underlying CoD. At populational level, the PCVA estimated cause specific mortality fractions (CSMF) may be considered close to the fractions pointed by the gold standard. The CSMF-accuracy was 0.81 and the chance corrected CSMF-accuracy was 0.49.

Conclusions

The PCVA was efficient in attributing some causes individually and proved effective in estimating the CSMF, which indicates that the method is useful to establish public health priorities.

Improving methods to measure comparable mortality cause (IMMCMC) gold standard verbal autopsy dataset

Objectives

Gold standard cause of death data is critically important to improve verbal autopsy (VA) methods in diagnosing cause of death where civil and vital registration systems are inadequate or poor. As part of a three-country research study—Improving Methods to Measure Comparable Mortality by Cause (IMMCMC) study—data were collected on clinicopathological criteria-based gold standard cause of death from hospital record reviews with matched VAs. The purpose of this data note is to make accessible a de-identified format of these gold standard VAs for interested researchers to improve the diagnostic accuracy of VA methods.

Data description

The study was conducted between 2011 and 2014 in the Philippines, Bangladesh, and Papua New Guinea. Gold standard diagnoses of underlying causes of death for deaths occurring in hospital were matched to VAs conducted using a standardized VA questionnaire developed by the Population Health Metrics Consortium. 3512 deaths were collected in total, comprised of 2491 adults (12 years and older), 320 children (28 days to 12 years), and 702 neonates (0–27 days).

Improving cause of death certification in the Philippines: implementation of an electronic verbal autopsy decision support tool (SmartVA auto-analyse) to aid physician diagnoses of out-of-facility deaths

Background

The majority of deaths in the Philippines occur out-of-facility and require a medical certificate of cause of death by Municipal Health Officers (MHOs) for burial. MHOs lack a standardised certification process for out-of-facility deaths and when no medical records are available, certify a high proportion of ill-defined causes of death. We aimed to develop and introduce SmartVA Auto-Analyse, a verbal autopsy (VA) based electronic decision support tool in order to assist the MHOs in certifying out-of-facility deaths.

Method

We conducted a stakeholder consultation, process mapping and a pre-test to assess feasibility and acceptability of SmartVA Auto-Analyse. MHOs were first asked to conduct an open-ended interview from the family members of the deceased, and if they were not able to arrive at a diagnosis, continue the interview using the standardised SmartVA questionnaire. Auto-Analyse then presented the MHO with the three most likely causes of death. For the pilot, the intervention was scaled-up to 91 municipalities. We performed a mixed-methods evaluation using the cause of death data and group discussions with the MHOs.

Results

Of the 5649 deaths registered, Auto-Analyse was used to certify 4586 (81%). For the remaining 19%, doctors believed they could assign a cause of death based on the availability of medical records and the VA open narrative. When used, physicians used the Auto-Analyse diagnosis in 85% of cases to certify the cause of death. Only 13% of the deaths under the intervention had an undetermined cause of death. Group discussions identified two themes: Auto-Analyse standardized the certification of home deaths and assisted the MHOs to improve the quality of death certification.

Conclusion

Standardized VA combined with physician diagnosis using the SmartVA Auto-Analyse support tool was readily used by MHOs in the Philippines and can improve the quality of death certification of home deaths.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-021-10542-0.

The epidemiological transition in Papua New Guinea: new evidence from verbal autopsy studies

BACKGROUND

Recent economic growth in Papua New Guinea (PNG) would suggest that the country may be experiencing an epidemiological transition, characterized by a reduction in infectious diseases and a growing burden from non-communicable diseases (NCDs). However, data on cause-specific mortality in PNG are very sparse, and the extent of the transition within the country is poorly understood.

METHODS

Mortality surveillance was established in four small populations across PNG: West Hiri in Central Province, Asaro Valley in Eastern Highlands Province, Hides in Hela Province and Karkar Island in Madang Province. Verbal autopsies (VAs) were conducted on all deaths identified, and causes of death were assigned by SmartVA and classified into five broad disease categories: endemic NCDs; emerging NCDs; endemic infections; emerging infections; and injuries. Results from previous PNG VA studies, using different VA methods and spanning the years 1970 to 2001, are also presented here.

RESULTS

A total of 868 deaths among adolescents and adults were identified and assigned a cause of death. NCDs made up the majority of all deaths (40.4%), with the endemic NCD of chronic respiratory disease responsible for the largest proportion of deaths (10.5%), followed by the emerging NCD of diabetes (6.2%). Emerging infectious diseases outnumbered endemic infectious diseases (11.9% versus 9.5%). The distribution of causes of death differed across the four sites, with emerging NCDs and emerging infections highest at the site that is most socioeconomically developed, West Hiri. Comparing the 1970-2001 VA series with the present study suggests a large decrease in endemic infections.

CONCLUSIONS

Our results indicate immediate priorities for health service planning and for strengthening of vital registration systems, to more usefully serve the needs of health priority setting.

Automated verbal autopsy: from research to routine use in civil registration and vital statistics systems

BACKGROUND

The majority of low- and middle-income countries (LMICs) do not have adequate civil registration and vital statistics (CRVS) systems to properly support health policy formulation. Verbal autopsy (VA), long used in research, can provide useful information on the cause of death (COD) in populations where physicians are not available to complete medical certificates of COD. Here, we report on the application of the SmartVA tool for the collection and analysis of data in several countries as part of routine CRVS activities.

METHODS

Data from VA interviews conducted in 4 of 12 countries supported by the Bloomberg Philanthropies Data for Health (D4H) Initiative, and at different stages of health statistical development, were analysed and assessed for plausibility: Myanmar, Papua New Guinea (PNG), Bangladesh and the Philippines. Analyses by age- and cause-specific mortality fractions were compared to the Global Burden of Disease (GBD) study data by country. VA interviews were analysed using SmartVA-Analyze-automated software that was designed for use in CRVS systems. The method in the Philippines differed from the other sites in that the VA output was used as a decision support tool for health officers.

RESULTS

Country strategies for VA implementation are described in detail. Comparisons between VA data and country GBD estimates by age and cause revealed generally similar patterns and distributions. The main discrepancy was higher infectious disease mortality and lower non-communicable disease mortality at the PNG VA sites, compared to the GBD country models, which critical appraisal suggests may highlight real differences rather than implausible VA results.

CONCLUSION

Automated VA is the only feasible method for generating COD data for many populations. The results of implementation in four countries, reported here under the D4H Initiative, confirm that these methods are acceptable for wide-scale implementation and can produce reliable COD information on community deaths for which little was previously known.

Robustness of the Tariff method for diagnosing verbal autopsies: impact of additional site data on the relationship between symptom and cause

Background

Verbal autopsy (VA) is increasingly being considered as a cost-effective method to improve cause of death information in countries with low quality vital registration. VA algorithms that use empirical data have an advantage over expert derived algorithms in that they use responses to the VA instrument as a reference instead of physician opinion. It is unclear how stable these data driven algorithms, such as the Tariff 2.0 method, are to cultural and epidemiological variations in populations where they might be employed.

Methods

VAs were conducted in three sites as part of the Improving Methods to Measure Comparable Mortality by Cause (IMMCMC) study: Bohol, Philippines; Chandpur and Comila Districts, Bangladesh; and Central and Eastern Highlands Provinces, Papua New Guinea. Similar diagnostic criteria and cause lists as the Population Health Metrics Research Consortium (PHMRC) study were used to identify gold standard (GS) deaths. We assessed changes in Tariffs by examining the proportion of Tariffs that changed significantly after the addition of the IMMCMC dataset to the PHMRC dataset.

Results

The IMMCMC study added 3512 deaths to the GS VA database (2491 adults, 320 children, and 701 neonates). Chance-corrected cause specific mortality fractions for Tariff improved with the addition of the IMMCMC dataset for adults (+ 5.0%), children (+ 5.8%), and neonates (+ 1.5%). 97.2% of Tariffs did not change significantly after the addition of the IMMCMC dataset.

Conclusions

Tariffs generally remained consistent after adding the IMMCMC dataset. Population level performance of the Tariff method for diagnosing VAs improved marginally for all age groups in the combined dataset. These findings suggest that cause-symptom relationships of Tariff 2.0 might well be robust across different population settings in developing countries. Increasing the total number of GS deaths improves the validity of Tariff and provides a foundation for the validation of other empirical algorithms.

Monitoring progress in reducing maternal mortality using verbal autopsy methods in vital registration systems: what can we conclude about specific causes of maternal death?

Reducing maternal mortality is a key focus of development strategies and one of the indicators used to measure progress towards achieving the Sustainable Development Goals. In the absence of medical certification of the cause of deaths that occur in the community, verbal autopsy (VA) methods are the only available means to assess levels and trends of maternal deaths that occur outside health facilities. The 2016 World Health Organization VA Instrument facilitates the identification of eight specific causes of maternal death, yet maternal deaths are often unsupervised, leading to sparse and generally poor symptom reporting to inform a reliable diagnosis using VAs. There is little research evidence to support the reliable identification of specific causes of maternal death in the context of routine VAs. We recommend that routine VAs are only used to capture the event of a maternal death and that more detailed follow-up interviews are used to identify the specific causes.

Effect of Empiric Anti-Mycobacterium tuberculosis Therapy on Survival Among Human Immunodeficiency Virus-Infected Adults Admitted With Sepsis to a Regional Referral Hospital in Uganda

Background

Mycobacterium tuberculosis is the leading cause of bloodstream infection among human immunodeficiency virus (HIV)–infected patients with sepsis in sub-Saharan Africa and is associated with high mortality rates.

Methods

We conducted a retrospective study of HIV-infected adults with sepsis at the Mbarara Regional Referral Hospital in Uganda to measure the proportion who received antituberculosis therapy and to determine the relationship between antituberculosis therapy and 28-day survival.

Results

Of the 149 patients evaluated, 74 (50%) had severe sepsis and 48 (32%) died. Of the 55 patients (37%) who received antituberculosis therapy, 19 (35%) died, compared with 29 of 94 (31%) who did not receive such therapy (odds ratio, 1.34; 95% confidence interval [CI], .56–3.18; P = .64). The 28-day survival rates did not differ significantly between these 2 groups (log-rank test, P = .21). Among the 74 patients with severe sepsis, 9 of 26 (35%) who received antituberculosis therapy died, versus 23 of 48 (48%) who did not receive such therapy (odds ratio, 0.58; 95% CI, .21–1.52; P = .27). In patients with severe sepsis, antituberculosis therapy was associated with an improved 28-day survival rate (log-rank test P = .01), and with a reduced mortality rate in a Cox proportional hazards model (hazard ratio, 0.32; 95% CI, .13–.80; P = .03).

Conclusions

Empiric antituberculosis therapy was associated with improved survival rates among patients with severe sepsis, but not among all patients with sepsis.

On the estimation of population cause-specific mortality fractions from in-hospital deaths

BACKGROUND

Almost all countries without complete vital registration systems have data on deaths collected by hospitals. However, these data have not been widely used to estimate cause of death (COD) patterns in populations because only a non-representative fraction of people in these countries die in health facilities. Methods that can exploit hospital mortality statistics to reliably estimate community COD patterns are required to strengthen the evidence base for disease and injury control programs. We propose a method that weights hospital-certified causes by the probability of death to estimate population cause-specific mortality fractions (CSMFs).

METHODS

We used an established verbal autopsy instrument (VAI) to collect data from hospital catchment areas in Chandpur and Comilla Districts, Bangladesh, and Bohol province, the Philippines, between 2011 and 2014, along with demographic covariates for each death. Hospital medical certificates of cause of death (death certificates) were collected and mapped to the corresponding cause categories of the VAI. Tariff 2.0 was used to assign a COD for community deaths. Logistic regression models were created for broad causes in each country to calculate the probability of in-hospital death, given a set of covariate values. The reweighted CSMFs for deaths in the hospital catchment population, represented by each hospital death, were calculated from the corresponding regression models.

RESULTS

We collected data on 4228 adult deaths in the Philippines and 3725 deaths in Bangladesh. Short time to hospital and education were consistently associated with in-hospital death in the Philippines and absence of a disability was consistently associated with in-hospital death in Bangladesh. Non-communicable diseases (excluding stroke) and stroke were the leading causes of death in both the Philippines (33.9%, 19.1%) and Bangladesh (46.1%, 21.1%) according to the reweighted method. The reweighted method generally estimated CSMFs that fell between those derived from hospitals and those diagnosed by Tariff 2.0.

CONCLUSIONS

Statistical methods can be used to derive estimates of cause-specific probability of death in-hospital for Bangladesh and the Philippines to generate population CSMFs. In regions where hospital death certification is of reasonable quality and routine verbal autopsy is not applied, these estimates could be applied to generate cost-effective and robust CSMFs for the population.

Assessing the quality of medical death certification: a case study of concordance between national statistics and results from a medical record review in a regional hospital in the Philippines

Background

Medical certificates of cause of death (MCCOD) issued by hospital physicians are a key input to vital registration systems. Deaths certified by hospital physicians have been implicitly considered to be of high quality, but recent evidence suggests otherwise. We conducted a medical record review (MRR) of hospital MCCOD in the Philippines and compared the cause of death concordance with certificates coded by the Philippines Statistics Authority (PSA).

Methods

MCCOD for adult deaths in Bohol Regional Hospital (BRH) in 2007–2008 and 2011 were collected and reviewed by a team of study physicians. Corresponding MCCOD coded by the PSA were linked by a hospital identifier. The study physicians wrote a new MCCOD using the patient medical record, noted the quality of the medical record to produce a cause of death, and indicated whether it was necessary to change the underlying cause of death (UCOD). Chance-corrected concordance, cause-specific mortality fraction (CSMF) accuracy, and chance-corrected CSMF were used to examine the concordance between the MRR and PSA.

Results

A total of 1052 adult deaths were linked between the MRR and PSA. Median chance-corrected concordance was 0.73, CSMF accuracy was 0.85, and chance-corrected CSMF accuracy was 0.58. 74.8% of medical records were deemed to be of high enough quality to assign a cause of death, yet study physicians indicated that it was necessary to change the UCOD in 41% of deaths, 82% of which required addition of a new UCOD.

Conclusions

Medical records were generally of sufficient quality to assign a cause of death and concordance between the PSA and MRR was reasonably high, suggesting that routine mortality statistics data are reasonably accurate for describing population level causes of death in Bohol. While overall agreement between the PSA and MRR in major cause groups was sufficient for public health purposes, improvements in death certification practices are recommended to help physicians differentiate between treatable (immediate) COD and COD that are important for public health surveillance.

Electronic supplementary material

The online version of this article (10.1186/s12963-018-0178-0) contains supplementary material, which is available to authorized users.

Comparing tariff and medical assistant assigned causes of death from verbal autopsy interviews in Matlab, Bangladesh: implications for a health and demographic surveillance system

BACKGROUND

Deaths in developing countries often occur outside health facilities, making it extremely difficult to gather reliable cause of death (COD) information. Automated COD assignment using a verbal autopsy instrument (VAI) has been proposed as a reliable and cost-effective alternative to traditional physician-certified verbal autopsy, but its performance is still being evaluated. The purpose of this study was to compare the similarity of diagnosis by Medical Assistants (MA) in the Matlab Health and Demographic Surveillance System (HDSS) with the SmartVA Analyze 1.2 (Tariff 2.0) diagnosis.

METHODS

This study took place between January 2011 and April 2014 in Matlab, Bangladesh. MA with 3 years of medical training assigned COD to Matlab residents by reviewing the information collected using the Population Health Metrics Research Consortium (PHMRC) long-form VAI. Smart VA Analyze 1.2 automatically assigned COD using the same questionnaire. COD agreement and cause-specific mortality fractions (CSMFs) were compared for MA and Tariff.

RESULTS

Of the 4969 verbal autopsy cases reviewed, 4328 were adults, 296 were children, and 345 were neonates. Cohen's kappa was 0.38 (0.36, 0.40) for adults, 0.43 (0.38, 0.49) for children, and 0.27 (0.22, 0.33) for neonates. For adults, the top two COD for MA were stroke (29.6%) and ischemic heart diseases (IHD) (14.2%) and for Tariff these were stroke (32.0%) and IHD (14.0%). For children, the top two COD for MA were drowning (33.5%) and pneumonia (13.2%) and for Tariff these were also drowning (36.8%) and pneumonia (12.4%). For neonates, the top two COD for MA were birth asphyxia (41.2%) and meningitis/sepsis (22.3%) and for Tariff these were birth asphyxia (37.0%) and preterm delivery (30.9%).

CONCLUSION

The CSMFs for Tariff and MA showed very close agreement across all age categories but some differences were observed for neonate preterm delivery and meningitis/sepsis. Given the known advantages of automated methods over physician certified verbal autopsy, the SmartVA software, incorporating the shortened VAI questionnaire and Tariff 2.0, could serve as a cost-effective alternative to Matlab MA to routinely collect and analyze verbal autopsy data in a HDSS to generate essential population level COD data for planning.

The quality of medical death certification of cause of death in hospitals in rural Bangladesh: impact of introducing the International Form of Medical Certificate of Cause of Death

Background

Accurate and timely data on cause of death are critically important for guiding health programs and policies. Deaths certified by doctors are implicitly considered to be reliable and accurate, yet the quality of information provided in the international Medical Certificate of Cause of Death (MCCD) usually varies according to the personnel involved in certification, the diagnostic capacity of the hospital, and the category of hospitals. There are no published studies that have analysed how certifying doctors in Bangladesh adhere to international rules when completing the MCCD or have assessed the quality of clinical record keeping.

Methods

The study took place between January 2011 and April 2014 in the Chandpur and Comilla districts of Bangladesh. We introduced the international MCCD to all study hospitals. Trained project physicians assigned an underlying cause of death, assessed the quality of the death certificate, and reported the degree of certainty of the medical records provided for a given cause. We examined the frequency of common errors in completing the MCCD, the leading causes of in-hospital deaths, and the degree of certainty in the cause of death data.

Results

The study included 4914 death certificates. 72.9% of medical records were of too poor quality to assign a cause of death, with little difference by age, hospital, and cause of death. 95.6% of death certificates did not indicate the time interval between onset and death, 31.6% required a change in sequence, 13.9% required to include a new diagnosis, 50.7% used abbreviations, 41.5% used multiple causes per line, and 33.2% used an ill-defined condition as the underlying cause of death. 99.1% of death certificates had at least one error. The leading cause of death among adults was stroke (15.8%), among children was pneumonia (31.7%), and among neonates was birth asphyxia (52.8%).

Conclusion

Physicians in Bangladeshi hospitals had difficulties in completing the MCCD correctly. Physicians routinely made errors in death certification practices and medical record quality was poor. There is an urgent need to improve death certification practices and the quality of hospital data in Bangladesh if these data are to be useful for policy.

Electronic supplementary material

The online version of this article (10.1186/s12913-017-2628-y) contains supplementary material, which is available to authorized users.