Validity and usability testing of a health systems guidance appraisal tool, the AGREE-HS

Background

Health systems guidance (HSG) provides recommendations to address health systems challenges. No tools exist to inform HSG developers and users about the components of high quality HSG and to differentiate between HSG of varying quality. In response, we developed a tool to assist with the development, reporting and appraisal of HSG – the Appraisal of Guidelines for Research and Evaluation–Health Systems (AGREE-HS). This paper reports on the validity, usability and initial measurement properties of the AGREE-HS.

Methods

To establish face validity (Study 1), stakeholders completed a survey about the AGREE-HS and provided feedback on its content and structure. Revisions to the tool were made in response. To establish usability (Study 2), the revised tool was applied to 85 HSG documents and the appraisers provided feedback about their experiences via an online survey. An initial test of the revised tool’s measurement properties, including internal consistency, inter-rater reliability and criterion validity, was conducted. Additional revisions to the tool were made in response.

Results

In Study 1, the AGREE-HS Overview, User Manual, quality item content and structure, and overall assessment questions were rated favourably. Participants indicated that the AGREE-HS would be useful, feasible to use, and that they would apply it in their context. In Study 2, participants indicated that the quality items were easy to understand and apply, and the User Manual, usefulness and usability of the tool were rated favourably. Study 2 participants also indicated intentions to use the AGREE-HS.

Conclusions

The AGREE-HS comprises a User Manual, five quality items and two overall assessment questions. It is available at agreetrust.org.

Electronic supplementary material

The online version of this article (10.1186/s12961-018-0334-9) contains supplementary material, which is available to authorized users.

The cost-effectiveness of preventing mother-to-child transmission of HIV in low- and middle-income countries: systematic review

BACKGROUND

Although highly effective prevention interventions exist, the epidemic of paediatric HIV continues to challenge control efforts in resource-limited settings. We reviewed the cost-effectiveness of interventions to prevent mother-to-child transmission (MTCT) of HIV in low- and middle-income countries (LMICs). This article presents syntheses of evidence on the costs, effects and cost-effectiveness of HIV MTCT strategies for LMICs from the published literature and evaluates their implications for policy and future research.

METHODS

Candidate studies were identified through a comprehensive database search including PubMed, Embase, Cochrane Library, and EconLit restricted by language (English or French), date (January 1st, 1994 to January 17th, 2011) and article type (original research). Articles reporting full economic evaluations of interventions to prevent or reduce HIV MTCT were eligible for inclusion. We searched article bibliographies to identify additional studies. Two authors independently assessed eligibility and extracted data from studies retained for review. Study quality was appraised using a modified BMJ checklist for economic evaluations. Data were synthesised in narrative form.

RESULTS

We identified 19 articles published in 9 journals from 1996 to 2010, 16 concerning sub-Saharan Africa. Collectively, the articles suggest that interventions to prevent paediatric infections are cost-effective in a variety of LMIC settings as measured against accepted international benchmarks. In concentrated epidemics where HIV prevalence in the general population is very low, MTCT strategies based on universal testing of pregnant women may not compare well against cost-effectiveness benchmarks, or may satisfy formal criteria for cost-effectiveness but offer a low relative value as compared to competing interventions to improve population health.

CONCLUSIONS AND RECOMMENDATIONS

Interventions to prevent HIV MTCT are compelling on economic grounds in many resource-limited settings and should remain at the forefront of global HIV prevention efforts. Future cost-effectiveness analyses can help to ensure that pMTCT interventions for LMICs reach their full potential by focussing on unanswered questions in four areas: local assessment of rapidly evolving HIV MTCT options; strategies to improve coverage and reach underserved populations; evaluation of a more comprehensive set of MTCT approaches including primary HIV prevention and reproductive counselling; integration of HIV MTCT and other sexual and reproductive health services.