Is knowledge retained by healthcare providers after training? A pragmatic evaluation of drug-resistant tuberculosis management in China

OBJECTIVES

Considering the urgent need of training to improve standardised management of drug-resistant infectious disease and the lack of evidence on the impact of training, this study evaluates whether training participants' knowledge on multidrug-resistant tuberculosis (MDR-TB) is improved immediately and a year after training.

SETTING AND PARTICIPANTS

The study involved 91 MDR-TB healthcare providers (HCPs), including clinical doctors, nurses and CDC staff, who attended a new MDR-TB HCP training programme in Liaoning and Jiangxi provinces, China.

MAIN OUTCOME MEASURES

A phone-based assessment of participants' long-term retention of knowledge about MDR-TB management was conducted in July 2017, approximately 1 year after training. The proportion of correct responses in the long-term knowledge assessment was compared with a pretraining test and an immediate post-training test using a χ2 test. Factors influencing participants' performance in the long-term knowledge assessment were analysed using linear regression.

RESULTS

Across both provinces, knowledge of definitions of drug-resistant TB, standardised MDR-TB case detection protocols and laboratory diagnosis was improved 1 year after the training by 14.5% (p=0.037), 32.4% (p<0.001) and 31% (p<0.001) relative to pretraining. However, compared with immediately after training, the knowledge of the three topics declined by 26.5% (p=0.003), 19.8% (p=0.018) and 52.7% (p<0.001) respectively in Jiangxi, while no significant decline was observed in Liaoning. Additionally, we found that obtaining a higher score in the long-term knowledge assessment was associated with longer years of clinical experience (coefficient=0.51; 95 CI% 0.02 to 0.99; p=0.041) and attending training in Liaoning (coefficient=0.50; 95% CI 0.14 to 0.85; p=0.007).

CONCLUSION

Our study, the first to assess knowledge retention of MDR-TB HCPs 1 year after training, showed an overall positive long-term impact of lecture-style group training on participants' knowledge. Knowledge decline 1 year after training was observed in one province, Jiangxi, and this may be partly addressed by targeted support to HCPs with fewer years of clinical experience.

Designing evaluation studies to optimally inform policy: what factors do policy-makers in China consider when making resource allocation decisions on healthcare worker training programmes?

BACKGROUND

In light of the gap in evidence to inform future resource allocation decisions about healthcare provider (HCP) training in low- and middle-income countries (LMICs), and the considerable donor investments being made towards training interventions, evaluation studies that are optimally designed to inform local policy-makers are needed. The aim of our study is to understand what features of HCP training evaluation studies are important for decision-making by policy-makers in LMICs. We investigate the extent to which evaluations based on the widely used Kirkpatrick model - focusing on direct outcomes of training, namely reaction of trainees, learning, behaviour change and improvements in programmatic health indicators - align with policy-makers' evidence needs for resource allocation decisions. We use China as a case study where resource allocation decisions about potential scale-up (using domestic funding) are being made about an externally funded pilot HCP training programme.

METHODS

Qualitative data were collected from high-level officials involved in resource allocation at the national and provincial level in China through ten face-to-face, in-depth interviews and two focus group discussions consisting of ten participants each. Data were analysed manually using an interpretive thematic analysis approach.

RESULTS

Our study indicates that Chinese officials not only consider information about the direct outcomes of a training programme, as captured in the Kirkpatrick model, but also need information on the resources required to implement the training, the wider or indirect impacts of training, and the sustainability and scalability to other settings within the country. In addition to considering findings presented in evaluation studies, we found that Chinese policy-makers pay close attention to whether the evaluations were robust and to the composition of the evaluation team.

CONCLUSIONS

Our qualitative study indicates that training programme evaluations that focus narrowly on direct training outcomes may not provide sufficient information for policy-makers to make decisions on future training programmes. Based on our findings, we have developed an evidence-based framework, which incorporates but expands beyond the Kirkpatrick model, to provide conceptual and practical guidance that aids in the design of training programme evaluations better suited to meet the information needs of policy-makers and to inform policy decisions.