Developing a framework for evaluation: a Theory of Change for complex workplace mental health interventions

Beyond image: evaluating the perceived effectiveness of workplace mental health programs in the insurance sector

BACKGROUND

Mental health programs in the workplace have gained increasing attention as organizations strive to support employee well-being. However, the effectiveness and reception of these initiatives from the employee perspective still need to be studied.

METHODS

A qualitative study used 30 semi-structured interviews with individuals from large insurance companies in Portugal. It focused on their perceptions of workplace mental health programs, stress, and time management training. The participants ranged in age from 25 to 55 years (average 38), with a slightly more significant proportion of women, reflecting the gender distribution in the Portuguese insurance sector. The sample had an average of 10 years of industry experience, from entry-level to middle-management roles. Thematic analysis with an audit trail and thick description was used to analyse data.

RESULTS

The findings revealed a significant disconnect between the intended benefits of these programs and employee experiences. Many viewed the initiatives as primarily beneficial for the company's image rather than employee well-being. A common concern was that workload remained unchanged after training, leading employees to prioritize work over program participation. Additional key findings included generational differences in program perception, with younger employees expressing more frustration over the lack of follow-through. Most participants reported that initiatives failed to address the root causes of stress. More than half of the interviewees felt a lingering stigma around discussing mental health at work, whereas a majority mentioned increased stress due to rapid industry digitalization. Notably, many participants perceived a significant gap between upper management's promotion of mental health initiatives and their understanding of daily employee challenges.

CONCLUSIONS

These findings highlight a critical gap between implementing workplace mental health programs and their perceived value among employees. Organizations need to reevaluate their mental health initiatives to ensure that they provide tangible benefits and address underlying workplace stressors. Future research should explore strategies to better align mental health programs with the insurance sector's employee needs and work realities.

Towards an assessment of psychosocial work factors in a multi-level mental health intervention in the workplace: results from the MENTUPP pilot-study

BACKGROUND

Mental health in the workplace is a growing concern for enterprises and policy makers. MENTUPP is a multi-level mental health intervention implemented in small and medium size enterprises from three work sectors in nine countries. This pilot study aimed to evaluate the feasibility, delivery, and instruments for the MENTUPP intervention to inform the planning of a clustered randomized controlled trial.

METHODS

We administered items from the Copenhagen Psychosocial Questionnaire and the Danish Work Environment Cohort Study measuring psychosocial workplace factors. The questionnaire was answered by 382 participants at baseline, of which 98 participants also answered after six months at follow-up. We calculated mean scores of 19 psychosocial factors at baseline and conducted repeated measures ANOVAs to assess differences in eight psychosocial factors at follow-up. We also examined whether outcomes differed between work sectors and job positions at follow-up.

RESULTS

The construction sector and workers with no or a lower leadership role reported more negative working environment factors at baseline. We observed a statistically significant decline in social support from colleagues and social community at work, and a marginally significant decline in justice at work. For the rest of the constructs, we did not observe statistically significant changes.

CONCLUSIONS

We found significant differences in psychosocial work environment factors among work sectors and job positions at baseline. Contrary to our hypotheses, three psychosocial work environment factors decreased at follow-up. Possible explanations are the utilization of specific psychosocial factors as resources to cope with psychosocial stressors, high participant expectations that were not met by the intervention, insufficient time for structural changes, or the intervention prompting critical evaluations of the work environment. These findings will inform the design and implementation of the forthcoming clustered randomized controlled trial, where they will also be further investigated to validate their significance.

The effects of different types of organisational workplace mental health interventions on mental health and wellbeing in healthcare workers: a systematic review

OBJECTIVE

To determine if and which types of organisational interventions conducted in small and medium size enterprises (SMEs) in healthcare are effective on mental health and wellbeing.

METHODS

Following PRISMA guidelines, we searched six scientific databases, assessed the methodological quality of eligible studies using QATQS and grouped them into six organisational intervention types for narrative synthesis. Only controlled studies with at least one follow-up were eligible.

RESULTS

We identified 22 studies (23 articles) mainly conducted in hospitals with 16 studies rated of strong or moderate methodological quality. More than two thirds (68%) of the studies reported improvements in at least one primary outcome (mental wellbeing, burnout, stress, symptoms of depression or anxiety), most consistently in burnout with eleven out of thirteen studies. We found a strong level of evidence for the intervention type "Job and task modifications" and a moderate level of evidence for the types "Flexible work and scheduling" and "Changes in the physical work environment". For all other types, the level of evidence was insufficient. We found no studies conducted with an independent SME, however five studies with SMEs attached to a larger organisational structure. The effectiveness of workplace mental health interventions in these SMEs was mixed.

CONCLUSION

Organisational interventions in healthcare workers can be effective in improving mental health, especially in reducing burnout. Intervention types where the change in the work environment constitutes the intervention had the highest level of evidence. More research is needed for SMEs and for healthcare workers other than hospital-based physicians and nurses.

Implementing a complex mental health intervention in occupational settings: process evaluation of the MENTUPP pilot study

BACKGROUND

According to the Medical Research Council (MRC) framework, the theorisation of how multilevel, multicomponent interventions work and the understanding of their interaction with their implementation context are necessary to be able to evaluate them beyond their complexity. More research is needed to provide good examples following this approach in order to produce evidence-based information on implementation practices.

OBJECTIVES

This article reports on the results of the process evaluation of a complex mental health intervention in small and medium enterprises (SMEs) tested through a pilot study. The overarching aim is to contribute to the evidence base related to the recruitment, engagement and implementation strategies of applied mental health interventions in the workplace.

METHOD

The Mental Health Promotion and Intervention in Occupational Settings (MENTUPP) intervention was pilot tested in 25 SMEs in three work sectors and nine countries. The evaluation strategy of the pilot test relied on a mixed-methods approach combining qualitative and quantitative research methods. The process evaluation was inspired by the RE-AIM framework and the taxonomy of implementation outcomes suggested by Proctor and colleagues and focused on seven dimensions: reach, adoption, implementation, acceptability, appropriateness, feasibility and maintenance.

RESULTS

Factors facilitating implementation included the variety of the provided materials, the support provided by the research officers (ROs) and the existence of a structured plan for implementation, among others. Main barriers to implementation were the difficulty of talking about mental health, familiarisation with technology, difficulty in fitting the intervention into the daily routine and restrictions caused by COVID-19.

CONCLUSIONS

The results will be used to optimise the MENTUPP intervention and the theoretical framework that we developed to evaluate the causal mechanisms underlying MENTUPP. Conducting this systematic and comprehensive process evaluation contributes to the enhancement of the evidence base related to mental health interventions in the workplace and it can be used as a guide to overcome their contextual complexity.

TRIAL REGISTRATION NUMBER

ISRCTN14582090.

Outcome assessment of a complex mental health intervention in the workplace. Results from the MENTUPP pilot study

OBJECTIVE

Multicomponent interventions are recommendable to achieve the greatest mental health benefits, but are difficult to evaluate due to their complexity. Defining long-term outcomes, arising from a Theory of Change (ToC) and testing them in a pilot phase, is a useful approach to plan a comprehensive and meaningful evaluation later on. This article reports on the pilot results of an outcome evaluation of a complex mental health intervention and examines whether appropriate evaluation measures and indicators have been selected ahead of a clustered randomised control trial (cRCT).

METHODS

The MENTUPP pilot is an evidence-based intervention for Small and Medium Enterprises (SMEs) active in three work sectors and nine countries. Based on our ToC, we selected the MENTUPP long-term outcomes, which are reported in this article, are measured with seven validated scales assessing mental wellbeing, burnout, depression, anxiety, stigma towards depression and anxiety, absenteeism and presenteeism. The pilot MENTUPP intervention assessment took place at baseline and at 6 months follow-up.

RESULTS

In total, 25 SMEs were recruited in the MENTUPP pilot and 346 participants completed the validated scales at baseline and 96 at follow-up. Three long-term outcomes significantly improved at follow-up (p < 0.05): mental wellbeing, symptoms of anxiety, and personal stigmatising attitudes towards depression and anxiety.

CONCLUSIONS

The results of this outcome evaluation suggest that MENTUPP has the potential to strengthen employees' wellbeing and decrease anxiety symptoms and stigmatising attitudes. Additionally, this study demonstrates the utility of conducting pilot workplace interventions to assess whether appropriate measures and indicators have been selected. Based on the results, the intervention and the evaluation strategy have been optimised.

Implementation and evaluation of a multi-level mental health promotion intervention for the workplace (MENTUPP): study protocol for a cluster randomised controlled trial

BACKGROUND

Well-organised and managed workplaces can be a source of wellbeing. The construction, healthcare and information and communication technology sectors are characterised by work-related stressors (e.g. high workloads, tight deadlines) which are associated with poorer mental health and wellbeing. The MENTUPP intervention is a flexibly delivered, multi-level approach to supporting small- and medium-sized enterprises (SMEs) in creating mentally healthy workplaces. The online intervention is tailored to each sector and designed to support employees and leaders dealing with mental health difficulties (e.g. stress), clinical level anxiety and depression, and combatting mental health-related stigma. This paper presents the protocol for the cluster randomised controlled trial (cRCT) of the MENTUPP intervention in eight European countries and Australia.

METHODS

Each intervention country will aim to recruit at least two SMEs in each of the three sectors. The design of the cRCT is based on the experiences of a pilot study and guided by a Theory of Change process that describes how the intervention is assumed to work. SMEs will be randomly assigned to the intervention or control conditions. The aim of the cRCT is to assess whether the MENTUPP intervention is effective in improving mental health and wellbeing (primary outcome) and reducing stigma, depression and suicidal behaviour (secondary outcome) in employees. The study will also involve a process and economic evaluation.

CONCLUSIONS

At present, there is no known multi-level, tailored, flexible and accessible workplace-based intervention for the prevention of non-clinical and clinical symptoms of depression, anxiety and burnout, and the promotion of mental wellbeing. The results of this study will provide a comprehensive overview of the implementation and effectiveness of such an intervention in a variety of contexts, languages and cultures leading to the overall goal of delivering an evidence-based intervention for mental health in the workplace.

TRIAL REGISTRATION

Please refer to Item 2a and registration ISRCTN14104664. Registered on 12th July 2022.