Process Evaluation in Health Communication Media Campaigns: A Systematic Review

Objective. The objective is to examine the scope of health communication media campaign process evaluation methods, findings, and dissemination practices. Data Source. A systematic review of peer-reviewed literature was conducted using database searches. Study Inclusion and Exclusion Criteria. Published studies on process and implementation evaluation of health campaigns with a media component were included. Exclusion criteria included not health, non-empirical, no media campaign, or a focus on other evaluation types. Data Extraction. Articles were assessed for general campaign information, theory use, and details about process evaluation plan and procedures. Data Synthesis. A coding scheme based on 9 process evaluation best practice elements (e.g., fidelity and context) was applied. Process evaluation methods, measures, and reporting themes were synthesized. Results. Among 691 unique records, 46 articles were included. Process evaluation was the main focus for 71.7% of articles, yet only 39.1% reported how process evaluation informed campaign implementation strategy. Articles reported 4.39 elements on average (SD = 1.99; range 1-9), with reach (87.0%) and recruitment (73.9%) described most frequently, yet reporting was inconsistent. Further, the level of detail in reporting methods, theory, and analysis varied. Conclusions. Process evaluation provides insight about mechanisms and intervening variables that could meaningfully impact interpretations of outcome evaluations; however, process evaluations are less often included in literature. Recommendations for evidence-based process evaluation components to guide evaluation are discussed.

Health Care Providers as Key Change Agents in the Community Intervention Trial for Smoking Cessation (COMMIT)

The Community Intervention Trial for Smoking Cessation (COMMIT) is a multi-center collaborative trial sponsored by the National Cancer Institute. COMMIT utilizes multiple, community-based channels to facilitate quit attempts among heavy cigarette smokers. The “health care provider channel” is important, in that physicians, dentists, and other health care providers can effect changes in smoking behavior at both the individual and community level. There are over 1,600 primary care physicians and general practice dentists in the COMMIT intervention communities. This article describes the conceptual basis for the health care provider activities; results of a survey of Community attitudes and behaviors regarding smoking and health care; specific training and intervention activities; and the role of health care providers as community change agents in the smoking cessation arena.

Media Advocacy and Public Education in the Community Intervention Trial to Reduce Heavy Smoking (COMMIT)

The Community Intervention Trial (COMMIT) is designed to reduce the rate of heavy smoking in eleven pairs of North American communities over an eight-year period. The intervention, consisting of a minimum of fifty-one activities, is being implemented through local community boards and task forces. This article describes the goals and activities for the public education task force with a specific focus on “media advocacy,” an innovative use of mass media that follows more closely political activist models than traditional public service models. Two brief case studies are presented to illustrate some applications of media advocacy. The reasons for relatively infrequent use of media advocacy are discussed.

Evaluation Plan for the Community Intervention Trial for Smoking Cessation (COMMIT)

The National Cancer Institute is sponsoring the Community Intervention Trial for Smoking Cessation (COMMIT), a multi-center research project designed to test the value of a community-based effort to promote smoking cessation. The trial involves eleven matched pairs of communities with random assignment of one community per pair to the intervention or to the comparison condition. This article reviews the rationale and methodology of the COMMIT evaluation plan which is organized into four components: 1) outcome assessment, monitoring changes in community smoking patterns; 2) impact assessment, measuring the effect of the COMMIT intervention on mediating factors thought to be important in facilitating changes in community smoking behavior (e.g., social norms supporting nonsmoking); 3) process assessment, monitoring the quality and timeliness of intervention delivery; and 4) economic assessment, estimating the cost effectiveness of the intervention.

Introduction to the Community Intervention Trial for Smoking Cessation (COMMIT)

The Community Intervention Trial for smoking cessation (COMMIT) is sponsored by the National Cancer Institute and involves eleven pairs of communities in North America. COMMIT emphasizes a partnership between the eleven research institutions and their respective intervention communities in developing the structures needed to implement the intervention protocol. We summarize the epidemiological data and describe the prior community interventions that set the stage for COMMIT, and discuss how COMMIT may inform state-wide tobacco reduction demonstration programs. An overview of the articles that describe the COMMIT intervention and evaluation plan is presented.

Principles of Community Organization and Partnership for Smoking Cessation in the Community Intervention Trial for Smoking Cessation (COMMIT)

The Community Intervention Trial for Smoking Cessation (COMMIT) has adopted a community approach to smoking cessation. State-of-the-art interventions that have proven efficacious for smoking cessation are delivered to smokers through community-based organizations. An innovative adaptation of community organization methods accommodated the need for a standardized protocol with the flexibility required for diverse and unique communities. The unique characteristics of the eleven intervention communities are examined with a focus on differences in size, location, availability and importance of the intervention channels, and other factors that were important for community mobilization. Initial results of the mobilization process are summarized. Although there were some differences in the structures formed and the time required to complete the initial project activities, all eleven intervention sites were mobilized around the COMMIT goals and activities.

Process evaluation of a store-based environmental obesity intervention on two American Indian Reservations

Obesity and other diet-related chronic diseases are widespread in American Indian communities. Inadequate access to healthy food on many reservations has led to a high-fat, high-sugar diet. The purpose of this paper is to report on the results of the process evaluation of a food store-based program to improve diet on two American Indian reservations. Process data were collected from 11 intervention stores to document the implementation of the Apache Healthy Stores (AHS) program. Process evaluation instruments recorded the stocking of promoted foods, presence of in-store communication materials, implementation of and participation in the cooking demonstrations and taste tests, and the transmission of mass-media messages. At the store level, the program was implemented with a high level of dose and reach, and a moderate to high level of fidelity. At the community level, the AHS program was implemented with a moderate degree of fidelity and dose. At the individual level, the cooking demonstrations and taste tests reached a large number of community members with a high dose. Implementing the AHS program on multiple levels (store, community, individual) was challenging, and differed between levels. Overall, improvements were seen from start to finish as program staff monitored, documented and responded to barriers to implementation. Process data will be tied to outcomes and will be useful for the planning of future store-based programs.

Reducing drinking and related harms in college: evaluation of the "A Matter of Degree" program

OBJECTIVES

To examine the effects of a multisite environmental prevention initiative, the "A Matter of Degree" (AMOD) program, on student heavy alcohol consumption and resultant harms at ten colleges.

METHODS

A quasi-experimental longitudinal analysis of alcohol consumption and harms was employed, using repeated cross-sectional survey data from the Harvard School of Public Health College Alcohol Study (CAS). Areas examined included seven measures of alcohol consumption, thirteen measures of alcohol-related harms, and eight measures of secondhand effects of alcohol use by others. Comparisons were conducted on self-reported behavior of students for the ten AMOD sites in aggregate and by level of program implementation, with students at 32 comparison colleges in the CAS, for each outcome.

RESULTS

No statistically significant change was found in the overall ten-school AMOD program for outcome measures of interest from baseline (1997) to follow-up (2001). However, there was variation in the degree of environmental program development within AMOD during the intervention period. A pattern of statistically significant decreases in alcohol consumption, alcohol-related harms, and secondhand effects was observed, reflecting minor to more substantial changes across measures among students at the five program colleges that most closely implemented the AMOD model of environmental change. No similar pattern was observed for the low implementation sites or at 32 comparison colleges.

CONCLUSIONS

While there was no change in the ten AMOD schools in study measures, significant although small improvements in alcohol consumption and related harms at colleges were observed among students at the five AMOD sites that most closely implemented the environmental model. Fidelity to a program model conceptualized around changing alcohol-related policies, marketing, and promotions may reduce college student alcohol consumption and related harms. Further research is needed over the full course of the AMOD program to identify critical intervention components and elucidate pathways by which effects are realized.

A meta-analysis of the effect of mediated health communication campaigns on behavior change in the United States

A meta-analysis was performed of studies of mediated health campaigns in the United States in order to examine the effects of the campaigns on behavior change. Mediated health campaigns have small measurable effects in the short-term. Campaign effect sizes varied by the type of behavior: r=.15 for seat belt use, r=.13 for oral health, r=.09 for alcohol use reduction, r=.05 for heart disease prevention, r=.05 for smoking, r=.04 for mammography and cervical cancer screening, and r=.04 for sexual behaviors. Campaigns with an enforcement component were more effective than those without. To predict campaign effect sizes for topics other than those listed above, researchers can take into account whether the behavior in a cessation campaign was addictive, and whether the campaign promoted the commencement of a new behavior, versus cessation of an old behavior, or prevention of a new undesirable behavior. Given the small campaign effect sizes, campaign planners should set modest goals for future campaigns. The results can also be useful to evaluators as a benchmark for campaign effects and to help estimate necessary sample size.

Methodologic advances and ongoing challenges in designing community-based health promotion programs

Community intervention trials continue to attract researchers as potential ways to achieve widespread, long-term change in health behaviors. The first generations of community studies were somewhat unsophisticated in design and analysis, and their promise may have been overstated. As design and analysis issues were better defined, as secular trends caught up with the behaviors that researchers were trying to change, or as other unknown variables affected community studies, small effects of interventions were observed in community trials. Discussions were held in professional meetings and reported in the literature: Should community trials be discontinued? In general, the answer was a qualified no. In this paper, we briefly review some of the many advances made in community intervention trials, and address in more detail the challenges ahead.

Community interventions for reducing smoking among adults

BACKGROUND

Since smoking behaviour is determined by social context, the best way to reduce the prevalence of smoking may be to use community-wide programmes which use multiple channels to provide reinforcement, support and norms for not smoking.

OBJECTIVES

To assess the effectiveness of community interventions for reducing the prevalence of smoking.

SEARCH STRATEGY

We searched the Cochrane Tobacco Addiction Group specialised register, MEDLINE (1966-August 2001) and EMBASE (1980-August 2001) and reference lists of articles.

SELECTION CRITERIA

Controlled trials of community interventions for reducing smoking prevalence in adult smokers. The primary outcome was smoking behaviour.

DATA COLLECTION AND ANALYSIS

Data were extracted by one person and checked by a second.

MAIN RESULTS

Thirty two studies were included, of which seventeen included only one intervention and one comparison community. Only four studies used random assignment of communities to either the intervention or comparison group. The population size of the communities ranged from a few thousand to over 100,000 people. Change in smoking prevalence was measured using cross-sectional follow-up data in 27 studies. The estimated net decline ranged from -1.0% to 3.0% for men and women combined (10 studies). For women, the decline ranged from -0.2% to + 3.5% per year (n=11), and for men the decline ranged from -0.4% to +1.6% per year (n=12). Cigarette consumption and quit rates were only reported in a small number of studies. The two most rigorous studies showed limited evidence of an effect on prevalence. In the US COMMIT study there was no differential decline in prevalence between intervention and control communities, and there was no significant difference in the quit rates of heavier smokers who were the target intervention group. In the Australian CART study there was a significantly greater quit rate for men but not women.

REVIEWER'S CONCLUSIONS

The failure of the largest and best conducted studies to detect an effect on prevalence of smoking is disappointing. A community approach will remain an important part of health promotion activities, but designers of future programmes will need to take account of this limited effect in determining the scale of projects and the resources devoted to them.

Measuring community mobilization in the Seattle Minority Youth Health Project

This article assesses the validity and reliability of the approach used to measure community mobilization in the Seattle Minority Youth Health Project (MY Health), a neighborhood-based program to prevent drug use, violence, teen pregnancy, and sexually transmitted diseases (STDs). Two constructs were measured: neighborhood cooperation in solving problems, and sense of pride and identification with the neighborhood. The convergent validity of the measurement approach was assessed by comparing several independent measures of community mobilization generated from surveys of key neighborhood leaders, youth, and parents. For the neighborhood cooperation construct, correlations were uniformly positive across measures from different surveys and statistically significant about a quarter of the time. The correlations for the neighborhood pride construct were weaker and generally not statistically significant. Interrater reliability was low for all of the surveys, possibly reflecting varying ideas about what community mobilization meant among survey respondents.

Implications of the results of community intervention trials

This paper examines the results of population-level interventions conducted in three settings: entire communities, worksites, and schools. Four major conclusions are discussed: (a) Directions for the next generation of community-based interventions include targeting multiple levels of influence; addressing social inequalities in disease risk; involving communities in program planning and implementation; incorporating approaches for "tailoring" interventions; and utilizing rigorous process evaluation. (b) In addition to randomized controlled trials, it is time to use the full range of research phases available, from hypothesis generation and methods development to dissemination research. (c) The public health research agenda may have contributed to observed secular trends by placing behavioral risk factors on the social and media agendas. (d) The magnitude of the results of community intervention trials must be judged according to their potential public health or population-level effects. Small changes at the individual level may result in large benefits at the population level.

Using process data to explain outcomes. An illustration from the Child and Adolescent Trial for Cardiovascular Health (CATCH)

The purpose of this article is to illustrate the use of process evaluation for understanding study outcomes. Data from the Child and Adolescent Trial for Cardiovascular Health (CATCH), a large school-based field trial, are used. Teacher characteristics, measures of classroom curriculum implementation, and competing influences are linked to changes in dietary knowledge, intentions, and self-efficacy of students in the intervention schools. Multiple regression analyses indicate that teacher characteristics did not predict program implementation. Teacher characteristics and program fidelity, or the number of modifications made to the classroom curriculum during implementation, had direct and independent effects on student outcomes.

Evaluation of health education programs: current assessment and future directions

Recently there has been an increase in the different types of strategies used in health education interventions, including an emphasis on broadening programs focused on individual behavior change to include larger units of practice. There has also been an increasing critique of the traditional physical science paradigm for evaluating the multiple dimensions inherent in many interventions. Additionally, there is a growing recognition of the importance of involving multiple stakeholders in designing, implementing, and evaluating interventions. Each of these factors carries specific evaluation challenges. With the overall aim of strengthening the evaluation of health education programs, this article aims to (a) present conceptual and technical design issues and options, (b) describe different approaches to evaluation, (c) highlight evaluation approaches that have been effective, (d) critique the limitations of traditional evaluation approaches, (e) examine promising approaches and implications for future evaluations, and (f) provide recommendations for evaluation designs, data collection methods, roles, responsibilities, and principles for evaluating interventions.

Creating capacity through health education: what we know and what we don't

Researchers in health education have rarely convened to outline research priorities in the field. This article discusses the results of a meeting to develop a research agenda aimed at creating capacity to promote and maintain health. Salient research findings related to individual and community health are summarized and priority issues for future research are presented.

Community intervention trial for smoking cessation (COMMIT): II. Changes in adult cigarette smoking prevalence

OBJECTIVES

COMMIT (Community Intervention Trial for Smoking Cessation) investigated whether a community-level multichannel intervention would decrease the prevalence of adult cigarette smoking and increase quitting, with heavy smokers (> or = 25 cigarettes per day) receiving the highest priority.

METHODS

One community within each of 11 matched community pairs (10 in the United States, 1 in Canada) was randomly assigned to intervention. Baseline (1988) and final (1993) telephone surveys sampled households to determine prevalence of smoking behavior.

RESULTS

Among the target population aged 25 to 64 years, there was no intervention effect on heavy smoking prevalence, which decreased by 2.9 percentage points in both intervention and comparison communities. Overall smoking prevalence decreased by 3.5 in intervention communities vs 3.2 in comparison communities, a difference not statistically significant, while the mean quit ratios were 0.198 versus 0.185, respectively, a difference of 0.013 (90% test-based confidence interval = -0.003, 0.028).

CONCLUSIONS

Results are consistent with the cohort analysis reported separately, although the more powerful cohort design showed a statistically significant intervention effect upon light-to-moderate smokers. This community-based intervention did not have a significant impact on smoking prevalence beyond the favorable secular trends. In future efforts, additional strategies should be incorporated and rigorously evaluated.

Community Intervention Trial for Smoking Cessation (COMMIT): I. cohort results from a four-year community intervention

OBJECTIVES

The primary hypothesis of COMMIT (Community Intervention Trial for Smoking Cessation) was that a community-level, multi-channel, 4-year intervention would increase quit rates among cigarette smokers, with heavy smokers (> or = 25 cigarettes per day) of priority.

METHODS

One community within each of 11 matched community pairs (10 in the United States, 1 in Canada) was randomly assigned to intervention. Endpoint cohorts totaling 10,019 heavy smokers and 10,328 light-to-moderate smokers were followed by telephone.

RESULTS

The mean heavy smoker quit rate (i.e., the fraction of cohort members who had achieved and maintained cessation at the end of the trial) was 0.180 for intervention communities versus 0.187 for comparison communities, a nonsignificant difference (one-sided P = .68 by permutation test; 90% test-based confidence interval (CI) for the difference = -0.031, 0.019). For light-to-moderate smokers, corresponding quit rates were 0.306 and 0.275; this difference was significant (P = .004; 90% CI = 0.014, 0.047). Smokers in intervention communities had greater perceived exposure to smoking control activities, which correlated with outcome only for light-to-moderate smokers.

CONCLUSIONS

The impact of this community-based intervention on light-to-moderate smokers, although modest, has public health importance. This intervention did not increase quit rates of heavy smokers; reaching them may require new clinical programs and policy changes.

Design of process evaluation within the Child and Adolescent Trial for Cardiovascular Health (CATCH)

Process evaluation complements outcome evaluation by providing data to describe how a program was implemented, how well the activities delivered fit the original design, to whom services were delivered, the extent to which the target population was reached, and factors external to the program that may compete with the program effects. The process evaluation system used in the Child and Adolescent Trial for Cardiovascular Health (CATCH) is presented in this paper. The conceptual model underlying the CATCH process evaluation system is described, and process measures and data collection protocols are reviewed. Functions of process evaluation data in the trial include: (1) describing the implementation of the program, (2) quality control and monitoring, and (3) explaining program effects. The importance of incorporating process evaluation into final outcome analyses and assessments of program impact is emphasized.