Analysis of clustered data in community psychology: with an example from a worksite smoking cessation project

Consequences of ignoring clustering in linear regression

Background

Clustering of observations is a common phenomenon in epidemiological and clinical research. Previous studies have highlighted the importance of using multilevel analysis to account for such clustering, but in practice, methods ignoring clustering are often employed. We used simulated data to explore the circumstances in which failure to account for clustering in linear regression could lead to importantly erroneous conclusions.

Methods

We simulated data following the random-intercept model specification under different scenarios of clustering of a continuous outcome and a single continuous or binary explanatory variable. We fitted random-intercept (RI) and ordinary least squares (OLS) models and compared effect estimates with the “true” value that had been used in simulation. We also assessed the relative precision of effect estimates, and explored the extent to which coverage by 95% confidence intervals and Type I error rates were appropriate.

Results

We found that effect estimates from both types of regression model were on average unbiased. However, deviations from the “true” value were greater when the outcome variable was more clustered. For a continuous explanatory variable, they tended also to be greater for the OLS than the RI model, and when the explanatory variable was less clustered. The precision of effect estimates from the OLS model was overestimated when the explanatory variable varied more between than within clusters, and was somewhat underestimated when the explanatory variable was less clustered. The cluster-unadjusted model gave poor coverage rates by 95% confidence intervals and high Type I error rates when the explanatory variable was continuous. With a binary explanatory variable, coverage rates by 95% confidence intervals and Type I error rates deviated from nominal values when the outcome variable was more clustered, but the direction of the deviation varied according to the overall prevalence of the explanatory variable, and the extent to which it was clustered.

Conclusions

In this study we identified circumstances in which application of an OLS regression model to clustered data is more likely to mislead statistical inference. The potential for error is greatest when the explanatory variable is continuous, and the outcome variable more clustered (intraclass correlation coefficient is ≥ 0.01).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-021-01333-7.

Rationale and design: telepsychology service delivery for depressed elderly veterans

Background

Older adults who live in rural areas experience significant disparities in health status and access to mental health care. "Telepsychology," (also referred to as "telepsychiatry," or "telemental health") represents a potential strategy towards addressing this longstanding problem. Older adults may benefit from telepsychology due to its: (1) utility to address existing problematic access to care for rural residents; (2) capacity to reduce stigma associated with traditional mental health care; and (3) utility to overcome significant age-related problems in ambulation and transportation. Moreover, preliminary evidence indicates that telepsychiatry programs are often less expensive for patients, and reduce travel time, travel costs, and time off from work. Thus, telepsychology may provide a cost-efficient solution to access-to-care problems in rural areas.

Methods

We describe an ongoing four-year prospective, randomized clinical trial comparing the effectiveness of an empirically supported treatment for major depressive disorder, Behavioral Activation, delivered either via in-home videoconferencing technology ("Telepsychology") or traditional face-to-face services ("Same-Room"). Our hypothesis is that in-homeTelepsychology service delivery will be equally effective as the traditional mode (Same-Room). Two-hundred twenty-four (224) male and female elderly participants will be administered protocol-driven individual Behavioral Activation therapy for depression over an 8-week period; and subjects will be followed for 12-months to ascertain longer-term effects of the treatment on three outcomes domains: (1) clinical outcomes (symptom severity, social functioning); (2) process variables (patient satisfaction, treatment credibility, attendance, adherence, dropout); and (3) economic outcomes (cost and resource use).

Discussion

Results from the proposed study will provide important insight into whether telepsychology service delivery is as effective as the traditional mode of service delivery, defined in terms of clinical, process, and economic outcomes, for elderly patients with depression residing in rural areas without adequate access to mental health services.

Trial registration

National Institutes of Health Clinical Trials Registry (ClinicalTrials.gov identifier# NCT00324701).

Getting the big picture in community science: methods that capture context

Community science has a rich tradition of using theories and research designs that are consistent with its core value of contextualism. However, a survey of empirical articles published in the American Journal of Community Psychology shows that community scientists utilize a narrow range of statistical tools that are not well suited to assess contextual data. Multilevel modeling, geographic information systems (GIS), social network analysis, and cluster analysis are recommended as useful tools to address contextual questions in community science. An argument for increased methodological consilience is presented, where community scientists are encouraged to adopt statistical methodology that is capable of modeling a greater proportion of the data than is typical with traditional methods.

Statistical and Substantive Inferences in Public Health: Issues in the Application of Multilevel Models

Multilevel statistical models have become increasingly popular among public health researchers over the past decade. Yet the enthusiasm with which these models are being adopted may obscure rather than solve some problems of statistical and substantive inference. We discuss the three most common applications of multilevel models in public health: (a) cluster-randomized trials, (b) observational studies of the multilevel etiology of health and disease, and (c) assessments of health care provider performance. In each area of investigation, we describe how multilevel models are being applied, comment on the validity of the statistical and substantive inferences being drawn, and suggest ways in which the strengths of multilevel models might be more fully exploited. We conclude with a call for more careful thinking about multilevel causal inference.

Demand characteristics of residential substance abuse treatment programs

PURPOSE

This study examined the objective demand characteristics of treatment programs in which substance abuse patients, or psychiatric patients, were residing. It also examined associations of objective demand with substance abuse patients' perceived expectations for functioning during treatment and patients' in-program participation.

METHODS

A total of 994 patients living in 79 programs took part.

RESULTS

When patients had a substance abuse rather than a psychiatric problem, objective demand was higher: program policies had higher requirements for functioning and more resident control; programs offered fewer health-treatment services; and the physical design provided fewer safety features and social-recreational aids. Compared to substance abuse patients in low-demand programs, patients in high-demand programs perceived the program to have higher expectations, in that the treatment climate exerted more press to develop relationships, set goals, and be organized. Patients in high-demand programs engaged more in self-initiated activities and participated more in treatment services and program-organized events. Substance abuse patients' activity and participation levels were determined jointly by the level of demand and by the expectations for patients' expressiveness and self-understanding of their personal problems.

IMPLICATIONS

The findings illustrate the importance of considering objective indices of demand in conjunction with perceived expectations to improve patients' treatment outcomes.

Individual and contextual predictors of involvement in twelve-step self-help groups after substance abuse treatment

Drawing on ecological and narrative theories of self-help groups, this study tests a multilevel model predicting self-help group involvement among male veterans who received inpatient substance abuse treatment. Following K. Maton (1993), the study moves beyond the individual-level of analysis to encompass variables in the treatment and post-treatment social ecology. Surveys administered to patients (N = 3,018) and treatment staff (N = 329) assessed these predictor domains and self-help group involvement 1 year after discharge. A hierarchical linear model fit to the data indicates that greater involvement in 12-step groups after discharge is predicted by the compatibility between personal and treatment belief systems. The implications of these findings for efforts to facilitate transitions between inpatient professional treatment and community-based self-help groups are discussed.

Multilevel analysis in public health research

Over the past few years there has been growing interest in considering factors defined at multiple levels in public health research. Multilevel analysis has emerged as one analytical strategy that may partly address this need, by allowing the simultaneous examination of group-level and individual-level factors. This paper reviews the rationale for using multilevel analysis in public health research, summarizes the statistical methodology, and highlights some of the research questions that have been addressed using these methods. The advantages and disadvantages of multilevel analysis compared with standard methods are reviewed. The use of multilevel analysis raises theoretical and methodological issues related to the theoretical model being tested, the conceptual distinction between group- and individual-level variables, the ability to differentiate "independent" effects, the reciprocal relationships between factors at different levels, and the increased complexity that these models imply. The potentialities and limitations of multilevel analysis, within the broader context of understanding the role of factors defined at multiple levels in shaping health outcomes, are discussed.

Looking for the disorder in conduct disorder

Besides their well-known externalizing behavior, children with conduct disorder (CD) often have additional impairments outside the criteria for the CD diagnosis. In a 5-year study of 984 treated children (ages 5-17 years), those with CD had an average of 2.2 primary diagnoses. Children with CD showed the worst problem and impairment scores in comparison with 11 common diagnoses. Compared with other treated children, children with CD achieved worse scores on 14 of 15 syndromes, including internalizing problems such as withdrawal and major depression. The average child with CD had larger relapse scores in the 1.5- to 3-year period after admission to treatment. This pattern, pervasive at intake and chronic in course, resembles a global disability more than a circumscribed problem managed with a narrow range of treatments specific to it.

Social support in a worksite smoking intervention. A test of theoretical models

To better understand how social support operates in smoking cessation, three models of support were tested: main effect, stress-buffering, and indirect. Employees from 61 worksites received (a) self-help manuals (SH); (b) self-help manuals and incentives (I); or (c) self-help manuals, incentives, and social support groups (G). At 24 months, results suggest that the main effect model of social support was operating in this study. That is, social support had a direct influence on smoking cessation. The group intervention significantly enhanced positive partner support across all time points, and partner support facilitated quitting smoking. Higher levels of appraisal support also significantly predicted successful quitting. The beneficial effects of social support were sustained throughout the process of quitting and long-term maintenance.

A longitudinal assessment of teacher perceptions of parent involvement in children's education and school performance

This study examines the ways in which parental involvement in children's education changes over time and how it relates to children's social and academic functioning in school. Teachers provided information on parent involvement and school performance for 1,205 urban, kindergarten through third-grade children for 3 consecutive years. They rated the following four dimensions of parent involvement: frequency of parent-teacher contact, quality of the parent-teacher interactions, participation in educational activities at home, and participation in school activities. As predicted, the frequency of parent-teacher contacts, quality of parent-teacher interactions, and parent participation at school declined from Years 1 to 3. Every parent involvement variable correlated moderately with school performance and parent involvement in Years 1 and 2, and accounted for a small, but significant amount of variance in Year 3 performance after controlling for initial performance level. Participation in educational activities at home predicted the widest range of performance variables. Results suggest that enhancing parental involvement in children's schooling relates to improvements in school functioning.

A review and analysis of the clinical and cost-effectiveness studies of comprehensive health promotion and disease management programs at the worksite: 1995-1998 update (IV)

The fourth in a series of critical reviews, this study examines the clinical and cost outcome evaluations of 10 worksite health promotion programs that were comprehensive, multifactorial, and directed at risk management. The studies, conducted between 1994 and 1998, indicate favorable clinical and cost outcomes and suggest characteristics of worksite interventions that may be critical for effectiveness.

Social, community, and preventive interventions

Psychology can and should be at the forefront of participation in social, community, and preventive interventions. This chapter focuses on selective topics under two general areas: violence as a public health problem and health promotion/competence promotion across the life span. Under violence prevention, discussion of violence against women, youth violence, and child maltreatment are the focal points. Under health and competence promotion, attention is paid to the prevention of substance abuse and HIV/AIDS. We highlight a few significant theoretical and empirical contributions, especially from the field of community/prevention psychology. The chapter includes a brief overview of diversity issues, which are integral to a comprehensive discussion of these prevention efforts. We argue that the field should extend its role in social action while emphasizing the critical importance of rigorous research as a component of future interventions.

Interaction effects of client and treatment program characteristics on retention: an exploratory analysis using hierarchical linear models

This paper applied a hierarchical linear modeling approach to explore the interaction effects of treatment program and client characteristics on client retention in treatment for drug users. Program characteristics included services provision, funding sources, and staff-client gender congruence, and client characteristics included gender, age at admission, and drug use level prior to admission. The same model was applied separately to three modalities: residential, methadone maintenance, and outpatient drug-free programs. Data were obtained from 59 treatment programs and 3,764 of their clients who had discharge records. The most noteworthy significant interaction effect detected was program's funding source and client's gender on treatment retention in the outpatient drug-free modality. For example, female clients remained less time in the programs that accepted only public funding than in the programs that accepted both public and private funding. Male clients remained in the treatment an average of 25.3 fewer days than female clients in drug-free programs that only accepted public fund, but stayed about the same time as females if the programs received mixed funding.

Tobacco, drug, and HIV preventive media interventions

Media-based health promotion interventions can reach large segments of the population and lower barriers to participation in programs. By collaborating with media specialists, we can develop interventions that might be broader, more publicized, and ultimately more likely to change behaviors and attitudes within the community. The present paper describes a series of media interventions involving smoking cessation, drug abuse prevention, stress reduction, weight control, and HIV/AIDS prevention. The process by which I became involved in these interventions is discussed, and the implications of developing these types of large-scale behavioral interventions are discussed.

Context, composition and heterogeneity: using multilevel models in health research

This paper considers the use of multilevel models in health research. Attention focuses on the structure and potential of such models and particular consideration is given to their use in elucidating the importance of contextual effects in relation to individual level social and demographic factors in understanding health outcomes, health-related behaviour and health service performance. Four graphical typologies are used to outline the questions that multilevel models can address and the paper illustrates their potential by drawing on published examples in a number of different research areas.

Clinical and cost outcomes of multifactorial, cardiovascular risk management interventions in worksites: a comprehensive review and analysis

This paper is a critical review of the clinical and cost outcome evaluation studies of multifactorial, comprehensive, cardiovascular risk management programs in worksites. A comprehensive international literature search conducted under the auspices of the National Heart, Lung and Blood Institute identified 17 articles based on 12 studies that examined the clinical outcomes of multifactorial, comprehensive programs. These articles were identified through MEDLINE, manual searches of recent journals, and through direct inquiries to worksite health promotion researchers. All studies were conducted between 1978 and 1995, with 1978 being the date of the first citation of a methodologically rigorous evaluation. Of the 12 research studies, only 8 utilized the worksite as both the unit of assignment and as the unit of analysis. None of the studies analyzed adequately for cost effectiveness. Given this limitation, this review briefly considers the relevant worksite research that has demonstrated cost outcomes. Worksite-based, multifactorial cardiovascular intervention programs reviewed for this article varied widely in the comprehensiveness, intensity, and duration of both the interventions and evaluations. Results from randomized trials suggest that providing opportunities for individualized, cardiovascular risk reduction counseling for high-risk employees within the context of comprehensive programming may be the critical component of an effective worksite intervention. Despite the many limitations of the current methodologies of the 12 studies, the majority of the research to date indicates the following: (1) favorable clinical and cost outcomes; (2) that more recent and more rigorously designed research tends to support rather than refute earlier and less rigorously designed studies; and (3) that rather than interpreting the methodological flaws and diversity as inherently negative, one may consider it as indicative of a robust phenomena evident in many types of worksites, with diverse employees, differing interventions, and varying degrees of methodological sophistication. Results of these studies reviewed provide both cautious optimism about the effectiveness of these worksite programs and insights regarding the essential components and characteristics of successful programs.