Comparing Web and Mail Protocols for Administering Hospital Consumer Assessment of Healthcare Providers and Systems Surveys

OBJECTIVE

The objective of this study was to compare results of using web-based and mail (postal) Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) data collection protocols.

RESEARCH DESIGN

Patients who had been hospitalized in a New England Hospital were surveyed about their hospital experience. Patients who provided email addresses were randomized to 1 of 3 data collection protocols: web-alone, web with postal mail follow-up, and postal mail only. Those who did not provide email addresses were surveyed using postal mail only. Analyses compared response rates, respondent characteristics, and patient-reported experiences.

SUBJECTS

For an 8-week period, patients were discharged from the study hospital to home.

MEASURES

Measures included response rates, characteristics of respondents, 6 composite measures of their patient experiences, and 2 ratings of the hospital.

RESULTS

Response rates were significantly lower for the web-only protocol than the mail or combined protocols, and those who had not provided email addresses had lower response rates. Those over 65 were more likely than others to respond to all protocols, especially for the mail-only protocols. Respondents without email addresses were older, less educated, and reported worse health than those who provided email addresses. After adjusting for respondent differences, those in the combined protocol differed significantly from the mail (postal) only respondents on 2 measures of patient experience; those in the web-only protocol differed on one. Those not providing an email address differed from those who did on one measure.

CONCLUSION

If web-based protocols are used for HCAHPS surveys, adjustments for a mode of data collection are needed to make results comparable.

The Effect of the Familiarity of a Survey Sender on Response Outcomes in a Large-Scale Survey of Emergency Medical Services Agencies

Survey response is higher when the request comes from a familiar entity compared to an unknown sender. Little is known about how sender influences response to surveys of organizations. We assessed whether familiarity of the sender influences response outcomes in a survey of emergency medical services agencies. Emergency medical services agencies in one U.S. state were randomly assigned to receive survey emails from either a familiar or unfamiliar sender. Both deployment approaches were subsequently used nationwide, with each state selecting one of the two contact methods. Experimental results showed that requests from the familiar sender achieved higher survey response (54.3%) compared to requests from the unfamiliar sender (36.9%; OR: 2.03; 95% CI: 1.23, 3.33). Similar results were observed in the subsequent nationwide survey; in states where the familiar sender deployed the survey, 62.0% of agencies responded, compared to 51.0% when the survey was sent by the unfamiliar sender (OR: 1.57; 95% CI: 1.47, 1.67). The response difference resulted in nearly 60 additional hours of staff time needed to perform telephone follow-up to nonrespondents. When surveying healthcare organizations, surveyors should recognize that it is more challenging to obtain responses without a pre-established relationship with the organizations.

Methods for Improving Representativeness in a Web Survey on Sexual Assault Among College Students

Many colleges and universities conduct web-based campus climate surveys to understand the prevalence and nature of sexual assault among their students. When designing and fielding a web survey to measure a sensitive topic like sexual assault, methodological decisions, including the length of the field period and the use or amount of an incentive, can affect the representativeness of the respondent sample leading to biased or imprecise estimates. This study uses data from the Campus Climate Survey Validation Study (CCSVS) to assess how the interaction between field period length and survey incentive amount affects nonresponse, sample representativeness, and the precision of survey estimates. Research suggests that using robust incentives gives potential survey respondents a reason to complete the survey beyond their intrinsic motivation to do so. Likewise, extending the field period gives more time to people who may be less intrinsically motivated to complete the survey. Both serve to increase sample size and representativeness, minimize bias, and improve estimate precision. Schools, however, sometimes lack the time and/or resources for both a robust incentive and a lengthy field period, and this study examines the extent to which the potential negative impacts of not using one can be mitigated by the presence of the other. Findings indicate that target response rates can be achieved using a smaller incentive if the field period is lengthy but, even with a lengthy field period, the use of a smaller incentive can result in biased estimates due to a lack of representativeness. Conversely, when a robust incentive is used and weights are developed to adjust for nonresponse, a shorter field period will not have a significant impact on point estimates, but the estimates will be less precise due to fewer respondents participating in the survey.

Reducing speeding in web surveys by providing immediate feedback

It is well known that some survey respondents reduce the effort they invest in answering questions by taking mental shortcuts - survey satisficing. This is a concern because such shortcuts can reduce the quality of responses and, potentially, the accuracy of survey estimates. This article explores "speeding," an extreme type of satisficing, which we define as answering so quickly that respondents could not have given much, if any, thought to their answers. To reduce speeding among online respondents we implemented an interactive prompting technique. When respondents answered faster than a minimal response time threshold, they received a message encouraging them to answer carefully and take their time. Across six web survey experiments, this prompting technique reduced speeding on subsequent questions compared to a no prompt control. Prompting slowed response times whether the speeding that triggered the prompt occurred early or late in the questionnaire, in the first or later waves of a longitudinal survey, among respondents recruited from non-probability or probability panels, or whether the prompt was delivered on only the first or on all speeding episodes. In addition to reducing speeding, the prompts increased response accuracy on simple arithmetic questions for a key subgroup. Prompting also reduced later straightlining in one experiment, suggesting the benefits may generalize to other types of mental shortcuts. Although the prompting could have annoyed respondents, it was not accompanied by a noticeable increase in breakoffs. As an alternative technique, respondents in one experiment were asked to explicitly commit to responding carefully. This global approach complemented the more local, interactive prompting technique on several measures. Taken together, these results suggest that interactive interventions of this sort may be useful for increasing respondents' conscientiousness in online questionnaires, even though these questionnaires are self-administered.

Informed Consent for Web Paradata Use

Survey researchers are making increasing use of paradata - such as keystrokes, clicks, and timestamps - to evaluate and improve survey instruments but also to understand respondents and how they answer surveys. Since the introduction of paradata, researchers have been asking whether and how respondents should be informed about the capture and use of their paradata while completing a survey. In a series of three vignette-based experiments, we examine alternative ways of informing respondents about capture of paradata and seeking consent for their use. In all three experiments, any mention of paradata lowers stated willingness to participate in the hypothetical surveys. Even the condition where respondents were asked to consent to the use of paradata at the end of an actual survey resulted in a significant proportion declining. Our research shows that requiring such explicit consent may reduce survey participation without adequately informing survey respondents about what paradata are and why they are being used.

Increasing Respondents' Use of Definitions in Web Surveys

Survey respondents may misinterpret the questions they are asked, potentially undermining the accuracy of their answers. One way to reduce this risk is to make definitions of key question concepts available to the respondents. In the current study we compared two methods of making definitions available to web survey respondents - displaying the definition with the question text and displaying the definition when respondents roll the mouse over the relevant question terms. When definitions were always displayed they were consulted more than when they required a rollover request. The length of the definitions did not affect how frequently they were used under either method of display. Respondents who completed training items designed to encourage definition use actually requested definitions less often, suggesting that they may value minimal effort over improved understanding. We conclude that at least for small numbers of questions, providing definitions with the question is likely to be the more effective approach than rollovers or hyperlinks.