Record Details

Replication Data for: A Text-As-Data Approach for Using Open-Ended Responses as Manipulation Checks

Harvard Dataverse (Africa Rice Center, Bioversity International, CCAFS, CIAT, IFPRI, IRRI and WorldFish)

View Archive Info
 
 
Field Value
 
Title Replication Data for: A Text-As-Data Approach for Using Open-Ended Responses as Manipulation Checks
 
Identifier https://doi.org/10.7910/DVN/WXIRQN
 
Creator Ziegler, Jeffrey
 
Publisher Harvard Dataverse
 
Description Abstract: Participants that complete online surveys and experiments may be inattentive, which can hinder researchers’ ability to draw substantive or causal inferences. As such, many practitioners include multiple factual or informational closed-ended manipulation checks to identify low-attention respondents. However, closed-ended manipulation checks are either correct or incorrect, which allows participants to more easily guess and it reduces the potential variation in attention between respondents. In response to these shortcomings, I develop an automatic and standardized methodology to measure attention that relies on the text that respondents provide in an open-ended manipulation check. There are multiple benefits to this approach. First, it provides a continuous measure of attention, which allows for greater variation between respondents. Second, it reduces the reliance on subjective, paid humans to analyze open-ended responses. Last, I outline how to diagnose the impact of inattentive workers on the overall results, including how to assess the average treatment effect of those respondents that likely received the treatment. I provide easy-to-use software in R to implement these suggestions for open-ended manipulation checks.
 
Subject Social Sciences
text-as-data
open-ended responses
survey experiments
manipulation checks
online respondent attention
 
Contributor Ziegler, Jeffrey