Beyond Baseline and Follow-up : The Case for More T in Experiments
The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one fo...
| Main Author: | |
|---|---|
| Format: | Policy Research Working Paper |
| Language: | English |
| Published: |
2012
|
| Subjects: | |
| Online Access: | http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20110425104143 http://hdl.handle.net/10986/3403 |
| Summary: | The vast majority of randomized
experiments in economics rely on a single baseline and
single follow-up survey. If multiple follow-ups are
conducted, the reason is typically to examine the trajectory
of impact effects, so that in effect only one follow-up
round is being used to estimate each treatment effect of
interest. While such a design is suitable for study of
highly autocorrelated and relatively precisely measured
outcomes in the health and education domains, this paper
makes the case that it is unlikely to be optimal for
measuring noisy and relatively less autocorrelated outcomes
such as business profits, household incomes and
expenditures, and episodic health outcomes. Taking multiple
measurements of such outcomes at relatively short intervals
allows the researcher to average out noise, increasing
power. When the outcomes have low autocorrelation, it can
make sense to do no baseline at all. Moreover, the author
shows how for such outcomes, more power can be achieved with
multiple follow-ups than allocating the same total sample
size over a single follow-up and baseline. The analysis
highlights the large gains in power from ANCOVA rather than
difference-in-differences when autocorrelations are low and
a baseline is taken. The paper discusses the issues involved
in multiple measurements, and makes recommendations for the
design of experiments and related non-experimental impact evaluations. |
|---|