id okr-10986-3403
recordtype oai_dc
spelling okr-10986-34032021-04-23T14:02:09Z Beyond Baseline and Follow-up : The Case for More T in Experiments McKenzie, David AUTOCORRELATION BOOTSTRAP CHOLESTEROL CLINICAL TRIALS CONFIDENCE INTERVALS CORRELATIONS COVARIANCE DEVELOPMENT ECONOMICS DEVELOPMENT POLICY DEVELOPMENT RESEARCH DIARRHEA ECONOMETRICS ECONOMIC OUTCOMES ECONOMICS ECONOMICS RESEARCH EQUATIONS ESTIMATORS EXPERIMENTAL IMPACT EVALUATION EXPERIMENTAL IMPACT EVALUATIONS EXPERIMENTAL STUDIES EXPERIMENTS EXTERNALITIES FIELD EXPERIMENTS FINANCIAL CRISIS FIXED COSTS HEADACHES HYPOTHESES INCOME INVENTORY LAW OF LARGE NUMBERS LEAST SQUARES REGRESSION MARGINAL COST MEASUREMENT ERRORS MEDICINE PHYSICAL HEALTH PRECISION RANDOMIZATION RESEARCH METHODOLOGY RESEARCH WORKING PAPERS RESEARCHERS SAMPLE SIZE SIGNIFICANCE LEVEL STANDARD DEVIATION STATA TIME SERIES TREATMENT VALIDITY VARIABILITY WATER TREATMENT WEALTH The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations. 2012-03-19T18:01:51Z 2012-03-19T18:01:51Z 2011-04-01 http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20110425104143 http://hdl.handle.net/10986/3403 English Policy Research working paper ; no. WPS 5639 CC BY 3.0 IGO http://creativecommons.org/licenses/by/3.0/igo/ World Bank Publications & Research :: Policy Research Working Paper The World Region The World Region
repository_type Digital Repository
institution_category Foreign Institution
institution Digital Repositories
building World Bank Open Knowledge Repository
collection World Bank
language English
topic AUTOCORRELATION
BOOTSTRAP
CHOLESTEROL
CLINICAL TRIALS
CONFIDENCE INTERVALS
CORRELATIONS
COVARIANCE
DEVELOPMENT ECONOMICS
DEVELOPMENT POLICY
DEVELOPMENT RESEARCH
DIARRHEA
ECONOMETRICS
ECONOMIC OUTCOMES
ECONOMICS
ECONOMICS RESEARCH
EQUATIONS
ESTIMATORS
EXPERIMENTAL IMPACT EVALUATION
EXPERIMENTAL IMPACT EVALUATIONS
EXPERIMENTAL STUDIES
EXPERIMENTS
EXTERNALITIES
FIELD EXPERIMENTS
FINANCIAL CRISIS
FIXED COSTS
HEADACHES
HYPOTHESES
INCOME
INVENTORY
LAW OF LARGE NUMBERS
LEAST SQUARES REGRESSION
MARGINAL COST
MEASUREMENT ERRORS
MEDICINE
PHYSICAL HEALTH
PRECISION
RANDOMIZATION
RESEARCH METHODOLOGY
RESEARCH WORKING PAPERS
RESEARCHERS
SAMPLE SIZE
SIGNIFICANCE LEVEL
STANDARD DEVIATION
STATA
TIME SERIES
TREATMENT
VALIDITY
VARIABILITY
WATER TREATMENT
WEALTH
spellingShingle AUTOCORRELATION
BOOTSTRAP
CHOLESTEROL
CLINICAL TRIALS
CONFIDENCE INTERVALS
CORRELATIONS
COVARIANCE
DEVELOPMENT ECONOMICS
DEVELOPMENT POLICY
DEVELOPMENT RESEARCH
DIARRHEA
ECONOMETRICS
ECONOMIC OUTCOMES
ECONOMICS
ECONOMICS RESEARCH
EQUATIONS
ESTIMATORS
EXPERIMENTAL IMPACT EVALUATION
EXPERIMENTAL IMPACT EVALUATIONS
EXPERIMENTAL STUDIES
EXPERIMENTS
EXTERNALITIES
FIELD EXPERIMENTS
FINANCIAL CRISIS
FIXED COSTS
HEADACHES
HYPOTHESES
INCOME
INVENTORY
LAW OF LARGE NUMBERS
LEAST SQUARES REGRESSION
MARGINAL COST
MEASUREMENT ERRORS
MEDICINE
PHYSICAL HEALTH
PRECISION
RANDOMIZATION
RESEARCH METHODOLOGY
RESEARCH WORKING PAPERS
RESEARCHERS
SAMPLE SIZE
SIGNIFICANCE LEVEL
STANDARD DEVIATION
STATA
TIME SERIES
TREATMENT
VALIDITY
VARIABILITY
WATER TREATMENT
WEALTH
McKenzie, David
Beyond Baseline and Follow-up : The Case for More T in Experiments
geographic_facet The World Region
The World Region
relation Policy Research working paper ; no. WPS 5639
description The vast majority of randomized experiments in economics rely on a single baseline and single follow-up survey. If multiple follow-ups are conducted, the reason is typically to examine the trajectory of impact effects, so that in effect only one follow-up round is being used to estimate each treatment effect of interest. While such a design is suitable for study of highly autocorrelated and relatively precisely measured outcomes in the health and education domains, this paper makes the case that it is unlikely to be optimal for measuring noisy and relatively less autocorrelated outcomes such as business profits, household incomes and expenditures, and episodic health outcomes. Taking multiple measurements of such outcomes at relatively short intervals allows the researcher to average out noise, increasing power. When the outcomes have low autocorrelation, it can make sense to do no baseline at all. Moreover, the author shows how for such outcomes, more power can be achieved with multiple follow-ups than allocating the same total sample size over a single follow-up and baseline. The analysis highlights the large gains in power from ANCOVA rather than difference-in-differences when autocorrelations are low and a baseline is taken. The paper discusses the issues involved in multiple measurements, and makes recommendations for the design of experiments and related non-experimental impact evaluations.
format Publications & Research :: Policy Research Working Paper
author McKenzie, David
author_facet McKenzie, David
author_sort McKenzie, David
title Beyond Baseline and Follow-up : The Case for More T in Experiments
title_short Beyond Baseline and Follow-up : The Case for More T in Experiments
title_full Beyond Baseline and Follow-up : The Case for More T in Experiments
title_fullStr Beyond Baseline and Follow-up : The Case for More T in Experiments
title_full_unstemmed Beyond Baseline and Follow-up : The Case for More T in Experiments
title_sort beyond baseline and follow-up : the case for more t in experiments
publishDate 2012
url http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20110425104143
http://hdl.handle.net/10986/3403
_version_ 1764386927673344000