Can We Trust Shoestring Evaluations?

Can We Trust Shoestring Evaluations?

Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, ask...

Full description

Bibliographic Details
Main Author: Ravallion, Martin
Format: Policy Research Working Paper
Language:English
Published: 2012
Subjects:
AGRICULTURE
ANTIPOVERTY PROGRAM
CLINICS
CORRELATIONS
DATA COLLECTION
DEBT
DEVELOPMENT POLICY
DEVELOPMENT PROJECTS
DEVELOPMENT RESEARCH
DWELLING
ECONOMIC RESEARCH
ECONOMIC SURVEYS
ESTIMATORS
EXTERNALITIES
GENDER
GROWTH MODELS
HOUSEHOLD SURVEYS
HOUSEHOLDS
HOUSING
HYGIENE
IMPACT EVALUATION
INCOME
INTEREST RATE
INTERVENTION
INTERVENTIONS
INTERVIEWS
LABOR MOBILITY
LEARNING
LIVING CONDITIONS
LIVING STANDARDS
LONGITUDINAL DATA
MARGINAL COST
OCCUPATION
POVERTY REDUCTION
PRODUCTION COSTS
PRODUCTION INPUTS
PROGRAM EVALUATION
PROGRAMS
QUANTITATIVE DATA
QUESTIONNAIRES
RELIABILITY
RESEARCH METHODS
RESEARCH WORKING PAPERS
SAFETY
SELECTION BIAS
SOCIAL SCIENCE
SURVEY DATA
SURVEY DESIGN
TARGETING
TREATMENT EFFECTS
URBAN AREAS
VILLAGES
WEIGHTING
WELFARE LEVELS
Online Access:http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20120305094659
http://hdl.handle.net/10986/3269
id okr-10986-3269
recordtype oai_dc
spelling okr-10986-32692021-04-23T14:02:08Z Can We Trust Shoestring Evaluations? Ravallion, Martin AGRICULTURE ANTIPOVERTY PROGRAM CLINICS CORRELATIONS DATA COLLECTION DEBT DEVELOPMENT POLICY DEVELOPMENT PROJECTS DEVELOPMENT RESEARCH DWELLING ECONOMIC RESEARCH ECONOMIC SURVEYS ESTIMATORS EXTERNALITIES GENDER GROWTH MODELS HOUSEHOLD SURVEYS HOUSEHOLDS HOUSING HYGIENE IMPACT EVALUATION INCOME INTEREST RATE INTERVENTION INTERVENTIONS INTERVIEWS LABOR MOBILITY LEARNING LIVING CONDITIONS LIVING STANDARDS LONGITUDINAL DATA MARGINAL COST OCCUPATION POVERTY REDUCTION PRODUCTION COSTS PRODUCTION INPUTS PROGRAM EVALUATION PROGRAMS QUANTITATIVE DATA QUESTIONNAIRES RELIABILITY RESEARCH METHODS RESEARCH WORKING PAPERS SAFETY SELECTION BIAS SOCIAL SCIENCE SURVEY DATA SURVEY DESIGN TARGETING TREATMENT EFFECTS URBAN AREAS VILLAGES WEIGHTING WELFARE LEVELS Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring," method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings. 2012-03-19T17:29:27Z 2012-03-19T17:29:27Z 2012-03-01 http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20120305094659 http://hdl.handle.net/10986/3269 English Policy Research working paper ; no. WPS 5983,Paper is funded by the Knowledge for Change Program (KCP) CC BY 3.0 IGO http://creativecommons.org/licenses/by/3.0/igo World Bank Publications & Research :: Policy Research Working Paper The World Region The World Region
repository_type Digital Repository
institution_category Foreign Institution
institution Digital Repositories
building World Bank Open Knowledge Repository
collection World Bank
language English
topic AGRICULTURE
ANTIPOVERTY PROGRAM
CLINICS
CORRELATIONS
DATA COLLECTION
DEBT
DEVELOPMENT POLICY
DEVELOPMENT PROJECTS
DEVELOPMENT RESEARCH
DWELLING
ECONOMIC RESEARCH
ECONOMIC SURVEYS
ESTIMATORS
EXTERNALITIES
GENDER
GROWTH MODELS
HOUSEHOLD SURVEYS
HOUSEHOLDS
HOUSING
HYGIENE
IMPACT EVALUATION
INCOME
INTEREST RATE
INTERVENTION
INTERVENTIONS
INTERVIEWS
LABOR MOBILITY
LEARNING
LIVING CONDITIONS
LIVING STANDARDS
LONGITUDINAL DATA
MARGINAL COST
OCCUPATION
POVERTY REDUCTION
PRODUCTION COSTS
PRODUCTION INPUTS
PROGRAM EVALUATION
PROGRAMS
QUANTITATIVE DATA
QUESTIONNAIRES
RELIABILITY
RESEARCH METHODS
RESEARCH WORKING PAPERS
SAFETY
SELECTION BIAS
SOCIAL SCIENCE
SURVEY DATA
SURVEY DESIGN
TARGETING
TREATMENT EFFECTS
URBAN AREAS
VILLAGES
WEIGHTING
WELFARE LEVELS
spellingShingle AGRICULTURE
ANTIPOVERTY PROGRAM
CLINICS
CORRELATIONS
DATA COLLECTION
DEBT
DEVELOPMENT POLICY
DEVELOPMENT PROJECTS
DEVELOPMENT RESEARCH
DWELLING
ECONOMIC RESEARCH
ECONOMIC SURVEYS
ESTIMATORS
EXTERNALITIES
GENDER
GROWTH MODELS
HOUSEHOLD SURVEYS
HOUSEHOLDS
HOUSING
HYGIENE
IMPACT EVALUATION
INCOME
INTEREST RATE
INTERVENTION
INTERVENTIONS
INTERVIEWS
LABOR MOBILITY
LEARNING
LIVING CONDITIONS
LIVING STANDARDS
LONGITUDINAL DATA
MARGINAL COST
OCCUPATION
POVERTY REDUCTION
PRODUCTION COSTS
PRODUCTION INPUTS
PROGRAM EVALUATION
PROGRAMS
QUANTITATIVE DATA
QUESTIONNAIRES
RELIABILITY
RESEARCH METHODS
RESEARCH WORKING PAPERS
SAFETY
SELECTION BIAS
SOCIAL SCIENCE
SURVEY DATA
SURVEY DESIGN
TARGETING
TREATMENT EFFECTS
URBAN AREAS
VILLAGES
WEIGHTING
WELFARE LEVELS
Ravallion, Martin
Can We Trust Shoestring Evaluations?
geographic_facet The World Region
The World Region
relation Policy Research working paper ; no. WPS 5983,Paper is funded by the Knowledge for Change Program (KCP)
description Many more impact evaluations could be done, and at lower unit cost, if evaluators could avoid the need for baseline data using objective socio-economic surveys and rely instead on retrospective subjective questions on how outcomes have changed, asked post-intervention. But would the results be reliable? This paper tests a rapid-appraisal, "shoestring," method using subjective recall for welfare changes. The recall data were collected at the end of a full-scale evaluation of a large poor-area development program in China. Qualitative recalls of how living standards have changed are found to provide only weak and biased signals of the changes in consumption as measured from contemporaneous surveys. Importantly, the shoestring method was unable to correct for the selective placement of the program favoring poor villages. The results of this case study are not encouraging for future applications of the shoestring method, although similar tests are needed in other settings.
format Publications & Research :: Policy Research Working Paper
author Ravallion, Martin
author_facet Ravallion, Martin
author_sort Ravallion, Martin
title Can We Trust Shoestring Evaluations?
title_short Can We Trust Shoestring Evaluations?
title_full Can We Trust Shoestring Evaluations?
title_fullStr Can We Trust Shoestring Evaluations?
title_full_unstemmed Can We Trust Shoestring Evaluations?
title_sort can we trust shoestring evaluations?
publishDate 2012
url http://www-wds.worldbank.org/external/default/main?menuPK=64187510&pagePK=64193027&piPK=64187937&theSitePK=523679&menuPK=64187510&searchMenuPK=64187283&siteName=WDS&entityID=000158349_20120305094659
http://hdl.handle.net/10986/3269
_version_ 1764386711033348096

Similar Items