DRG Impact Evaluation Retrospective: Learning from Three Generations of Impact Evaluations

Initiated in response to a 2008 National Academies of Science report, the DRG Center's pilot program of impact evaluations has completed or is close to completing 27 evaluations as of March 2021. The retrospective aims to derive lessons learned and provide evidence-based recommendations for future DRG Center evaluations. Impact evaluations are distinct from traditional performance evaluations and monitoring, as they measure a counterfactual for an intervention and make causal inferences about its impact. In Haiti, an impact evaluation demonstrated that a program was working and should be scaled up, while in the Caribbean, an evaluation found that previous project reviews were incorrect. Impact evaluations frequently provide better measures of outcomes and changes in those outcomes over time, relative to earlier studies that did not employ counterfactual reasoning rigorously. Despite significant achievements, the DRG Center's impact evaluation program encountered several challenges. The objective and intended use of the evaluations were often not well defined, and several evaluations failed to test an intervention with an adequately robust theory of change. Implementation challenges, including inadequate implementing partner buy-in and conflict between implementers and evaluators, also hindered the evaluations. The DRG Center and its partners lacked strategies to move from a conflictive to a cooperative relationship between evaluators and implementing partners. The retrospective highlights the importance of impact evaluations in informing existing projects, future projects, strategies, and general knowledge. However, there is considerable variation in the usefulness of impact evaluations, with several factors contributing to this variation. Survey and case study evidence show that impact evaluation reports are often produced too late to inform decision making, and dissemination of reports is not always widespread. The retrospective offers several key recommendations for the DRG Center's approach to impact evaluations going forward. The Center should build from its previous impact evaluation program, rather than abandon it or shift to an entirely different model. Missions and the DRG Center should make greater use of formal evaluability assessments, with an emphasis on defining the objective of a resulting impact evaluation. Contracting should include a better-defined evaluation objective that clarifies stakeholder roles with specific provisions for implementing partners, evaluators, and academic principal investigators. Instituting these recommendations should encourage a more nimble but far-reaching impact evaluation approach, with a clear objective, carried out by a well-coordinated evaluation team. This would make possible more targeted dissemination and use both during and after a project. Dissemination and use would be further enhanced through increasing the accessibility and actionability of the findings report, involving USAID staff in crafting recommendations for Agency strategy and programming, and creating a central repository for posting research products.