Meta-evaluation of quality and coverage of USAID evaluations 2009-2012

The study builds on USAID's practice of periodically examining evaluation quality to identify opportunities for improvement. It covers USAID evaluations completed between January 2009 and December 2012. During this four-year period, USAID launched an ambitious effort called USAID Forward, which aims to integrate all aspects of the Agency's programming approach, including program and project evaluations, into a modern, evidence-based system for realizing development results. A key element of this initiative is USAID's Evaluation Policy, released in January 2011. The meta-evaluation on which this volume reports systematically examined 340 randomly selected evaluations and gathered qualitative data from USAID staff and evaluators to address three questions: (1) to what degree have quality aspects of USAID's evaluation reports, and underlying practices, changed over time; (2) at this point in time, on which evaluation quality aspects or factors do USAID's evaluation reports excel and where are they falling short; and (3) what can be determined about the overall quality of USAID evaluation reports and where do the greatest opportunities for improvement lie. The overall picture of evaluation quality at USAID from this study is one of improvement over the study period, with strong gains emerging on key factors between 2010 and 2012. The number of evaluations per year increased, and the quality of evaluation reports has improved. While this portrait is largely positive, the study also identified evaluation quality factors, or standards, that USAID evaluation reports do not yet meet. On several core evaluation quality standards -- such as clear distinctions among evaluation findings, conclusions, and recommendations -- performance was found to be below USAID standards. Other significant deficiencies included the small percentage of evaluations that indicated that an evaluation specialist was a member of the evaluation team, which USAID has required for the better part of a decade, and low ratings on the presence of sex-disaggregated data at all results levels -- not simply for input level activities. Low ratings were also found for several evaluation standards introduced in the 2011 Evaluation Policy, but this may simply reflect slow uptake or lack of awareness of standards. (Excerpt, modified)