Assessing the Quality of Education Evaluations

The assessment achieves two related objectives: identifying strengths and weaknesses in the quality of USAID-funded education evaluations and codifying best practices in designing, implementing, and reporting on evaluations for use by USAID Missions, partners, and the education sector at large. The evaluation quality assessment tool developed as part of the quality review process is based on internationally recognized best practices in evaluation, USAID policies, and standards in the evaluation practice in the US. The tool is structured around seven aspects of quality that can be applied to any evaluation, regardless of type. These aspects include conceptual framing, openness and transparency, cultural appropriateness, robustness of methodology, validity, reliability, and cogency. The tool assesses various aspects of evaluation quality, including the research/evaluation questions included in the report, the study hypotheses, the evaluation questions' appropriateness for the intervention's conceptual framework, the report's acknowledgment of existing relevant research, and the report's openness about study limitations. The tool also evaluates the report's cultural appropriateness, including steps taken to ensure that study questions and methodology are informed by local stakeholders, are culturally relevant, and contextually appropriate. The assessment tool is intended for USAID Missions, USAID Washington, and USAID partners involved in evaluating USAID-funded education programming. The tool could also be useful for other sectors of international development in helping improve the strength of evidence generated by evaluations. The Office of Education plans to implement a review of evaluation quality on a periodic basis as part of the larger evaluation synthesis study. The assessment tool is based on internationally recognized best practices in evaluation and USAID policies. The tool's framework is based on Assessing the Strength of Evidence in the Education Sector (DFID, 2015; produced by Building Evidence in Education Donor Working Group). The tool's development involved revisions based on comments from reviewers, and it is intended to be used by various stakeholders, including USAID Missions, partners, and the sector at large. The assessment tool evaluates various aspects of evaluation quality, including conceptual framing, openness and transparency, cultural appropriateness, robustness of methodology, validity, reliability, and cogency. The tool assesses the report's research/evaluation questions, study hypotheses, evaluation questions' appropriateness, report's acknowledgment of existing research, and report's openness about study limitations. The tool also evaluates the report's cultural appropriateness, including steps taken to ensure that study questions and methodology are informed by local stakeholders, are culturally relevant, and contextually appropriate. The assessment tool is a critical component of the quality assessment of USAID-funded evaluations in the education sector. The tool provides a framework for evaluating the quality of evaluations and identifying areas for improvement. The tool's development and implementation are intended to support the improvement of evaluation quality and the generation of robust evidence in the education sector.