Three Keys to Getting Evaluation Questions Right

A set of clear, realistic questions linked to the evaluation purpose can result in detailed findings and actionable recommendations that improve program outcomes. Conversely, questions that are unwieldy, unrealistic, or difficult to understand can compromise even the best methodology. Research conducted by The Cloudburst Group at the request of USAID's Bureau for Democracy, Human Rights, and Governance (DRG) identified three key elements where performance evaluation questions commonly fall short: feasibility, scope, and clarity. To improve performance evaluation questions, USAID staff should focus on asking questions that are feasible to answer with available methodologies, while also considering the scope and clarity of the questions. Feasibility is a critical consideration in developing evaluation questions. Evaluation questions that are too complex or require data sources that go beyond the scope of most performance evaluations can lead to findings with low levels of confidence and potentially incorrect conclusions. Traditional performance evaluations typically combine formative and summative evaluations, which can answer questions about processes and outcomes, comparative questions, and questions about program assumptions and theories of change. However, they cannot answer causal questions about program effects, which require an impact evaluation. Questions that are feasible under most performance evaluation methodologies include those about program processes, outcomes for program participants, program assumptions and theories of change, comparative questions, bottlenecks, challenges, and opportunities, and lessons learned. However, questions about outcomes for indirect participants or non-program participants, and comparisons to other similar programs, require data sources that go beyond the scope of most performance evaluations. To ensure that evaluation questions are focused and relevant, USAID staff should identify the decisions the evaluation will inform and write questions aimed at providing this information. This involves prioritizing collecting information about the program components where there is most uncertainty, and limiting the number of questions to 3-5 per evaluation. Additionally, questions that require in-depth comparisons to other donors, organizations, or programs can be valuable, but require more resources to answer. Clarity is also a critical element of successful evaluation questions. If evaluation questions and terms used are not clear, evaluation teams might not provide the answers commissioners expect. To ensure questions are clear and precise, USAID staff should express the question itself as a single clear, precise sentence free of jargon, define all terms in the question, and use compound questions and subquestions sparingly and thoughtfully. By focusing on feasibility, scope, and clarity, USAID staff can develop evaluation questions that are effective in informing decision-making and improving program outcomes. This involves prioritizing questions that are feasible to answer with available methodologies, focusing on specific aspects of the program, and ensuring that questions are clear and precise.