Facilitating Impact Evaluations: Recommendations from the LAC Reads Evaluations

Mathematica conducted the evaluations, which reveal several factors that facilitate successful randomized impact evaluations. These factors include engaging stakeholders and partners on the ground, being nimble and creative, evaluating validated programs, using national data to improve cost-efficiency and relevance, and using multisite approaches and multiple comparisons to generate generalizable results. Designing impact evaluations before rolling out interventions is crucial to ensure a robust design. Random assignment designs are considered the most rigorous method for estimating causal effects, but they require careful planning and stakeholders' buy-in. The LAC Reads evaluations demonstrate the importance of working closely with implementing partners and stakeholders to determine the evaluation design before project rollout. This allows evaluators and implementers to share information and discuss necessary adjustments to the evaluation design or program implementation before the intervention begins. Being flexible and creative is essential to transform challenges into opportunities. Despite adequate planning, program activities or delivery may change due to unforeseen circumstances. For example, education projects may face teacher strikes, changes in national curriculum, or the rollout of other similar projects implemented in targeted schools. If the context or program changes, so too must evaluation strategies. The LAC Reads evaluations show that being flexible can enable evaluators to identify new learning opportunities and incorporate additional or new contrasts into the study design. Evaluating validated programs can lead to clearer findings. When planning to evaluate an intervention, consider whether it has been adapted to the context, aligns with needs of prospective beneficiaries, and represents a meaningful contrast to prevailing practices or business-as-usual services. Programs that are not validated may run into challenges due to local context that constrain implementation and limit improvements in learning. Piloting interventions can flag risks related to their feasibility, or barriers to implementation and obtaining local stakeholders' buy-in. The LAC Reads evaluations demonstrate that piloting interventions can facilitate smooth rollout and implementation fidelity. Multisite evaluations with multiple comparisons maximize learning opportunities. Multisite evaluation designs are particularly useful to examine the impact of an intervention across a range of contexts, build a knowledge base rapidly, and increase confidence that the findings are broadly applicable. The LAC Reads evaluations show that multisite evaluations can be particularly useful when examining the impact of an intervention across regions and countries, where the counterfactual varies. Collectively, multisite evaluations may be suggestive of the value of key components or approaches. In Honduras, the evaluation of the impacts of providing support to schools using summative and formative instruction began working with the implementing partner and the USAID Mission on the evaluation design nearly a year before the program rolled out. This allowed the evaluation team to present the rationale for the evaluation design to program staff during their initial training and gain their buy-in. As a result, when program staff learned that formative assessments were inadvertently being distributed to schools in the control group, they alerted the evaluation team and the Ministry of Education immediately, which enabled the team to take the assessments out of circulation and ensure the integrity of the evaluation design. The Amazonía Lee evaluation was complicated by the national government's rollout of a similar intervention in the same region. Thanks to the collaborative relationship the local research partner had established with the regional educational authorities, when the partner recognized the threat to the integrity of the evaluation, they were able to facilitate an agreement to implement the government intervention only in the control schools. This step preserved the randomly assigned groups for the Amazonía Lee evaluation. By maintaining the integrity of the random assignment, the research team turned this challenge into an opportunity to test Amazonía Lee's effectiveness against Soporte Pedagógico, the flagship education quality improvement program from Peru's Ministry of Education. The two evaluations of LAC Reads interventions that were previously tested in very similar contexts in the same country led to clearer, more actionable results than did the ones that had been piloted in different countries and required more adaptation. The teacher training model tested in the Amazonía Lee evaluation had been designed and implemented in other regions of Peru by the Universidad Peruana Cayetano Heredia prior to the evaluation. In contrast, although the Leer Juntos, Aprender Juntos intervention had been implemented in other developing countries, it was not piloted in Guatemala and Peru before the implementation for the impact evaluation. Findings indicate that it did not have an impact on learning outcomes in either of these countries.