Methodological Insights for Impact Evaluations

Mathematica conducted randomized controlled trials to evaluate the impact of these interventions. The evaluations generated methodological insights that can inform researchers, implementers, and donors on how to improve evidence-based decision-making. Baseline data are crucial in impact evaluations, as they can be used to adjust impact estimates and increase statistical power. Researchers calculate the minimum detectable effect (MDE), which is the smallest true effect that a study can detect. The MDE is influenced by the sample size, and other factors such as baseline data can also increase statistical power. For example, in one of the Amazonía Lee sites, the MDE decreased by 44% when baseline test scores were included, from 9 correct words read per minute to 5 words per minute. The LAC Reads evaluations used student-level pre-intervention data on key reading skills to adjust impact estimates and increase precision. The studies gained additional precision by adjusting for school infrastructure characteristics, such as access to potable water, working restroom facilities, library, and internet connectivity, as well as household-level data, such as number of rooms in the house, highest grade attained by mother, access to key services, and assets. Adjusting for baseline student reading test scores provided the largest gains in statistical power, with MDE reductions ranging from 20% to 44%. Adjusting impacts for school-level characteristics proved to be a low-cost approach to increase statistical power for several of the LAC Reads evaluations. For example, in Guatemala, the MDE decreased by 12% when school-level infrastructure characteristics were included in the regression, from 7 correct words read per minute to 6 words per minute. Collecting these data added little cost, as each LAC Reads evaluation included visits to schools to obtain student test data or interview teachers. However, adding household data provided minimal gains in power after including baseline test scores and did not provide sufficient power gains to be cost-effective. In the case of the Leer Juntos, Aprender Juntos evaluation, household data provided minimal gains in statistical power after baseline test scores were included. Collecting data on family and household characteristics is often costly and may not be worth the cost, particularly if there is not much variation in the socio-demographic characteristics of the households in the evaluation sample. Individual-level random assignment increases statistical power, but group randomization can reduce contamination and facilitate program participation. Individual-level randomization may appeal to program implementers, as it enables them to serve as many children as possible while also generating information about impact. However, individual-level random assignment can be challenging for interventions rolled out at the school level, and it may not be advisable in other settings where it is difficult to prevent a subset of potential beneficiaries from accessing the intervention. The counterfactual matters in impact evaluations, as the benefit to the treatment group is compared to a contrast or counterfactual. What the intervention is compared to matters, and the LAC Reads evaluations estimated the impact of a program compared to prevailing practice or to the existing education system in certain situations. The evaluations also compared one program to another or to no program at all, as in the case of the Espacios para Crecer program in Nicaragua. For example, the evaluations of Leer Juntos, Aprender Juntos in Peru and Guatemala showed that the teacher training and coaching component of the program had positive effects in Peru but almost no effects in Guatemala.