Peer Review of the MISTI Survey and Evaluation Methodology

The project aimed to assess the quasi-experimental impact evaluation methods designed to evaluate the impacts of this development programming on Afghans' perceptions of stability. Seven key challenges were identified during the review of MISTI data and existing reports. The first two challenges stem from difficulties faced by the evaluation team in coordinating with the implementing partners, which led to problems in identifying intervention villages and understanding the implementing partners' theories of change. The third and fourth challenges are data-related, namely a lack of comprehensive historical data on development programming and a lack of a credible metric for measuring support for the Taliban as compared to the Afghan government. Challenges 5 and 6 relate to technical implementation, specifically the difficulties faced in identifying appropriate control villages and developing a defensible metric of stability. The final challenge (7) was a design challenge in that the villages benefiting from the stability programming may not be necessarily representative of the overall population, threatening the external validity of the MISTI evaluation for other stability-focused programming in Afghanistan or elsewhere in the world. The MISTI data collection effort likely provides an effective tool for measuring the direct impacts of USAID programming conducted during 2012-2015. This means that MISTI should be able to provide credible estimates of governance programs on governance outcomes, economic programs on economic outcomes, and so on. Although the evaluation approach presented in existing MISTI reports has several empirical limitations, most are surmountable given the range and quality of data collected, making a credible impact evaluation of direct program impacts feasible. However, it is less clear whether MISTI will be able to provide credible estimates of the impact on perceptions of stability. Although these outcomes have been the focus of the MISTI evaluation, they are essentially indirect potential outcomes of the various programs put into place. The evaluation lacks a clearly delineated theory of change for each stabilization program to explain how the programs could influence perceptions of stability. Without such a framework, it would be difficult to interpret a positive result even if that was the result obtained. The primary tools used to measure the perceptions of stability, the stability index and endorsement experiment, are unlikely to be well suited for measuring either stability or relative support for the Taliban. The stability index is poorly defined, combining fairly disparate elements that do not add up to a clear construct for 'stability.' The available data from the endorsement experiment suggests that the approach was unable to capture individuals' support for non-state actors. With regard to stability, there is still scope for analyzing this important outcome, for example, by considering component parts of the index that are well defined. A key lesson learned from this evaluation is the importance of external coordination in planning and carrying out an impact evaluation in a developing, unstable country context like Afghanistan. This would involve close coordination with both the implementing partners and other development, security, and international organizations. This coordination would mean that evaluators and implementers are in communication to ensure, for example, that unambiguous definitions of intervention and control groups are established and that the evaluation design is aligned with the implementing partners' theories of change.