Avoiding Pitfalls in Performance Evaluations

However, many PEs involve a team of researchers conducting numerous interviews in the field over three to four weeks, which can lead to accuracy, reliability, and usability issues with findings. The Addressing Learning and Evaluation Challenges (ALEC) initiative of USAID's Bureau for Democracy, Human Rights, and Governance examined seven common challenges in PEs and offered guidance to address them. Site selection is a critical aspect of PEs, as it can significantly impact the accuracy and representativeness of findings. ALEC guidance recommends that site selection be based on the evaluation questions and analytical goals of the evaluation. This involves clearly identifying the population of potential localities and considering representative or purposive site selection approaches. Convenience sampling should be avoided, as it can lead to inaccurate or biased results. Respondent selection is another key aspect of PEs, as it can affect the quality and reliability of data. ALEC guidance recommends that study participants be selected using representative or purposive approaches that are best positioned to answer evaluation questions. This involves identifying and mapping types and populations of potential participants and tracking nonresponse and its potential implications on findings. Social desirability bias (SDB) is a common challenge in PEs, as program participants may respond positively to questions about their satisfaction with a program, regardless of their true beliefs. ALEC guidance recommends that evaluators offer confidentiality, use thoughtful question wording, and create a trusting atmosphere in interviews. Potential bias should be flagged in notes and taken seriously in data analysis and report writing, including using more objective sources of data to complement interview data. Qualitative data capture is a critical aspect of PEs, as interview notes are a key data source for findings and conclusions. ALEC guidance recommends that evaluators use a systematic method to capture and organize data, clearly articulated in their work plan or design. Record interviews when appropriate, and have a plan to consistently capture a near-verbatim record of each qualitative event. LP staff should have access to the ET's notes and conduct quality spot checks. Qualitative data analysis is a complex process that requires a systematic, documentable, and somewhat replicable approach to ensure biases and difficulties in collaboration and quality control. ALEC guidance recommends that evaluators use and document a systematic approach for arriving at findings using all data sources. For analysis of qualitative data, this should include structured thematic or content analysis. LP staff should provide oversight ensuring that analysis plans are carried out faithfully. Evidentiary support for findings is essential in PEs, as qualitative evidence can appear anecdotal and unconvincing. ALEC guidance recommends that evaluators cite the source of evidence in a way that provides basic information about the source while maintaining confidentiality. Only quantify qualitative data when using highly structured instruments on a large sample. Findings should be based on multiple data points. Clarity of findings to facilitate use is a critical aspect of PEs, as PE reports are often lengthy and key points risk being buried or never read by intended users. ALEC guidance recommends that evaluators provide a summary of the question response at the outset and use bolded topic sentences throughout the narrative to summarize individual findings. LP staff should review drafts using a checklist to ensure well-written reports. Commissioners play a crucial role in ensuring the success of PEs, and ALEC guidance offers several recommendations for commissioners to avoid common pitfalls. Planning and realistic timelines are essential, as many challenges could be reduced with planning and more realistic timelines. Commissioners should plan to request information about program locations and participants ahead of time, as this will enable ETs to select sites and respondents thoughtfully. SOWs should be used to require and incentivize ALEC guidance, and commissioners should ensure that questions are specific, clear, answerable, and that their responses will be useful and utilized. Commissioners should also be aware of their biases and be open to negative or null findings, as these can provide valuable insights for program improvement.