Review of the quality of A.I.D. evaluations FY 1987 and FY 1988

Missions and offices during FY87 and FY88 to examine their overall quality and compliance with the A.I.D. Handbook. Among the report's key findings are (1) over half (53%) of the evaluations were conducted solely by contractors, while A.I.D. and host country personnel participated in 29% and 27% of them, respectively; (2) on average, less than 50% of the recommendations could be acted upon by the A.I.D. Mission or Bureau that sponsored the evaluation, the rest being directed to those actually implementing the project; and (3) regarding special concerns, detailed discussion was given to women in development issues in just 9% of the evaluations, and to environmental issues in 8%. Sustainability issues were discussed in 36%. Other findings are listed below. (1) The majority (89%) of the reports evaluated single projects, 10% more than one project, and 1% nonproject assistance. (2) On certain key measures (e.g., completeness, methodology), evaluations prepared for the Asia/Near East and Latin America/Caribbean Bureaus were rated highest. (3) Interim evaluations accounted for 69% of the total. (4) Regarding their primary focus, 64% of the reports addressed outputs, 28% addressed purpose, and 2% addressed goals. (5) Complete data were available on outputs in 51% of the evaluations, purposes in 19%, and goals in 4%. (6) Possibly because of the short time taken to conduct the evaluations (on average about a month), data collection relied heavily on key informant interviews and to a lesser extent on site visits, with little or no use made of focus group or community interviews, formal or informal surveys, or direct observation. (7) Of reports using analytical methods, 50% analyzed trend data, 23% analyzed cost-effectiveness in detail, and 11% used comparison or control groups. (8) Some 60% contained information on financial monitoring and 79% on program monitoring; of these, 66% of the projects evaluated were given high marks for financial monitoring and 54% for program monitoring. (9) A total of 59% of the reports called for improvements in the project monitoring, evaluation, or management information systems. (10) Included in 66% of the evaluations was the required ES or PES containing a section for listing actions to be taken on the basis of the evaluation. (11) Some 18% of the evaluations noted that recommendations from prior evaluations had not been implemented. (12) Cost data, which were provided for 45% of the evaluations, indicate a mean cost per evaluation of $37,450, with 17% costing less than $10,000 and 20% costing over $60,000. (Author abstract, modified)