Future priorities for USAID in data collection, monitoring, and evaluation : a survey of population, health, and nutrition cooperating agencies (CAs) -- background paper

Agency for International Development (USAID) is developing an umbrella activity to address future needs in data collection, monitoring, and evaluation. The new activity represents the continuation of long-standing work by USAID in these program areas. In January 1996, a survey was conducted to solicit information and advice from organizations and experts in population, health, and nutrition. Results are presented in this paper. Those interviewed were asked to respond to questions about data collection, evaluation methodologies, program monitoring, and evaluation on the following topics: (1) their organization"s activities, whether these activities were included in their project"s work scope, levels of funding, and their project"s comparative advantage in carrying out these activities; (2) their organization"s activities in developing host-country capacity, and needs and priorities for future training; (3) host-country and other donor agency partners in these activities; (4) outstanding needs and priorities as well, as the most appropriate mechanisms for data collection, activities for performance monitoring, and research designs for evaluation; (5) lessons learned in the development of data collection, monitoring, and evaluation methodologies and how these could be applied in future work; (6) challenges to sustainability, the types of activities that are most or least sustainable, and the balance between the development of sustainable capacity in these program areas and the need for timely, high-quality data; and (7) lessons learned and the best mechanisms for increasing use of data in program planning and evaluation. Despite the wide range of projects among the three sectors of population, health, and nutrition, there was surprising consensus about future needs and priorities in data collection, monitoring, and evaluation. Respondents indicated that information and data in population, health, and nutrition are needed for policy change and reform, program planning, management, resource allocation, and program sustainability. Clients for this information include host-country institutions and donor organizations. Most respondents saw the needs of the host countries and the local program managers as primary. All projects in USAID/PHN are involved to some degree in data collection, program monitoring, and evaluation as well as dissemination, although some specialize in different aspects such as data collection or applied research. For most projects, the data collection, monitoring, and evaluation activities were part of the original project design. All projects have developed, tested, and used a variety of tools and methods for these purposes. While there appears to be considerable overlap in what the PHN organizations are doing, there are also numerous opportunities for sharing approaches, methods, tools, and information across sectors and projects. For some projects, their total budgets can be attributed to data collection, monitoring, and evaluation; while for others, a relatively small percentage is attributed to these activities. Overall, considerable resources are being invested in these program areas. All projects train host-country nationals and most of this training occurs in-country in workshops or as on-the-job training. For various reasons, the training activities generally do not result in institutionalization. Respondents identified the critical need to convince donors and senior host-country officials of the importance of good evaluation and monitoring since sufficient resources are generally not allocated to these activities or to the longer-term needs for institution building in these program areas. Information needed for future work in monitoring and evaluation will address multiple functions such as determining effectiveness and cost effectiveness of management and particular interventions, as well as stimulating policy reform. These multiple functions suggest a systems approach rather than an intervention-by-intervention collection of data. There will be a growing need for disaggregated information at the sub-national level for management and planning, and this need implies more emphasis on routine administrative record-keeping systems in host countries. Similarly, USAID"s need to monitor performance suggests more attention to routine monitoring activities. Given the apparent need for routine administrative systems and the current reliance on surveys to supply critical data, attention needs to be given to assessing the cost effectiveness of routine systems versus surveys, as well as the feasibility of supplying good information on a sustainable basis. Donors need to continue to work together to define uniform, small sets of indicators to eliminate duplication of effort, to simplify data needs, and to ensure that host-country programs and technical assistance contractors are not overwhelmed by data demands. Following from the work of the EVALUATION Project, USAID needs a mechanism to bring together those working on indicator development in the PHN sectors to facilitate coordination and dissemination. The continuing need to test indicators for reliability, validity, and feasibility can probably best be carried out by those projects and programs with special technical expertise. For assessing impact of health and family planning interventions, USAID should support the collection of core data (including information on the financing and organization of service delivery systems, the extent of health problems, and the nature of health behaviors). Some longitudinal studies that follow a panel of clients and service sites should also be supported to assess impact. For both types of data collection, the supply and demand information should be combined and analyzed together. In the future, there is less need for the development of new methods and tools for monitoring and evaluation. More attention should be directed to disseminating adequate information about the methods and tools that exist so they can be adapted to different settings and for different uses. Whether USAID"s support of data collection, monitoring, and evaluation can result in sustainable host-country capacity in the future may require modifying the Agency"s approach not only in these specific program areas, but also in the overall PHN sector. The key issues for sustainability include a broad systems approach in the PHN field, meeting the needs of host-country policy makers and program managers, working more closely with other donors, and addressing the costs of work in these program areas. The paper concludes with a short list of recommendations that are, for the most part, embodied in the points cited above. In addition, there are several annexes that include more information from Cooperating Agencies on: methods and tools used in data collection, monitoring, and evaluation (Annex D); lessons in data use including project-specific examples (Annex E); and sustainability and institutionalization (Annex F). (Author abstract)