World Health Organization
 
Regional Committee
 
Fifty-fifth Session
11-13 September 2002

SEA/RC55/14

2 August 2002

Review of the Intercountry Programme

Joint Evaluation of the Supplementary
Intercountry Programmes (ICP-II)

REPORT

CONTENTS

Annex - Persons Met by the Joint Evaluation Team
 

1. BACKGROUND

The decision to undertake the "Joint Evaluation of the Supplementary Intercountry Programme (ICP-II)" reflects the conclusions and recommendations of the Regional Committee and high-level regional bodies culminating in:

The recommendation by the High Level Task Force for Intercountry Collaboration (HLTF) to the Regional Director that the supplementary intercountry programme be evaluated and reported to the 55^th session of the Regional Committee.

The evaluation of the supplementary intercountry programme is an integral component of the WHO managerial process. It reflects the Organization-wide evaluation requirements as outlined in the "Programme Management in WHO, Programme Evaluation Broad Guidance for 2002–2003 (WHO/BMR January 2002)," "Framework for programme evaluation (EB 107/INF.DOC./3)" and the "Framework for the Evaluation of Priority Supplementary Intercountry Programmes 2002–2003 (SEA/PDM/HLTF/Meet.3/5)".

The evaluation will strengthen the dialogue among countries and the Regional Office. It is a tool for adapting the WHO technical co-operation to meet the challenges created by the regionalisation of public health problems and assisting Member Countries to address common concerns collectively. The evaluation will serve to provide feedback on the current supplementary intercountry programme and furnish timely input to the development of the 2004–2005 programme.

2. OBJECTIVE OF THE EVALUATION

The objective of the evaluation was to critically evaluate the Supplementary Intercountry Programme for 2000–2001 and 2002–2003 and assess them on the following:

	appropriateness of the mechanisms and approaches;
	adequacy of the resources for the purposes to be achieved including whether attention had been given to the "appropriate definition" of the problem and to designing programmes that address the problem in an optimal manner;
	relevance, efficiency and effectiveness of the programme;
	complementarity of the programme in terms of how it supports, enables and reinforces without duplicating other WHO collaborative programmes and health sector development activities of the Member Countries as well as those of their other development partners, and
	sustainability and replicability.

It must be emphasised that the focus of the evaluation is the supplementary intercountry programme. The 2001 joint evaluation of two supplementary intercountry programmes noted the difficulties in attempting to isolate the ICP-II components from the other factors affecting the WHO collaborative programme. Recognizing these difficulties, the current evaluation did not attempt to evaluate the country’s own programmes in these areas nor the joint country/WHO efforts supported through the WHO country budget.

3. METHODOLOGY

3.1 Selection of Countries Visited and Composition of the Evaluation Team

The basis for selecting countries to participate in the evaluation was the extent to which supplementary intercountry programme activities targeted those countries. On that basis Indonesia, Thailand and Sri Lanka were selected to be visited by the evaluation teams.

As suggested by the High Level Task Force on Intercountry Collaboration, a joint team comprising representatives from (i) Member Countries, either focal points for programmes supported by ICP-II or high-level officials familiar with the programmes, and (ii) the Regional Office (SEARO) undertook the evaluation.

The country representatives nominated by the Member Countries on following the Regional Director’s invitation included:

	Mr Ahmed Salih, Director, International Health Section, Ministry of Health, Maldives
	Dr Jagvir Singh, Assistant Director-General (International Health), of Health Services, Ministry of Health and Family Welfare, India
	Dr Wistianto Wisnu, Chief, Bureau of Planning and Financing, National Agency of Drug and Food Control, Ministry of Health, Indonesia

Members of the joint evaluation team from the Regional Office were:

	Dr Than Sein, Director, Department of Evidence and Information for Policy,
	Dr Sawat Ramaboot, Social Change and Noncommunicable Diseases
	Dr S. Puri, Programme Development Officer

3.2 Process

Building upon the "Framework for the Evaluation of Priority Supplementary Intercountry Programmes 2002–2003 (SEA/PDM/HLTF/Meet.3/5)" an evaluation protocol was developed. The protocol, together with background documentation on the supplementary intercountry programmes for 2000–2001 and 2002-2003 formed the basis for evaluation.

Joint evaluation teams consisting of one country member and a SEARO member visited each of the selected countries. The country visits lasting three to four working days took place during 17-26 June 2002. Each team interviewed officials and staff involved in programme implementation in the Ministry of Health and/or other focal ministries, and the WHO Country Office. The teams reviewed relevant WHO, and country documents.

The Regional Office prepared a preliminary draft report based on the teams’ individual country analysis that was later reviewed by team members from the countries. The final report will be submitted to the Regional Director for his consideration and submission later to the 55^th session of the Regional Committee.
 

4. ANALYSES OF THE PROGRAMME

4.1 Appropriateness of the Mechanisms and Approaches

Overall, the approaches adopted by the supplementary intercountry programmes for the 2000–2001 and 2002–2003 biennia are appropriate. In general, programmes correctly defined the issues and identified suitable outcomes to be achieved by the end of the biennium (expected contributions). The planned outcomes optimally addressed the problems while recognizing the role and limitations of the supplementary intercountry programmes. Moving from outcomes to the means by which they are to be achieved it was observed that the "products" and "activities" are appropriate in general though there are some aspects that need to be addressed.

The countries recognized the use of regional consultations as an effective mechanism for developing regional strategies for meeting country needs, based on "best practices." They also perceived regional consultations as an effective means of ensuring the technologies introduced are suitable for the countries involved and that the programme interventions are socio-culturally compatible; two aspects critical to the sustainability of a programme.

Regional consultations are further seen as facilitating country acceptance of the regional strategies. Examples of this approach include the development of regional strategies for injury prevention; noncommunicable disease surveillance; blood safety; tobacco control and cessation of tobacco use; community-based mental health and prevention; and control of blindness. However, it is too often the case that advance notification of regional consultations is insufficient to permit the selection of appropriate participants in accordance with national procedures. Related issues are the selection of inappropriate participants for reasons unrelated to insufficient notification and participants that fail to communicate the meeting’s discussions and recommendations to programme colleagues.

Despite its effectiveness, the intercountry programme places too much emphasis on regional consultations to the exclusion of other means for achieving programme outcomes. The extensive use of regional consultations and intercountry meetings places inordinate demands on national officials; their frequent attendance at WHO meetings adversely affects programme implementation. To minimize such disruptions countries often select less appropriate individuals to participate in these consultations.

The emphasis on regional consultations to the exclusion of other mechanisms was particularly the case in terms of introducing "new concepts and methods" to address regional priorities and multi-country problems. In such instances, there is a need to supplement regional consultations with direct Regional Office technical support to countries through Regional Advisors or STC/STPs as well as ensuring continued follow-up from the Regional and WHO Country offices. It was noted that such support was too often lacking.

The countries appreciated the involvement of their representatives in the preparation of the intercountry programme through the High Level Task Force for Intercountry Collaboration. However, it was felt that the level of representation was perhaps too high particularly when the Task Force shifted focus from policy, priorities and expected contributions to the more detailed planning of the products and activities, the means through which the expected contributions are achieved.

To address these concerns, officials at the technical and operational levels, particularly those involved with the WHO country programme, should have a role in the Task Force’s development of products and activities. Their more formal involvement in the planning of the intercountry programme beyond their commenting at key decision points in the planning process would help ensure the appropriateness of products and activities; and greater complementarity of country and intercountry work plans while minimizing unnecessary duplication (see 4.4 Complementarity of the programme).

4.2 Adequacy of Resources

Adequacy is viewed in terms of whether the allocated resources are sufficient for achieving the planned outcomes (expected contribution). Implicit is the assumption that attention has been given to an appropriate definition of the problem and to designing programmes that address the problems optimally.

In general, the programmes adequately defined the problems and designed relevant outcomes, products and activities; though there were some limitations as noted in the discussion of the "Appropriateness of the mechanisms and approaches." However, some of the outcomes expected to flow from the programme appeared to be over-ambitious given the level of resources allocated to the supplementary intercountry programme.

The number of programme areas supported through the supplementary intercountry programme increased from ten in 2000–2001 to fourteen in 2002–2003, though the budget allocation remained constant. Without significant inputs from other sources it is questionable whether WHO’s technical support could achieve all of the programme’s planned outcomes.

Within this context, some of the supplementary intercountry programmes were seen as "seed monies" for triggering resources from national budgets and/or other development partners. As exemplified by efforts to strengthen capacity for IMCI implementation, the intercountry programme was able to help mobilize further resources to address critical problems in the area of IMCI. The World Bank and UNICEF have assumed financing for most IMCI activities in Indonesia after the supplementary intercountry programme provided initial programme support.

4.3 Relevance, Efficiency and Effectiveness of the Programme

The countries participating in the evaluation perceive the supplementary intercountry programme as being relevant to the Region as a whole while recognizing that not all areas supported by programmes will have a direct impact on each of the SEAR countries. In the case of Indonesia, the intercountry programme addresses the six priority areas identified in its WHO Country Cooperation Strategy, with ten of the 14 supplementary intercountry programmes directly responding to country needs. For Sri Lanka, all ten of the supplementary intercountry programmes in 2000–2001 were relevant with ten of the 14 relevant in 2002-2003.

The relevance of the programme is attributed in part to specific criteria used in selecting areas to be included in the supplementary intercountry programme. Specifically, the criteria included whether the areas addressed common country priorities identified in the ten SEAR WHO Country Cooperation Strategies and were high priority health problem(s) of regional importance as identified by the Health Ministers, Health Secretaries or the Regional Committee.

An unintentional consequence of employing the above selection criteria was that some major programmes such as tuberculosis, malaria and AIDS, which have WHO technical staff in place at the country level, received supplementary intercountry support. Whereas areas such as mental health, and noncommunicable disease surveillance in which there is less country level capacity did not receive the level of support they warrant.

In assessing efficiency, emphasis was placed on the actual versus planned use of budgets and changes in work plans with particular attention given to reasons for changes. In general, the supplementary intercountry programme was implemented within the given allocations with all funds obligated by 31^st December 2001. There were a number of programme changes in the second year of the biennium when it was found that some activities could not be achieved as planned.

The ability to assess the effectiveness of the 2000–2001 supplementary intercountry programme was severely limited due in part to difficulties in distinguishing its components from the other factors affecting the WHO collaborative programme. Nonetheless, significant achievements were recorded in the areas including but not limited to:

	communicable diseases surveillance and response (case–definition guidelines issued for major communicable diseases; health staff trained in epidemiological surveillance and disease outbreak investigation and control);
	"tobacco free initiative" (a common regional understanding and support on most of the issues of the Framework Convention on Tobacco Control; support for drafting tobacco control legislation and sensitizing multiple sectors for the need for tobacco control);
	IMCI (training a core of IMCI facilitators enabling countries to expand national IMCI programmes to an increasing number of districts);
	"making pregnancy safer" (regional model for MPS developed in 6 high MMR countries in SEAR);
	gender mainstreaming (regional strategy and tools for gender mainstreaming in health developed and disseminated; a technical consultation in GMS in Health which developed plans of action to be taken at regional and country levels to implement the strategy).
	"prevention of blindness" (Vision 2020 strategy).

It is too early to determine the effectiveness of the 2002–2003 supplementary intercountry programme as the evaluation was undertaken seven months into the biennium. The difficulties in distinguishing the effect of the supplementary intercountry programme from other WHO and national inputs will still be an issue in 2002–2003. However, the structure of the 2002–2003 work plans, which includes planned expected contributions, that is, outcomes to be achieved at the end of the biennium, indicators, targets and baseline, will facilitate the assessment process.

4.4 Complementarity of the Programme

Taken as a whole, the supplementary intercountry programme effectively complemented – supported, enabled, facilitated or reinforced without duplicating – the WHO country programmes, those of Member States and their other development partners. Complementarity was strongest in those areas where country programmes are less well-established, for example IMCI, "tobacco free initiative," mental health, noncommunicable diseases, gender mainstreaming and "making pregnancy safer." In these areas, the intercountry programme generally focused on the preliminary steps for "rolling-out" effective programmes emphasizing the more basic programme issues: programme advocacy; development of regional strategies, models and guidelines that would be adapted for country implementation; regional training of core country staff etc.

In areas where programmes are better established and enjoy broad-based support such as communicable diseases, there was a greater tendency for intercountry programme activities to overlap or duplicate those supported by the WHO country or national programmes. The differences in complementarity may be attributable to the greater need of the "better established programmes" for tighter coordination between country and regional levels in the preparation of work plans. These programmes generally receive wide-ranging support from various levels of WHO, their own national governments and their development partners; hence the greater potential for overlap and duplication of effort.

Difficulties have been encountered in assuring that the responsible individuals involved in the development of the supplementary intercountry programme, WHO country and Regional Office programmes reviewed the related work plans at the key decision points in their formulation and provided their counterparts with appropriate and timely input. At the completion of each phase, the Regional Office shared the supplementary intercountry work plans with the WR’s office and national counterparts to ensure the work plans for the intercountry and country work plans were mutually supportive and avoided unnecessary overlap and duplication. Nonetheless, the input received was minimal, owing to:

	an acute misunderstanding regarding the purpose and objectives of the supplementary intercountry programme on the part of national officials and WHO country and Regional Office staff;
	the time demands imposed on the WHO country staff and national counterparts by the development of the WHO country work plans; and
	difficulties in effectively synchronising the parallel development of the supplementary intercountry programme, WHO country and Regional Office work plans.

At times, during implementation, complementarity was eroded because of inadequate coordination in the reprogramming of intercountry and country work plans. Though the Regional Office submitted the final supplementary intercountry work plans to the Member States prior to the Regional Committee, national officials at the operational level did not receive the plans.

4.5 Sustainability and Replicability

Many of the programme areas addressed in both biennia, particularly communicable disease-related programmes, appear to be moving towards sustainability. These programmes tended to be "older" and better established which, over the years, had attracted resources from WHO as well as the countries’ other development partners. Nonetheless, the countries noted that such programmes would continue to benefit from supplementary intercountry support for the development of regional strategies for introduction/adaptation of new technologies and approaches.

Newly introduced programmes, which have only recently become WHO priorities are still mainly dependent upon intercountry inputs; such programmes include: noncommunicable disease control; mental health and substance abuse; injury prevention; healthy environment and lifestyles; blood safety and clinical technologies, etc. Often there is insufficient follow-up and support from the countries to carry on the initiatives with few national counterparts to undertake activities. However, the programmes generally have been designed in a manner that over time would enable the countries to sustain them after major financial, managerial and technical assistance has been terminated. That is, there is an emphasis on adapting technologies to be suitable to the Region and countries, making the programmes socio-culturally compatible, and developing local managerial and technical capacity. Nonetheless, it is too early to determine whether these programmes will prove to be sustainable.

The decision of the 54^th session of the Regional Committee that areas chosen for support through the supplementary intercountry programme should continue for a minimum of two biennia was seen as increasing the probability of programme sustainability and of achieving the desired programme goals. However, such expectations were tempered by the recognition that the limited resources available for the supplementary intercountry programme are spread over too many programme areas.

Regarding the capacity of countries to duplicate the intercountry programmes’ processes and benefits in new locations after their effectiveness has been demonstrated, there are good examples of replicability including IMCI, DOTS and the "tobacco control initiative."