- D. EVALUATION
- 119. TC evaluation efforts focused on introducing the "Integrated Evaluation Framework", as recommended by the 1995 TACC. This system is based on the logical framework approach for project cycle management commonly used by most technical co-operation agencies.
- 120. Evaluation is fully meaningful and relevant only when a consistent logical framework is followed throughout the project life cycle (from the planning stage through project design and implementation). The integrated evaluation framework was applied in most of the exercises carried out in 1996 and provided a means of drawing global conclusions with regard to the overall performance of a multiplicity of projects by adding up ratings on corresponding issues.
The Integrated Evaluation Framework
- 121. The logical framework for project evaluation provides a tool to analyze the cause-effect relationships that should exist between the main elements of project design: overall goal, project objectives, outputs, inputs and activities. It compares actual achievements with pre-established performance indicators that describe goal, objective and outputs in operationally measurable terms such as target group, quantity and quality, time and place. The results obtained concerning efficiency, effectiveness, impact, relevance and sustainability were qualitatively rated using four categories: High Satisfaction; Acceptable; Low Satisfaction; and Not Achieved.
- EVALUATION FINDINGS
- 122. Effectiveness defines the degree to which a project achieves its principal objectives, while impact relates to the magnitude of the benefits that result. By these standards about 80% of the budget disbursed on the 74 projects reviewed in 1996 rated High or Acceptable Satisfaction.
- 123. Using the methodology of the integrated evaluation framework, the following findings have contributed to understanding the benefit of the logical framework for Agency TC activities (evaluation report numbers are identified within brackets):
- - Two of the four Model Projects evaluated were satisfactorily oriented towards achieving their objectives. These two projects, PER/7/003 - Nuclear Techniques to Improve Child Nutrition (IAEA-PDE-96/02) and SLR/9/005 - Strengthening the Nuclear Safety Regulatory Body (IAEA-FER-96/02), have embodied the Model Project criteria to a maximum extent. They addressed national priorities and fitted within on-going national or/and donor funded programmes; they addressed the root cause of the existing problems and provided key support to address such constraints; and they adhered to high quality design standards.
- - The other two Model Projects evaluated, CPR/5/009 - Industrial Scale Irradiation of Rice and Other Foodstuffs and MLI/5/014 - Field Performance of Selected Mutants of Sorghum and Rice, (IAEA-FER-96/01) posed some concerns with regard to achieving their objectives. In overall terms, such concerns resulted from incorrect assumptions about project management skills on the counterpart side. By addressing these constraints affecting project performance, the risk of not achieving the objectives can be minimized.
- - The technical assistance and co-operation provided by the Agency to the Islamic Republic of Iran have represented a key contribution to the significant socio-economic impact achieved by the related national programmes in areas of basic human needs and radiation protection, as well as in other applications of nuclear science and technology. The availability of teams of qualified staff in most areas of Agency assistance is to be highlighted. The attention given to date to building up nuclear regulatory functions was however not sufficient.
- - The review conducted on 19 projects in selected sectors of agriculture in four countries in Sub-Saharan Africa showed that although the agronomic research supported by these 19 projects had achieved important results in some of the countries, the emphasis that was placed on delivering laboratory infrastructure has not proved sustainable or cost-effective in all cases. However, some cost-effective approaches such as utilizing the analytical services available within the country or elsewhere, had also been embodied in some projects. Additionally, the on-going regional project "Increasing Food Security in Sub-Saharan Africa" provides an opportunity for meaningful utilization of the isotope analysis equipment previously supplied to project participants.
- - The effectiveness of the training course programme would greatly increase if its design were based on a systematic assessment of training demands, with clear definition of training objectives tailored to specific audiences. Assessment of the competence achieved should also be systematically incorporated . Reference is made to the current Policy in Radiation Protection Education and Training and to the Systematic Approach to Training in conjunction with the training activities related to nuclear power, developed in the Departments of Nuclear Safety and Nuclear Energy respectively as highly positive trends. Similar rationales were encouraged in other sectors of Agency's training activities.
- 124. Within the integrated evaluation framework, efficiency was defined as the sum of the following factors: (a) project design quality, measured by the appropriateness and linkage of inputs and the activities that transform them into outputs; (b) project implementation, measured by adherence to schedule; timeliness and fitness-for-purpose of inputs, and budget utilization; and (c) management performance, measured by the ability to monitor progress and take corrective actions. Cost-effectiveness, as a result of combining the issues mentioned above, was assessed by the utilization of outputs, and when meaningful, by comparing the approach adopted with other options.
- 125. Of the total budget disbursed on the projects evaluated in 1996, about 60% went to projects ranking in the "High Satisfaction" category and 24% on those rated "Acceptable Satisfaction". The remaining 16% went to projects rated "Low Satisfaction", mainly because of concerns about design adequacy or cost-effectiveness of some delivered inputs.
- 126. Some of the shortcomings in those projects rated least efficient are assessed as follows:
- - For some projects the equipment component was not balanced with adequate expertise, which in retrospect proved necessary. Additionally, some equipment was not strictly required to achieve the core objectives. As shown by the in-depth evaluations, such equipment appears to have been underutilized. Unavailability of local funds for repair and maintenance of equipment, as well as unaffordable consumables, also contributed to low utilization. In some cases, delivery of hardware to stations or institutions that lacked conditions for either utilizing it or for conducting the intended research resulted both in research failure and underutilization of equipment.
- - Some shortcomings have been noticed in planning and/or programming expert services. The duration of expert services was sometimes not designed to meet the specific needs. Additionally, the output of expert services was sometimes limited to travel reports informing or providing recommendations to the Agency. Fully specifying the services to be provided by experts and documenting the expert advice provided (such as conceptual or detail design or drawings),needs greater attention.
EVALUATION OF COUNTRY PROGRAMME FRAMEWORKS
- 127. In the first quarter of 1996 an evaluation was requested by the senior management of the TC Department to assess the adequacy of the CPF approach and the progress achieved to date. The findings and conclusions of this evaluation have been formulated after extensive discussions with both in-house staff and external evaluation consultants.
- 128. As a result of the evaluation, the CPF guidelines have been revised feeding back the experience to date and providing measurable success criteria to demonstrate the benefits of the CPF process. One success criterion is that the definition and agreement on the project opportunities resulting from the CPF process must result in a specific and detailed plan of actions leading to further project formulations.