Evaluation Policy

Prepared by Evaluation and Audit Unit, June 2022

The Department of Foreign Affairs and Trade’s Statement of Strategy highlights that evaluation contributes to an open and accountable Department, and contributes to effective communication with citizens, other stakeholders and beneficiaries of the Department’s work. Evaluation provides independent objective assessment and insight regarding corporate performance and generates evidence to inform decisions on policies, strategies and programmes. This policy aims to strengthen accountability and learning through evaluation in the Department

The development of this policy has been consultative and has taken account of the literature and best practice in evaluation and has benefitted from input and review by external experts.

The purpose of this policy is to define the overall framework for evaluation in the Department, to promote a culture of evaluation across the organisation, and to facilitate conformity of evaluation in the Department with internationally accepted norms and standards. The policy describes what evaluation is, the types of evaluative work we undertake, criteria regarding what and how to evaluate and core evaluation principles, as well as the respective roles and responsibilities of the Evaluation and Audit Unit and other business units within the Department.

The operationalisation of this policy is informed and guided by the:

1. Public Spending Code [1];
2. OECD Development Assistance Committee Evaluation Criteria [2];
3. multi-year Evaluation and Audit Unit Strategy; and
4. Evaluation Operations Manual of the Department of Foreign Affairs.

The principal audience for the policy is senior management and staff, in order to inform best practice in evaluation approaches for officers involved in developing, planning, managing and implementing the Department’s policies, strategies, programmes or other initiatives. It is also intended to be of use to others outside of the Department who wish to inform themselves on our approach to and use of evaluation. The policy recognises our accountability to citizens and communities involved in our programme areas, and that these have a stake in our evaluations, beyond their role as active participants in the evaluation processes.

Evaluation is the systematic assessment of the design, implementation or results of an initiative for the purposes of learning or decision-making. For the Department of Foreign Affairs, evaluation is used to assess its policies, strategies, programmes, projects and other initiatives to generate evidence that provides both accountability for public funding and learning to inform strategic and operational decision making. The principle of independence is essential as it contributes to the credibility and legitimacy of the findings, recommendations and conclusions. Guided by the principles of independence and impartiality; credibility; utility and participation, evaluations serve two main purposes:

Accountability

To show and share the results of an investment with key stakeholders.

Learning

To support decision-making and improvements in policy, strategy, programme, and project implementation by uncovering what works, what does not work, and why.

This Policy identifies three types of evaluation and who has responsibility for them:

Evaluation types and examples

Centralised evaluations

Centralised evaluations are undertaken or commissioned directly by the Evaluation and Audit Unit. They are usually carried out for the Management Board or one of its divisions and can involve a number of business units.

Examples: high-level evaluations or reviews of a policy theme or approach, strategy, major programme or initiative.

Spending reviews

Spending reviews are ordinarily undertaken by the Evaluation and Audit Unit and a business unit or division in the Department. They are carried out in conjunction with the Irish Government Economic and Evaluation Service in the Department of Public Expenditure, Infrastructure, Public Service Reform and Digitalisation and in accordance with the Public Spending Code. [3]

Examples: cost-effectiveness reviews, value for money reviews and spending reviews of an area of expenditure.

Decentralised evaluations

Decentralised evaluations are led by business units or missions. They may be commissioned and managed by business units and missions regarding policies, strategies, including mission strategies, programmes and projects for which they have responsibility and oversight.

Examples: evaluations or reviews of smaller programmes, projects, outcome or output areas, or studies on niche business needs.

Business units and missions are encouraged to use the following set of considerations in coming to a decision to undertake an evaluation:

Considerations on deciding what and when to undertake an evaluation

Use these questions to help decide whether an evaluation should be undertaken, and when it should take place.

1. Importance to senior management [4]

• Is the evaluation likely to inform high-level, strategic decision-making by senior management?
• Can the evaluation make a significant contribution to achieving the Department’s high-level goals?

2. Importance to the business unit or mission

• Is there clarity about how the evaluation will contribute to the work and effectiveness of the business unit or mission?

3. Contribution to the evidence base

• Will the evaluation address an important evidence gap?
• Will the evaluation contribute to knowledge and learning?

4. Financial scale

• Is the area a major focus of expenditure?
• Will the evaluation outcome influence future funding?

5. Inherent risk

• Does the area contain a substantial level of risk?
• Could the evaluation contribute to the management and mitigation of this risk?

6. Innovation

• Is the evaluation innovative in its approach?
• Does it extend the reach of evaluation into new areas of the Department?

7. Feasibility

• Is necessary data available and have baselines been established?
• Are relevant external partners and interested parties available to participate?
• Are the time and resources adequate for a proper consideration of the selection evaluation criteria?

8. Timeliness

• Will the evaluation findings be available in time to inform a key decision?

9. Accountability

• Is there a clear need to demonstrate upward or downward accountability? [5]

10. Complementarity

• Has the business unit or mission established that there is no learning available from other bodies of work carried out recently?

An evaluation will typically be framed around a number of key questions for an intervention area of a business unit or mission of the Department. An intervention can be a policy, strategy, programme, project or other activity. Best practice is that these questions are structured with reference to the six OECD-DAC Evaluation Criteria [6]:

OECD-DAC Evaluation Criteria (OECD 2020)

Relevance

Is the intervention doing the right things?

Coherence

How well does the intervention fit?

Efficiency

How well are resources being used?

Sustainability

Will the benefits last?

Impact

What difference does the intervention make?

Effectiveness

Is the intervention achieving its objectives?

An evaluation may use a selection of these criteria and, as appropriate, additional or alternative criteria may be used or developed. Further details can be found in the Evaluation Operations Manual or from officers in the Evaluation and Audit Unit (EAU).

A communications plan that captures the dissemination of the evaluation report should be developed at the planning stage of the evaluation. The strategy should include: identification of who needs to receive what information; when it is required; how it is to be communicated (format, medium, etc.); and who is responsible for dissemination.

Evaluation and review final reports will normally be published on the Department’s websites. In addition, they can be actively shared, as appropriate, through other media and Departmental networks.

When conducting an evaluation, a number of principles and norms [7] should be observed. These are outlined in the Evaluation Operations Manual and include the OECD principles of independence and impartiality; credibility; utility and participation. Other norms include internationally agreed goals and targets; ethics; transparency; human rights and gender equality and professionalism.

Evaluations will be conducted with the highest ethical standards, integrity and respect for the beliefs, manners and customs of the social and cultural environment. The “do no harm” principle is to apply. Guidance and standards in relation to the Department’s 2021 Safeguarding Policy are to be drawn on.

Appendix 1 provides a summary of the process and timeline (indicative) of a typical EAU managed evaluation.

This evaluation policy takes account of the governance and institutional arrangements in the Department and wider Government.

Role and responsibilities of the Evaluation and Audit Unit

The mission of the Evaluation and Audit Unit is to support, protect and enhance the delivery of the Department’s Statement of Strategy and Foreign Policies by providing objective assessment, assurance, advice and insight regarding corporate performance, governance, risk management and internal control. The EAU reports directly to the Secretary General of the Department. Annually, and on an ongoing basis, the EAU consults with senior management personnel, business units and missions on their evaluation and review needs that are captured in a three-year rolling work plan. The work agreed for each year is captured in the EAU annual workplan that is discussed with the Secretary General and is approved by the Management Board.

The EAU is the constituent part of the Irish Economic and Evaluation Services Network (IGEES) for the Department.

In discharging its role, the EAU undertakes the following evaluation functions:

• Conducting and commissioning independent evaluations and reviews of central-level initiatives and areas of major expenditure;
• Helping ensure the Department meets evaluation obligations under the Public Spending Code;
• Facilitating acceptance and implementation of evaluation findings and recommendations, including maintaining a system to track the implementation of evaluation recommendations and reporting on implementation progress to the Audit Committee twice a year;
• Maximising use of evaluation and review results through communication and knowledge-sharing;
• Developing a culture of evaluation across the Department, including organising learning events, responding to particular requests for training and advisory support and communicating evaluation and review products;
• Providing Quality Assurance in relation to evaluation processes and reports
• Encouraging and providing advisory support to the evaluation and review work of business units and Missions (decentralised evaluation activity);
• Ensuring compliance with best practice in the evaluation, review and advisory roles;
• Developing and maintaining the Evaluation Operations Manual and supporting business units and missions in using same.
• Building and maintaining relationships with evaluation partners and peers;
• Participating in national and international evaluation networks and conferences;
• Reporting on evaluation work through its annual Evaluation and Audit Report to the Management Board and the Audit Committee.

Role and responsibilities of other entities

Heads of business units and missions support evaluation processes and ensure the application of evidence to performance in their policies, strategies, programmes and projects. They may plan and manage selected evaluations of direct relevance to their areas of responsibility (decentralised evaluations). They provide management responses to evaluations and follow-up information on the implementation of evaluation recommendations where appropriate.

The Secretary General and the Management Board in their respective roles provide oversight and support in relation to the implementation of this evaluation policy and resourcing the work of the EAU.

The Audit Committee of the Department provides an independent appraisal of the Department’s evaluation (and audit) arrangements as well as enhancing the effective operation of the evaluation (and audit) function within the Department. It also reviews the annual evaluation work plan and reviews all evaluations undertaken by the EAU.

In relation to the implementation of this policy, the EAU has the responsibility to monitor, review and support its implementation. This will be reflected in the EAU work plan and in its annual report. In addition, the implementation of this Policy will be the subject of an independent review every four years and for approval by the Management Board.

Evaluation process overview

The Evaluation and Audit Unit evaluation process has six stages.

1. Consultation

Indicative timeframe: 3 months

• EAU planning
• Stakeholder discussions
• Concept note development
• Terms of reference development
• Procurement of consultant

2. Planning and design

Indicative timeframe: 2 months

• Document review
• Reference group recruitment and meetings
• Protocol preparation with consultants
• Pilot

3. Implementation

Indicative timeframe: 1.5 months

• Implementation of methodology, for example stakeholder interviews, field trip or site visits, focus groups and surveys
• Emerging themes presentation to stakeholders

4. Analysis and reporting

Indicative timeframe: 1.5 months

• Analysis
• Stakeholder consultation for recommendations
• Draft report
• Feedback from stakeholder on factual inaccuracies
• Final report
• Management response
• Report to management

5. Accountability and learning

Indicative timeframe: 2 months

• EAU lesson learning workshop
• Dissemination
• Business unit learning
• Wider organisational learning

6. Post evaluation

Indicative timeframe: ongoing

• Implementation of recommendations
• Tracking of implemented recommendations

Values

• Trust
• Integrity
• Collaboration
• Objectivity
• Innovation

EAU mission

To support, protect and enhance the delivery of the Department’s mission by providing objective assessment, oversight, assurance, advice and insight regarding corporate performance, governance, risk management and internal control.

Policy guidance

• Global Island
• A Better World
• EAU evaluation policy
• OECD DAC
• UNEG

Quality assurance

• QA processes
• Reference group
• Evaluation manual
• Critical peer
• External experts
• EAU governance structure
• EMG/SMG
• Audit Committee