More Information on the Benefits and Types of Evaluation

What is Evaluation?

Program evaluation, as defined by the Australian Ministry of Finance is the systematic assessment of a program, or part of it,

  • firstly to assist managers and other decision makers to assess the appropriateness, efficiency and effectiveness of a program,
  • secondly to assess the case for establishment of new programs, or extensions to existing programs and
  • thirdly to decide whether the resources for the program should continue at current levels, be increased, reduced or discontinued.

Evaluation is not simply about assessing whether an initiative was a success or failure. Instead, evaluation is about creating the information and data about the initiative's success and why. Evaluation findings can lead to more effective and efficient program delivery.

Government of Newfoundland and Labrador's Policy on Evaluation defines Evaluation as the systematic collection and analysis of information on the performance of a policy, program or initiative to make judgments about relevance, progress or success and cost-effectiveness and / or to inform future programming decisions about design and implementation. Implementation of this Policy over time will build a public sector culture of continuous improvement and learning.

Benefits of Evaluation

Some of the benefits of evaluation include:

  • Enhancing the chance that the initiative's goals and objectives are being achieved
  • Determining value for money (i.e., allocated resources are yielding the greatest benefit for clients and stakeholders)
  • Identifying what components of an initiative work/do not work and why
  • Identifying areas that need improvement in order to provide the best service possible

While most people think about evaluation in the context of assessing programs and policies, building evaluation into operational processes can lead to better procurement decisions. The Government of Newfoundland and Labrador is in the process of reforming its procurement legislation and processes to allow for an assessment of value for money.

Types of Evaluation

There are several different types of evaluations depending on what is being evaluated and the purpose of the evaluation. All types of evaluation can be generally classified into the broad categories of formative and summative evaluations. Formative evaluations are used primarily to provide information for initiative improvement by examining the delivery of the initiative, its implementation, procedures, personnel, etc. Summative evaluations, in contrast, examine the initiative's outcomes and are used to provide information that will assist in making decisions regarding the initiative's adoption, continuation or expansion and can assist in judgments of the initiative's overall merit based on certain criteria. Some more Comprehensive evaluations combine both process and outcome questions.

Evaluability Assessment

Before a proper evaluation can take place it is often necessary to perform an evaluability assessment in order to assess the extent to which the initiative is ready to be evaluated and what type of evaluation would be most appropriate. The assessment aims to ascertain the needs, goals and objectives of the initiative and determine if a formal evaluation is warranted at this point in time or even whether the initiative is ready to be evaluated.

There are often several factors that prevent an initiative from being ready to be evaluated. These can include a failure to agree on goals, objectives and performance criteria, the goals and objectives are found to be unrealistic given the resources available, relevant information/data regarding performance is not available, or there is an unwillingness to change the initiative on the basis of evaluation information. An evaluability assessment will help bring these issues to light. The proper steps can then be taken so that a future evaluation becomes feasible. A more in-depth explanation and guide to completing an evaluability assessment can be found in the resource section at the end of this document.

The Policy on Evaluation requires that departments prepare annual or multi-year Departmental Evaluation Plans to identify priority evaluations and evaluation-related activities. A specific type of evaluability assessment process used to support departmental risk-based assessment of evaluation priorities is contained in the Guide to Developing a Risk-based Departmental Evaluation Plan.

Needs Assessment

A Needs Assessment can be useful for determining whether a problem or need exists within a community, organization or target group and then describing that problem. Recommendations can then be made for ways to reduce that problem. This process typically involves interviews and consultations with stakeholders as well as document reviews and research of relevant information. The Needs Assessment is an ongoing process and is helpful for assessing whether a new policy/program/intervention may be necessary. A more in depth explanation and guide to completing a needs assessment can be found in the resource section at the end of this document.

The existence of reliable data supporting a needs assessment is an important factor to justify major policy or program changes in departmental Cabinet submissions.

Implementation/Process/Formative Evaluation

Every initiative has a strategy or plan that dictates how it is intended to work. The initiative's theory states that if its plan is followed and implemented faithfully, then the intended outcomes will be achieved. A process evaluation can be conducted at any point in the initiative's lifecycle and is used to assess whether and to what degree this plan was followed and the extent to which early outcomes are achieved.

A process / formative evaluation involves collecting relevant information regarding the initiative's implementation on an ongoing basis and identifying any barriers that need to be overcome. Accurate and detailed information about the initiative and its activities and goals are a necessity in order to make the linkages between its various components and the achievement of outcomes. The results of a process evaluation can be used to make improvements to the initiative. If monitoring reports indicate material variances that cannot be explained, a process / formative evaluation may assist to determine the reasons and allow for mitigation strategies to be put in place to improve future performance.

Impact/Outcome/Summative Evaluation

An outcome / summative evaluation is useful for focusing on the results of an initiative, whether they are short-term, intermediate or long-term. This type of evaluation determines what changes, if any, occurred and if they are in line with the initiative's theory. An important aspect of this assessment is determining whether those outcomes occurred due to the initiative itself (impact or attribution), whether some change may have occurred without the program intervention (deadweight) or may have been achieved by other external factors (? Displacement). The findings from this type of evaluation can be used not only for making improvements to the initiative, but for summative decision making as well, that is will the program continue as is, expand, reduce or be eliminated.

The document "Developing an Accountability Framework Resource and Reference Guide" contains a section requiring departments to identify if and when formative or summative evaluations of new programs and major new policies will take place.

Program Review

A Program Review is a systematic overall assessment of an initiative's operations, processes and systems for the purpose of finding efficiencies, cost savings, opportunities for possible realignment with another level of government and/ or other delivery options. It is needed when there is an overriding concern with the initiative's relevance, operations and/or a need to find savings. Such a review process often involves a comprehensive review of an entire department or organization often with a budget reduction target assigned.

Efficiency Assessment (Cost/Benefit Analysis or Cost Effectiveness Analysis)

An efficiency assessment is used to determine the value or benefit of an initiative in relation to its cost. Whether the evaluation focuses on cost-benefit, cost-effectiveness or both depends on the evaluation's scope. A cost-benefit analysis seeks to compare the total costs of implementing an initiative to the total net benefits, while cost-effectiveness analyses assess the value-for-money of an initiative based on the costs required to produce various outcomes. Typically, this type of evaluation is recommended after the initiative has been in place for a period of time so that actual outcome data is available. A more in-depth explanation and guide to completing an efficiency assessment can be found in the resource section at the end of this document.

Common Barriers to Evaluation

It is common for people to have some reservations regarding evaluation of their programs or initiatives since there are several things that can disrupt the evaluation process or impact on the results. Some of the most commonly encountered barriers to an effective evaluation are:

  • Lack of Management Support

    If not initiated or supported by the Minister, Deputy Minister and other senior officials, evaluations will often not receive the support and resources necessary to conduct a proper evaluation. Personnel associated with the initiative are often expected to collect additional data and perform the analyses in addition to their original duties. The result is often a poorly constructed and conducted evaluation with unreliable information on which to base decisions.

  • Lack of Skills and Resources

    Lack of skills and resources in the collection, analysis, and interpretation of data fosters incomplete or inaccurate evaluation results. Many organizations simply do not have the necessary time, systems or personnel with the skills to construct interview or data collection instruments, or analyze and interpret the data. This could result in false assumptions and conclusions from inaccurate, missing or irrelevant data.

  • Lack of Relevant Data

    Lack of relevant data can render the evaluation useless. Unless data collection instruments and methods are carefully planned from the beginning of the evaluation period, missing, inconsistent, and untimely data will result in an incomplete evaluation. Certain evaluation questions will not be answered and the inaccurate data can render the results meaningless.

  • Fear of Consequences

    Fear that the results of the evaluation will suggest or recommend elimination of or significant changes to the initiative will often become a detriment to the process. More realistically, an evaluation, even when negative, will usually lead to the improvement or refinement of the existing program.

Promoting Proper Evaluation

A culture of effective evaluation can be fostered by encouraging the participation of employees and primary stakeholders in the development process, making the scope and purpose of the evaluation clear from the start and providing sufficient support from managers and supervisors. The creation of proper and easy to use evaluation tools will also ensure the process goes much more smoothly.