1993 Report of the Auditor General of Canada
Chapter 8—Program Evaluation in the Federal Government—The Case for Program Evaluation
Assistant Auditor General: Paul Ward
Responsible Auditor: Maria Barrados
Program Evaluation in the Federal Government: The Case for Program Evaluation
8.1 The results from program evaluation provide an important measure of the value obtained from government programs and expenditures. The Government of Canada has formally required the evaluation of the effectiveness of its programs and policies for the past 16 years. By 1983 the system was in place. Development of the function over the past ten years has fallen short of expectations set for it, at a time when information on the effectiveness of government activities could prove to be particularly useful.
8.2 The function's potential to provide reliable analytic information has been recognized by many managers in government, and by Cabinet and Parliament. Evaluation of program results not only can provide information to improve management but also should provide essential accountability information within the public service and outside, to Parliament and the public.
8.3 Program evaluation was expected to meet needs for information: to support decision making about resource allocation; to help Canadians determine the value obtained with their tax dollars; and to enable public servants to take responsibility for results rather than process. Such information is urgently needed if decisions to deal with ineffective program expenditures are going to be made.
8.4 Our audit found that the story of program evaluation in the Government of Canada is one of high expectations and great potential that have been only partly fulfilled.
8.5 The system's results are often disappointing. Program evaluations frequently are not timely or relevant. Many large-expenditure programs have not been evaluated under the policy. Despite policy requirements for evaluating all regulatory measures, half have been evaluated, although other reviews have taken place in many departments.
8.6 The development of program evaluation over the last ten years has been primarily in departments' evaluation units. More progress is required in developing program evaluation systems that respond effectively to the interest in effectiveness information shown by Cabinet, Parliament and the public.
8.8 There were high expectations for program evaluation. It was expected to support key functions of government - allocation of resources, efficient and effective operations, and accountability reporting. In addition, program evaluation was expected to serve many different information needs, including those of managers of government programs, managers in central agencies, Parliament and the public.
8.9 Program evaluation is intended to have a strategic focus that periodically assesses the performance of policies and programs, providing results that can be used reliably by decision makers. Program evaluation involves applying systematic research methods drawn from many different disciplines to assess performance, particularly effectiveness. The evaluation results are presented in a report. By answering questions about what has been accomplished, program evaluation can provide an important measure of the value obtained from government programs and expenditures.
8.10 The government established program evaluation to identify whether its initiatives have been successful, whether they should be continued, or whether there are more cost-effective alternatives. The federal government is responsible for large expenditures on programs - with a budget of $161 billion for 1993-94. The expenditures involve over 1500 components, each with its own objectives. In addition, the government has in place a large number of tax expenditures and selective tax measures that also have specific purposes. Program evaluation can be the source of objective information on the complex operations of government, and can provide a measure of the value being obtained.
Audit Scope and Approach8.11 Our audit examined program evaluation in the Government of Canada. The purpose was to assess the government's success in implementing its program evaluation policy and to ascertain its progress since our last government-wide audit of program evaluation in 1983.
8.12 The audit included four areas.
- We looked at effectiveness measurement in 11 major government program activities. Ten of these activities were examined as part of value-for-money audits carried out in the past three years. Exhibit 8.1 lists the program activities, the departments and the years when our value-for-money audits were reported. Four of these activities involved shared responsibilities among departments (paragraphs 8.100 and 8.101). While representing a range of government programs, this is not a statistically drawn sample. In addition to the value-for-money audits, we examined the Department of Finance's revised approach to evaluating tax measures.
- We audited the functioning and the evaluation output of 42 program evaluation units in departments across government that had reported completing at least one program evaluation from 1985-86 to 1991-92. The 42 units include all the larger federal departments except Treasury Board Secretariat, and a number of smaller agencies, departmental corporations and commissions. These organizations were responsible for almost all government program expenditures and major tax expenditures in fiscal year 1991-92.
- We examined the management and operational activities related to program evaluation that the Evaluation and Audit Branch of the Office of the Comptroller General of Canada had carried out up to 31 March 1993.
- We also reviewed reports and analyses describing program evaluation in other government jurisdictions outside Canada, to provide a comparative assessment of how evaluation has evolved in some other advanced industrialized countries.
8.14 Effectiveness measurement in government can be found in various forms, including policy analysis, special studies, organizational reviews, and royal commissions. However, program evaluation is the instrument that the government has put in place and defined in policy that sets out standards and methodology for formal effectiveness measurement. This formalization can ensure that adequate attention is given to systematic measurement, that standards are maintained and that the results are used throughout government and made publicly available. While recognizing that program review can take several forms, government policy identifies program evaluation as the key form of disciplined review of program performance.
8.15 Program evaluation is the focus of the present audit. Evaluation takes a systematic approach to measuring the results of a program, with the objective of producing relevant and reliable information.
8.16 Since the program evaluation policy does not apply to Crown corporations, this audit does not include the evaluation of the effectiveness of these organizations. Chapter 4 describes how Crown corporations report on their own effectiveness.
8.17 The results of this audit examination are reported in three chapters. Chapter 8 assesses the potential that program evaluation holds for examining questions of effectiveness in the federal government. Chapter 9 examines the implementation of program evaluation in departments and the types of results produced. Chapter 10 examines progress in putting in place a system for program evaluation in the federal government, and some lessons from the experience of other governments.
Audit Criteria8.18 The detailed audit criteria are set out in our audit guide on Auditing of Effectiveness Measurement, Reporting and Use . They are summarized in three main criteria contained in the guide:
- Departments and agencies should measure and report effectiveness and these procedures should reflect the state of the art and be cost-justified.
- The results of program effectiveness measurement should be considered for use in making decisions.
- Departments and agencies should have an organizational capability and management practices to measure program effectiveness in an ongoing and/or periodic way.
- Government practice should conform to its policy for program evaluation.
Audit Observations8.20 Our examination of the government's program evaluation function and the implementation of its evaluation policy produced the findings that follow.
Program Evaluation as an Essential Part of Government Management8.21 Program evaluation has the potential to answer three pressing needs for information: first, to help make the decisions about resource allocation that could contribute to controlling the deficit; second, to help Canadians determine the value obtained with their tax dollars; and third, to enable public servants to take responsibility for results rather than process.
Program evaluation findings as a key tool in resource allocation decisions8.22 The need for good effectiveness information is more urgent now than ever. In the 1990s, program evaluation should be seen as crucial to the management of government expenditures, because it can help to arrive at informed decisions aimed at controlling growth of the public debt. In addition, program evaluation provides useful information to Canadians about the effects of programs and what has been achieved with tax dollars at a time when taxes are taking a greater proportion of national income.
8.23 Canada has had large deficits for most of the last 20 years, resulting in a federal net debt of $423 billion as of March 1992. There is widespread concern over this growing debt. Chapter 5 provides the results of a study by this Office on debt and deficit.
8.24 For more than a decade, the government has been concerned about restraining its spending. In successive years it has imposed reductions in the person-year and operating budgets of many departments. However, the reductions have not yet reduced the annual deficit significantly or slowed the growth in debt. At the same time, the government is under pressure to spend more on both domestic and international programs in response to issues like unemployment, environmental protection, increasing numbers of seniors, more expensive health care, and demands like peacekeeping and humanitarian aid. Reducing the deficit in the face of such demands calls for difficult choices about which programs should be funded and to what extent.
8.25 Many factors affect the decisions that ministers must ultimately make about programs. However, program evaluation can aid their decision making by providing objective and reliable information that will enable decision makers to identify programs that:
- work well.
- are no longer needed because of changed conditions.
- do not make the desired difference because they are not reaching their objectives. This may be because the programs do not accomplish what was expected, or because they have negative side effects that counterbalance the desired accomplishments.
- can be replaced by other programs that will achieve the objectives more cost-effectively.
8.27 In the present climate of restraint, the public wants better information on the achievements of government programs. Canadians see their taxes going up, and the proportion spent on actual government services has generally been declining. They are concerned about the way continually high annual deficits and growing debt will affect their standard of living and that of future generations. Program evaluation can inform the public and Parliament about whether programs achieve the expected results, and thus help them judge the value obtained from government activities.
Evaluation findings as a necessary tool for managing results8.28 Like the private sector, the public sector is seeking new ways to improve its operations by placing more emphasis on quality and on meeting the needs of its client groups. Cabinet is changing its structure and organization. Ministers and public service managers are also seeking to renew management as part of the reform initiative. To make organizational change effective, managers must be clear about what they are trying to achieve; learn from past experience; and make improvements.
8.29 An essential part of change is assessing what has been achieved, what has not been achieved, and what other, unanticipated, things have happened. Managers have to be committed to measuring results against a basis for comparison, such as objectives, and be prepared to identify what is not working and take corrective action.
8.30 The information produced and used by managers to bring about effective renewal can provide information for Parliament on the results of government operations. Such information supports Parliament's scrutiny of government operations.
8.31 Program evaluation is not the only form of results-based measurement available to managers. It is often used in conjunction with such measures of performance as resources used, efficiency of operations, and output produced. These measures can be useful but, alone, provide only a partial reading of success. It is evaluation that can bring all the relevant factors together to conclude whether accomplishments were adequate and cost-effective, and what changes are needed for significant improvement.
8.32 Program evaluation is an established tool that allows managers to study the strengths and weaknesses of their programs, how well they meet expectations and their cost-effectiveness. It requires a management culture that values the assessment that program evaluation provides, and that learns from it by making necessary changes and adjustments.
8.33 Government renewal initiatives aim to shift the emphasis in government activities from administration and control systems to achieving results in delivering programs and services. As part of this, authority for the day-to-day running of programs is to be deregulated and decentralized to managers who are then accountable for delivering results that meet established goals.
8.34 Instead of following prescribed procedures, managers will be expected to judge their best course of action, within the parliamentary control framework. This approach to management relies on measuring performance to ensure that the right things are being delivered in the right way. Managers will be expected to make these changes in a time of reduced budgets and increasing demand for services.
8.35 A management culture that evaluates programs and activities with the objective of improving efficiency and effectiveness is needed more than ever, at all levels of government management - in departments that deliver programs and services directly, in central agencies, and in the Cabinet itself. Such evaluations not only would improve management but also could provide essential accountability information within the public service and outside, to Parliament and the public.
Evaluation findings as a necessary tool for accountability8.36 Government ministers are given responsibility for a wide range of diverse activities. They are obliged to report to Parliament on the way they have exercised their stewardship. Chapter 6 of our 1992 Report outlined principles for this stewardship reporting: an important component is accounting for the effectiveness of policies, programs and activities.
8.37 In the private sector, objectives are measured essentially in terms of profit, market share and return on equity. These are mostly reported in financial terms. They constitute the benchmarks against which a business's performance is measured.
8.38 In the public sector, financial accounts are also prepared. However, the nature and complex array of government activities do not lend themselves to the same profit-oriented measures of effectiveness used in the private sector, given that the goals of government programs are frequently stated in non-financial terms. Since effectiveness information is critical for managing these activities, other forms of measurement and reporting are needed.
8.39 Without such information, program managers, taxpayers and Parliament cannot know whether programs are achieving what was intended and what, if anything, has been changed as a result of implementing the program. Program evaluation can provide information and analysis to address these issues.
8.40 Under public service renewal, accountability for the use of public funds should focus on results, not simply on compliance with systems and procedures. Results will need to be measured - and program evaluation will be a necessary measurement tool.
8.41 Program evaluation's potential to provide reliable analytic information has been recognized by many managers in government, and by Cabinet and Parliament. However, our audit found that the story of program evaluation in the Government of Canada is one of high expectations and great potential that have been only partly fulfilled.
The Program Evaluation System in the Government of Canada8.42 Since 1977, government policy has required the establishment of a capacity in federal departments to conduct program evaluation, and has required them to evaluate their programs. Exhibit 8.2 provides an outline of the evolution of the government's evaluation system.
8.43 The federal government's requirements for program evaluation are set out in a central policy by the Treasury Board. The policy follows a two-pronged approach: self-evaluation by the departments responsible for implementing the programs; and a centrally led process that includes a framework for evaluation planning and priorities, technical assistance to departments in implementing studies, monitoring of the quality of evaluations undertaken by departments, and overseeing centrally requested evaluations.
8.44 Program evaluation studies are produced almost exclusively in departments. As described in Chapter 9, these studies are oriented to meeting the interests of departmental management. The central leadership of program evaluation, as described in Chapter 10, needs strengthening.
8.45 Departments expended a total of $28.5 million on the program evaluation function in 1991-92, which included expenditures on 263 person-years. The expenditures also included contract budgets; operational and management expenses; and resources transferred to departmental evaluation units for specified evaluations. In a limited number of instances, program evaluations are also undertaken outside of the formal evaluation function. For example, the Canadian International Development Agency spends, through its geographic branches, approximately $4.0 million annually in conducting evaluations of its projects, as described in Chapter 12.
8.46 The Evaluation and Audit Branch, reporting to the Comptroller General, is responsible for centrally led responsibilities. Approximately one half of the Branch's 32 person-years and $2.9 million budget are committed, as of April 1993, to program evaluation-related duties. This amount is 33 percent lower than in 1991-92.
The policy requirements8.47 In 1991, Treasury Board revised and reissued its policy on program evaluation. The revised policy calls for a more strategic approach, focussing the limited resources for evaluation on priority areas, where the need for information on the results that programs are achieving clearly justifies the cost of evaluating them.
8.48 Deputy heads of departments are accountable for implementing the program evaluation policy in their organizations. The policy requires that all programs or program instruments be considered for evaluation and that evaluations be carried out where they are material and cost-effective. It also requires that a senior manager, independent from line management, be assigned the responsibility for the day-to-day managing of evaluation.
8.49 The purpose of program evaluation, as set out in the policy, is to provide information on three basic evaluation issues, determining whether a program is:
- Relevant: continues to be consistent with department and government-wide priorities and to address realistically an actual need.
- Successful: is effective in meeting its objectives, within budget and without significant unwanted outcomes.
- Cost-effective: is the most appropriate and efficient means for achieving the objectives, relative to alternative design and delivery approaches.
8.51 The program evaluation policy is intended to apply government-wide, reflecting the need for reliable information to determine the "bottom line" results across the government's program activities. For evaluation purposes, a departmental program is any set of departmental activities directed to achieving common programming objectives. Departmental programs may involve one or more program instruments. Exhibit 8.3 shows the list of instruments to which the government's program evaluation requirements apply. The policy also applies to the full range of departments, departmental corporations and other agencies as defined in section 2 of the Financial Administration Act .
8.52 An important but frequently unused feature of the policy, as described in the Program Evaluation Policy Guidelines, is the requirement to develop evaluation frameworks. An evaluation framework sets out the plan to collect performance data as an activity is being implemented. Such data would provide ongoing measures of performance and much of the necessary information for planned evaluations.
8.53 In addition, the government's program evaluation policy specifies how the function is to be organized within the current structure of government - indicating the roles and responsibilities of both individual departments and the Treasury Board, including the Office of the Comptroller General, which has been merged into the Treasury Board Secretariat since July 1993.
8.54 The central agency functions located in the Comptroller General's Evaluation and Audit Branch include monitoring and providing quality assurance on departments' evaluations of their own programs so that users may have confidence in evaluation findings for decision-making purposes. The Comptroller General is also required to report to deputy ministers and the President of the Treasury Board on program evaluation performance across government. In addition, the Evaluation and Audit Branch is responsible for co-ordinating departmental evaluation planning with the needs of Treasury Board and other Cabinet committees for information on program performance, and for overseeing any evaluations undertaken for Cabinet committees.
Examples of the issues examined by evaluation8.55 The results from program evaluations should provide information on whether the programs are achieving their objectives. Some objectives are stated in terms of meeting the needs of clients, for example, in programs established for seniors, farm safety nets, and worker retraining. Other objectives are stated in terms of desired program effects on industries, regions, communities, groups and individuals.
8.56 Examples of the types of information that can be provided by program evaluation are presented in Exhibits 8.4 and 8.5. Exhibit 8.4 illustrates examples of issues that can be addressed in evaluating energy megaprojects. In response to the recommendation in our 1992 Report (Chapter 14, paragraph 14.78) that the Department of Energy, Mines and Resources evaluate the effects of the energy megaprojects, the Department is considering these issues in planning the evaluation.
8.57 Exhibit 8.5 shows the kinds of issues addressed in a program evaluation of the Cape Breton Investment Tax Credit.
8.58 As the examples indicate, there is a wide range of possible effects - both direct and indirect - where information is needed. This entails measuring not only positive effects but also any negative and unintended effects. For example, programs aimed at encouraging additional investment activity (such as the Cape Breton Investment Tax Credit) may end up supporting firms that would have undertaken the activity anyway, without government assistance. Such effects would be unintended but must be taken into account in arriving at an overall net assessment of the program's results.
Stakeholder Interest8.59 The main responsibility for the operations of the Canadian government's program evaluation system is with deputy heads. They are responsible for ensuring that program evaluation is conducted in their departments. Since few program evaluations have been carried out by the central agencies of government, it is essentially the line departments that are concerned with evaluating whether government program expenditures are achieving their objectives. Yet our audit found that there are interests beyond the immediate concerns of line departments that are not consistently being met by program evaluation.
Cabinet interest in program evaluation8.60 Based on information made available to us by the staff of the Comptroller General, Cabinet has maintained an ongoing interest in program evaluation information for more than ten years. On average, requests for evaluation studies were identified about ten to twenty times a year. The Comptroller General's staff did not have available a measure of compliance with these requests.
8.61 Aside from requests for special studies, Cabinet has also turned to program evaluation from time to time with broader requests to assess the effectiveness of programs. For example, in 1985 the Ministerial Task Force on Program Review referred to program evaluation as a source of information to make extensive recommendations on government programs.
8.62 The Task Force examined a wide range of existing government programs and had two major objectives - better service to the public and improved management of government programs. Among the areas where it sought advice were cases of duplication among programs, programs that might be eliminated, and programs whose scope could be reduced. The Task Force indicated that it had made significant attempts to assess programs based on the findings in program evaluation reports from departments. However, as a user of program evaluation information, the Task Force found the material provided to them by government evaluators did not satisfy their needs.
8.63 Cabinet also turned to program evaluation to provide feedback on the results of regulatory programs. The government's comprehensive statement of federal regulatory policy in its 1986 Regulatory Process Action Plan placed priority on reforming ineffective or economically counterproductive regulations. To achieve this goal, all regulatory programs were to be evaluated for efficiency and effectiveness at least once every seven years.
8.64 Based on estimates of the total regulatory activities in government, our audit of the evaluations produced by departmental program evaluation units (Chapter 9) estimated that only about 53 percent of regulatory activities had been evaluated during the period 1985-86 to 1991-92. Government has turned to other review mechanisms to assess the effectiveness of many of its regulatory programs (see Chapters 25, 26, 27) despite the policy obligations placed on program evaluation.
Growth in parliamentary committee interest in program evaluation8.65 In parallel with a growing interest among the Canadian public in the results of government expenditures, parliamentary committees have shown increasing interest in using evaluations in their scrutiny of government budgets and programs.
8.66 In its 1991 Report on the Program Evaluation System in the Government of Canada, the Senate Standing Committee on National Finance said that "a more systematic use of program evaluation findings could provide the National Finance Committee with a powerful approach to its reviews of the Estimates and, in particular, could enable it to make concrete proposals for improvements to government programs." However, the current system did not adequately provide the information the Committee felt was required.
8.67 Since 1986, the Public Accounts Committee (PAC) has also demonstrated a strong interest in evaluation as a way of obtaining specific information on the effectiveness of expenditures, particularly of tax expenditures. In its 26 June 1986 Report to the House, the Committee recommended that the Department of Finance develop a comprehensive program evaluation capability and ensure that all tax expenditures be subject to program evaluation.
8.68 More recently, the PAC stepped up its interest in program evaluation by requesting a range of evaluation studies on issues that it considered of pressing importance. For instance, in 1991 the Committee directed the Department of Finance to evaluate specific tax measures, including the Goods and Services Tax and the new tax treatment of retirement savings. The Department modified its evaluation plans to take into account the Committee's request. In 1993 the Committee asked the Department of Employment and Immigration to submit a detailed calendar of its evaluation work program.
8.69 Finally, in 1993 the Sub-committee on Regulations and Competitiveness of the House of Commons Standing Committee on Finance proposed changes aimed at giving Parliament a greater role in evaluating regulatory measures. Program evaluation under the Treasury Board policy was viewed largely as an aid to top managers in departments rather than an aid to Parliament. The Sub-committee recommended that standing committees in each subject area undertake reviews of regulatory programs. It said that one of the ways these reviews could be triggered was by the completion of a program evaluation. The committee felt that systematic program evaluations are a necessary step in moving toward improved accountability and more rational decision making about regulations.
Central agency interest in program evaluation8.70 The formal responsibilities of the Secretary of the Treasury Board and the Comptroller General are set out in the policy on program evaluation. The approach they have taken to implementing evaluation is to focus on issues at the departmental level and to rely on initiatives in the departments to meet the interests of the various stakeholders in evaluation.
8.71 The Comptroller General approached his responsibilities for program evaluation by developing and communicating an evaluation policy, providing advice on plans and providing guidance on evaluation methods, procedures and standards. The policy and supporting standards and methodologies provide a sound basis for the conduct of program evaluation in government. We used them as criteria in our audit. The concerns that we identify in Chapters 9 and 10 are related, in large part, to how these policies and standards have been implemented.
8.72 The Comptroller General provided professional development for the program evaluation community and actively promoted program evaluation throughout government. However, his staff acted primarily as a facilitator and co-ordinator, relying on persuasion rather than specific direction. We found no examples of program evaluations conducted or led by Treasury Board Secretariat and/or the staff of the Comptroller General since our last audit in 1983.
8.73 We also found that Treasury Board decisions seldom provided departments with the impetus to undertake program evaluations. In 1989-90, for example, Treasury Board issued a total of 2243 decisions, many of which relate to strictly administrative matters. In only 18 cases did it request program evaluations or approve resources in support of evaluation.
8.74 By 1993, departments had complied with ten of the 18 requests. In three of the eight cases where no evaluations had been produced, evaluations were no longer considered by departments to be required. In the remaining five, evaluations either were under way or were to be undertaken later.
Demonstrated Potential for Providing Good Effectiveness Information8.75 The findings of some individual program evaluation studies demonstrate the system's potential to provide sound effectiveness information on both strategic and operational issues. Our audit of the evaluations of selected programs found some well-supported findings contained in evaluation studies, which identified savings, led to informed management decision making and supported accountability. The following paragraphs give some examples of what evaluations have achieved.
8.76 Identified potential cost savings and increased cost-effectiveness. The Department of Transport completed an evaluation of its Canadian Coast Guard Search and Rescue activity in 1993. The evaluation looked at, among other things, the costs and benefits of the more than 70 vessels that the Coast Guard devotes primarily to search and rescue. These range from small inshore rescue boats to large cutters. The evaluation found that the costs of the six largest vessels more than equalled the costs for the entire remaining fleet ($35.6 million compared to $32.2 million). Yet these large vessels accounted for fewer than ten percent of the lives saved by the primary fleet from 1988 to 1990. On 1 March 1993, the Deputy Minister of Transport approved a recommendation that the Coast Guard consider decommissioning these 600-class vessels, or give them assignments apart from search and rescue to contribute to their cost-effectiveness.
8.77 Found initiative not cost-effective. The Department of Finance's evaluation of the Cape Breton Investment Tax Credit estimated that the amount of investment qualifying for the tax credit was $2.1 billion. Of this amount, however, the evaluation assessed that only 19 percent at most (or about $400 million) was incremental - that is, would not have been invested in Cape Breton without the 60 percent tax credit. The results of the study contributed to the decision to discontinue the tax credit.
8.78 Helped improve cost-effectiveness and service to clients. The Department of Employment and Immigration's Job Development Program is designed to assist workers who are experiencing employment difficulties, by providing a combination of formal training and direct job experience. It has a number of options to which applicants may be assigned. The evaluation found that the relatively inexpensive Job Finding Club is a cost-effective initiative. It also found that the program has helped social assistance recipients, particularly males in the Severely Employment-Disadvantaged option, to find jobs. These and other related findings on the relative effectiveness of various options for specific types of clients will enable program officers to make better judgments when assigning clients to projects. Another finding was that projects with private sponsors are more effective than those with public sector sponsors. New program directives will emphasize the use of private sector sponsors where appropriate.
8.79 Identified consequences of change in policy. The Department of Finance's evaluation of tobacco taxes indicated that, in the short run, a ten percent increase in the real price of cigarettes would lead to a nine percent decline in their consumption. However, consumption would later climb back, to some extent. The evaluation also developed a technique to measure unintended side effects of the impact of taxes on smuggling: a ten percent increase in tobacco taxes would lead to at least a ten percent increase in the amount of tobacco exported - an amount apparently destined to be smuggled back into the Canadian market. For instance, the evaluation concluded that the market for smuggled tobacco cost the federal government between $555 million and $695 million in foregone revenues in 1991.
8.80 Assessed savings resulting from a project initiative. An evaluation of the Locomotive Rehabilitation Program (Pakistan Railways), conducted by the Canadian International Development Agency, found that the total savings from the rehabilitation of 42 locomotives of Pakistan Railways was about $21.6 million. This figure was based on a comparison with the alternative of purchasing new locomotives, and took into account the shorter working life of the rehabilitated engines.
8.81 These specific examples demonstrate the potential of evaluation for providing reliable information to serve a range of information needs. Unfortunately, the quality demonstrated in these evaluations is not found consistently across the system.
Inconsistent delivery of essential information8.82 Program evaluation is oriented primarily to meeting the needs of departmental managers. As Chapter 9 shows, much of this evaluation effort is directed to smaller activities and toward operational questions, with the result that reliable evaluation findings on major government programs and activities are not always available in a timely manner.
8.83 In the 11 major programs where we examined effectiveness measurement, we found that evaluation did not consistently produce timely and relevant studies important to support decision making in program development or to provide ministers and Parliament with accountability information on major expenditures.
8.84 For example, in the case of the Agricultural Safety Net Programs (Department of Agriculture), there was a lack of timely evaluation findings available when major program redesign was under way. There were also problems with late timing in the evaluations of the Petroleum Incentives Program and the Canadian Exploration and Development Incentives Program (Department of Energy, Mines and Resources), and the Canada Pension Plan Retirement Benefits (Department of National Health and Welfare). In the case of the Specific Claims Program in the Department of Indian Affairs and Northern Development, at the time of our audit little evaluation had been undertaken although the program had been in operation since the early 1970s. The Department has since indicated that it is proceeding to evaluate this program.
Need to turn to other sources of effectiveness evaluation8.85 Program evaluation is described in policy as "the key form of disciplined review of the performance of programs." Effectiveness studies outside of program evaluation do not fall under the requirements of the established policy and the standards for quality and reliability.
8.86 In the 11 programs we examined, we asked program managers how they determined the program's effectiveness. We found that program evaluation was the most commonly used form of effectiveness study. In five of the 11 programs, however, managers turned to select special studies for input to policy development.
8.87 Special studies were undertaken instead of program evaluation (or instead of updating out-of-date program evaluations) to arrive at important program decisions, in such cases as the redesign of the Agricultural Safety Net Programs by the Department of Agriculture and the phase-out of the Petroleum Incentives Program by the Department of Energy, Mines and Resources. Managers also turned to Service Line Reviews in the Department of Supply and Services; and to the Royal Commission on the Ocean Ranger Marine Disaster, conducted in 1986, which examined search and rescue activities.
8.88 Given the policy requirement to evaluate regulatory programs, we expected major programs in these areas to be evaluated on a priority basis. The government's Regulatory Process Action Plan called for all regulatory programs to be evaluated for efficiency and effectiveness at least once every seven years, a requirement that was partially met (see paragraph 8.64).
8.89 A number of significant areas in regulatory activities, such as medical devices, atomic reactor regulation, atomic fuel facilities and material, and frontier oil and gas lands administration were not evaluated under the policy but were reviewed, in whole or in part, by other mechanisms that do not provide the same assurance on consistency of approach. The government may wish to consider including these reviews as part of the evaluation process. As noted in paragraph 8.69, the House of Commons Sub-committee on Regulations and Competitiveness indicated in 1993 that program evaluations should be playing an important role as part of any wider review process to be undertaken.
Often incomplete coverage of tough evaluation questions8.90 We examined how well evaluations of the 11 government programs addressed each of the issues of cost-effectiveness, program success and program relevance.
8.91 Consistent with findings reported for all program evaluations (See Chapter 9, paragraph 9.63), in only about half of the programs did evaluations deal with program relevance and cost-effectiveness. Even then, the evaluations often did not examine key questions about the programs' relevance. In most of the programs, evaluations examined program success. However, even when evaluation addressed all three issues, the treatment, in our opinion, often was incomplete. Evaluation issues important to providing a balanced assessment of the program were not included. These are often tough questions, but essential to challenging whether a program should continue.
Uneven quality of analysis reduces credibility and usefulness8.92 Evaluators are expected to develop and apply methodologies appropriate to the evaluation issues included in the scope of their studies. This enables them to produce valid and reliable findings to support their conclusions. Inconsistency in the reliability of the data reduces confidence in the usefulness of evaluation measurements. We examined the quality of evaluation in the 11 programs we audited and found it to be uneven. Even within a given study, the reliability of the findings was not consistent. We identified only two programs that had been evaluated consistent with standards of quality.
8.93 We have already noted that we found certain aspects of evaluation measurement that were well done and that provided important information on program results. Such findings provide a demonstrated potential for what the evaluation function can achieve when it produces work of quality.
8.94 However, we also found instances of studies that produced results of questionable reliability. Inconsistent quality reduces users' confidence in even reliable results.
8.95 For example, the evaluation of the 1987 Special Canadian Grains Program, in conjunction with the Western Grain Stabilization Act , by the Department of Agriculture (see our 1991 Report) concluded that the program had proven to be very effective because total payments from both safety net programs to farmers resulted in 75 percent of them having an income greater than they would have had in 1985 (pre-trade war). However, these data also raise the possibility that the cost to the government may well have been significantly higher than necessary. The Department did not attempt to assess this possibility.
8.96 Elsewhere, in the evaluations of Selective Tax Measures undertaken by the Department of Finance, the conclusions of the evaluation study of the Disability Tax Credit concerning the average out-of-pocket expenses covered by the tax credit are, in our opinion, of questionable reliability. This is particularly with respect to measuring the tax credit's effects in compensating severely disabled individuals. A full description of the assessment of the evaluation and the Department's response was provided to the Public Accounts Committee in April 1993.
8.97 Further, in National Health and Welfare's evaluation of the New Horizons program for seniors, we found shortcomings in sampling and in information collection. This program provides many small grants to seniors to alleviate loneliness and promote social activities. Part of the evaluation methodology involved a survey of a sample of project directors rather than typical participants, who were not surveyed. As a result, the methodology used in the program evaluation cannot support the evaluation's conclusion that the initiative had a positive effect on the lives of seniors.
Lack of systematic approach to multidepartment evaluations8.98 In our 1983 government-wide audit of program evaluation, we noted that no procedure was specified for conducting evaluations of programs that involve more than one department. Heads of evaluation, at the time, were uncertain about how to proceed and what their responsibilities were. The result was that multidepartment programs were not subject to the same systematic evaluation scrutiny as programs administered wholly within single departments. We recommended that the government improve procedures to ensure that these multidepartment programs are identified and periodically evaluated, where reasonable and appropriate to do so.
8.99 Our present audit found that, ten years later, the government has not yet implemented a systematic approach to multidepartment evaluations. In four of the 11 programs we examined, the issue of multidepartment evaluations was considered important. In all four cases, partial evaluations had been undertaken; there was still no adequate government procedure in place to provide an overall assessment of the effectiveness of the programs.
8.100 In the case of government measures to support public and private pension provisions, the Department of Finance and the Department of National Health and Welfare undertook separate evaluations. The National Health and Welfare study considered the anti-poverty effectiveness of Old Age Security and the Guaranteed Income Supplement, whereas the Finance study analyzed the distributional effects of the pension system. In the case of the Department of Supply and Services Acquisition program (government procurement), the Department recognizes that it evaluates only a portion of the program's effects in achieving national objectives such as regional development. Treasury Board's Procurement Review Policy co-ordinates evaluation responsibilities for the achievement of national objectives through procurement. In particular, the Department of Industry, Science and Technology has the lead responsibility for evaluating industrial and regional development objectives. No overall evaluation of the program's national objectives has been done to date, although evaluation is planned.
8.101 Elsewhere, in the case of the Special Import Measures Act (SIMA) , National Revenue - Customs and Excise evaluated the operational aspects of the program. The Department of Finance is responsible for the policy of the program. At the time of our audit, due to ongoing trade negotiations, which will have policy implications, the Department of Finance had not yet completed an effectiveness study. The Department indicated that it will intensify its work toward this end as the outcome of the negotiations becomes clear. Similarly, for search and rescue activities the main departments involved, such as National Defence and Transport, individually have undertaken evaluations focussing on their own activities. However, there has been no comprehensive evaluation of the overall effectiveness and cost-effectiveness of the government's search and rescue activities over the past ten years.
8.102 The division of responsibilities among departments for a single program places special requirements on the government to ensure that there is an approach in place to identify multidepartment programs and periodically evaluate them in a comprehensive way. Such a requirement needs to be taken into account as part of an improved approach to planning, as described in Chapter 10.
Lack of priority given to evaluating large-expenditure programs8.103 With increasing concern about the level of debt and deficit, and Parliament's expressed interest in having effectiveness information on government programs to assist it in the scrutiny of budgets, it would be expected that larger program expenditures would be of particular concern for evaluation. This has not been the case.
8.104 We reviewed continuing government programs with expenditures in excess of one billion dollars. These are the kinds of programs that we expected would receive priority attention for evaluation. In 1991-92, there were 16 programs whose expenditures totalled $124.5 billion (see Exhibit 8.6 ). Only two of the programs had been evaluated fully. We considered a program to have sizeable evaluation coverage if a sizeable number of its significant components were assessed or if, in our opinion, some important evaluation questions, but not all, had been addressed. We determined that about one third of the programs had sizeable evaluation coverage. The remainder had received limited or no evaluation.
8.105 For the Maritime Forces program, the Department of National Defence evaluation plans showed limited evaluation for the period under review, while more evaluation was done for Land Forces. However, the Department indicates that, since the Canadian Forces are unified and there is an integrated Canadian Forces/DND organization, the common service activities of the major programs (Maritime, Land and Air) are assessed through all the evaluations undertaken.
8.106 Exhibit 8.6 presents the coverage by program evaluations under the Treasury Board Policy. Not included are other studies or reviews such as the 1986 Commission of Enquiry on Unemployment Insurance.
8.107 Of the six programs that had sizeable evaluation coverage, the evaluations of three, in our view, included a sufficient share of components to be considered sizeable evaluations. The evaluations of the remaining three programs considered to have sizeable evaluation coverage addressed some important program issues but not all. For example, in the Employment activities, evaluations of most of the elements were undertaken but did not provide an assessment of their contributions to the objective of the whole program: to promote the effective and efficient functioning of the labour market.
8.108 Some action is being taken to fill some of these gaps. The Department of Finance has started an evaluation of its debt management program. In October 1992, it completed the first year's report on the debt management evaluation (the first phase of a series of planned reports) and submitted it to the Public Accounts Committee in December 1992. This was in response to our recommendation in 1988 and subsequent requests from the Public Accounts Committee. It is also undertaking a review, jointly with provinces, of programs such as Fiscal Equalization and Stabilization Payments arrangements, as part of the formal review process announced in the 1991 Budget to examine the need for reform of the system of major transfers to the provinces. The Department of Employment and Immigration is undertaking an evaluation of Unemployment Insurance Benefits.
8.109 Further, the Department of National Health and Welfare has indicated that it is now conducting an evaluation of a portion of the Health Insurance Contributions program and the Canada Pension Plan. The Department of Finance has begun an evaluation of the reform of tax assistance for retirement savings. In addition to completing an evaluation of the Bureau of Pension Advocates, the Veterans Affairs Portfolio approved an evaluation assessment (a plan for the evaluation) of the Pension Program in 1991 and is in the process of completing this evaluation report.
8.110 With programs such as Post-secondary Education, Fiscal Equalization and Stabilization, and Health Insurance Contributions, new mechanisms involving close consultation and co-operation with provincial governments may need to be developed to ensure that the gaps in evaluation coverage are filled. The Department of Finance is of the opinion that existing consultative mechanisms are sufficient for evaluation purposes.
Government approach to planning evaluations8.111 We found significant shortcomings in the overall approach to planning (see Chapter 10). The shortcomings result from not clearly identifying priority areas for evaluation, and not following through to ensure that the capacity to respond is in place and that quality studies are produced on a government-wide basis. Further, the overall requirements are expected to be met by individually developed plans, as described in Chapter 9.
8.113 The program evaluation policies, standards and methodologies were designed to provide effectiveness information to departmental program managers as well as to Parliament, ministers and central agency managers. However, the focus has remained almost exclusively on the departmental units. The shortcomings in the system are not in the policy and supporting guidance, but in the approach to implementing them. The will to make the system work more broadly appears to be lacking.
8.114 The commitment to effective program evaluation requires collecting and analyzing information on the performance of programs, and making decisions in light of this information. Program evaluation provides the capacity to bring together information gathered as part of program management with information from other sources, to present a complete picture of the continuing effectiveness of a program.
8.115 Government is faced with difficult questions about the effectiveness of its programs and expenditures. Great urgency is attached to many of these questions. Program evaluation can provide sound information to help find the answers to these questions, but considerable improvement needs to be made to increase the usefulness of evaluation. We have made specific recommendations in Chapter 9 to improve the operation of departmental evaluation units. Other recommendations are directed to the Treasury Board Secretariat and staff of the Comptroller General in Chapter 10.