This Web page has been archived on the Web.

1993 Report of the Auditor General of Canada

Assistant Auditor General: Paul Ward
Responsible Auditor: Maria Barrados

Main Points

The Program Evaluation System: Making It Work

10.1 Program evaluation is at a crucial stage. Although there are examples of success, it has not developed fully into an effective government-wide system. Strong leadership and greater effort are needed to improve the quality and relevance of evaluations, to meet information requirements outside of departments and to change the management culture toward insisting on the availability and use of broad-based effectiveness information.

10.2 The essential components of a program evaluation system are in place. But improvements need to be made in the coverage, relevance and reliability of program evaluations.

10.3 For the evaluation system to work effectively, key evaluations carried out by evaluation groups in their own departments need formal outside review and assessment, to ensure objectivity.

10.4 The relevance and timeliness of evaluations are enhanced if they are linked into the major decisions and initiatives of government related to resource allocation, program and policy reviews and accountability reporting. These links need to be strengthened.

10.5 Improvement is needed in monitoring and quality assurance by the Comptroller General. Improvement is also needed in the information provided to parliamentarians on the results of evaluations.

10.6 Although the program evaluation system has not responded well to the general requirements, evaluation units have responded to departmental needs and to specific requests by the Public Accounts Committee. Evaluation managers have also taken corrective action in response to specific audit recommendations.

10.7 If the present approach cannot be made to work through reform, Parliament may wish to consider more radical solutions to obtaining timely, relevant and reliable effectiveness information.

Introduction

10.8 Program evaluation is at a crucial stage in its development. Although there are examples of success, in the main it has not lived up to its potential, particularly in meeting expectations beyond those of operational management in departments. In addition, the government is gradually losing the capacity to do evaluation because of reductions in the resources available for it. These reductions come at a time when the effectiveness of current expenditures is becoming a major concern: there is a shift toward a more results-oriented public service and a growing public interest in holding government accountable for its performance.

10.9 In Chapter 9 we identified a number of areas for improvement in the operation of departmental program evaluation units. Action in these areas would significantly improve the functioning of part of the program evaluation system. The changes would not address other aspects of the overall system that can make an important difference to its effectiveness.

10.10 A number of recent examinations of the government program evaluation system have noted the need for corrective action and have identified specific factors that can make an important contribution to its success. Successful change can take place only by addressing how the system can be organized to provide relevant, timely and reliable information for all major stakeholders, as described in Chapter 8.

10.11 The Senate Committee on National Finance held hearings in 1990 with the major participants in the Canadian government system of program evaluation. They included senior managers responsible for evaluation in departments, and representatives of Treasury Board Secretariat, the Office of the Comptroller General and the Office of the Auditor General. In its report analyzing the current evaluation system (tabled in January 1991), the Committee identified a number of considerations that are important to successful evaluation.

10.12 The Committee was concerned about the objectivity of program evaluation, particularly as a source of reliable information for ministers. However, it recognized that there is a trade-off between the objectivity and independence that may be characteristic of evaluators from outside departments and the knowledge and access to information found among department personnel. Further, a balance needs to be struck between the desire to decentralize authority and the need for central control of an activity like program evaluation. The Committee believed that, in a rational world, program evaluation could support assessing government programs by measures of priority, and the lowest-priority items could be cut when the budgetary situation demanded.

10.13 A January 1993 report of the Sub-committee on Regulation and Competitiveness of the House of Commons Standing Committee on Finance argued that program evaluation conducted under the Treasury Board evaluation policy is largely an aid to serve top management in departments. Evaluations were not seen as a means by which "outsiders" (Parliament or citizens) would gain the kind of insight into the performance of regulatory programs that would enable them to hold to account those responsible (notably the ministers).

10.14 The Committee thought that systematic program evaluations were a necessary step toward improved accountability and more rational decision making about regulation, but that external review or audit of evaluation reports was essential. External review could be carried out by publishing program evaluation reports for public scrutiny, by auditing selected evaluation reports, or by general parliamentary review.

10.15 In his study of administrative reform from 1962 to 1991, commissioned by our Office, A. W. Johnson observed that it was generally accepted that departmental accountability to Parliament be based, in part, on the department's own evaluations of program effectiveness, and on the Auditor General's audit of the quality of those evaluations. The success of this approach would depend on the technical capacity to solve difficult analytical problems; the ability to carry out objective, independent evaluations of politically sensitive issues; and the willingness of ministers to have criticisms of their programs made public. He concluded that the evaluation system had not succeeded in meeting these challenges.

10.16 In the larger context of Parliament and government, there are potential constraints for program evaluation. The constraints often are inherent in our system of government and set limits on change. Within these limits, there is stakeholder interest and a number of opportunities for improving the current system.

Audit Scope and Approach

10.17 The overall objectives of our government-wide audit of program evaluation are described in Chapter 8, paragraph 8.11. As part of this government-wide audit, reported in Chapters 8, 9 and 10, we reviewed other studies, particularly recent parliamentary reviews, and experience with program evaluation in other countries. The purpose of the review was to identify, in light of the findings from the audit examination, approaches that have been suggested or used elsewhere to improve the effectiveness of government program evaluation.

10.18 Specifically, we sought to identify practices and procedures that would support improvement in the objectivity, relevance, timeliness, and use of evaluations. However, such practices and procedures must be feasible and practical in an environment of fiscal restraint, and must build on existing experience and infrastructure.

Audit Observations

Strengths to Build on

A system in place
10.19 The essential components of a program evaluation system are in place. Departments have established program evaluation units, as described in Chapter 9, paragraph 9.17. Central responsibility is in place with the Secretary of the Treasury Board and the Comptroller General. The Auditor General provides an external review of the systems and procedures for measuring the effectiveness of programs, where measurement could appropriately and reasonably be done.

The standards to be achieved are in place
10.20 The government has published a set of working standards for evaluation of programs in federal departments and agencies, in support of the program evaluation policy. The policy requires that "relevant, credible and objective" information be available from program evaluation.

10.21 Responsibility for meeting the standards lies with deputy heads and the responsible managers. Among other requirements, the standards call for evaluation of programs by departments, using techniques that produce credible and objective findings based on valid and reliable data collection. For an evaluation to be objective, others must be able to derive the findings using the same data and analysis; to be credible, the results usually should be based on multiple lines of evidence. A further step to protect objectivity is the requirement that responsibility for the "day-to-day" management of evaluation be independent from line management, as discussed in Chapter 9, paragraph 9.21.

Experience of other countries with program evaluation
10.22 We examined the approaches that other countries have taken to evaluation. Although there are differences in the structuring and development of evaluation functions, evaluation is a tool increasingly used throughout the world. Governments are turning to program evaluation as a way to improve financial management and accountability. Legislative audit offices, within their mandates, are using program evaluation methodologies to assess the effectiveness of government expenditures. It is also being used by international financial institutions to assess the effectiveness of their performance.

10.23 Some countries, although interested in the practice of evaluation, are first focussing their efforts on putting in place sound financial accounting procedures and systems, including, in some cases, the ability to assign expenditures to programs. We looked in greater detail at program evaluation in three countries where the practice is established: the United States, Australia, and the United Kingdom. A description of each of the systems is presented in Exhibits 10.1 to 10.3. (see Exhibit 10.1 , 10.2 , 10.3 )

10.24 Our aim was to identify management practices and procedures put in place by these governments to increase the objectivity, relevance and timeliness of program evaluations. We also examined the steps taken to change the culture and approach to evaluation as a means of increasing its acceptance and use. A range of approaches is identified and contrasted with the experience in the Canadian federal government. Our objective was to identify suggestions that could be used to strengthen the Canadian federal system of program evaluation.

10.25 Evaluation practice is the most clearly established in the United States of America, with program evaluation supporting government and legislative decisions. The development of program evaluation in Australia and the United Kingdom is more recent and is linked to government initiatives to improve financial management. Analysts have identified areas for improvement in each system. Despite interest in program evaluation in the United Kingdom, it is the least developed there, as a formal system, of the countries we examined. Our analysis identified as factors of particular importance the use of quality assessment or external review, the linking of program evaluation to significant decision making, and strong leadership.

Strengthening Evaluations through External Review

10.26 The Canadian system, as designed, is oriented to the concerns of departmental management and, as shown in Chapters 8 and 9, responds mostly to management concerns. Yet there is the expectation that the department will provide performance information from its evaluation units to Treasury Board Secretariat, Cabinet committees and Parliament. Can this expectation reasonably be met?

10.27 In any evaluation, many choices need to be made. Some relate to the part of a program or activity to be evaluated and the evaluation questions to be addressed, as discussed in Chapters 8 and 9. There are additional choices about aspects that are important, specific tests, strategies for data collection, methods of analysis and the interpretation and presentation of findings.

10.28 Each choice requires professional judgment and has the potential to compromise objectivity and credibility. We are concerned about the selectivity in evaluations that has given them more focus on operations and smaller activities. This type of evaluation is generally viewed as less useful to interested stakeholders outside departments. The result has often been the perception that program evaluations are "tailored" or that their findings are "dampened down". The dilemma is whether a system of departmental evaluations oriented to meeting departmental requirements can also meet external needs. What type of external evaluative capacity would complement the internal focus of the present system?

Different approaches to external evaluation or review
10.29 There is some capacity for external evaluation or external review in all three countries we looked at. The strongest system for external review is found in the United States. Independence is achieved through a system of evaluation in which the executive branch and the legislative branch have separate agencies for evaluation, each responding to its own needs. The executive branch evaluations are carried out in departments and agencies.

10.30 The General Accounting Office responds to requests by Congress for evaluation, initiates some evaluation on its own, and comments on the quality of evaluations done by the executive branch. The General Accounting Office also prepares reports on the state of the evaluation system. About 80 percent of its work is at the request of Congress.

10.31 Australia, Canada and the United Kingdom do not have external evaluation as is found in the United States. Nor do their legislative audit offices have a broad mandate to conduct program evaluations. These countries rely on their audit offices to provide limited assurance that the evaluation function is working as it should. The extent and type of work done varies among them with their mandates and approaches to value-for-money auditing. The Auditor General in Australia, for example, has issued a number of reports with recommendations for improving the operations of the program evaluation system, and specific reports in a number of portfolio sectors.

10.32 In Australia, as in Canada, program evaluation is expected to serve not only departmental needs but also the needs of central agencies, Cabinet and thereby Parliament. This is achieved by giving responsibility for program evaluation to departments (portfolios in Australia), with some co-ordination and review by a central agency. Central agencies provide some independence from specific departmental concerns.

10.33 This balance of interests is needed if the concerns about maintaining credibility and objectivity are to be understood clearly and supported actively by departmental management. The Australian Audit Office concluded that, if evaluation is to make an effective contribution to improving public sector performance, it is vital that the Department of Finance accept fully an oversight responsibility for monitoring, reporting, and insisting on adherence to the framework.

10.34 Another form of review is public scrutiny of published reports. In the United States almost all evaluations are public documents, often provided directly to Congress in response to its requests. In the United Kingdom the results of evaluations often are internal to the department, but an increasing number are being published, and summaries included in departmental annual reports. In Australia as in Canada, there is a formal requirement for public reporting and accountability. In Australia, departments' annual reports and portfolio program performance statements, which are tabled annually in Parliament, are intended to become major vehicles for providing summary information on program evaluation. In addition there is a requirement that the full report on major portfolio evaluations normally be made public.

Need to improve external review in the Canadian federal government evaluation system
10.35 As we noted in Chapter 9, the Comptroller General was assigned responsibility for assessing evaluation practice and monitoring the performance of departments in meeting the requirements of the policy. He was to provide deputy heads with his assessment of the quality of their evaluation practice and to report on government-wide evaluation performance to the President of the Treasury Board. We found that the Comptroller General has carried out only a part of these responsibilities.

10.36 The Comptroller General did not report formally to deputy heads or to the President of the Treasury Board. Instead, he raised matters of importance with them on an exception basis. As a result, this mechanism does not, in our opinion, provide sufficient assurance of the objectivity and reliability of evaluation through independent review from outside the departments.

10.37 In addition to being responsible for reviewing the quality of individual evaluations, the Comptroller General is responsible for assessing the overall performance of departmental evaluation practice. The assessments were part of the annual Departmental Management Assessment process managed by the Treasury Board Secretariat. On an exception basis, assessments were provided to appropriate managers. Where specific undertakings with management were identified, they were monitored by staff. As a result, assessment coverage was selective.

10.38 For improvements to be made, it is important to identify shortcomings clearly. The assessments we examined were general and not very analytical or critical. For nine of the 11 programs whose effectiveness measurement we examined, we compared our assessment of departmental evaluation performance to the assessments by the Comptroller General. We found that the assessments by the Comptroller General did not clearly identify the important areas for improvement that we identified as the result of our audit work.

10.39 The Public Accounts Committee reviewed the Auditor General's 1978 Report, and recommended to the House that all effectiveness evaluations be tabled in the House of Commons within 60 days of their completion. The government responded that it was committed to making program evaluations publicly available. However, by 1983 only one program evaluation report had been tabled.

10.40 Since that time, the Access to Information Act was proclaimed and Part IIIs of the Estimates were put in place. The Senate Committee on National Finance, as part of its inquiry on program evaluation in 1990, still found that the information provided on evaluations was inadequate.

10.41 In 1991 the Committee found that the list of completed program evaluations given to the Public Accounts Committee was not enough, since it provided only a list of titles and did not provide information about specific evaluations. Similarly, the Committee found that the Estimates did not provide enough information on program evaluation. As a result, it solicited information directly from departments that had completed evaluations. The Comptroller General subsequently published portions of the information collected. There has been little progress in making available to Parliament the results of program evaluation in a way that is useful to parliamentarians and the public.

10.42 In our 1992 Report we recommended changes to the Part IIIs of the Estimates to improve reporting on global stewardship of duties and obligations. Stewardship reporting would include a report on how a department did in meeting its objectives - information available from program evaluation. As we reported in 1992, there is considerable room for more openness by management about its global stewardship role.

10.43 The January 1993 Report of the Sub-committee on Regulation and Competitiveness of the Standing Committee on Finance concluded that it was essential to create procedures for an external review or audit of evaluation reports. One such mechanism would involve the publication and public scrutiny of the government's evaluation reports. Another mechanism would be to have all evaluations tabled in Parliament and referred automatically to the appropriate subject-area standing committees.

10.44 A system of evaluation by departmental evaluation units requires some form of check on the objectivity and credibility of its output. In Canada, we rely on external review by Parliament, on public disclosure, and on review by the external auditor. These mechanisms are in addition to a requirement for commitment among evaluators and evaluation managers to maintain established standards.

10.45 For the system to function effectively, all parts have to be working. In Chapter 9, we identified improvements that deputy heads and evaluation managers need to make. Improvements are also needed in monitoring, quality assurance and reporting by the Comptroller General, and in making information on the results of evaluations available to Parliament.

10.46 The government should ensure that the responsibilities of the Comptroller General for monitoring, quality assurance and reporting are carried out, as set out in the policy on program evaluation.

10.47 The government should ensure that Parliament is provided with the results of program evaluation in a form that Members find useful.

Being Part of the Mainstream

10.48 Evaluation offers the potential to improve management operations, identify areas of savings and ineffective expenditure, and support accountability. To meet this potential, evaluation needs to be an integral part of the management practices and culture of government. Its role should be well defined and clearly integrated into new initiatives and established management systems such as planning. The government approach to planning evaluation needs to establish a means of linking the requirements of individual units with those across government and those of other interested stakeholders outside government.

Proximity to major government initiatives in other countries
10.49 In reviewing the experience of other countries, we examined how evaluation is integrated into government. In the United States the General Accounting Office responds to requests from, and reports to, Congress. Recently enacted requirements for performance reporting include a definition of program evaluation, and a requirement that evaluation plans be included in strategic plans and their results included in performance reports. The Office of Management and Budget uses evaluation results to support decisions to continue, increase, or reduce proposed program funding during budget formulation.

10.50 In the United Kingdom, the reforms of public sector management make managers accountable for the achievement of performance targets. The scope of those performance assessment regimes was recently broadened by the introduction of customer service targets under the Citizens' Charters. Program evaluation, while not neglected, has received less emphasis than the more managerial and routine instruments associated with service planning and annual reporting.

10.51 In Australia, evaluation is an important part of rationalizing program budgeting procedures. The Australian Department of Finance, through its contribution to the evaluation planning process and through its advice to Cabinet during the budgetary process, requests evaluations that will contribute to future budget rounds in a timely manner. The Australian National Audit Office found, in an audit of evaluation's contribution to the budgetary process, a "reasonable level of overall usage at that stage of development of evaluation." Not as many evaluations were available as expected, but there were some notable uses of evaluation in budget deliberations. The Audit Office observed that systematic and vigorous linkages between evaluation and the central budget process have not been widely applied by administrations abroad or in Australian states. It concluded that it is crucial that the approach be carefully balanced, communicated and consistently applied.

10.52 Evaluations of the effectiveness of program expenditures can be a requirement built into all new expenditures. In the United Kingdom, the evaluation policy requires that program evaluation be built into all new policy initiatives and all proposals arising from policy reviews. So far, audit and academic work shows an uneven pattern of achievement.

10.53 In Australia, all proposals for major expansion or new programs are expected to be accompanied by an evaluation plan. In this way, evaluation is to provide information for subsequent scrutiny of budget expenditures. The Australian Audit Office found, in looking at the process over 1990-92, examples of "evaluation's use as a pointer to the need for significant changes, redirection and better targeting in the distribution of resources."

Need to strengthen the link to expenditure decisions in Canada
10.54 In Canada, some effort has been made to link evaluation to the government's planning and budgeting process that has resulted in limited success, as described in Chapters 8 and 9. Program evaluation was to be linked to an operational plan framework prepared by each department. This framework sets out objectives, statements of intended results, performance indicators and planned use of resources for each planning element or part of a program. The findings of evaluations were one piece of information to be included in the assessments of proposed policy and program changes. In practice, we found that these linkages are weak because of the focus of evaluation on small program units and on operational issues.

10.55 Other government initiatives setting out the accountability agreements between deputy heads and the Treasury Board, in the form of memoranda of understanding under Increased Ministerial Authority and Accountability (IMAA) or the Shared Management Agenda, also implied a need for evaluations. However, our findings in Chapter 9 show a decline in the capacity for evaluation.

10.56 In Canada, for most new or substantially modified programs, there has been a requirement in submissions to Treasury Board that an evaluation framework be developed as early as possible in a program's design or implementation. This requirement was spelled out in the 1981 Guide on the Evaluation Function .

10.57 An evaluation framework sets out a plan to collect and interpret baseline data, ongoing program performance indicators and a tentative plan for future evaluation of the program. We found that departments are not consistently meeting the requirements set out twelve years ago for frameworks. For example, we note in Chapter 11 that the Canadian Aboriginal Economic Development Strategy program agreed on a framework only after the start of the initiative.

10.58 Treasury Board should ensure that evaluation frameworks are developed and included as part of submissions to Treasury Board and that procedures to follow up on their implementation are established.

Need to establish government-wide planning of program evaluation
10.59 Evaluation has great potential. It cannot be marginal to the major concerns of government. For evaluation to contribute, it needs to be linked to key government processes. All programs cannot be evaluated simultaneously, and considerable lead times can be involved in conducting evaluations (as much as two years for more complex evaluations). Program evaluation, like other functions of government, is under budgetary pressure; scarce evaluation resources must be used in the most efficient and effective way. Program evaluation is expected to meet the needs of managers in their departments as well as the strategic requirements of government and Parliament, with timely and good-quality program evaluation studies. Evaluation effort needs to be turned to the priority concerns of government.

10.60 As shown in Chapter 9, the operation of evaluation units is oriented primarily to meeting the requirements of departmental managers. To meet government-wide evaluation needs and those of other stakeholders, requirements have to be clearly identified and steps taken to ensure that priority needs for effectiveness information are met. As Chapters 8 and 9 have shown, quality and quantity of evaluations need to be improved, to assess many of the activities of government in order to meet key stakeholders' needs for information.

10.61 We expected the Comptroller General to set out the government's priority needs for program evaluation. Further, we expected that, if individual departmental plans did not address those needs, steps would be taken to have the evaluations done. A well-functioning system of program evaluation would be expected to provide otherwise unavailable information on the results of priority programs and policies.

10.62 We found that the Comptroller General's staff do not have a systematic approach to planning for government-wide or central evaluations. They keep track of central requests for evaluations and, on an ongoing basis, communicate those requests to departments. However, they do not use the information to establish government-wide plans for responding to the needs.

10.63 The Government Program Evaluation Plan is not considered to be an instrument for conveying particular evaluation demands, nor is it a mechanism to sort out specific evaluation priorities. Rather, it is a vehicle for setting out general expectations for the function and assessing the overall capacity of departmental evaluation to respond. As a result, it does not serve as a mechanism for co-ordinating departmental evaluation plans with central needs and with the needs of other departments.

10.64 The Comptroller General has not ensured that government needs for information from program evaluation are addressed by departments in their individual plans. Departments have not consistently submitted plans. Rather, Comptroller General staff have relied on ongoing liaison with departments. In addition, they have not played a leadership role in identifying areas for multidepartment evaluations. Rather, they have relied on the planning processes of individual departments, contributing to poor evaluation coverage of major government-wide programs and initiatives.

10.65 Comptroller General staff have, on occasion, organized briefing meetings between a departmental evaluation unit and the central agencies to discuss the planning of evaluations. They have also participated in 12 interdepartmental evaluations in recent years. However, these initiatives were not part of a systematic planned approach. They have encouraged the inclusion of evaluation requirements in IMAA agreements. However, we found that these practices resulted in uneven success. For example, in six of the seven IMAA agreements that included evaluation plans, the evaluation plans had not been fully implemented.

10.66 An approach by government is needed to ensure that priority areas and government-wide concerns are covered in individual departments' evaluation plans. If they are not, a centrally led approach will be needed to ensure that high-quality, timely evaluations are done.

10.67 The Office of the Secretary of the Treasury Board and Comptroller General should develop the capacity to identify systematically the government priorities for evaluation. It should review individual departments' evaluation plans to identify gaps in the coverage of the identified priorities. At the same time, it should identify gaps in the evaluation coverage of multidepartment programs. Where there are significant gaps it should establish a mechanism to ensure that high-quality evaluations are conducted in a timely manner.

Strengthening the Leadership of the Function

10.68 From 1977 to 1983, progress was made in establishing the infrastructure to carry out evaluations and putting in place supporting policies, methodologies and guidance. However, the function has not developed into an effective government-wide system. Our audit identified a number of areas where the intent of the evaluation policy, directions and guidance were not followed. Strong leadership is needed to improve the operations of the program evaluation system, and to strengthen the commitment to making sound effectiveness information an integral part of government decision making.

The need for a strong focal point
10.69 The program evaluation policy assigned formal responsibility for leadership and support of the evaluation function to the Comptroller General. In our opinion, the Comptroller General placed greater emphasis on supporting the function than on leading it.

10.70 A strong central focus is necessary in a system that is decentralized and oriented toward serving departments, but that is also intended to serve the needs of those outside the departmental structure. Such a focus would set central requirements for evaluation, provide quality assurance and champion the cause of evaluation. Evaluation managers need strong support to meet these requirements both inside and outside of their management structure.

10.71 Evaluation managers do not feel that they have been receiving such support. In 1990 focus group discussions, program evaluation managers were generally supportive of the Office of the Comptroller General. However, they felt that the support they received was often indecisive and could not be counted on to defend the function "in a crunch with senior managers to reduce function staffing and resources." Many expressed the view that the Office of the Comptroller General's strength in "getting the function going" did not carry over into "helping ongoing functions."

10.72 Subsequent focus groups two years later, as part of an internal operational review, were more critical. These focus groups were held with heads of audit and evaluation because, increasingly, the two functions are being managed jointly. Heads of audit and evaluation said they were uncertain whether they could get assistance from the Office of the Comptroller General because of the varying quality and experience of its staff responsible for liaison with practitioners. The merger of liaison responsibilities for audit and evaluation at the Office of the Comptroller General has contributed to this situation. At the time of our audit, only one of the 22 branch officers had experience in both audit and evaluation.

10.73 In our opinion, the Comptroller General did not fully exercise the leadership described in the policy. Such leadership would develop the function by building on its strengths, identifying areas for improvement and initiating corrective action.

10.74 We found little evidence of self-evaluation or of performance measurement systems by evaluation managers, as described in Chapter 9. The staff of the Comptroller General put in place a database containing measures of departmental evaluation performance based on data provided by managers of evaluation units. As part of the audit work reported in Chapter 9, we verified the database measures for 1985 to 1991. We found a number of weaknesses in the data.

10.75 The Secretary of the Treasury Board and the Comptroller General should ensure that information on evaluation activity and performance is valid and reliable.

10.76 If properly implemented, the recommendation in 10.46 to ensure monitoring, quality assurance and reporting, and the recommendation in 10.67 to put in place system-wide evaluation planning, will support the leadership role of the Comptroller General in program evaluation. On 25 June 1993, the Prime Minister announced that the responsibilities and functions of the Office of the Comptroller General, including the Evaluation and Audit Branch, would be integrated into the Treasury Board. As a result, the evaluation function is being re-examined. This presents an opportunity for the Office of the Secretary of the Treasury Board and Comptroller General to re-establish its leadership of program evaluation.

Building demand among stakeholders
10.77 Essential to a process of successful evaluation is an interest in receiving evaluation results and finding the results credible and relevant to the questions being addressed. In addition, information should be presented in a clear, readable style.

10.78 In a parliamentary system, the political will of the majority sets the agenda. It has been argued that this results in an unwillingness on the part of ministers to present negative findings on the effectiveness of their own policies and programs. Yet this inherent constraint may not be as great as some have argued. Parliamentarians, including Cabinet ministers, have demonstrated an interest in obtaining the results of sound evaluations, as shown in Chapter 8.

Responsiveness to specific parliamentary requests
10.79 Departmental managers are constrained by limited resources and, often, by conflicting priorities. The program evaluation system has not responded well to the general requirement for program evaluation to provide information for scrutinizing budget estimates or regulatory programs, as shown in Chapter 8. However, departments have been responsive to providing evaluations when specific requests have been made by Parliament or suggestions made as part of our audits.

10.80 For example, following a series of hearings on tax measure evaluation, the House of Commons Standing Committee on Public Accounts prepared a report to the House listing five tax measures that it wished to see evaluated: the Goods and Services Tax (GST), the new tax treatment of retirement savings, the tax treatment of capital gains, the tax treatment of flow-through shares and the Canadian Exploration Incentives Program.

10.81 In December 1991, the Department of Finance responded by submitting a work plan setting out the tax measures that it would evaluate in the coming years, including possible methods of evaluating the GST and the treatment of retirement savings. In addition, the evaluation of flow-through shares was to be undertaken in 1993. The capital gains tax would begin with an evaluation of the lifetime capital gains exemption in 1993 and 1994. The Department of Finance indicated that the Canadian Exploration Incentives Program was the responsibility of the Department of Energy, Mines and Resources and its evaluation was already under way. In early 1993, the Department of Finance submitted a framework to the Committee for evaluating the different GST components.

Responsiveness to recommendations for improvements in the evaluation of specific programs
10.82 Our audit also found that the government's evaluation system responded to audit recommendations for improvements in the evaluation of specific programs. Some of the steps taken in response to recommendations follow.

10.83 Improve timing and range of questions addressed. In response to concerns raised in our 1991 audit of the Department of Agriculture regarding the evaluation of Farm Safety Net Programs, the Department produced timely evaluation frameworks for the new generation of government measures under the recently enacted Farm Income Protection Act . Although the evaluation framework for the Gross Revenue Insurance Program and the Net Income Stabilization Account requires further work by the Department, the evaluation framework prepared for the Farm Support and Adjustment Measures (FSAM) represents a significant improvement over the Department's earlier evaluation activities in this area. In the framework, the Department is indicating that it intends to address comprehensively the program's effects on the agricultural sector. The Department has indicated that the evaluation is under way.

10.84 Increase emphasis on reliability of findings and indicators. In the Department of Supply and Services, we identified problems with the quality of earlier evaluations of the Department's Acquisition and Major Crown Projects programs (government procurement), particularly with respect to reliability of findings and of evaluation indicators used. The Department has recognized these problems and has started to take corrective action. Its evaluation of its Electronic Catalogue and Order-taking System, and the framework it has developed to evaluate its Open Bidding Service, though partial in nature, represent a significant improvement over earlier activities.

10.85 Start the evaluation of a complex and technically difficult area. The Department of Finance's program activities related to the management of the national debt cover a large component of the federal expenditure. The Auditor General's reports of 1988 and 1991 recommended evaluation of this complex borrowing program. These concerns were supported by the Public Accounts Committee. In response to our recommendations, the Department agreed to carry out the evaluation over three to five years. In October 1992, it produced a first report on its approach to debt management evaluation and submitted it to the Committee in December 1992.

Conclusion

10.86 The need for sound effectiveness information is greater than ever. Yet the system is not meeting this need. There is potential in the current approach to evaluation that has not been exploited. Throughout the three chapters on our audit of program evaluation, we have identified where corrective action should be taken in managing departmental evaluation units and in managing the entire evaluation system.

10.87 Greater effort is needed to make the existing program evaluation policy work. More effort needs to be directed at improving the quality and relevance of evaluation products, at changing the management culture to seek effectiveness information and to use it in making decisions, and at developing a system that meets the needs of stakeholders inside and outside of government, as well as those of departmental managers.

10.88 If the present approach cannot be made to work, Parliament may wish to consider more radical solutions to obtain more timely, relevant and reliable effectiveness information.

10.89 The reorganization of the Office of the Comptroller General and its integration into the Treasury Board Secretariat present the government with the opportunity to renew and revitalize its program evaluation efforts.