This Web page has been archived on the Web.

1993 Report of the Auditor General of Canada

Assistant Auditor General: Paul Ward
Responsible Auditor: Maria Barrados

Main Points

Program Evaluation in Departments: The Operation of Program Evaluation Units

9.1 In 1991-92, only $28.5 million was spent on program evaluation across all federal government departments. Yet program evaluation is charged with considering for evaluation all the programs and activities of government. It evaluated about one quarter of government expenditures from 1985-86 to 1991-92, far short of original expectations that all programs would be evaluated over five years. Regulatory programs were all to be evaluated over seven years, but about 53 percent actually were. The program evaluation capability established by the federal government in the early 1980s is still in place, but its strength is declining. Fewer resources tend to produce fewer studies.

9.2 Program evaluation is located in departments and is the responsibility of the deputy heads. Priority has been given to meeting the needs of departmental managers. As a result, evaluations examine smaller program units or lower-budget activities and focus on operational performance. They are less likely to challenge the existence of a program or to evaluate its cost-effectiveness.

9.3 The government's use of program evaluation studies often is difficult to demonstrate. Although the savings from program evaluation have been small overall, a few individual studies have led to significant savings. However, deputy heads seldom have used evaluation to carry out studies that could lead to significant savings.

9.4 Management of individual units in a number of departments should be improved and be better integrated into the management of the department. Some units demonstrate good practices. These units are integrated into the management of their departments and support the operational interests of deputy heads.

Introduction

9.5 The government system of program evaluation relies almost exclusively on the units in departments to provide evaluation studies. The Comptroller General and the Secretary of the Treasury Board set out policy and methodology to guide and support the work of these units.

9.6 The policy requires departments to carry out program evaluations. The deputy heads are accountable for the planning, conduct, and reporting of program evaluation studies. In placing the responsibility with deputy heads, Treasury Board expected evaluation results to be produced for, and used by, senior managers.

9.7 The 1991 Treasury Board policy on program evaluation requires that departments establish an infrastructure headed by a senior manager who is independent from program operations and responsible for the conduct of evaluations for the department. Conducting high-quality evaluation studies requires an adequate budget and appropriate professional evaluation resources. Decisions on the strength of the evaluation function, the programs to be evaluated and the evaluation questions to be addressed are made in departments, under the direction of the deputy head.

9.8 In addition to developing the evaluation policy, the Comptroller General has provided Working Standards for the Evaluation of Programs in Federal Departments and Agencies , and other guidance on conducting evaluations. The Comptroller General publishes newsletters and occasional papers discussing innovations in evaluation methods and practice.

9.9 Our Office carried out government-wide examinations of program evaluation in 1978 and 1983. The 1978 study found few successful attempts at program evaluation. By 1983, we found that departments had made considerable progress in establishing an infrastructure and building the capacity to carry out evaluations. Deputy ministers found evaluations useful but felt it was too early to assess their cost-effectiveness.

9.10 Our 1983 audit identified a number of areas for improvement. We observed that the government initially encouraged the use of study findings in the ongoing management of programs, in order to develop experience and enhance the credibility of program evaluation with management. In response to the audit, the government confirmed its aim to ensure the production of relevant and reliable information on the performance and relative cost-effectiveness of programs. Its main approach would be to integrate program evaluation into the improving management practices in each department.

Audit Scope and Objectives

9.11 The overall objectives of our government-wide audit of program evaluation are described in Chapter 8, paragraph 8.11. This chapter examines the operation of program evaluation in departments. Our purpose was to assess the implementation, products and use of program evaluation in departments and the progress made since 1983. Specifically, we examined:

  • evaluation infrastructure - the resources dedicated to evaluation and the way it is organized;
  • the products of evaluation units - the scope and quality of evaluation studies;
  • the uses made of evaluation studies, and the associated savings; and
  • the management of program evaluation units.
9.12 The Comptroller General's staff monitors the performance of program evaluation units and maintains a database of performance information, including information on program evaluation resources, products, use and contribution to cost savings. In January 1992 they surveyed evaluation units to update their performance information. After consultation with our Office, they designed the survey to meet their own data requirements and to support the information needs of our audit.

9.13 We contacted each program evaluation unit included in the survey, to confirm and supplement the information we obtained from the Office of the Comptroller General. Based on this work, we constructed an expanded database that provides the basis for our audit observations and conclusions. Included were 42 evaluation units that are responsible for evaluating most of the program expenditures and tax measures of government.

9.14 In conducting our examination, we:

  • interviewed the heads of evaluation units about the structure, role, planning and management of program evaluation;
  • obtained documentation to confirm data from the Office of the Comptroller General on program evaluation resources, the uses made of evaluations and the cost savings resulting from program evaluation;
  • reviewed program evaluation reports completed between 1 April 1985 and 31 March 1992 to assess the scope of the studies and the format of the reports; and
  • obtained additional information on the activity structure and expenditures in the departments concerned.
9.15 Statistical analyses were used to identify patterns in the data as a basis for our conclusions. These analyses were conducted on the data for all departments combined. The conclusions and recommendations therefore apply to program evaluation units overall. Estimates of savings and use of evaluations reflect our judgment based on material provided to us during the audit. In one department we did not obtain final concurrence on the estimate of savings to be entered into the aggregate analysis. However, this would not have changed our conclusions.

9.16 Our 1983 government-wide audit of program evaluation examined the organization and management of program evaluation units and the planning, conduct, reporting and use of program evaluation studies in 19 departments. We compared our 1983 findings with information about the same 19 departments obtained during the present audit, to assess how the function has changed over time.

Audit Observations

Evaluation Infrastructure

9.17 There were 42 active program evaluation units that made up the system of federal program evaluation and that were monitored under the Treasury Board policy. These units spent $28.5 million on program evaluation in 1991-92 and included all major departments, with the exception of Treasury Board Secretariat. There was little change in the number of evaluation units between 1983, when 41 departments and agencies had established a basic structure for program evaluation, and 1991-92.

9.18 Most evaluation managers report to directors general or assistant deputy ministers who are also responsible for internal audit or some other management function, such as corporate planning.

9.19 Program evaluation and internal audit increasingly are managed jointly. A common arrangement is to have a director general managing an audit and evaluation division with separate audit and evaluation units, as is the case in the Department of Agriculture (see Exhibit 9.1 ). Approximately half of the program evaluation units have adopted some form of this arrangement since 1989-90.

9.20 Even though program evaluation and internal audit often are jointly managed, their work remains distinct. The two groups operate under independent standards and policies of Treasury Board. (Additional detail on internal audit can be found in Chapter 7.) Only a few studies are carried out jointly by internal audit and program evaluation units.

9.21 Consistent with Treasury Board policy, program evaluation managers are independent of program responsibility centres and report to the deputy head, often through senior managers with other administrative responsibilities.

9.22 The Department of Finance has implemented an organizational approach where the management responsibility for evaluation is independent of policy development but evaluators can be drawn from the policy areas. The Department undertook this reorganization in an effort to deploy its resources in the most efficient and effective way.

9.23 The Department of Finance was the only major department in 1983 without an evaluation unit. The Tax Measures Evaluation Division was established in 1987, in response to concerns of the House of Commons Standing Committee on Public Accounts and this Office about the need for effectiveness information on tax expenditures. When the Department of Finance subsequently announced its new approach to evaluation in 1991, there was concern that it would not maintain its capacity to evaluate tax measures.

9.24 However, we found that the Department has maintained a capability to carry out evaluations, although it dedicates fewer resources to them and places greater reliance on contracting. The Department has confirmed its commitment to evaluate tax measures in a systematic way. Because the new organizational arrangement has been in place for only a short time, our audit conclusions on the efficacy of the new approach are preliminary. The evaluation function under the new approach is being closely monitored by the Public Accounts Committee and this Office.

The resources allocated to program evaluation are declining
9.25 The ability to carry out program evaluations depends on the availability of professional evaluators and staff, supplemented by contract resources. Treasury Board guidance suggests that departments take full advantage of the benefits of professional evaluation resources outside the government, such as those in private sector consulting firms, university-based institutes and consulting groups, and public policy advisory groups.

9.26 Overall, there has been a loss of resources allocated to program evaluation. Based on the 19 departments audited in 1983, in-house staff has been reduced, initially offset by increased contracting resources. Later there was a reduction in contract dollars.

9.27 Expenditures on contracts for program evaluations in all departments decreased significantly from 1989-90 to 1991-92, by 28 percent. In-house staff levels were unchanged over the same period (see Exhibit 9.2 ). However, total program expenditures have not decreased over this period. Thus, the work to be accomplished has not decreased, but the ability to carry out evaluations using contract resources has declined.

9.28 A reduced ability to carry out evaluations results from the reduction in contracting resources and from changes over the past ten years in the expertise required of the function. In 1983, we noted the need for uniquely trained professionals who are knowledgeable about government and trained in evaluation. We had found a number of program evaluation units where the number of staff with program evaluation knowledge and skills was insufficient.

9.29 In the 1980s, program evaluation managers came to rely more on outside consultants than on in-house staff to carry out evaluations. In our comparison group of departments from the 1983 audit, in-house evaluation staff declined from 1982-83 to 1989-90 by ten percent. At the same time, expenditures on contracting nearly doubled, from $3.3 million to $6.5 million. Between 1989-90 and 1991-92, however, contract resources decreased without a compensating increase in staff resources. Contracting levels in 1991-92 for these 19 departments were closer to 1982-83 levels (see Exhibit 9.3 ). Greater reliance was placed on contractors in the 1980s. The need is changing from contract management to conducting in-house evaluations, with fewer people.

Evaluation managers emphasize supporting departmental management
9.30 Locating program evaluation units in departments and agencies gives more immediacy to the needs of departmental managers than to broader government or public interest in evaluations. We discussed the role of program evaluation with all 42 heads of program evaluation units.

9.31 Evaluation managers attached the greatest importance to helping management resolve operational issues and improve programs. They placed a much lower priority on the role of evaluation in challenging existing programs to support resource allocation decisions, and on evaluating large program units to support accountability to Parliament.

9.32 Consistent with their desire to assist program managers, many evaluation managers viewed non-evaluation work as important to establishing the credibility of the function and fostering the use of evaluation findings. Evaluators can contribute usefully to other departmental management processes. Because of the pressure on evaluation resources, we asked evaluation managers to estimate the amount of their resources that were used in other management activity. A quarter of evaluation managers had difficulty distinguishing between evaluation and non-evaluation work and documenting the effort devoted to non-evaluation activities.

9.33 Managers of 33 program evaluation units indicated that evaluation resources were sometimes allocated to a variety of work whose focus was not effectiveness measurement. Such work included reviewing program management and operations, participating in policy and program development and analyses, and providing technical support such as statistical analysis.

9.34 Less than ten percent of total expenditures for program evaluation in 1991-92 went to non-evaluation work. The estimate includes departments where program evaluation resources were combined with resources for other functions, such as audit and corporate review. As an average figure this should not be a concern, but in individual cases it can be as high as one third of evaluation resources, which would mean that far fewer evaluations could be carried out.

Not all departments plan systematically
9.35 Almost all managers responsible for evaluation in departments prepare evaluation plans for the deputy head. However, not all prepare multi-year plans and only 17 do multi-year plans along with a separate annual plan. The program evaluation policy calls for multi-year plans to be submitted to the Comptroller General for his information. Plans are generally respected.

9.36 Program evaluation is to address priority concerns where it can make the greatest contribution. Evaluation plans are vital in allocating scarce resources to a range of potential evaluation projects that can involve lead times of up to two years.

9.37 In the 11 major program activities where we examined effectiveness measurement (described in Chapter 8, paragraph 8.12), we found that departments do not always undertake a systematic analysis of the full array of their programs to select the most suitable candidates for effectiveness measurement. Planning considerations might include a concern with activities involving large amounts of money (expenditures, revenues, or tax expenditures), requests from Parliament and Cabinet, or major policy and program initiatives.

9.38 An example of a reasonable attempt at systematic planning was that of the Department of Finance in its first three-year evaluation plan. Although the second three-year plan did not follow the same rigorous approach, the Department subsequently reaffirmed its commitment to evaluate tax measures systematically. The Department is sharing its plans with the Public Accounts Committee and has made changes in the plans at the Committee's request.

9.39 The Department of Finance's first plan developed a basic list of program components and undertook a systematic analysis for selection purposes, based on a set of seven criteria (see Exhibit 9.4 ). By this means the Department was able to establish a planning process that took into account a range of competing considerations when allocating scarce evaluation resources among programs. Applying criteria of this type can provide a risk analysis of the consequences of not evaluating particular programs.

9.40 Three of the other departments we examined also have developed their own criteria for selecting and/or scheduling program components to be evaluated (National Defence, Employment and Immigration, and National Health and Welfare).

9.41 Improvements can be made in a number of departments in planning to allocate scarce evaluation resources to areas of greatest priority. Although the basis for establishing priorities can vary by department, specifying the basis for selecting areas to evaluate makes choices and priorities explicit.

9.42 The decisions on the areas to be evaluated and the types of studies to be undertaken are made in each department under the direction of the deputy head, with limited central direction. Major changes in the operation of the evaluation function in departments cannot be made without the support of the deputy head. Evaluators and evaluation managers are responsible for implementing the plans and producing the evaluation studies.

Evaluation Outputs

The number of program evaluations is declining
9.43 Program evaluation units provide a number of different outputs, including advice to managers, evaluation frameworks, evaluation assessments and evaluation reports. The evaluation reports are one of the most visible outputs, since they are to be used outside departments as well as inside. Treasury Board requires that these reports be written and presented clearly, in anticipation of an external readership.

9.44 Evaluation reports can vary widely with respect to such factors as complexity of the issues evaluated and methodologies used. Often they are written too technically to be easily understood by the general reader. As a measure of output, the number of reports produced does not take into account the variation in their content or quality, but does indicate the volume of the primary product being produced by decreasing program evaluation resources. Even if other evaluation products were to increase, it is important that the flow of reports be maintained, as they reflect the completion of evaluation studies and the reporting of their findings. Evaluation managers were asked to show us the studies they had completed. Our estimates are based on the sum of the studies we were shown in the course of our audit.

9.45 The number of program evaluations completed each year has been declining since 1987-88, as contracting resources have declined. In 1987-88, program evaluation units produced 99 reports, compared with 80 in 1991-92. (see Exhibit 9.5 )

Low evaluation coverage of program expenditures
9.46 The decision on what and how much to evaluate in each department is the responsibility of the deputy head. The 1977 Treasury Board policy suggested that departments evaluate each program at least once every three to five years, so that 100 percent of program expenditures would be evaluated over five years. This proved to be an unachievable target. The revised 1991 policy suggested that all programs be considered for evaluation within six years. It would be reasonable to expect a level of evaluation across government that demonstrated a clear commitment to implementing the intent of the policy.

9.47 The revised policy requires that all programs be considered for evaluation, although evaluations should be carried out only where material and cost-effective. There are requirements for evaluation associated with Treasury Board decisions, Increased Ministerial Authority and Accountability (IMAA) memoranda of understanding with Treasury Board, and other policies such as the Regulatory Process Action Plan, as described in Chapter 8. That policy, for example, required that all regulatory programs be evaluated every seven years.

9.48 We found that evaluation studies conducted over a seven-year period covered approximately 24 percent of the government's program expenditures in 1991-92. If expenditures on servicing the national debt are included, only 18 percent of expenditures were covered in seven years. Evaluation coverage can be estimated in different ways. Our estimates are based on best estimates, agreed by managers, of expenditures evaluated as a proportion of budgetary expenditures reported in the Public Accounts.

9.49 As a test of responsiveness to the 1986 general regulatory policy requirement, we examined the evaluation coverage of regulatory activities. We estimate that 53 percent of regulatory program expenditures have been evaluated over seven years. Although this estimate is higher than the overall rate of coverage of 18 percent to 24 percent, it falls far short of the policy requirement that all regulatory programs be evaluated. Revised 1992 Treasury Board policy restates the requirement to determine the effectiveness of regulatory programs, but is less specific in its requirements for coverage.

9.50 Program evaluation studies have examined a comparatively small proportion of government expenditures. The extent of coverage of other significant areas, such as the regulatory programs, did not meet policy requirements. Government has frequently turned to other review mechanisms, such as special studies, regulatory reviews or royal commissions, rather than to program evaluation. These mechanisms often have other objectives in addition to measuring program or policy effectiveness, as illustrated in Chapter 13 on the implementation of the Agri-Food Policy Review.

Program evaluations focus on smaller units of government activities or programs
9.51 In 1983 we found that, in drafting evaluation plans, departments focussed on smaller programs or less difficult aspects of larger ones. In some cases, the evaluation of larger and more complex programs was deferred. We found in the present audit that the focus continues to be on smaller program units.

9.52 Government programs are organized into activities and sub-divisions of activities for planning, budgeting and reporting purposes. The same activity structure is reflected in aggregate form in the Estimates submitted to Parliament and in the accompanying Part III accountability reports. The smallest program unit reported is typically the sub-activity.

9.53 Program evaluation results could support parliamentary scrutiny of the Estimates. However, ideally those results would need to relate to the reporting units in the activity structure. Of the 610 program evaluations completed from 1985-86 to 1991-92, 533 evaluated a portion of the activity structure. The remaining evaluations examined departmental operations or other matters that could not be attributed to a part of the activity structure, such as evaluations of governing legislation. We found that 75 percent of evaluations looked at program components much smaller than the sub-activity. As a result, the conclusions of these specifically focussed studies would be of only limited use for broader-based accountability reporting to Parliament.

9.54 Even when evaluations cover all the sub-units of an activity, the understanding of the total program that they provide can be incomplete, as illustrated by the evaluations of Employment activities in the Department of Employment and Immigration, described in Exhibit 9.6 . These evaluations responded to the needs of departmental management but not to parliamentary interest in the overall effectiveness of the program. Program evaluations at the sub-activity level or below provide only a partial assessment of the total activity's effectiveness.

9.55 Another measure of a program's size is expenditure. We found that programs and activities with larger expenditures receive less evaluation attention than those with smaller expenditures. The rate of coverage of programs with expenditures of over $250 million is less than half that of programs spending $250 million or less (see Exhibit 9.7 ). Yet programs with expenditures of over $250 million account for 78 percent of total federal government program expenditures. Because of its focus on smaller program expenditures, program evaluation is not addressing the effectiveness of the bulk of government expenditures.

Program evaluations focus on matters related to operational effectiveness
9.56 The policy statements and guidance on the conduct of evaluations set out the basic issues that program evaluation studies are expected to address. These issues, set out in 1981, were simplified and redefined in the 1991 revised policy as follows:

  • Relevance (formerly Program Rationale): Does the program make sense?
  • Success (formerly Impacts and Effects and Objectives Achievement):
    What happened as a result of the program? Has the program achieved what was expected?
  • Cost-effectiveness (formerly Alternatives): Are there better ways of achieving the results?
9.57 Not all program evaluations are required to address all these issues. The selection of the issues examined is determined in the planning stage, and should be strongly influenced by the nature of the program or activity being evaluated and by the expected use of the evaluation results.

9.58 We reviewed all program evaluation reports completed from 1985-86 to 1991-92, to determine whether operational issues had been addressed and which basic evaluation issues had been evaluated. Operational issues would include the suitability and efficiency of the program's day-to-day operations, rather than the program's results.

9.59 Operational issues are a major focus of program evaluations. We found that over three quarters of evaluations addressed operational issues as part of the evaluation.

Evaluation studies are less likely to evaluate questions of the relevance and cost-effectiveness of programs
9.60 We would expect the focus on particular evaluation issues to vary, depending on the anticipated use of the evaluation. If the use was to challenge the future of a program, the focus would tend to be on questions of relevance (does the program make sense?) and cost-effectiveness (are there better ways of achieving the same results?) However, if the intended use was to provide program understanding and accountability information, the focus would tend to be on questions of success (what has happened as a result of the program and has it achieved what was expected?)

9.61 We identified the basic evaluation issues that appeared in the terms of reference for each evaluation, or that appeared to be reported in the evaluation conclusions or recommendations.

9.62 Attention to questions of program relevance has decreased since 1983. Three quarters of the studies we examined in 1983 addressed relevance, while the same departments addressed this issue in just over half of the studies they completed from 1985-86 to 1991-92. This trend is consistent with the focus on operational issues and is away from challenging policy effectiveness.

9.63 Overall, of the three basic evaluation issues, evaluation studies were most likely to address questions of success. Close to 80 percent examined issues of success to some extent, while about half the evaluations sought to determine whether there were cost-effective alternatives and just over half whether the program was still relevant (see Exhibit 9.8 ). Program evaluations in departments are directed even less than in 1983 to challenging the future direction of programs rather than examining past performance.

The Use of Evaluation Studies

Findings from program evaluations are relevant mostly to the operational effectiveness of programs
9.64 The results of program evaluation are expected to serve different stakeholders who have different information requirements (see Chapter 8). Affected by technical feasibility and limits of time and budgets, evaluations inevitably are selective in the questions they address and the topics selected for examination. The type of information produced by evaluation studies indicates which of the potential uses described in the evaluation policy they can serve. We would expect evaluations to provide information to support:

  • accountability, by examining questions of program success;
  • program and policy decision making, by examining questions of continued relevance and cost-effectiveness; or
  • operational effectiveness, by examining questions of program delivery, such as quality of service to clients and their assessment of programs.
9.65 We examined the program evaluations for the 11 programs we audited to assess whether the information provided was relevant to program/policy decision making, accountability to Parliament and operational effectiveness, and whether the studies provide reasonable, partial or adequate information. To provide useful information, not only do the relevant evaluation issues need to be covered but the information presented needs to be reasonably complete.

9.66 We found that evaluation is less likely to be an important source of information in support of program and policy decisions addressing questions of continued relevance and cost-effectiveness. Evaluations are more likely to provide information for accountability purposes but it is often partial.

9.67 The most complete information available is related to operational effectiveness, the way a program is working. This information is useful for administering programs and supporting government initiatives to reduce red tape and streamline regulations. It also can assist in refocussing the day-to-day management of programs to achieve results. The program evaluation of the Special Import Measures Act demonstrates the usefulness of this type of program evaluation. Yet the impacts and effects of the program have not been measured (see Exhibit 9.9 ).

The use of program evaluation frequently cannot be demonstrated
9.68 Deputy heads are responsible for having evaluations conducted and using the resulting information to confirm, modify or recommend major reforms of programs or to discontinue them. We would expect to find evidence that program evaluations are used by senior managers for any of these three purposes.

9.69 We examined 168 studies where uses had been identified in the January 1992 survey by the Comptroller General, or by departments during our examination.

9.70 We asked departments to provide documentary evidence in support of the evaluations' primary uses, and we sought to confirm whether the evaluations had contributed to program termination, reform or modification, or had led to improved understanding of the programs. Departments at times found it difficult to provide documentation at the standard we required within the time frame of the audit. Exhibit 9.10 sets out our criteria for use. Documentation was still being provided as long as 11 months after our initial request.

9.71 Applying the requirement to provide evidence demonstrating the use of studies, we were able to confirm the uses identified by the departments or by the Comptroller General's staff for 53 percent of the evaluations. This figure underestimates the overall use, in that we did not seek confirmation of uses other than those suggested by the Comptroller General or departments. Neither were we able to recognize the informal sharing of information that occurs as part of the evaluation process, and the input of evaluators into management decision making after a formal evaluation is completed. Still, when so much of the value gained appears to be shared informally, it is difficult to be certain how useful the evaluations are for providing information to others outside the departments.

Where used, evaluations contribute most often to operational decisions or improving program understanding
9.72 Program evaluations for which we found documented confirmation of use contributed most frequently to improved understanding of the programs and to program modifications (see Exhibit 9.11 ). In 20 percent of the evaluations we examined, departments provided evidence of operational changes made to improve program delivery, often in ways not visible to program clients. To confirm the use of the evaluation, we sought evidence that the program evaluation had clearly recommended or identified program changes, that it was a factor considered by management in changing the program and that the changes had actually occurred.

9.73 For another 23 percent, we obtained documents confirming that the departments had used the evaluation findings to promote better understanding of the programs, by reporting findings in Main Estimates Part III or by some other mechanism. No specific changes were made to these programs as a result of the evaluations, but program evaluation resulted in a better understanding of the programs and their rationale among line managers and clientele, and in better information for reporting on program performance. We did not attempt to identify any further benefits associated with improved understanding.

9.74 A much smaller proportion of the studies we examined were confirmed as having contributed to significant reforms, producing major changes to program delivery. These changes were visible to program clients and reoriented the programs. Few evaluations resulted in program termination.

Use of evaluations increases with senior management involvement
9.75 We examined further the evaluations that clearly had been used, to determine what factors were associated with greater use. Evaluations that dealt with operational questions were the most likely to have a demonstrated use. We observed that 62 percent of studies that addressed operational issues had a demonstrated use, compared to 34.5 percent of studies that addressed only other issues.

9.76 Long-range planning for program evaluation allows a department to look ahead and make strategic choices about the focus and timing of evaluations. Adequate follow-up procedures help ensure that appropriate actions are taken on any decisions the deputy head makes as a result of the evaluation. Evaluations are more likely to have a demonstrated use if there is a multi-year plan for program evaluation in place ( Exhibit 9.12 ).

9.77 We also found indications that evaluations were more likely to have a demonstrated use in departments where a senior-level committee, such as an audit and evaluation committee or an executive committee, monitored the implementation of evaluation findings. All departments have some form of follow-up procedure. However, 14 departments do not have follow-up by a committee of senior managers. Follow-up that includes a senior committee, such as an audit and evaluation committee, provides the basis for closing the loop with senior managers who are also responsible for approving evaluation plans. Closer integration of program evaluation into the senior management process in the department was associated with demonstrable use of evaluation.

Savings Resulting from the Use of Evaluation Studies

9.78 Program evaluations are used at times to modify programs or, in a few cases, to reform them in ways that result in savings. Program evaluation can also be used to confirm the value of programs. Although program evaluations are not necessarily designed to result in savings, they can produce findings that can result in significant savings. Program evaluation managers were asked to provide confirmation of cost savings identified by the Comptroller General as a result of evaluations completed during the period 1986-87 to 1991-92. Some program managers identified additional examples. We sought to confirm the cost-savings estimates with supporting evidence that would link an evaluation study with management action that resulted in savings. Most program evaluation managers do not attempt to maintain information systematically on savings resulting from evaluation studies. The estimates included both savings resulting from program terminations and the reallocation of resources from one program to another or within a program.

Limited savings from evaluation studies confirm potential
9.79 Because of the lack of documentation on savings and on the results of evaluation studies in many departments, we were able to confirm in only 17 of the 39 examples provided that evaluations contributed to savings. Most of these savings were under $5 million. Total savings for the period 1986-87 to 1991-92 were estimated at $212 million. However, a large proportion of this figure was accounted for by one classified study, at the Department of National Defence. In some instances, including the single study associated with larger savings, the studies were only one factor contributing to a decision that resulted in cost savings. The one study demonstrates the contribution that program evaluation can make to large program savings.

9.80 Our audit of effectiveness measurement in 11 government programs identified another form of saving not included by the Comptroller General. The Department of Finance's 1990 evaluation of the effects of the Cape Breton Investment Tax Credit was instrumental in the decision not to renew this measure, which involved more than $500 million in tax revenue foregone during the period it was in effect.

The Management of Program Evaluation Units

Quality of evaluation studies needs improvement
9.81 The 1983 audit found that a substantial number of the evaluation studies had significant weaknesses in the methods used to carry out planned work, particularly with respect to measuring program effectiveness. As reported in Chapter 8, we continue to identify weaknesses in the methods used to conduct evaluations. We examined the quality of evaluations in the 11 programs we audited and found it to be uneven. Even within a given study, the reliability of the findings was not consistent.

9.82 Evaluators are expected to develop and apply methodologies appropriate to the evaluation issues included in the scope of their studies. This enables them to produce valid and reliable findings to support their conclusions. Inconsistency in the reliability of the data reduces confidence in the usefulness of the evaluation.

9.83 Decisions affecting the quality of individual studies are made in the departments. It is increasingly accepted that the most cost-effective approach to achieving quality is to build it into the product instead of relying primarily on after-the-fact quality assurance. We found that insufficient attention has been paid to building quality into evaluation products.

9.84 In addition, Treasury Board guidelines made the Comptroller General responsible for quality assessment across the evaluation system. His staff is required to assess the quality of key evaluations and the quality of departmental evaluation practice against standards. We also found that, in carrying out his quality assurance responsibilities, the Comptroller General has not clearly identified and reported difficulties in meeting standards of quality.

9.85 Staff of the Comptroller General did not assess the quality of key evaluations against the working standards. Rather, they produced summaries. These summaries were a source of information for liaison officers' assessments of departments and for their database on the program evaluation function. Staff of the Comptroller General feel that, through their participation in steering and advisory committees for key evaluations, they provide regular input on evaluation quality. Given the difficulties we identified with a number of evaluations, we believe a more systematic approach to quality assurance should be taken by the staff of the Comptroller General, as described in Chapter 10. This approach would be consistent with the requirements of the policy.

Need to improve management practices
9.86 In many departments, steps are needed to improve management practices with respect to program evaluation. Some evaluation units are stronger than others. All program evaluation managers and deputy heads responsible for program evaluation need to be sure that their units maximize their contribution to meeting departmental needs and, as noted in Chapters 8 and 10, to government-wide initiatives.

9.87 With limited evaluation resources, only a select number of evaluations can be carried out. Criteria need to be established that set out the basis for selecting evaluation topics and areas for examination as part of other strategic and performance planning. If departmental units are to contribute to government-wide concerns, in developing plans they should give greater attention to evaluating programs with larger expenditures and evaluating larger program units. As part of departmental planning, the link should be made between the government's strategic requirements and the interests of the evaluation managers. Programs should be selected for evaluation so that evaluations add value to departmental and overall government effectiveness.

9.88 Deputy heads should ensure in planning that all activities are considered for evaluation and that criteria for selecting areas for evaluation are clearly set out and followed. They should also ensure that evaluations address the more significant departmental issues, as well as government priority areas, in a timely manner.

9.89 The Comptroller General has provided guidance on integrating evaluations into senior management decision making in departments. Guidance, rather than policy direction, offers suggestions on past practices that have proven beneficial. In particular, our audit supports the importance of senior management commitment, demonstrated by involvement in the systematic planning of evaluations on a multi-year basis, in monitoring the implementation of evaluation recommendations and in following up on corrective action. Regular departmental audit and evaluation committee meetings, chaired by the deputy heads, can ensure that priority issues are evaluated and findings acted on. The departments that implemented these practices were better able to demonstrate greater use of evaluation studies.

9.90 Deputy heads should be involved in approving and ensuring the use of multi-year evaluation plans, monitoring the implementation of evaluation recommendations and following up on corrective action.

9.91 We had difficulty confirming the uses and cost savings claimed for program evaluation studies. Managers of program evaluation in many instances were not able to provide clear evidence to demonstrate the effectiveness of their outputs. Monitoring by a central group of the Comptroller General's staff cannot be expected to serve as a substitute for this individual management responsibility. We found few systematic approaches in place to provide a basis for evaluation managers to assess the performance of their units, both to improve their management and to account for their own responsibilities.

9.92 Program evaluation managers should put in place mechanisms for objectively assessing and demonstrating the value obtained from the evaluation efforts of their units.

9.93 All users of evaluation results want to be assured that they can rely on the evaluation findings. It is the responsibility of evaluation managers to take appropriate steps to build quality into their evaluations. Some managers have used advisory committees, others have used specialists on contract, and others have used peer review. The Comptroller General has a role in quality assurance for a limited number of key evaluations, but evaluation managers ultimately are responsible for the quality of their products.

9.94 Program evaluation managers should ensure that they have the required management processes in place to build quality into their evaluation products.

9.95 The infrastructure to carry out evaluations has been in place since the early 1980s. Over time, the function is evolving. The ability to produce high-quality evaluation studies depends, to a great extent, on the skills of professional evaluators - either on staff or on contract. For a time, reliance was placed on outside contractors to carry out evaluations. Now, increasingly, in-house staff will be required to conduct the evaluations themselves. This represents a significant change in their responsibilities and raises a question of whether the required professional skills are available.

9.96 Deputy heads and program evaluation managers should assess the skill requirements of their evaluation units against existing staff skills to ensure that the required professional capacity is in place, and take appropriate steps to fill identified gaps.

Conclusion

9.97 Questions are being raised about the cost-effectiveness of many support functions in government. Program evaluation has demonstrated that its studies can contribute to the review, modification and reform of government programs, and can contribute to savings. However, deputy heads have not fully exploited the potential of the function.

9.98 The resources dedicated to program evaluation are relatively small - both staff and contract resources - in proportion to the expenditures they are to evaluate. Further, these comparatively modest resources are dispersed across government departments. The outputs from the investment also are comparatively modest. The units in departments are primarily directed to supporting the evaluation interests of deputy heads and senior departmental officials. These units are not working equally well. Although a few strong units in government produce quality studies that respond to the interests of the deputy head, others need to improve.

9.99 The program evaluation system at present relies on results from program evaluation units in departments. Yet in meeting departmental requirements, the interests of other stakeholders have not been met well, as shown in Chapter 8. With its focus on operational issues and informal process, the contribution of evaluation in departments has not been sufficiently impressive to prevent the loss in evaluation capacity over the past seven years.

9.100 There are limits to how much can be accomplished, with the current resources, by changing the practice of evaluation in departments. The challenge is to respond to other stakeholders' needs for improved effectiveness information, as shown in Chapter 8. Evaluation units in departments can play an important part but will require additional interest and support, as described in Chapter 10.