This Web page has been archived on the Web.

1983 Report of the Auditor General of Canada

Synopsis

Development of Program Evaluation

3.1 In the 1960s, the awareness that program effectiveness lies at the heart of good public sector management led increasingly to a recognition that formal procedures to measure the effectiveness of public programs were necessary. This happened for a variety of reasons, but two are noteworthy. First, the state of the art in social research had advanced to the point where useful measurement of attainment of program objectives was possible. Second, a growing concern for value for money in complex and expensive public programs led to a rising demand for proof that the expenditures were cost-effective.

3.2 The Government of Canada began to place a growing emphasis on program evaluation in the late 1960s when departments and agencies were encouraged to establish central Planning and Evaluation Divisions by having Treasury Board make resources and person-years available to them. The departments responded. As a 1974 Treasury Board study noted, they had established planning and evaluation units involving approximately 3,500 person-years. However, that study also found that little program evaluation was being done. Most of the effort was apparently being directed to policy analysis and planning.

3.3 In 1977-78, we conducted a government-wide study of program evaluation, looking at 23 programs in 18 departments and agencies. We also found few successful attempts to conduct program evaluations. During that audit, the Government issued a Policy Circular (Treasury Board, 1977-47) which required all departments and agencies to establish procedures to evaluate systematically the efficiency and effectiveness of their programs.

3.4 In 1978, the Office of the Comptroller General was established and given functional responsibility for implementing this new policy. Since then, it has developed a policy framework to guide and structure departmental evaluation functions. This has been published in two documents: Guide on the Program Evaluation Function (May 1981); and Principles for the Evaluation of Programs (September 1981). By establishing a team of liaison officers to give guidance and advice to departments, the Office of the Comptroller General has also taken an active role in encouraging and assisting departments to implement the policy framework.

3.5 It is not easy to establish a new function like program evaluation on a consistent basis throughout government. In this case, the task was made more difficult because many senior managers were sceptical about the value of program evaluation, and there was a shortage of qualified program evaluators. To deal with this, the Office of the Comptroller General set as its first target getting evaluation units established in the larger departments and agencies and encouraging them initially to attempt useful program evaluations of small programs, or those of limited scope, to gain experience and to enhance credibility with management.

Audit Focus

3.6 The purpose of this audit was to report on the progress made in establishing program evaluation in the federal government over the five years since our 1978 Report. To do this, we selected a sample of 19 departments and agencies across government and audited their program evaluation functions. In each case, we looked at two areas:

    - the infrastructure for program evaluation, including its policy, plans, resources and management, to assess the degree to which the organizational elements necessary to maintain a functioning and productive unit were in place;
    - the conduct, reporting and use of program evaluations to assess the degree to which they have been carried out in accordance with government guidelines, reported in a balanced and fair way to the appropriate officials, and used by them.
3.7 Our audit criteria were developed from two sources: The Report of the Public Accounts Committee, tabled in the House in July 1980, in which it endorsed the five basic criteria set by our Office for auditing evaluations; and the Office of the Comptroller General's policy framework which was used to provide a more detailed elaboration of these criteria.

3.8 Our observations are divided into three parts. The first two deal with the findings of our audit of program evaluation in 19 departments and agencies. The last section deals with matters of a government-wide nature affecting program evaluation.

Organization and Management for Program Evaluation

3.9 In auditing the development of program evaluation functions in 19 departments and agencies, we found that the government has made significant progress in establishing a program evaluation function in its departments and agencies. All the departments and agencies we audited have established corporate program evaluation units. In 1982-83, the 19 units used 168 person-years of staff time and spent just over $3 million contracting with outside consultants for professional and special services. Only two of the departments did not have deputy-approved policies for program evaluation, although one of these has subsequently been approved.

3.10 To ensure that program evaluations are carried out systematically and on a cyclical basis, departments are required to develop a long-term plan to evaluate all their programs. Sixteen of the 19 departments had approved long-term evaluation plans, 2 had draft plans and 1 had no plan.

3.11 There were a number of areas in which the management practices of the evaluation units could be improved. In particular, these concerned the control of projects, both with respect to timing and resources. Further, in many units, adequate documentation for studies was not maintained, nor was information available to some managers to enable them to account for resources used on evaluations.

Program Evaluation Initiatives

3.12 By contrast with our findings in 1978, we found that departments have made considerable progress in planning and carrying out corporate program evaluations. Seventeen of the 19 departments have evaluation initiatives under way, and 15 of these have completed at least one program evaluation study. Many have completed more. Overall, we found that 6 evaluation frameworks, 119 evaluation assessments, and 86 program evaluation studies had been completed.

3.13 In those departments with relatively more experience and in which we were able to identify a trend, we observed improvements in the quality of evaluations undertaken. However, we also identified a number of areas where further improvements are required. A significant proportion of evaluation assessments did not form an adequate basis for sound advice to the deputy for planning the evaluation study. In a number of cases, limitations in the study could be traced back to weaknesses in the assessment.

3.14 A substantial number of the evaluation studies had significant weaknesses in the methods used to carry out planned work, particularly with respect to measuring program effectiveness. We found poorly designed questionnaires; unreliable data; data that were incomplete and/or lacked objectivity; inadequately specified sample designs; and samples that were too small for the purpose intended, or biased. As a result of these problems, approximately half the studies which attempted to measure the effectiveness of programs were unable to adequately attribute outcomes to activities.

3.15 It is important to note that most study reports included discussions of qualifications of the findings, although, in some instances, these were not as complete as they should have been.

3.16 Even in those cases where difficulties were encountered in planning and carrying out assessments and studies, it was often the case that parts of these produced information that is sound and that departmental managers found useful.

3.17 The 43 studies that we audited in depth covered a wide range of topics, and most addressed at least one of the four basic program evaluation issues:

    - over three-quarters attempted to address program rationale, raising questions about the continuing relevance of, or need for, the program;
    - three-quarters attempted to measure the degree to which the program had achieved one or more of its goals and objectives;
    - a similar proportion tried to determine the extent to which the observed changes had occurred as a result of the program; and
    - half attempted to assess the degree to which there were cost-effective alternatives to the program.
3.18 The recommendations arising from these studies covered an equally wide range. They dealt with matters such as changing program support activities, program design, and the size of the program. In most instances these recommendations have been accepted by senior management and acted upon.

3.19 We found that, in almost all instances, evaluation reporting has followed procedures set out in the departmental evaluation policy. All departmental policies require reporting to the deputy, and nearly all require reporting to other responsible levels of program management. Instances of failure to report evaluations have been infrequent, and steps have been taken to ensure appropriate reporting. However, we also found that the reporting of studies in the Strategic Overviews under the Government's new Policy and Expenditure Management System could be improved.

Government-wide Issues

3.20 We used the program evaluation guidelines issued by the Office of the Comptroller General as a basis for expanding and elaborating on the audit criteria developed by our Office for our 1978 study and endorsed by the Public Accounts Committee in 1980. We believe that these guidelines provide a useful basis for departments to organize, plan and conduct program evaluation work. Also, as noted earlier, the implementation strategy followed by the Office of the Comptroller General involved an initial emphasis on establishing evaluation units in the major departments and agencies and on encouraging these departments to conduct evaluations, even of a limited nature. In our opinion, this has been a reasonable way in which to proceed. However, we noted several areas where the policy framework may need to be modified or extended.

3.21 The current policy framework for program evaluation has the department and the deputy head as the central focus of program evaluation activity. However, many important programs are delivered in a way that involves more than one department. Further, the basic structure of the Policy and Expenditure Management System stresses the interdepartmental nature of program decision making.

3.22 Although current policy and guidelines recognize the existence of interdepartmental programs, they fail to specify procedures to be followed in conducting evaluations of them. The consequence of this is that interdepartmental programs are not systematically being subjected to the same type of orderly review and evaluation as programs administered wholly within single departments and agencies.

3.23 Most of the agencies which had not developed an infrastructure for evaluation were small. As part of its implementation strategy, the Office of the Comptroller General has only recently given full attention to the way in which evaluations of these agencies' programs should be carried out.

3.24 Crown corporations are being used by the Government to achieve public policy objectives, and funds for these purposes are being provided through the Estimates. We found that, in general, public policy objectives of Crown corporations were not subject to program evaluation, nor were they being scheduled for program evaluation. As of 31 March 1983, five Crown corporations were establishing program evaluation functions in liaison with the Office of the Comptroller General.

3.25 In its review of the 1978 Auditor General's Report, the Public Accounts Committee recommended to the House, in July 1980, that all effectiveness evaluations be tabled in the House of Commons within 60 days of their completion. The Government responded, in October 1980, through the President of the Treasury Board, that:

With enactment of the Access to Information Bill as currently proposed, and approval of a draft Treasury Board policy on the documentation of program evaluation studies, such information would be publicly available. Some procedure, perhaps tabling of evaluation reports within 60 to 90 days of their completion by the responsible Minister, will have to be established to ensure easy access by Members of Parliament.
3.26 On the basis of our audit work we found that, since 1980, only one program evaluation report has been tabled in the House.

Summary

3.27 While we were sharply critical of the situation we observed in 1978, we also stated that "the recent emphasis the Government of Canada has placed on program evaluation could, if developed and sustained, put it among the leaders in the field." Leadership does not come easily. The task which the Government has set for itself is to make program evaluation an integral part of public sector management. This requires no less than a commitment on the part of the Government to critically examine the success of its own programs and policies.

3.28 We found that very real progress has been made. Most of the major departments and many of the agencies have the basic infrastructure for evaluation in place. Many of these are actively attempting to carry out evaluations. However, the quality of the evaluations needs to be improved.

3.29 To achieve these improvements in quality, the government's commitment must be made evident through requests for the evaluation of its programs, appropriate staffing of departmental evaluation branches, and the distribution of evaluation reports.

3.30 Program evaluation requires asking fundamental questions about a program's existence. Requests for evaluations should make explicit what the program is trying to achieve and against what it should be evaluated.

3.31 In getting the results achieved thus far, one of the major problems the Office of the Comptroller General and departments have faced has been a shortage of appropriately trained and qualified evaluators. This shortage continues, and if the progress and momentum achieved thus far are to be maintained, the development of a cadre of qualified professional program evaluators will be one of the major challenges to be met.

3.32 Finally, the quality of evaluations should improve with experience. This process requires that evaluation reports be widely distributed. Broader scrutiny will help ensure higher quality products. It will also help to ensure that lessons learned in one study will be available to build on in the next.

The Government's Response to the Auditor General's 1983 Audit Report on Program Evaluation

The Government's approach to evaluation of programs aims at ensuring that relevant and reliable information is produced on the continuing need for, performance of, and relative cost-effectiveness of programs. in its approach to the establishment of the program evaluation function, the Government has been mindful of the need to ensure procedures for the conduct and consideration of evaluations which enhance their credibility and actual use, as well as being mindful of the significant cost of conducting studies. Accordingly, the approach taken to establishing the program evaluation function is one of integrating evaluation within the improving management practices in departments and with the Policy and Expenditure Management System, while the approach to conducting evaluations is one which balances the need for objectivity with the need to link evaluation closely with line management of government programs.

This audit, coming midway through the planned initial development phase, is a timely progress report on the approach the Government has taken, the accomplishments to date and the main work yet to be done. The findings, taken as a whole, represent to the Government an endorsement of its approach and confirm the substantial progress achieved. The recommendations, in general, are helpful and supportive of the Government's efforts and, with a few exceptions, outline a program of work largely consistent with the Government's plans in this area.

The infrastructure in terms of evaluation policies, plans and organizations is in place in most of the major departments and agencies throughout the Federal Government. Furthermore, the Office of the Comptroller General is working now with smaller departments and agencies and Crown corporations to assist them in developing an appropriate approach to evaluating their programs. As of September 30, 1983 a total of 90 departments and agencies were liaising with the Office on program evaluation, 19 of which were Crown corporations.

The audit identified problems in the quality of a number of evaluation studies produced between 1980 and March 1983. It also notes significant improvements in quality over this time period, especially in departments which have had relatively more experience in conducting studies. This confirms the Government's view and, notwithstanding the noted progress, this is an area where further work is required. Among the several recommendations aimed at improving quality is the recommendation that further steps be taken to ensure the availability of qualified evaluators. This is not only a question of acquiring technical skills. Fully qualified evaluators require a combination of technical and managerial skills and a thorough knowledge of programs and program management in the Federal Government. Such knowledge and skills are fully obtained only through appropriate experience. Accordingly, while the need is recognized and is being addressed, progress will be gradual. The Office of the Comptroller General, for its part, offers an ongoing series of seminars and workshops on evaluation, is consulted on most senior staffing actions in evaluation and has recently distributed a discussion paper on human resources management in the evaluation function to departments and agencies. The paper includes a number of proposals concerning the development of evaluation personnel.

Although adequate infrastructure and appropriate quality are required for producing good quality evaluations, the Government's central concern is that the findings of the studies be used in the ongoing management of government. As this report notes, some action has been taken on the recommendations of most of the studies produced to date. This audit finding is particularly welcome and likely reflects the approach several departments have taken in their initial evaluations. In particular, approaches which involve both senior management and line managers in the evaluation and which focus on issues on which departments can act appear to be the most successful.

This audit report includes a number of recommendations relating to the availability of evaluation findings to Parliament. At this time, it appears that the most effective and appropriate way for reporting such information to Parliament would be to include the key findings of evaluations, where relevant, in the Part III of each department's Estimates. This would provide for relevant findings on the effectiveness of programs to be presented in a concise manner to Members of Parliament. Access to information legislation, of course, provides for any member of the public to request a particular evaluation study. Accordingly, it would seem that a requirement to report also in departmental annual reports would be unnecessary, as would be, perhaps, other procedures to table reports in the House. With these developments, it may be appropriate for the Public Accounts Committee to consider the need for tabling all evaluation study reports, as had been recommended prior to these developments.

Finally, the audit report recommends that there is a need for an improvement of procedures to conduct interdepartmental evaluations. Such evaluations have taken place in the past and several are now under way. Existing procedures of the Cabinet Committee system do provide for the identification of the need for such evaluations, but few have been called for, perhaps due to the need first to demonstrate the success of evaluation on less complex issues. The Office of the Comptroller General is currently preparing discussion papers on evaluation in several interdepartmental areas and will be exploring further procedures to facilitate such evaluations whenever there is a demonstrated need for them by an agency which can act on the findings.

This audit of the program evaluation function in the Federal Government has been useful and its findings provide a valuable information base for deciding the future directions the Government will be taking in developing and enhancing the evaluation capability of departments and agencies. As confirmed by this audit report, the continued development and indeed existence of this evaluation capability will require a continual monitoring by the Office of the Comptroller General. Increasing attention will be devoted toward improving the quality of evaluation products and to ensuring that the evaluation findings are used by all levels of management within government.

Introduction

3.33 A basic tenet of good management is the use of feedback on the achievement of objectives. A good manager sets objectives, defines specific goals in relation to them, and monitors achievements accordingly. Adjustments can then be made to work more efficiently and effectively toward the desired ends. In the private sector, goals and objectives are primarily defined in terms of profit, market share, and return on equity. Their measurement is carried out and reported in financial terms. These yield the well known "bottom line".

3.34 In government, measuring achievement is not as straightforward. Departments and agencies deliver a vast array of goods and services to a wide variety of clients. What most of these services have in common is that the objectives to be achieved, and the degree to which these are met, are often unclear and seldom easily measured.

3.35 The increasing emphasis on value for money in public sector management has led to the view that, if improvements in the economy, efficiency and effectiveness of government programs are to be achieved, systematic procedures need to be used to more precisely understand program activities and to measure program impacts and effects.

3.36 Program evaluation has as its hallmark a concern with measuring client needs, program implementation, and program effects to assist in the more efficient and effective management of the program. In this way, it represents an attempt to provide the public service manager with the bottom line which his private sector counterpart takes for granted. Program evaluation in the federal government refers specifically to the periodic, independent and objective assessment of programs to determine the adequacy of objectives, design and results.

3.37 Some form of evaluative activity has always been undertaken in the federal government. Program managers and policy makers have collected different types of information so as to make judgements about the effectiveness, efficiency and worth of programs. The distinguishing characteristic of recent federal government efforts is the policy framework established for systematic planning, conducting and reporting program evaluation studies.

Audit Purpose

3.38 In Chapter 5 of our 1978 Report, following a government-wide study of program effectiveness evaluation in government, we concluded that:

A review of 23 programs in 18 departments has disclosed few successful attempts to evaluate the effectiveness of programs. The scope and quality of effectiveness evaluation will have to be increased significantly before management, the Government and Parliament, each with its respective interests, can be reasonably informed on the achievements of public programs.
3.39 Our 1978 Report, however, also sounded a note of optimism for the future of federal government evaluation activities:

... it can be said that the recent emphasis the Government of Canada has placed on program evaluation could, if developed and sustained, put it among the leaders in the field.
3.40 Five years have passed since those observations were made and, in this audit, we report on the extent to which progress has been made toward establishing program evaluation in the government.

Development of Program Evaluation

3.41 Over the last decade, the Government has attempted to establish program evaluation in departments and agencies. Exhibit 3.1 represents a chronology of key events. Over that time, three approaches have been taken.

Exhibit not available

3.42 First, in the late 1960s and early 1970s, there were attempts to encourage the development of planning and evaluation units in departments and agencies by providing staff and resources. Under Treasury Board sponsorship, evaluations were carried out to show that they could be done. However, in 1974, a Treasury Board study found that little progress had been made in evaluating programs.

3.43 Second, in 1977, the Government issued a policy circular (TB 1977-47) requiring departments and agencies to carry out program evaluations systematically. There was some response to this policy statement, with some departments establishing corporate program evaluation units and, in those departments that already had ongoing evaluation functions, there was an increase in activity.

3.44 Third, in 1980, the Office of the Comptroller General began an effort to establish program evaluation throughout the government. Its Program Evaluation Branch has taken a variety of steps to help ensure that federal departments and agencies establish evaluation units to plan and carry out program evaluation studies. The Branch has provided departments with detailed guidelines for planning and carrying out program evaluations, advice and assistance on doing evaluation work, and training courses to enhance the knowledge and skills of departmental evaluation staff. Moreover, the Branch is involved in assisting departments in the classification and staffing of program evaluation positions.

3.45 In 1981, the Office of the Comptroller General issued a set of detailed guidelines providing a policy framework for program evaluation and setting clear expectations for program evaluation work in government departments. The Guide on the Program Evaluation Function addresses three fundamental questions: How should a program evaluation unit be organized in a government department? What should be the focus of the work? How should this work be carried out? These questions are to be addressed in the form of a departmental program evaluation policy, program evaluation component structure and program evaluation plan.

    - The departmental program evaluation policy should describe the organization of program evaluation in the department, as well as the procedures to be followed for carrying out evaluation work.
    - Program evaluation components are the groupings of resources, activities, and outputs related to a common objective or set of objectives, the focus of evaluation work.
    - The departmental program evaluation plan presents a schedule for the cyclical evaluation of the program evaluation components.
3.46 Three types of evaluation initiatives are covered in the guidelines. The relationship among these is shown in Exhibit 3.2. These are as follows:

Exhibit not available

    - an evaluation assessment, which serves as a proposal and plan for an evaluation study;
    - the program evaluation study; and
    - an evaluation framework, a planning document that is to be completed during the design and implementation of all new, significantly modified or, where appropriate, renewed programs.
3.47 The government policy framework for evaluation has been implemented in departments with the assistance of staff from the Comptroller General's Office. Liaison officers have been assigned to work with government departments and agencies, providing advice and assistance on how to plan and carry out useful and credible evaluations. The Policy Branch has worked on extending the framework and acting as a link between the departments' evaluation functions and the respective Cabinet envelope committee secretariats.

3.48 In implementing the policy framework, the Office of the Comptroller General has established the following priorities:

    - Initially, to assist in establishing an evaluation organization in the major departments and agencies of the Government.
    - Then to encourage departments and agencies to begin to carry out program evaluation studies, even if only of a limited nature.
    - Then to turn attention to the smaller agencies and to determine what arrangements for evaluation would be most appropriate.
    - Over time, to extend the scope and raise the quality of the evaluations being produced by all departments and agencies.
3.49 The Comptroller General's objective is, by 1985-86, to see departments develop to the stage where they can regularly carry out useful and high quality program evaluations.

3.50 On the basis of our review of government program evaluation in western industrial societies, we have been unable to identify any other central government that has attempted to establish a policy framework for program evaluation comparable in scope to that established by the Canadian federal government.

Audit Scope

3.51 This audit covers program evaluation in the federal government, based on an examination of 19 departments. In them, we examined the organization and management of corporate program evaluation units as well as the planning, conduct, reporting and use of program evaluation studies.

3.52 Our examination of the corporate program evaluation units covered organizational arrangements, resources, evaluation policies and plans. We also examined evaluations completed by the corporate units as of 31 March 1983, as well as the planning and design of evaluations, data collection and analysis, quality control procedures, and the reporting and use of evaluations.

3.53 Two groups of departments and agencies are covered by this audit. The first group is made up of nine departments and agencies scheduled for audit reporting in 1983. The second group covers 10 departments and agencies where we followed up on audits of program evaluation carried out in earlier years. By covering these two groups in this audit, we are able to provide a broad and generally representative view of the progress made by the government in establishing program evaluation.

3.54 Exhibit 3.3 presents information about the 19 departments covered by this audit. It shows the audit reporting year(s), along with information about whether the department was covered by our 1978 audit of program effectiveness evaluation, and information on estimated resource expenditures for 1983-84. More detail on the departments which were subject to comprehensive audits this year can be found in the departmental chapters of this Report.

Exhibit not available

3.55 In addition to the audit work carried out in the 19 departments, we also examined activities carried out by the Office of the Comptroller General in implementing the Government's program evaluation policy framework. In this connection, we audited certain of the information maintained by the Office of the Comptroller General on the level of program evaluation activity across the 56 government departments and agencies with which it was liasing as of 31 March 1983. Based on our audit work, we found this information to be accurate and will present it throughout this chapter in conjunction with information obtained from our audit of 19 departments.

3.56 Our audit mandate, as stated in Section 7(2)(e) of the Auditor General Act, requires that we report to Parliament any instances in which we have observed that satisfactory procedures to measure and report on program effectiveness have not been implemented, where it would have been reasonable and appropriate to do so. The relationship between our mandate and the focus of this audit requires that two distinctions be made.

3.57 The first distinction is between the focus of program evaluation in the government and our audit mandate on program effectiveness measurement. As used in the government, program evaluations may cover a broader range of issues than those of program effectiveness, or not cover questions of program effectiveness at all.

3.58 The second distinction between our mandate and the scope of this audit concerns the possible ways in which effectiveness measurement can be carried out. These involve 1) periodic effectiveness evaluation studies done on an occasional and as-needed basis; 2) ongoing effectiveness monitoring systems; and/or 3) periodic program evaluations done on a systematic and cyclical basis. In each case, the focus may be on measuring and reporting program effectiveness. Usually, what differs is the relative frequency of taking measurements and the level and frequency of reporting results along with the nature of the organizational unit responsible for doing so.

3.59 The government emphasis has been placed on the third of these approaches - periodic program evaluation studies carried out on a cyclical basis by a corporate or department-wide evaluation unit with the deputy head as the primary client. Thus, we have focused the audit on the corporate program evaluation function and the evaluation initiatives completed by such units. As a result in this chapter, we are not reporting on evaluation-like activities done outside corporate evaluation units, nor on program effectiveness monitoring procedures.

Audit Criteria

3.60 The basic criteria that we use in auditing program evaluation were first published in the 1978 government-wide audit of program evaluation. These criteria were reviewed and endorsed by the Public Accounts Committee in their July 1980 Report. In October 1980, in a letter to the Chairman of the Committee, the President of the Treasury Board endorsed the report saying, "I have discussed its contents with my officers both in the Treasury Board and in the Office of the Comptroller General, and I am pleased to say that we consider the Committee's recommendations positive and helpful." Attached to his letter were annotations on a number of the Committee's recommendations. The only comment related to our criteria had to do with the issue of reporting evaluations. We discuss this matter later in this chapter.

3.61 The criteria are:

    - Program objectives and effects should be specified as precisely as possible.
    - Program objectives and effects which can be measured should be identified.
    - Procedures to measure program effectiveness should reflect the state of the art and be cost justified.
    - The results of effectiveness measurement should be reported.
    - Evaluations should be used to increase program effectiveness.
3.62 We have expanded and elaborated on these criteria, based on the Guide on the Program Evaluation Function issued by the Office of the Comptroller General, particularly with respect to the organization and management of the program evaluation function in departments.

Observations and Recommendations

Program Evaluation in Departments and Agencies

Organization and Management for Program Evaluation
3.63 We observed that the Government has made significant progress in establishing a program evaluation function in departments and agencies:

    - The Government has assigned functional responsibility for program evaluation to the Comptroller General, who has created a Program Evaluation Branch to provide leadership for the function.
    - The Office of the Comptroller General has developed an appropriate policy framework for program evaluation and presented it in the form of a Guide on the Program Evaluation Function , published in May 1981.
    - All the departments and agencies we audited have established corporate program evaluation units.
    - Most of these departments have developed deputy-approved policies and have prepared plans to evaluate their programs on a cyclical basis.
Organizational Requirements for Program Evaluation
3.64 The government policy framework for program evaluation requires the departmental deputy head to be the accountable client for program evaluations. It also requires that there be a designated manager responsible to the deputy head for the evaluation function.

3.65 Four basic principles, underlying the Government's policy framework, require that the evaluation function be:

    - comprehensive - all the programs delivered by a department are to be assessed for evaluation;
    - systematic - the way in which each of the departmental programs is considered for evaluation is to be orderly and based on a clearly articulated rationale;
    - cyclical - all departmental programs are to be considered for evaluation once every three to seven years; and
    - independent - evaluators are to be organizationally independent from the direct management of the evaluated programs.
3.66 To support a program evaluation function operating in accordance with these principles, the Government requires that an organizational capability for evaluation be established in all departments. It consists of three elements: a corporate program evaluation unit with assigned resources; a program evaluation policy; and a program evaluation planning structure. In addition, the operation of a viable evaluation function requires that each unit combine these elements with good management practices.

3.67 Program evaluation units. A corporate evaluation unit is to be established in each department and consist of a set of positions assigned responsibility for carrying out evaluation work. All 19 departments covered by our audit had established a program evaluation unit by appointing a manager.

3.68 Exhibit 3.4 shows the growth in the number of corporate evaluation units in the 19 departments audited. Only 2 of the 19 have had a corporate program evaluation unit continuously in existence over the 5 years since 1978. Fourteen departments have appointed a manager for corporate program evaluation since 1980.

Exhibit not available

3.69 Information maintained by the Office of the Comptroller General and presented in Exhibit 3.4 shows that, of the 56 departments and agencies with which they liase, 41 had established the basic organization for program evaluation.

Exhibit not available

    - The Department of Finance is the only major department in the government subject to Treasury Board circular 1977-47 that has not appointed a manager for program evaluation and has not established a program evaluation unit.
    - Many of the smaller agencies in government have yet to establish program evaluation units. Although program evaluation is as important in smaller as in larger agencies, the level of effort required and the organizational arrangements needed can differ.
3.70 During 1982-83, the 19 evaluation units had 201 person-years allocated to them and used 168 person-years of staff time. The median size of the units, based on the number of person-years used in 1982-83, is approximately five persons.

3.71 Program evaluation units typically supplement their own staff resources and expertise by contracting with outside consultants for services. The 19 units covered by the audit spent $3,274,000 on contracted services in 1982-83.

3.72 Program evaluation policy. A key element in the Government's approach to program evaluation is that departments issue a program evaluation policy statement, approved by the deputy head, presenting the basic procedures to be followed in carrying out the mandate, defining the key actors or committees in the process, and assigning responsibility and authority for the function.

3.73 Exhibit 3.5 presents information on the progress made at establishing program evaluation policies in the 19 audited departments. Seventeen had deputy-approved program evaluation policies in place as of 31 March 1983. Two departments did not have deputy-approved evaluation policies. In one of these, the draft policy has since been approved. In the other, the Secretariat of the Solicitor General, there was no program evaluation policy as of 31 March 1983.

Exhibit not available

3.74 As is evident from Exhibit 3.5, most of the departmental evaluation policies have been established in the past three years. Our findings are paralleled by the Office of the Comptroller General's government-wide information in Exhibit 3.5, showing that 38 of 56 departments and agencies have deputy-approved policies in place. A large proportion of departments and agencies without a program evaluation policy are smaller agencies.

Exhibit not available

3.75 We audited the 17 approved policies against the standards set for them by the Government. We found that, in general, these policies are capable of serving as a working basis for a program evaluation function.

3.76 The involvement of program managers in evaluations is important. We found that many of the departmental program evaluation policies required that this be done by setting up, for each study, an advisory committee that includes the program manager. Further, we found that the use of such committees was common in those departments which did not have this requirement in their policy.

3.77 Program evaluation plans. Another important element in the Government's approach to program evaluation is that all the program components in a department are to be identified for evaluation purposes, and a long-term evaluation plan is to be developed to specify the order in which the evaluations are to be undertaken. The development of evaluation plans helps ensure that the evaluation process will be comprehensive, systematic and cyclical.

3.78 Our audit focused on three planning tasks: the program evaluation component structure and related profiles describing each component; long-term evaluation plans; and the annual update of these plans, which sets out in detail the evaluation activities planned for the current year. Exhibit 3.6 shows for each of these tasks the number of departments meeting the requirement.

Exhibit not available

3.79 We found that 16 of the 19 departments had approved program evaluation component structures, and 2 departments had draft component structures awaiting Deputy Minister approval, as of 31 March 1983. The appropriate departmental activities were covered in all of the approved or draft evaluation component structures.

3.80 Evaluation plans should set out departmental evaluation priorities and schedule program components for evaluation accordingly. We found that 16 of the 19 departments had deputy-approved long-term evaluation plans covering all departmental programs. Two departments had developed draft long-term evaluation plans.

3.81 The existence of long-term plans in the majority of the 19 departments covered by our audit is also reflected across the 56 departments and agencies on which the government maintains information. There has been an increase in the number of long-term evaluation plans in these departments and agencies, from 9 in 1980, to 33 in 1983.

3.82 Progress has been made in the completeness and detail provided in long-term plans over the past three years. Long-term evaluation plans have been responsive to change. Specifically, plans have been revised to reflect:

    - changing priorities of the deputy head;
    - delays in completing evaluation initiatives;
    - more realistic expectations;
    - work completed to date; and
    - revisions to the evaluation component structure.
3.83 Eighteen of the 19 departments had either approved or draft annual updates for their long-term plans.

3.84 To develop experience and gain credibility, many departments, in drafting their long-term evaluation plans, have chosen to focus their early studies on smaller programs or on the less difficult aspects of larger ones. This was, given the available pool of skills, a reasonable way to proceed. In some cases, this has resulted in deferring the evaluation of large and complex programs.

3.85 Management practices. In addition to examining the organization of program evaluation, we also examined procedures for managing it. Generally, we found these to be consistent with departmental evaluation policies. However, we noted areas requiring further attention.

3.86 As program evaluation plans are implemented, appropriate arrangements need to be made to monitor progress, keep departmental management informed of significant developments and use the information obtained to modify subsequent plans.

    - In approximately half the 19 departments, the program evaluation units formally monitor the implementation of their plans and report progress to departmental management.
    - Approximately half the departments audited had summarized delays in completing evaluation initiatives and reported this to senior management. The others had not done so.
3.87 Approximately two-thirds of the departments audited have experienced delays in meeting scheduled completion dates for evaluation initiatives. A variety of reasons account for these delays:

    - staff shortage, turnover, or lack of suitably qualified staff;
    - expansion of scope subsequent to study initiation;
    - response to requests for additional work;
    - lack of funds for using outside resources; and
    - poor or overly ambitious planning.
3.88 The evaluation function is still relatively new and, therefore, it is reasonable to expect that difficulties will be encountered in developing realistic plans and meeting deadlines. We observed improvements in the planning undertaken by more experienced departments.

3.89 In a number of departments, the program evaluation unit was not designated as a separate financial responsibility centre. As a consequence, in some cases, the managers did not know what resources had been expended on the evaluation function. Further, in many units the allocation of resources to specific evaluation projects was not monitored, and managers were unable to provide data on the costs of studies. If managers are to be accountable for their evaluations and exercise proper control and planning with respect to resources, then proper accounts of dollars and time spent must be maintained.

3.90 Departments should ensure that program evaluation managers are given the information needed to keep track of time and resources expended on evaluation activities.

3.91 In many departments, the procedures for maintaining the working papers and documenting evaluations were inadequate. If proper records are not maintained, valuable information may be rendered unusable, lessons learned may be forgotten, and auditing may be more costly than it needs to be.

3.92 Departments should ensure that the documentation for evaluation studies is properly kept and safeguarded.

Program Evaluation Initiatives
3.93 Compared to our 1978 audit findings, departments have made considerable progress in planning and carrying out corporate program evaluations.

    - Seventeen of the 19 departments and agencies in the audit have evaluation initiatives in progress. The corporate evaluation units in 15 of these departments have completed at least one program evaluation study.
    - Many have completed more. In the audited departments, we observed:
      - 6 completed evaluation frameworks;
      - 119 completed evaluation assessments; and
      - 86 completed program evaluation studies.
    - Most of this has been accomplished in the last three years.
    - Of the 19 departments and agencies audited, the National Parole Board and the Solicitor General Secretariat had not attempted any corporate evaluation assessments or studies.
3.94 Although the number of corporate evaluations which have been done in a relatively brief period is an important achievement, we identified a number of areas where significant improvements are required:

    - The majority of evaluation assessments, particularly the early ones, did not form an adequate basis for sound advice to the deputy for planning the evaluation study. We observed a number of cases where this had resulted in limitations in the study which could be traced to problems or weaknesses in the assessment.
    - A substantial number of the evaluation studies had significant weaknesses in the methods used to achieve the work planned for them. This was particularly the case with respect to the measurement of program effectiveness. Approximately half these studies had inadequacies in design sufficient to preclude them from properly attributing results to programs.
    - It is important to note that, although many of the evaluation initiatives had limitations, most of the study reports have given some, but not always full, recognition to this.
    - We also note that, even in those cases where difficulties were encountered in planning and carrying out assessments and studies, it is often the case that parts of the studies have produced information that is sound and has been found useful by departmental managers.
3.95 Generally, there have been improvements in the quality of evaluation initiatives undertaken in those departments with relatively more experience and in which a trend can be examined.

Type and Extent of Evaluation Activities
3.96 We examined three types of evaluation initiatives: evaluation frameworks; evaluation assessments; and evaluation studies.

3.97 Evaluation frameworks. A program profile and an evaluation framework are to be prepared for all new or renewed programs. Frameworks describe how the program will be evaluated and are required by the policy framework to be completed when a new program is designed and implemented or when a renewed program is significantly modified. The purpose is to ensure that the information needed for the evaluation study will be available and that program objectives are clear.

3.98 Information maintained by the Office of the Comptroller General shows that only 17 evaluation frameworks have been completed in 56 government departments and agencies, with an increase from 1 in 1980-81 to 9 in 1982-83. We identified 6 completed frameworks in the 19 departments we audited.

3.99 Departments and agencies should develop evaluation frameworks in conjunction with the design and early implementation of all new, substantially modified programs, or where appropriate, renewed programs.

3.100 We selected three evaluation frameworks for audit. Two of these were explicitly requested by Cabinet in conjunction with program renewal, and the third was carried out after the program had been in existence for two years. We found that one of the frameworks provided only a general discussion of possible evaluation designs and failed to discuss data collection methods. The remaining two frameworks were adequate.

3.101 Evaluation assessments. Assessments involve analysing the program and its environment, identifying the specific evaluation questions to be considered and the extent to which they can be addressed in a study, and developing and costing options for carrying out the study.

3.102 The Office of the Comptroller General's information on 56 departments and agencies shows that 177 assessments have been completed, with a nine-fold increase in the number of completed assessments, from 11 in 1980-81 to 99 in 1982-83. The departments covered by our audit were responsible for 119 assessments. We audited 53 of these assessments in detail.

3.103 Although we found many examples of weaknesses in the evaluation assessments completed, we also observed improvement in the more recent assessments done in most of the departments.

3.104 However, most of the evaluation assessments we examined did not adequately describe program activities and assess them for plausibility in achieving stated objectives. That is to say, they did not examine the extent to which the desired effects could reasonably be expected to result from the activities of the program. If this is not properly done, the evaluation study can be seriously misdirected because resources would be used to measure the achievement of objectives which are not realistically linked to program activities.

3.105 Departments should ensure that program activities are adequately described and their plausibility assessed in evaluation assessments.

3.106 Many assessments failed to develop an appropriate range of issues into detailed and costed options for the deputy to consider. The result of this was that many studies were narrow in focus and, in some cases, missed key program evaluation questions.

3.107 Departments should ensure that evaluation assessments present an appropriate range of issues, questions and options for the deputy head to consider.

3.108 Evaluation studies. The focus of program evaluation in the government is on the program as a whole, on its impacts and effects, and on the relationships between impacts and effects and the outputs produced, the resources, and the activities undertaken. Program evaluation is intended to address broad issues about the very existence of the program in relation to changing conditions and government objectives. In this way, it is to be different from internal audit which focuses on internal management processes related to program activities and resources.

3.109 Information maintained by the Office of the Comptroller General indicates that 174 evaluation studies had been completed by corporate evaluation units in 56 federal departments and agencies. A major portion of this activity has taken place in the past two years. The number of studies completed by corporate evaluation units has increased from 31 in 1980-81, to 87 in 1982-83. The corporate evaluation units in departments covered by this audit completed 86 studies between 31 March 1979 and 31 March 1983, accounting for approximately 50 per cent of the population of studies identified by the Office of the Comptroller General. (See also Exhibit 3.7.)

Exhibit not available

3.110 In the audited departments and agencies that had completed an evaluation study, we reviewed 43 of these studies in depth. These covered a wide range of topics, and most addressed at least one of the four basic program evaluation issues:

    - over three-quarters attempted to address program rationale, raising questions about the continuing relevance of or need for the program;
    - three-quarters attempted to measure the degree to which the program had achieved one or more of its goals and objectives;
    - a similar proportion tried to determine the extent to which the observed changes had occurred as a result of the program; and
    - half attempted to assess the degree to which there were cost-effective alternatives to the program.
3.111 In most of those departments where there is a sufficient basis of experience to allow a comparison to be made, we observed significant improvement in the quality of the more recently completed studies.

3.112 We found a variety of problems with many of the studies audited. For example, we observed:

    - poorly designed questionnaires;
    - many studies using data which were unreliable, incomplete and/or lacking in objectivity;
    - many studies inadequately specifying the sample design; and
    - samples which were too small for the intended use or biased.
3.113 Most of these methodological problems could be traced directly to planning failures in the assessments. The most notable problems occurred as a result of failure to give sufficient consideration to the design of the study and its relationship to the questions to be answered. Even in those assessments where sufficient planning was done to assess the practicability of the evaluation study options presented, the work often fell short of the level of planning required to carry out a study on the selected option. Further, in some cases, the deputy's committee had designed its own option for which no detailed plan was developed. As a result, before the study was begun, further work would often be required to ensure that the designs selected would achieve what was required of them.

3.114 The impact of these problems was that approximately half the evaluations that attempted to assess the effectiveness of the program were unable to properly attribute the outcomes to the program's activities.

3.115 Before beginning an evaluation study, departments should fully specify the study design for the selected evaluation option.

3.116 To eliminate unnecessary information collection, minimize response burden, and ensure that professional and technical design standards are met, Government policy on information management requires that all new or substantially modified collections of information be approved by Treasury Board and registered in the Inventory of Federal Banks of Information before any field work starts. As part of this process, all proposals for such collections are required to be reviewed by Statistics Canada against information collection standards prior to approval and registration.

3.117 We found that 28 of the 43 studies examined required review, approval and registration under this government policy. However, fewer than half of those required to comply did so.

3.118 Departments should, as required, ensure that all proposals for new or substantially modified data collections, forming part of an evaluation study, are assessed by Statistics Canada and approved and registered by Treasury Board.

3.119 In addition to problems related to collecting primary data for evaluation, we found that approximately 80 per cent of the studies relied to a significant degree on administrative data. In doing so, many of these studies encountered problems because these data were incomplete, inaccurate, or poorly stored. Often data which would have been of great value in assessing program effectiveness and which could have been gathered at small marginal cost were not being gathered. Such data often represent the only practical way to assess with any precision the impacts which a program has had on its clients. Accordingly, these data bases should be given close attention well before they will be needed for program evaluation purposes. Additionally, if these data bases are well developed, the information in them will often be of great value to the line managers responsible for running the program. Where appropriate, evaluation frameworks could serve as an important means of identifying areas where such data can be developed.

3.120 If the evaluation function is to develop, it will require staff who have proper qualifications and experience. This includes persons knowledgeable about government procedures and trained in program evaluation. This combination is rare. We observed a number of evaluation units where there is an insufficient number of staff with program evaluation knowledge and skills.

3.121 The Office of the Comptroller General, in conjunction with departments and agencies, should develop and implement a plan to ensure an adequate and continuing supply of qualified evaluators.

3.122 Departments should take steps to fill available positions with more qualified program evaluators, and enable staff to take the training necessary to upgrade their skills to a reasonable level.

Report Quality
3.123 Although the design and conduct of evaluations may involve constraints and compromise in the methods used, the evaluation study report should be complete and accurate in describing the study implementation, limitations and results. Given the developing nature of the evaluation function, evaluation reports should fully document procedures so that future efforts can build on the experience and the information obtained.

3.124 Most of the study reports attempted to explain how the problems and limitations of the study affected the findings and the conclusions to be drawn from them. However, some failed to do so in a way which informed readers of significant qualifications on the results. In a small number of the studies, the findings were not presented in a sufficiently accurate and balanced way.

3.125 Where study reports do not present balanced and accurate findings, with the limitations discussed and conclusions appropriately qualified, there is the danger that the information presented may be misleading to management.

3.126 Departmental evaluation units should ensure that evaluation reports fully present study findings in a balanced manner and discuss the implications of limitations in the methods used for the interpretation of the findings.

Reporting and Using Evaluations
3.127 Departmental reporting. Deputy heads are the principal client for evaluations, and evaluation studies should be reported to them. Program managers, themselves decision makers, should also be provided with evaluation reports.

3.128 We found that, in almost all instances, evaluation reporting has followed procedures set out in the departmental evaluation policy. All departmental policies require reporting to the deputy, and nearly all require reporting to other responsible levels of program management. Instances of failure to report evaluations have been infrequent, and steps have been taken to ensure appropriate reporting.

3.129 Annual reports. Departmental annual reports serve to communicate the nature of departmental activities to Parliament and, more broadly, the public.

3.130 In its review of the 1978 audit, the Public Accounts Committee recommended to the government that "annual reports of departments contain non-technical summaries of effectiveness evaluations to highlight the impact of government programs."

3.131 We found only two departments whose annual reports had summaries of completed evaluations. Even in these cases, however, only the focus of the studies was described, not the findings. Several other departmental annual reports have mentioned that particular studies had been completed or were under way.

3.132 Other public reporting. The Public Accounts Committee also recommended that:

technical reports of effectiveness evaluations be available for critical review and comment and in particular, your Committee encourages review in learned journals in order to focus the informed commentary of the academic community on the technology of evaluation research.
3.133 Only in a few departments did we find a concerted effort to make evaluations available to the public. One department maintains an evaluation resource centre of completed studies and evaluation literature for use by persons both inside and outside the department and has distributed a large number of its evaluations on request. Four others have published information on some of their evaluations in a Statistics Canada journal of government survey activity.

3.134 The situation varied in the remaining departments. Some claimed that evaluations were available to the public on request. In most of these cases, there were few, if any, known requests, and the departments had done little to make known the availability of particular studies. Other departments maintained that these documents are for use only by the department.

3.135 The government should ensure that annual reports of departments contain non-technical summaries of program evaluations.

3.136 Strategic Overviews. The Policy and Expenditure Management System emphasizes Cabinet committees' decision-making and their role as evaluation users. The concerns of Cabinet committees are to be taken into account in planning evaluations. A summary of the findings of program evaluations and the changes proposed as a result of these findings are to be reported, where appropriate, to central agencies and Cabinet committees by means of the annual Strategic Overview. This reporting is meant to indicate briefly what the department considers significant about its evaluations and the actions taken or to be taken in light of them as they relate to the themes indicated in the Overview.

3.137 The Strategic Overview serves as a direct communication from the departmental minister, over whose signature the document is issued, to colleagues within the Cabinet committees. This type of reporting also helps forge a link between the evaluation function and the departmental strategic planning function. With this link, strategic planning can become an important user of evaluations.

3.138 We found that few of the audited departments reported findings of any evaluations in Strategic Overviews.

3.139 Departments should report in Strategic Overviews, when appropriate, the findings of evaluations and the changes proposed as a result of them.

3.140 Use of evaluations. In discussing the use made of evaluation studies, it is important to recognize the difficulty involved in determining the extent to which program decisions were influenced by the results of an evaluation study. A program evaluation finding or recommendation is seldom the sole factor influencing decisions.

3.141 We reviewed whether recommendations were made as a result of evaluations, whether they were considered for approval by deputies, and whether implementation plans were developed for approved recommendations. We also interviewed several deputy ministers about their views on the usefulness of information produced by evaluations.

3.142 The deputy ministers reported that they had found most of their evaluations to be useful, although they said that some had been more useful than others. Given the newness of program evaluation in most of their departments, the majority believed that it was too early to determine how cost effective program evaluation could eventually be.

3.143 In all but three of the studies we audited, recommendations had been made in the reports. In almost all cases, a decision was made by senior management to accept some or all the recommendations in completed evaluation studies. Where this was not done, it was usually because the evaluation was too recent for decisions to have been made.

3.144 In approximately one-half of the evaluations in which some or all recommendations were accepted, the department had prepared an implementation or action plan.

3.145 The recommendations coming out of the 43 audited evaluations fall into four broad categories which sometimes overlap. They involve changes in program size, program design and support operations, and the undertaking of further effectiveness studies, or developing effectiveness monitoring systems.

    - Changes in program size. Most of the evaluations did not recommend a change in program size. Only six recommended expanding the program, and three recommended reducing the program.
    - Changes in program design. Almost all studies made some recommendations to change aspects of program design. Among these recommended changes were those having to do with adding new program aspects, altering existing features and shifting emphasis from one part of the program to another.
    - Changes in support operations. Nearly all evaluations made recommendations for changes in support activities related to the program, such as changes in staffing levels or organization.
    - Further effectiveness information. Over one-third of the studies recommended that additional data on program effectiveness be gathered either through further studies or ongoing monitoring systems.

Government-wide Program Evaluation Issues

3.146 As noted previously we have used the program evaluation guidelines issued by the Office of the Comptroller General as a basis for expanding and elaborating on the audit criteria developed by our Office for our 1978 study. The Office of the Comptroller General has published these detailed guidelines for organizing, planning and conducting evaluations to give direction to departments in establishing a program evaluation function. In our opinion, this policy framework represents a reasonable standard for departments to strive to meet.

3.147 The implementation strategy followed by the Office of the Comptroller General involved an initial emphasis on establishing evaluation units in the major departments and agencies and on encouraging these departments to conduct evaluations, even of a limited nature. In our opinion, this has been a reasonable way in which to proceed.

3.148 As we observed earlier, the Office of the Comptroller General is becoming increasingly concerned with the focus and quality of the evaluation studies being carried out by departments and agencies. To this end, it issued the Principles for the Evaluation of Programs by Federal Departments and Agencies to give guidance to departments with respect to what would be required in a satisfactory program evaluation initiative - assessment and study. These represent important benchmarks against which the quality of an initiative can be assessed.

3.149 The responsibility for monitoring, on behalf of the Government, the extent to which the studies carried out by departments represent work of adequate quality and objectivity is assigned to the Office of the Comptroller General. However, in the early stages attention was focused primarily on the establishment of program evaluation functions and encouraging departments to attempt evaluations. Recently, the Office of the Comptroller General has been involved in more extensively challenging the information being produced by program evaluations. If program evaluation is to continue to develop, a strong central monitoring and quality assurance function will be required, both to support departmental efforts as well as to ensure the integrity of program evaluation information.

3.150 The Office of the Comptroller General should further develop the central monitoring and quality assurance function for program evaluation.

Interdepartmental Evaluation
3.151 The current policy framework for program evaluation has the department and the deputy head as the central focus of program evaluation activity. However, many important programs are delivered in a way that involves more than one department. Further, the basic structure of the Program and Expenditure Management System stresses the interdepartmental nature of program decision making.

3.152 We identified a minimum of 17 interdepartmental programs in the 19 departments included in the audit. It is evident from our review that some interdepartmental programs involve large expenditures and some are of broad public concern. For example, the Industrial and Labour Adjustment Program, which involves CEIC, Labour and DRIE, has been allocated $450 million to be spent from 1981-82 to 1984-85. The parole program, in addition to having a significant impact on the lives of inmates of Canadian prisons, also has important implications for the well-being of the general public. Other interdepartmental programs identified include CHIP (Canadian Home Insulation Program), involving the Canadian Mortgage and Housing Corporation and Energy, Mines and Resources; National Search and Rescue, involving the Departments of Transport, National Defence and Fisheries and Oceans; and Recruitment and Selection of Immigrants, involving Employment and Immigration and External Affairs.

3.153 Although present policy and guidelines recognize that interdepartmental programs exist, they fail to specify procedures to be followed in conducting evaluations of them.

3.154 Although a few initiatives to evaluate such programs were being taken, the practice was not widespread. In most departments, interdepartmental aspects of programs were not recognized in either the component profiles or the plans for evaluation. In many cases, when asked about this, heads of program evaluation units expressed concern or uncertainty as to how they should proceed and what their responsibilities were with respect to such programs.

3.155 A key problem in interdepartmental evaluations lies in identifying the position or committee accountable for the overall management or co-ordination of the program, the party who would then appropriately be the client for the study. An example of this can be found in looking at the implementation of the Parole Act.

3.156 The objectives of the parole program are found in the Parole Act (Section 10(1)): "(ii) the reform and rehabilitation of the inmate will be aided by the grant of parole, and (iii) the release of an inmate on parole would not constitute an undue risk to society." The program to accomplish these objectives is delivered primarily by the National Parole Board and The Correctional Service of Canada.

3.157 As part of our 1978 audit of the National Parole Board, we reported that an effectiveness evaluation of the federal parole system had not been conducted. We have followed up on progress in evaluating the parole program. Consistent with their areas of responsibility, both the National Parole Board and The Correctional Service of Canada still plan to evaluate only their respective intra-agency activities related to the parole program.

3.158 Evaluation approaches adopted by these two agencies, although they may be appropriate to their individual needs, do not seem to address the relationships of these particular activities to accountability for the overall effectiveness of the program. The legislative mandate for the parole program does not specifically identify who is to be held accountable for achieving these overall objectives or for evaluating the program. As of 31 March 1983, we found that no evaluation of the parole program had been conducted, although a related study of the program has been carried out by the Secretariat of the Department of the Solicitor General.

3.159 The Solicitor General Secretariat is responsible for co-ordinating policy development with the agencies as well as for improving policies and programs in the criminal justice system. The Secretariat has, in the past, conducted studies and evaluations on "programs" and legislation which cut across many organizations. Given the inter-agency nature of its responsibilities, the Secretariat may be the appropriate party to conduct or co-ordinate the evaluation of the overall effectiveness of the parole program, with the agencies continuing to evaluate their respective areas of responsibility.

3.160 The consequence of problems of this type is that interdepartmental programs are not systematically being subjected to the same type of orderly review and evaluation as those programs administered wholly within single departments and agencies. As these programs are often very major ones, this means that important activities risk not being evaluated.

3.161 The Government should ensure that departments and agencies identify the interdepartmental dimensions of their programs in their component profiles. The implications of these interdepartmental dimensions, both for the effective operation and the evaluation of the program, should be examined in assessments and frameworks.

3.162 The Government, in creating or renewing interdepartmental programs, should identify an organization responsible for evaluating the program. In doing this, the Government should identify who is accountable for the overall management of the program and, therefore, who is the client for the program evaluation.

3.163 With respect to existing interdepartmental programs, the Government should establish improved procedures to ensure that these programs are identified and periodically evaluated, where it is reasonable and appropriate to do so.

Small Agencies
3.164 The current policy framework requires that all departments and agencies establish a full evaluation infrastructure. However, in smaller agencies, it may not be cost-effective to establish an internal capability to plan and carry out program evaluations.

3.165 As noted earlier, we found that most of the agencies which had not developed an infrastructure for evaluation were among the smaller agencies. As part of its implementation strategy, the Office of the Comptroller General has only recently given full attention to the place of program evaluation in these agencies.

3.166 The Government should determine appropriate ways to develop a program evaluation capability for smaller entities.

Crown Corporations
3.167 One of the vehicles which the Government employs to achieve its public policy objectives is the Crown corporation. In Chapter 2 of our 1982 Report, we stated that although the corporate form, like tax expenditures, grants and contributions, or direct departmental intervention represents a means to achieve public policy objectives, the implications for the Government and Parliament in terms of direction, control and accountability are quite different.

3.168 Within the 19 departments and agencies we audited, we observed instances in which Crown corporations were being used by the Government to assist the departments to achieve their program objectives, and where funds flowed to these corporations through the departments' Estimates. In these circumstances, we would have expected that the departments would have made plans to evaluate the effectiveness of these expenditures. We found that the departments had not planned to evaluate the effectiveness of this public policy vehicle in their program evaluation plans.

3.169 Information maintained by the Comptroller General shows that, as of 31 March 1983, five Crown corporations were developing a program evaluation function within the Government policy framework in order to evaluate the achievement of their public policy objectives. We have been informed that, in four of the five cases, the client for the evaluation function in these corporations will be the chief operating officer of the corporation.

3.170 Last year, in paragraph 2.116, we recommended that:

Crown-owned corporations should report to Government on their plans, performance and achievement of goals. Government should set out such reporting requirements and ensure that reports are prepared as and when required. Parliament should be made aware of the reporting requirements and should have the opportunity to evaluate periodically these reports and consider them when considering changes in mandate or objectives of the corporation and relevant legislative, expenditure and other government proposals.
3.171 The Government's policy framework on program evaluation represents a reasonable approach to the evaluation of the performance of Crown corporations. One of the important first steps in implementing this would be to identify and clarify the public policy objectives of these corporations.

3.172 The Government should establish procedures to ensure that the public policy objectives of Crown corporations are evaluated and that the results are reported, when appropriate, to the Government and Parliament.

Reporting to Parliament
3.173 The revised Estimates are designed to provide Parliament with better information on the operations of government programs, including performance information on results. The specific medium for this information is the Program Expenditure Plan or Part III of the Estimates. In our view, Part III would be an appropriate place to present to Parliament the findings of program evaluations.

3.174 The Office of the Comptroller General should ensure that Part III of the Estimates refers to and incorporates the findings of evaluation studies that are pertinent to program performance and resource management.

Parliamentary Tabling
3.175 In its review of the 1978 Auditor General's Report, the Public Accounts Committee recommended to the House that all effectiveness evaluations be tabled in the House of Commons within 60 days of their completion. The government responded, through the President of the Treasury Board, that:

With enactment of the Access to Information Bill as currently proposed, and approval of a draft Treasury Board policy on the documentation of program evaluation studies, such information would be publicly available. Some procedure, perhaps tabling of evaluation reports within 60 to 90 days of their completion by the responsible Minister, will have to be established to ensure easy access by Members of Parliament.
3.176 On the basis of our audit work, we found that one program evaluation report has been tabled in the House.

3.177 The Access to Information Act was proclaimed on 1 July 1983.

3.178 The Government should establish procedures to ensure that the recommendation of the Public Accounts Committee with respect to tabling program evaluation reports is acted on.