2013 Spring Report of the Auditor General of Canada Chapter 1—Status Report on Evaluating the Effectiveness of Programs

2013 Spring Report of the Auditor General of Canada

Chapter 1—Status Report on Evaluating the Effectiveness of Programs

Main Points

Introduction

What we found in our 2009 audit
Focus of the audit

Observations and Recommendations

Meeting evaluation requirements

Two departments did not evaluate all ongoing grant and contribution programs, as required
Departments are making progress in evaluating other departmental programs
A third of evaluation reports are incomplete
Gaps remain in how the Treasury Board of Canada Secretariat monitors evaluation coverage in the departments
While progress has been made, the Treasury Board of Canada Secretariat still needs to provide more guidance to departments
Departments have major concerns about the evaluation requirements

Generating ongoing performance information

Departments are making progress in generating ongoing performance information
The Treasury Board of Canada Secretariat provided adequate support to help departments generate ongoing performance information
Weaknesses in ongoing performance measurement continue to limit evaluation

Using evaluation findings

Departments use evaluation findings to support improvement
Evaluation findings are not always available to support spending decisions

Improving the evaluation function

Departments have established evaluation committees
Departments obtain feedback from evaluation users
Capacity in departments has remained stable or increased
The Treasury Board of Canada Secretariat has increased its capacity

Conclusion

Summary of progress

About the Audit

Appendix—List of recommendations

Exhibits:

1.1—Lack of ongoing performance data affected evaluators’ ability to assess the performance of Canada’s maritime search and rescue program

1.2—Human Resources and Skills Development Canada considered evaluation when seeking funding for a new program

 

Performance audit reports

This report presents the results of a performance audit conducted by the Office of the Auditor General of Canada under the authority of the Auditor General Act.

A performance audit is an independent, objective, and systematic assessment of how well government is managing its activities, responsibilities, and resources. Audit topics are selected based on their significance. While the Office may comment on policy implementation in a performance audit, it does not comment on the merits of a policy.

Performance audits are planned, performed, and reported in accordance with professional auditing standards and Office policies. They are conducted by qualified auditors who

Performance audits contribute to a public service that is ethical and effective and a government that is accountable to Parliament and Canadians.

Main Points

What we examined

Evaluation is defined by the Treasury Board as “the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver programs or to achieve the same results.”

Federal departments have been required to evaluate their programs since the 1970s. In 2009, the Treasury Board issued a new Policy on Evaluation to strengthen the evaluation function in departments and agencies. Also in 2009, this Office issued a report that raised a number of concerns identified in our audit of evaluation practices in six departments and the leadership role played by the Treasury Board of Canada Secretariat. Our audit observations and recommendations addressed three issues in particular: the availability of ongoing performance information to support evaluation, departmental capacity to meet established requirements for evaluation, and use of evaluation findings and recommendations to support program improvement and expenditure management.

In this chapter, we looked at progress made by Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, Human Resources and Skills Development Canada, and the Treasury Board of Canada Secretariat in implementing our 2009 recommendations. We looked at how the departments responded to the expanded requirements of the 2009 Policy on Evaluation as they relate to evaluation coverage of programs and to ongoing performance measurement, as well as how the Treasury Board of Canada Secretariat supported departments to meet these new requirements.

Full implementation of the 2009 Policy is subject to a transition period that ends 31 March 2013. This audit was scheduled to provide assurance that implementation of the new requirements was being well managed during this period. We are also responding to a request from the House of Commons Standing Committee on Public Accounts to follow up on our 2009 audit within five years.

Audit work for this chapter was completed on 5 November 2012. More details on the conduct of the audit are in About the Audit at the end of this chapter.

Why it’s important

Governments are under continual pressure to spend money on a range of programs designed to serve particular needs of society. Many factors affect the decisions that governments must ultimately make about programs. Evaluations can aid their decision making by providing objective and reliable information that helps identify programs or components of programs working as intended, those no longer needed, and those not accomplishing the desired objectives. In addition, evaluations can help departments inform Parliament and taxpayers about the results they are delivering for Canadians.

What we found

The entities have responded. The entities agree with our recommendations. Their detailed responses follow the recommendations throughout the chapter.

Introduction

1.1 Federal departments have been required to evaluate their programs since the 1970s. The Treasury Board issued its first program evaluation policy in 1977 and has since modified that policy in 1991, 1994, 2001, and 2009. The Treasury Board defines program evaluation as “the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results.”

1.2 The Office of the Auditor General has conducted six audits of the federal government’s program evaluation function since 1978. Taken together, the Office’s findings in audits reported prior to 2009 identified the poor quality and insufficient impact of evaluations as ongoing concerns.

1.3 In 2009, the Treasury Board issued a new Policy on Evaluation to strengthen the evaluation function in departments and agencies. This policy was designed to ensure that departments create and use a comprehensive and reliable base of evaluation evidence to support their policy and expenditure management decisions, as well as to improve programs. Among other things, the new policy required departments to evaluate all direct program spending every five years. Direct program spending includes spending on ongoing programs of grants and contributions and other departmental programs. (It excludes public debt charges and major transfers to persons and to other levels of government.) These expanded requirements, along with the 2006 amendment to the Financial Administration Act requiring evaluation of all ongoing grant and contribution programs every five years, posed a significant challenge to departments’ evaluation capacity (both funding and staff).

1.4 The Directive on the Evaluation Function, which accompanies the 2009 Policy on Evaluation, expanded and strengthened requirements that program managers collect ongoing performance information for all programs, to support both management of these programs and periodic formal evaluation of their effectiveness.

What we found in our 2009 audit

1.5 In the fall of 2009, we reported our findings in Chapter 1, Evaluating the Effectiveness of Programs. That audit examined how evaluation units in six departments identified and responded to the various needs for effectiveness evaluations. The audit also looked at whether evaluation units had built the required capacity to respond to those needs. In addition, the audit looked at the Treasury Board of Canada Secretariat’s oversight and support role in monitoring and improving the evaluation function in the federal government. Because the audit period for that examination came before the introduction of the 2009 Policy, the audit did not assess compliance with the new policy. It did, however, consider the anticipated requirements of that policy.

1.6 The 2009 audit raised concerns about how evaluations were being conducted in the departments, and about the leadership role of the Secretariat. These observations led to six recommendations, all of which the entities accepted. The audit observations and recommendations addressed three issues in particular:

Related observations and recommendations addressed the Treasury Board of Canada Secretariat’s support to departments.

1.7 The 2009 audit also observed that implementing the (then) new requirement to evaluate all direct program spending would pose serious challenges. In 2009, departmental officials told us they were concerned about their capacity to respond to this new requirement. The 2009 audit also noted a shortage of experienced evaluators, as well as extensive use of contractors to complete evaluations.

1.8 We expressed the view in the 2009 audit that it would be important for departments to evaluate programs that were susceptible to significant change because of shifting priorities and circumstances, as these were the programs where departments could best use evaluation findings.

1.9 The House of Commons Standing Committee on Public Accounts held a hearing on the 2009 audit in February 2010, and issued a related report in September 2010. That report called for action plans in response to some of the recommendations, and included a request for the Office of the Auditor General to conduct a follow-up audit within five years. This current audit responds to that request.

Focus of the audit

1.10 The focus of our audit was to determine whether selected departments and the Treasury Board of Canada Secretariat have made satisfactory progress in implementing the recommendations in our 2009 audit of the evaluation function. To make this determination, we examined whether the departments

1.11 As part of our examination, we also looked at related support the Secretariat provided to improve the evaluation function.

1.12 The audit covered the three-year period from the 2009–10 fiscal year to the 2011–12 fiscal year. Focusing on this period allowed us to examine the first three years of implementation of the new Policy on Evaluation. Our examination of the evaluation of ongoing programs of grants and contributions covered the five-calendar-year period 2007 to 2011, as required by the Financial Administration Act following the introduction of this requirement in December 2006. We carried out detailed examination work in the Treasury Board of Canada Secretariat and in three of the six departments audited in 2009:

1.13 The audit did not examine the quality of evaluations, or whether evaluation findings were used to support strategic reviews of expenditures. More details about the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Meeting evaluation requirements

1.14 Legislation and government policies have established several critical requirements for evaluation.

Two departments did not evaluate all ongoing grant and contribution programs, as required

1.15 The Financial Administration Act requires departments to evaluate all ongoing grant and contribution programs every five years. We examined whether the audited entities met this requirement for the five-calendar-year period 2007 to 2011.

1.16 We found that the departments evaluated the following numbers of their ongoing grant and contribution programs during this five-year period:

1.17 By not evaluating all ongoing grant and contribution programs, Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada are not complying with the Financial Administration Act. Further, the departments may have funded programs that were not as effective or efficient as planned.

1.18 Recommendation. Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada should evaluate all of their ongoing grant and contribution programs every five years, as the Financial Administration Act requires.

Agriculture and Agri-Food Canada’s response. Agreed. Agriculture and Agri-Food Canada will increase its efforts to evaluate ongoing grant and contribution programs. The Department’s Five Year Evaluation Plan (2013–14 to 2017–18) will reflect full coverage of the Department’s ongoing grant and contribution programs as required by the Treasury Board Policy on Evaluation and the Financial Administration Act.

The Office of the Auditor General’s calculation of coverage does not include several departmental evaluation reports on the basis that they do not sufficiently demonstrate the relevance and effectiveness of each ongoing grant and contribution program or other departmental program. Going forward, the Department will ensure that all future evaluations sufficiently demonstrate that they address the key issues of relevance and effectiveness, in accordance with the Financial Administration Act and the five core effectiveness issues identified in the 2009 Directive on the Evaluation Function.

The Department will continue to work closely with the Treasury Board of Canada Secretariat’s Centre of Excellence for Evaluation and other federal evaluation units to identify the best approach for determining the level of effort required to ensure that evaluations meet the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation. The implementation date is June 2013.

Human Resources and Skills Development Canada’s response. Agreed. The Department will work collaboratively with the Treasury Board of Canada Secretariat’s Centre of Excellence for Evaluation and other federal evaluation units in addressing the challenges that departments face in meeting the requirement to evaluate ongoing grant and contribution programs every five years. Further, as part of the Department’s 2013–14 to 2017–18 Evaluation Plan, the Department will include an analysis of all current ongoing grant and contribution programs in the Department, and ensure that the evaluation schedule complies with the requirements of the Financial Administration Act. The level of effort, scope, and focus of evaluations will be determined in accordance with program materiality and risk.

Departments are making progress in evaluating other departmental programs

1.19 The 2009 Policy on Evaluation requires departments to evaluate all of their other departmental programs every five years. We examined the extent to which the audited entities were making progress in meeting this requirement. The requirement was subject to a four-year transition period that ends 31 March 2013.

1.20 Departments were not required to evaluate all other departmental programs until the five-year period starting 1 April 2013. However, they were required to demonstrate progress toward meeting this requirement or to include in their departmental evaluation plan a written, risk-based rationale to explain the department’s evaluation coverage choices. We examined the audited entities’ progress against this requirement during the three-year audit period, from the 2009–10 fiscal year to the 2011–12 fiscal year.

1.21 We found that the departments evaluated the following numbers of their other departmental programs during the audit period:

1.22 In our view, both Fisheries and Oceans Canada and Human Resources and Skills Development Canada are making satisfactory progress in evaluating their other departmental programs. However, with only one year remaining in the transition period, by evaluating an annual average of six percent of other departmental programs, Agriculture and Agri-Food Canada is at risk of not making satisfactory progress in this area. To meet the policy requirement to evaluate all direct program spending every five years starting 1 April 2013, all departments will have to evaluate, on average, 20 percent of their direct program spending per year.

1.23 The departments’ concerns about the challenges they face in evaluating all direct program spending are discussed in paragraphs 1.48–1.50.

A third of evaluation reports are incomplete

1.24 The 2009 Directive on the Evaluation Function requires evaluations to address five core issues if they are to be counted against the requirement of the 2009 Policy on Evaluation to evaluate all direct program spending. We examined all 20 evaluation reports that the three departments approved in the 2011–12 fiscal year to see whether they were complete, according to the Directive. We focused on these most recent reports because the departments most likely designed, carried out, and reported on these evaluations under the 2009 Policy.

1.25 We found that 11 of the 20 reports did not address all five core issues and therefore, according to the definition in the 2009 Policy, were incomplete. The core issues most often missed in the reports were whether the evaluated program

1.26 Departmental officials told us that six of these reports did not address all five core issues because these evaluations were designed (some as long ago as 2005) under the previous evaluation policy, which did not have the same completeness requirements. When we restricted our analysis to the 13 evaluations started since the 2009 Policy took effect, we found that 4 of 13 (31 percent) were incomplete. In our calculation of evaluation coverage, we included evaluations that did not address all five core issues because they were started before the 2009 Policy was implemented. In its reporting of evaluation coverage to date, the Treasury Board of Canada Secretariat followed the same approach.

1.27 Under the 2009 Policy, departments exercise flexibility in their choice of evaluation approaches in order to use evaluation resources in a more cost-effective manner. Departments are still required to meet all the Policy and Directive requirements, including completeness.

Gaps remain in how the Treasury Board of Canada Secretariat monitors evaluation coverage in the departments

1.28 The Treasury Board of Canada Secretariat is responsible for monitoring departments’ compliance with the 2009 Policy on Evaluation. At the departmental level, the Secretariat reports annually on its assessment of the evaluation function through the Management Accountability Framework. According to the Secretariat, the Management Accountability Framework is a performance management tool the federal government uses to improve accountability and management practices across departments and agencies in 15 areas of management, one of which is evaluation. For the 2009–10 through 2011–12 fiscal years, the Secretariat assessed four aspects of evaluation in departments, including the extent to which their evaluations covered direct program spending as the Financial Administration Act and the Policy on Evaluation require.

1.29 At the government-wide level, the Secretariat reports on its evaluation monitoring activities through an annual report on the evaluation function. This report does not provide data on individual departments. The first report was completed in 2011 for the 2009–10 fiscal year, and the second report was released in October 2012 for the 2010–11 fiscal year. The information on evaluation coverage, which is analyzed for the purposes of both the Secretariat’s annual report and the Management Accountability Framework assessments, is provided annually by departments. It consists primarily of departmental responses to a structured survey, as well as evaluation reports and departmental evaluation plans.

1.30 We examined how the Secretariat monitored the audited departments in implementing the 2009 Policy. We also examined the Secretariat’s Management Accountability Framework ratings for evaluation coverage of direct program spending by the three audited departments for the three-year period covered by the audit. We did not examine the ratings for the other aspects of evaluation.

1.31 We compared our findings with the ratings of evaluation coverage provided under the Management Accountability Framework for the three audited departments. We examined the extent to which the audited entities completed evaluations of all of their ongoing programs of grants and contributions as required by the Financial Administration Act, as well as of their other departmental programs as required by the Policy on Evaluation. As noted earlier, full implementation of the 2009 Policy is subject to a transition period that ends 31 March 2013. Our analysis compared the number of programs evaluated by the departments with the total number of programs subject to each requirement. We found that the Secretariat, in its analysis, compares estimated expenditures of evaluated programs to total direct program spending. Our comparison of these two approaches yielded reasonably similar results. Since departments are required to evaluate 100 percent of these programs, either approach should provide accurate information on evaluation coverage.

1.32 We found that the Management Accountability Framework ratings for evaluation coverage of direct program spending by the three departments were more positive than our audit findings. In particular, we found that Agriculture and Agri-Food Canada had the lowest evaluation coverage of both ongoing grant and contribution programs and other departmental programs, although the Secretariat gave it the highest possible rating for the 2010–11 and 2011–12 fiscal years.

1.33 Furthermore, the Secretariat’s ratings did not reflect non-compliance by Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada with the Financial Administration Act requirement to evaluate all ongoing programs of grants and contributions. Officials told us that the Secretariat’s rating was based not only on completed evaluations but also on planned evaluations for the period of the assessment. In our opinion, evaluations should only be counted for coverage once they are completed, because planned evaluations can be delayed or cancelled. If the coverage requirements assessed through the Management Accountability Framework are to serve the purpose of monitoring this element of evaluation in departments, the methodology needs to generate more accurate and reliable departmental ratings.

1.34 In 2009, we recommended that the Secretariat monitor the implementation of the new policy requirements. While the Secretariat gathers and analyzes key information on evaluation coverage, the resulting Management Accountability Framework ratings do not accurately reflect evaluation coverage. Thus, we conclude that the Secretariat’s monitoring of evaluation coverage in departments is unsatisfactory.

1.35 Recommendation. The Treasury Board of Canada Secretariat should review the tools it uses to monitor evaluation coverage to ensure that they provide an accurate perspective on departmental progress that can be considered in the Management Accountability Framework assessment process.

The Treasury Board of Canada Secretariat’s response. Agreed. The Treasury Board of Canada Secretariat continuously strives to improve the monitoring tools that support its responsibilities under the 2009 Policy on Evaluation for providing functional leadership for evaluation across government, including reporting on the health of the evaluation function across government. While the Management Accountability Framework addresses evaluation coverage, the Secretariat does not use Management Accountability Framework ratings on their own for the purpose of monitoring coverage.

The Secretariat will review the tools it uses for monitoring evaluation coverage to ensure that they provide an accurate perspective on departmental progress that can be considered in the Management Accountability Framework assessment process. The Secretariat will complete this action by the end of March 2013.

While progress has been made, the Treasury Board of Canada Secretariat still needs to provide more guidance to departments

1.36 According to the 2009 Policy on Evaluation, the Treasury Board of Canada Secretariat is to provide leadership, advice, and guidance to departments in the conduct, use, and advancement of evaluation practices. We examined how the Secretariat supported the audited departments in implementing the 2009 Policy on Evaluation.

1.37 Our 2009 audit recommended that the Treasury Board of Canada Secretariat regularly identify gaps in its evaluation-related support to departments and address these gaps. We found that before the Policy on Evaluation took effect in April 2009, the Secretariat used a consultant to identify any guidance the new policy required and to outline an action plan to address these needs. The Centre of Excellence for Evaluation at the Secretariat used the consultant’s findings as it established its priorities and reflected these findings in its business plans for the 2010–11 and 2011–12 fiscal years.

1.38 Secretariat officials consult regularly with evaluators in the federal government to support their efforts to identify and fill gaps in guidance. As noted in paragraph 1.29, the Secretariat prepares an annual report on the evaluation function and on the support the Secretariat provided to departmental evaluation units.

1.39 The Secretariat has provided evaluation-related guidance to departments in a number of areas, including

1.40 We conclude that the Treasury Board of Canada Secretariat has made satisfactory progress since 2009 in developing guidance and providing support to departments to address identified gaps in meeting evaluation requirements. However, we identified the following three areas where more guidance is needed.

1.41 Tracking evaluation coverage of direct program spending. It is important to have a reliable and consistent approach to tracking evaluation coverage of direct program spending. We examined how the three departments monitored their progress in meeting the coverage requirements and how the Treasury Board of Canada Secretariat supported them.

1.42 We found problems in how the three departments and the Secretariat tracked progress on evaluation coverage. Departmental officials described a number of challenges they encountered in tracking progress, including the following:

1.43 Since 2009, the Secretariat has been aware of the challenges departments have faced in tracking the extent to which they have complied with the requirements to evaluate all direct program spending. Secretariat officials told us that work is under way to refine how they monitor the extent of the evaluation coverage of direct program spending. Because this work is not complete three years after the Policy was implemented, we conclude that the Secretariat has made unsatisfactory progress on our 2009 recommendation that it help departments prepare to implement the new coverage requirements.

1.44 Assessing program efficiency and economy. The departments need further guidance on how to assess program efficiency and economy. Of the five core evaluation issues, departments have identified the fifth issue—demonstration of efficiency and economy—as particularly challenging, because it requires program-level information about both effectiveness and expenditures. This is a long-standing issue. The consultation and action plan that a consultant prepared for the Treasury Board of Canada Secretariat in 2009 also identified this as an issue requiring urgent guidance. Secretariat officials told us they were about to provide guidance on how to address program efficiency and economy.

1.45 Carrying out neutral assessments. The Policy on Evaluation requires departments to conduct “neutral assessments” of their evaluation functions at least every five years, but the Policy is not clear about how to achieve neutrality. Although none of the audited departments have completed a neutral assessment to date, they will be required to do so before the end of the 2013–14 fiscal year. In order for these assessments to be neutral and conducted consistently across departments, the Treasury Board of Canada Secretariat needs to provide guidance about their content and methods. We note that the Treasury Board requires departments to have assessments of internal audit functions conducted by qualified independent reviewers.

1.46 In summary. The Secretariat needs to develop guidance for departments on how to

1.47 Recommendation. The Treasury Board of Canada Secretariat should provide departments with clear guidance about how to track evaluation coverage of direct program spending, how to assess program efficiency and economy, and how to carry out neutral assessments of their evaluation functions to ensure that departments follow a consistent approach.

The Treasury Board of Canada Secretariat’s response. Agreed. In October 2012, the Secretariat issued draft guidance for departments on how to assess program efficiency and economy when conducting evaluations according to the 2009 Policy on Evaluation. The Secretariat plans to post finalized guidance on its website by the end of March 2013.

The Secretariat has begun developing additional guidance to expand upon its Guide to Developing a Departmental Evaluation Plan (released in June 2011), which advises departments on how to determine which programs should be evaluated so that coverage requirements are met and how to reflect evaluation coverage information in their plans. In particular, the additional guidance will support deputy heads in tracking whether they are meeting their accountabilities under the Financial Administration Act (section 42.1) for evaluating all ongoing programs of grants and contributions every five years. Further, the Secretariat is preparing guidance to departments on how to carry out neutral assessments of their evaluation functions. Both guidance efforts are expected to be completed by 31 March 2014.

Departments have major concerns about the evaluation requirements

1.48 In a consultation that the Treasury Board of Canada Secretariat carried out in 2010, a sample of nine deputy heads of departments expressed their concerns about their departments’ capacity to evaluate all direct program spending, as the Policy on Evaluation requires. Many of those consulted (the precise number was not reported) believed that their department could not meet this requirement. The majority expressed concern about the cost of maintaining a five-year cycle. They noted the challenge to their evaluation unit’s capacity posed by these requirements.

1.49 Officials in the three audited departments also expressed concerns during this audit about the requirements for evaluation coverage of all direct program spending and completeness of reports. In particular, they expressed the following concerns:

In our view, these concerns have merit and warrant analysis and consideration by the Secretariat.

1.50 We note that the Policy on Evaluation requires the Secretariat to ensure that the Policy itself is evaluated every five years. It also requires departments to conduct a neutral assessment of their evaluation functions every five years. Information from these two sources could provide a clear picture of how well departments are implementing the Policy, whether departments are meeting the objectives the Treasury Board set for the Policy, and whether departments are putting evaluation resources to the best possible use.

1.51 Recommendation. In evaluating the Policy on Evaluation by 2014, the Treasury Board of Canada Secretariat should, in consultation with departments, review the requirements to evaluate all direct program spending over a five-year cycle and to address all five core issues.

The Treasury Board of Canada Secretariat’s response. Agreed. By renewing the Policy on Evaluation in April 2009, the government strengthened its commitment to evaluating the value for money of federal programs. To allow departments time to expand their evaluation capacity, a gradual four-year implementation period for some coverage requirements was built into the policy. To meet a commitment it made at the time the policy was approved in 2009, the Secretariat launched a policy implementation review in March 2012, including extensive consultations with departments, and will complete this review by the end of the gradual implementation period on 31 March 2013.

Building on the knowledge gained through the policy implementation review, the subsequent evaluation of the 2009 Policy, which is required by the policy itself, will be conducted by the end of the 2013–14 fiscal year and will centre on key questions such as the coverage requirements and focus of evaluations.

Generating ongoing performance information

1.52 Information about ongoing program performance can serve two important purposes. It can help program managers in taking appropriate and timely action to improve the way they manage programs, and it can support periodic program evaluation. Our 2009 audit examined a sample of evaluations and found that inadequate information about ongoing program performance limited many evaluations (17 of 23) in their assessment of program effectiveness. We recommended that departments develop and implement action plans to ensure that they collect ongoing program performance information to support evaluation. The audited departments agreed with this recommendation. We note that in the Treasury Board of Canada Secretariat’s 2010 consultation with deputy heads, departments found it challenging to make their performance measurement frameworks operational.

1.53 The 2009 Directive on the Evaluation Function requires departments to develop and implement performance measurement strategies for their programs and to ensure that ongoing performance information is collected. To assess the progress the three departments made in developing and implementing ongoing performance measures for their programs, we selected a random sample of 54 programs across the three departments. For each program, we examined

For a performance measurement strategy to be considered complete, the Treasury Board of Canada Secretariat states that it should include

1.54 We also examined annual reports about the state of ongoing performance measurement, which the Directive on the Evaluation Function requires heads of evaluation units to prepare for their departmental evaluation committees. In our view, these reports have the potential to play an important role in advancing ongoing performance measurement in departments, by informing senior officials about what has been done and what remains to be done. Our recommendation concerning ongoing performance information is in paragraph 1.63.

Departments are making progress in generating ongoing performance information

1.55 We found that 44 of 54 programs had reasonably complete performance measurement strategies, as defined by the Treasury Board of Canada Secretariat. However, data sources were identified for only 29 of 54 programs, and explicit budgets for collecting performance data were included for only 4 of the 54. It is important that departments state, at the planning stage, what performance information they require. They also need to state how they will collect data, and budget accordingly.

1.56 Our examination of departmental progress in collecting ongoing performance information assessed whether departments were collecting data. We found that 48 of 54 programs were collecting some performance data.

The Treasury Board of Canada Secretariat provided adequate support to help departments generate ongoing performance information

1.57 The 2009 Policy on Evaluation requires the Treasury Board of Canada Secretariat to provide guidance to departments on generating information from ongoing performance measurement. We examined whether the Secretariat supported departments’ efforts in generating this information and found that the Treasury Board and the Secretariat introduced both policy and guidance to support departments’ efforts to generate ongoing performance information, including the following:

Weaknesses in ongoing performance measurement continue to limit evaluation

1.58 According to the Directive on the Evaluation Function, departments must ensure that they implement ongoing performance measurement department-wide, so credible and reliable ongoing performance information is available to effectively support the evaluation of programs.

1.59 Quality assurance is essential if ongoing performance measurement is to provide an accurate and complete picture of ongoing program performance. We examined whether the three departments had processes in place to monitor and assess the quality of data that is collected through ongoing performance measurement activities. We did not assess the quality of the information collected.

1.60 We found that while the audited departments undertake some quality assurance activities, most such activities are informal and departments do not track them. Two of the three departments do not have formal and systematic department-wide quality assurance processes. Furthermore, two of the three departments noted concerns relating to completeness and quality of data in their annual reports on the state of performance measurement.

1.61 As a result, important information about ongoing program performance was unavailable to the evaluators as they assessed program effectiveness, and thus they were required to either rely more on subjective and qualitative information or collect more data than would otherwise have been the case. We looked at the 20 evaluation reports the three departments approved in the 2011–12 fiscal year to determine whether limitations in the availability of ongoing performance information constrained the scope of the evaluations. We found that 14 of the 20 evaluations acknowledged such constraints (Exhibit 1.1).

Exhibit 1.1—Lack of ongoing performance data affected evaluators’ ability to assess the performance of Canada’s maritime search and rescue program

Fisheries and Oceans Canada is responsible, through the Canadian Coast Guard, for the maritime component of the country’s national search and rescue program. Major activities of this program include distress monitoring, communication, and rescue operations in Canadian territorial waters. In the 2010–11 fiscal year, the maritime search and rescue program had a $19.4 million budget.

In the 2011–12 fiscal year, the Department evaluated the program to address the five core issues that the Directive on the Evaluation Function requires. The evaluation reported limited and inconsistent performance data collection. In particular, the evaluation report noted that incidents involving search and rescue are under-reported nationwide in the program database, and the single service standard being tracked did not measure the time required for the rescuers to reach the rescue site.

The Department had raised the same concern in 2007. Because evaluators did not have accurate numbers about search and rescue incidents and lacked ongoing program performance information, they could not fully address the program’s effectiveness and efficiency.

(To read our audit on federal search and rescue activities, see Chapter 7 of this Report.)

1.62 We conclude that although departments have made satisfactory progress since 2009 in generating ongoing performance information, program evaluators have yet to fully benefit from increased availability and quality of ongoing performance data as they assess program effectiveness. This will be an important issue for the Secretariat to consider in its upcoming evaluation of the Policy.

1.63 Recommendation. The audited departments should complete performance measurement strategies for all programs, collect complete performance data, and implement quality assurance processes for ongoing performance information.

Agriculture and Agri-Food Canada’s response. Agreed. Agriculture and Agri-Food Canada’s performance measurement strategies that track and monitor performance have been developed for many departmental programs, including all our new Growing Forward 2 programs. The Department will continue to develop and implement systematic processes to measure the performance of all programs, including data collection and quality assurance processes. Performance measurement will be developed by March 2014 for all departmental programs, using approaches that consider the cost and expected benefits of performance information.

With the launch of the revised Treasury Board Policy on Evaluation in April 2009, the Department now undertakes an annual reporting process on the state of performance measurement to support evaluation.

Since June 2010, the Department has undertaken targeted activities to improve the quality of performance measures, to strengthen accountability for performance measurement, to clarify roles and responsibilities, and to build capacity throughout the Department to support performance monitoring and reporting.

Fisheries and Oceans Canada’s response. Agreed. Regarding the completion of performance measurement strategies, Fisheries and Oceans Canada is nearing completion of its Performance Measurement Action plan concerning the development of performance measurement strategies. Of the total 40 performance measurement strategies for the Department, 17 have been approved by the Chief Financial Officer, 9 are at the final program review and approval stage, and the remaining 14 are nearing completion (note that 7 of these are for internal services and hence not mandatory, but are under development). The implementation date is 30 January 2013.

Regarding the collection of complete performance data, implementation of the approved performance measurement strategies will start in the winter of 2013. The implementation date is 31 March 2014.

Regarding the implementation of quality assurance processes for ongoing performance information, by July 2013 the Evaluation Directorate will develop a risk-based methodology for monitoring the implementation of the performance measurement strategies. By 31 March 2014, the Evaluation Directorate will have started monitoring the implementation of performance measurement strategies, targeting 100 percent of high-risk strategies, 50 percent of medium-risk strategies, and 25 percent of low-risk strategies.

Human Resources and Skills Development Canada’s response. Agreed. Human Resources and Skills Development Canada will undertake an assessment in 2013 to determine the state of performance measurement strategies and data collection systems for all departmental programs, in order to assess the extent and significance of existing gaps and the appropriate corrective measures to be taken.

The Department acknowledges that performance measurement strategies that track and monitor performance have been developed for most departmental programs, but not all. Also, many departmental program areas have developed and used quality assurance processes to monitor and correct problems identified with the quality and completeness of performance measurement data. These practices will be encouraged for adoption by all program areas, where appropriate, based on an assessment of cost of investments in the data systems versus the expected benefit of the performance information.

Using evaluation findings

1.64 The 2009 Treasury Board Policy on Evaluation requires departments to use evaluation findings to inform decisions on program and policy improvement and program spending. Furthermore, our 2009 Fall Report recommended that the Treasury Board of Canada Secretariat address identified gaps in tools and guidance for departments, including any related to use of evaluation findings. In this audit, we examined two specific uses of evaluation findings: implementation of recommendations from evaluations, and reflection of evaluation findings in departments’ submissions to the Treasury Board for funding for new or renewed programs.

Departments use evaluation findings to support improvement

1.65 Recommendations from evaluations can lead to improvements in programs and policy, but only if departments implement them. The 2009 Policy on Evaluation requires deputy heads of departments to approve management responses to evaluation reports and related action plans, with the support of departmental evaluation committees.

1.66 We selected a random sample of completed evaluations across the three departments to determine the status of 99 recommendations and associated action plans in these evaluations. We found that 47 of these 99 recommendations and management responses to them were implemented, 50 were in progress, and 2 were abandoned as obsolete. We also found that all three departments have established structured processes, involving their departmental evaluation committees, to monitor implementation of plans to address recommendations from evaluations. Finally, we found that the Secretariat provided adequate tools and guidance to support departmental use of evaluation findings for policy and program improvement.

Evaluation findings are not always available to support spending decisions

1.67 An important use of evaluation findings, according to the Policy on Evaluation, is to support departmental submissions to the Treasury Board for program funding. We examined a random sample of 32 Treasury Board submissions from the three audited departments for program funding from the 2009–10 to 2011–12 fiscal years. This sample excluded submissions for funding for which we would not expect to find a program evaluation, such as those concerning capital acquisitions or settlement of lawsuits. We found that 10 of the 32 submissions (31 percent) reflected information from completed evaluations.

1.68 The use of evaluation findings we observed in our sample indicates that the departments are not fully realizing the potential benefits of presenting evaluation results in Treasury Board submissions. However, in one instance, Human Resources and Skills Development Canada did consider an evaluation of its International Trade and Labour Program before submitting a funding request for a new program (Exhibit 1.2).

Exhibit 1.2—Human Resources and Skills Development Canada considered evaluation when seeking funding for a new program

The federal government established the International Trade and Labour Program within Human Resources and Skills Development Canada in 2004 to promote respect for international labour rights and standards. The program provided grants and contributions to support domestic and international labour organizations and to help Canada’s trade partners meet international labour standards. Funding for the program totalled $15.1 million from the 2004–05 to the 2010–11 fiscal years.

Human Resources and Skills Development Canada evaluated the program in the 2009–10 fiscal year and issued a report in 2011. The evaluation confirmed the continuing relevance of the program and indicated it was making progress toward its expected outcomes. The evaluation included three recommendations, all of which program managers accepted and addressed in action plans.

In April 2012, the International Trade and Labour stream of the Labour Funding Program replaced the International Trade and Labour Program. Officials told us they took the 2009–10 evaluation of the International Trade and Labour Program into account when they sought Treasury Board funding approval for the new program. We confirmed that the funding submission included references to the International Trade and Labour Program evaluation, and noted reference to an evaluation of another related program. This is a good example of how evaluation findings can support program improvement and a decision on program funding.

Improving the evaluation function

1.69 Our 2009 audit recommended that departments implement systematic processes to identify and improve evaluations, and that the Treasury Board of Canada Secretariat monitor and support departments in this process, including providing tools and guidance as needed. The 2009 Policy on Evaluation requires departments to monitor their own compliance with the policy, and to take remedial actions as necessary. We examined whether such processes were implemented during the audit period, and whether the Treasury Board of Canada Secretariat monitored and supported departments to do so.

Departments have established evaluation committees

1.70 The 2009 Policy on Evaluation introduced a specific requirement that departments establish a departmental evaluation committee, composed of senior officials, to advise the deputy head about all evaluation and evaluation-related activities. The Policy also established the committee’s roles and responsibilities.

1.71 We found that all three departments have established departmental evaluation committees, and that these committees meet, on average, four times per year. We also found that these committees review their departments’ rolling five-year evaluation plans, as well as completed evaluation reports, and follow up on evaluation recommendations and related action plans. This senior-level involvement in evaluation planning, reporting, and use is important if evaluation is to support decision making and expenditure management in departments.

1.72 The 2009 audit also recommended that departments consider the merits of including external experts on their departmental evaluation committees, as is the practice on their audit committees. We also recommended that the Treasury Board of Canada Secretariat provide guidance in this regard.

1.73 We found that the Secretariat has provided guidance about the composition of departmental evaluation committees. The three audited departments have considered the merits of adding external members to their committees, and Agriculture and Agri-Food Canada has decided to do so.

Departments obtain feedback from evaluation users

1.74 Our 2009 audit observed that feedback from evaluation users can lead to improvements in the quality and value of evaluations. In 2009, we found that only one of the six audited departments obtained such feedback. In the current audit, we found that all three departments systematically survey managers of evaluated programs to determine whether the evaluations met their needs.

Capacity in departments has remained stable or increased

1.75 In our 2009 audit, we noted that while evaluation capacity (both funding and staff) in departments had increased from the 2004–05 fiscal year to the 2008–09 fiscal year, the audited departments nevertheless found it challenging to meet evaluation requirements. The 2009 Policy on Evaluation requires departments to ensure that adequate resources are allocated to the evaluation function in order to meet established requirements, including those related to evaluation coverage and completeness.

1.76 In this follow-up audit, we examined the capacity of the three departments, and found it had either remained stable or had increased from the 2009–10 fiscal year to the 2011–12 fiscal year. Although departmental officials told us they have the capacity to meet the requirement to evaluate 100 percent of direct program spending, the combined coverage and completeness requirements limit the extent to which they can put their evaluation resources to best use.

1.77 We noted that the audited departments have taken steps to define the evaluation expertise their staff require. This work was done at the departmental level, with support from the Treasury Board of Canada Secretariat. Departments are also developing the evaluation expertise of their staff.

The Treasury Board of Canada Secretariat has increased its capacity

1.78 The Centre of Excellence for Evaluation within the Treasury Board of Canada Secretariat is responsible for providing leadership, advice, and guidance about the conduct, use, and advancement of evaluation practices across the federal government. In our 2009 audit, we noted the need for sufficient resources and evaluation expertise in the Centre if it is to fulfill this leadership role. We recommended that the Secretariat ensure that it allocates sufficient resources to tasks that require evaluation expertise.

1.79 We examined the resources available to the Centre, and the expertise of its staff. We found that the Treasury Board of Canada Secretariat has demonstrated that the Centre has the capacity to support departments to meet evaluation requirements. The Centre has increased its staff since the 2009 audit, with most staff at the level of senior analysts. Funding for the Centre has been stable from the 2009–10 fiscal year to the 2011–12 fiscal year. In addition, the Centre has defined the evaluation expertise its staff require.

Conclusion

1.80 We concluded that the three audited departments and the Treasury Board of Canada Secretariat have made satisfactory progress in implementing the recommendations from our 2009 audit Evaluating the Effectiveness of Programs.

1.81 The audited departments established ongoing performance measurement strategies and are collecting some performance information for most programs. In addition, they have maintained or increased their capacity to evaluate their programs and are using findings and recommendations from evaluations to improve program performance. They have also established systematic processes to monitor and improve the function.

1.82 Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada did not meet the statutory requirement to evaluate all ongoing programs of grants and contributions during the five-calendar-year period 2007 to 2011. Furthermore, Agriculture and Agri-Food Canada was at risk of not making satisfactory progress against the related requirement in the 2009 Policy on Evaluation to evaluate all direct program spending, starting in 2013. We note that full implementation of the 2009 Policy is subject to a four-year transition period that ends 31 March 2013.

1.83 Departmental officials expressed concerns about Treasury Board requirements to evaluate all programs every five years, and to address the full range of evaluation issues in all evaluations. They indicated that these requirements limit their ability to put their evaluation resources to the best use. The Treasury Board of Canada Secretariat needs to consider these concerns as it carries out the required evaluation of the Policy in the 2013–14 fiscal year.

1.84 Although departments have made progress since 2009 in generating ongoing performance information, program evaluators noted constraints in their ability to address program effectiveness due to limited availability of ongoing performance information. As a result, departments are making decisions about programs and related expenditures with incomplete information about their effectiveness.

1.85 The Treasury Board of Canada Secretariat monitored departmental progress in implementing the 2009 Policy on Evaluation and in collecting ongoing program performance information, and provided guidance and support as needed. The Secretariat also demonstrated that its capacity is adequate to support the evaluation function. It followed a systematic approach to identify gaps in its support to departments to implement the 2009 Policy. However, the Secretariat needs to address gaps in its support related to assessing evaluation coverage, measuring program efficiency and economy, and carrying out neutral assessments of the function in departments.

1.86 We found that the Management Accountability Framework ratings of evaluation coverage for the three audited departments were more positive than our audit findings. Furthermore, the ratings did not reflect that two of the audited departments were at risk of not meeting the statutory requirement to evaluate all ongoing programs of grants and contributions. It is important for the Treasury Board of Canada Secretariat to ensure that these ratings of evaluation coverage are reliable for monitoring and decision making.

1.87 Federal departments and agencies have been required to evaluate their programs for more than 30 years. Implementation of the 2009 Policy on Evaluation has supported improvements in a number of areas. However, significant weaknesses continue to limit the contribution of evaluation to decision making in the government.

Summary of progress

The following table includes our assessment of progress measured against our recommendations in the Fall 2009 Auditor General Report, Chapter 1, Evaluating the Effectiveness of Programs—for the Treasury Board of Canada Secretariat and three of the six departments audited in 2009. Progress is considered either satisfactory or unsatisfactory.

Issue or recommendation from 2009 Progress
1.37 Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should develop and implement action plans to ensure that ongoing program performance information is collected to support effectiveness evaluation. Satisfactory
1.43 Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should consider the merits of including external experts on their departmental evaluation committees. Satisfactory
The Treasury Board of Canada Secretariat should provide guidance to departments in this regard. Satisfactory
1.48 Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada should implement systematic processes to determine whether their effectiveness evaluations are meeting government-wide requirements and internal corporate needs, and act on areas identified for improvement. Satisfactory
The Treasury Board of Canada Secretariat should monitor and provide any additional support it considers necessary for the implementation of these processes. Unsatisfactory
1.82 In developing tools, guidance, and support for departments, the Treasury Board of Canada Secretariat should regularly identify gaps that it needs to act on, develop plans to address these gaps, and act on these plans. Satisfactory
1.88 The Treasury Board of Canada Secretariat should ensure that it allocates sufficient resources to tasks that require evaluation expertise. Satisfactory
1.93 The Treasury Board of Canada Secretariat should help departments prepare to implement the new coverage requirements. During the transition period, the Secretariat should provide advice and guidance for effectiveness evaluation, focusing on programs where such evaluation can be put to best use. Unsatisfactory

Satisfactory—Progress is satisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

Unsatisfactory—Progress is unsatisfactory, given the significance and complexity of the issue, and the time that has elapsed since the recommendation was made.

About the Audit

All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, we also draw upon the standards and practices of other disciplines.

Objectives

The overall objective of the audit is to determine whether the selected departments and the Treasury Board of Canada Secretariat have made satisfactory progress in implementing recommendations from the 2009 chapter on evaluating the effectiveness of programs.

The audit sub-objectives were to determine whether

Scope and approach

Our audit focused on the Treasury Board of Canada Secretariat and three of the six departments audited for the 2009 chapter Evaluating the Effectiveness of Programs. We carried out the examination work using three lines of enquiry. They were designed to assess the departments’ progress in implementing recommendations from the 2009 audit. They also assessed how well the departments were conducting evaluations of ongoing programs of grants and contributions as required by the Financial Administration Act, and how well they responded to the new Policy on Evaluation’s expanded requirements for ongoing performance measurement and for evaluation of all direct program spending.

We examined the extent to which the departments are meeting the expanded requirements in the 2009 Policy in light of capacity constraints noted in the 2009 audit, and with related support that the Treasury Board of Canada Secretariat provides. We also examined progress the departments made in generating and using ongoing performance information to support evaluation, as well as the related support that the Treasury Board of Canada Secretariat provided. Finally, we examined departmental use of findings and recommendations from effectiveness evaluations to support policy and program improvement, as well as expenditure management, and we looked at the related support that the Treasury Board of Canada Secretariat provided. The audit did not examine the quality of evaluations or whether evaluation findings were used to support strategic reviews of expenditures.

Selection of entities. While the 2009 audit focused on six departments and the Treasury Board of Canada Secretariat, this audit focused on three departments—Agriculture and Agri-Food Canada, Fisheries and Oceans Canada, and Human Resources and Skills Development Canada—and the Treasury Board of Canada Secretariat. The main factors we considered in selecting the departments were the proportion of departmental spending on grants and contributions, and the Treasury Board of Canada Secretariat’s Management Accountability Framework ratings concerning the departments’ quality and use of program evaluation.

Use of random samples. This audit used two random samples across the three audited departments:

Both samples were sufficient in size to conclude on the sampled populations with a confidence level of 90 percent and a margin of error of +10 percent.

Criteria

Criteria Sources
To determine whether the selected departments and the Treasury Board of Canada Secretariat have made satisfactory progress in implementing recommendations from the 2009 chapter on evaluating the effectiveness of programs, we used the following criteria:

Departments prepare performance measurement strategies for all programs.

Departments systematically collect ongoing performance information, as planned in their performance measurement strategies.

Departments implement quality assurance processes for ongoing performance information to support effectiveness evaluation.

  • Policy on Management, Resources and Results Structures, Treasury Board, 2008 and 2010
  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies, Treasury Board of Canada Secretariat, 2010
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, Recommendation 1.37 and related departmental responses

Departments use the ongoing performance information collected to support effectiveness evaluation.

  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • Supporting Effective Evaluations: A Guide to Developing Performance Measurement Strategies, Treasury Board of Canada Secretariat, 2010
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, Recommendation 1.37 and related departmental responses

The Treasury Board of Canada Secretariat supports departments in generating information from ongoing performance measurement.

  • Policy on Evaluation, Treasury Board, 2009
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, recommendations 1.82 and 1.93 and related responses of the Secretariat

Departments meet established requirements for evaluation coverage of ongoing grant and contribution programs and other departmental programs, as well as for completeness of evaluation reports.

  • Financial Administration Act
  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • Guide to Developing a Departmental Evaluation Plan, Treasury Board of Canada Secretariat, 2011

Departments demonstrate that sufficient resources (funding and staff) are allocated to meet the coverage and completeness requirements of the 2009 Policy on Evaluation.

  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • Standard on Evaluation for the Government of Canada, Treasury Board, 2009
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, recommendations 1.43 and 1.48 and related departmental responses

The Treasury Board of Canada Secretariat demonstrates that it has the capacity (staff with evaluation expertise and funding) to support departments to meet established requirements for effectiveness evaluation.

  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, Recommendation 1.88 and related response of the Secretariat

The Treasury Board of Canada Secretariat supports departments to meet established requirements for effectiveness evaluation.

  • Policy on Evaluation, Treasury Board, 2009
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, recommendations 1.82 and 1.93 and related responses of the Secretariat

Departments use findings and recommendations from their effectiveness evaluations to support policy and program improvement, as well as expenditure management.

  • Policy on Evaluation, Treasury Board, 2009
  • Directive on the Evaluation Function, Treasury Board
  • Standard on Evaluation for the Government of Canada, Treasury Board, 2009

Departments develop and implement systematic processes to regularly identify gaps and to act on required improvements related to the use of effectiveness evaluations to support policy and program improvement as well as expenditure management.

  • Policy on Evaluation, Treasury Board, 2009
  • Standard on Evaluation for the Government of Canada, Treasury Board, 2009
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, Recommendation 1.48 and related departmental responses

The Treasury Board of Canada Secretariat provides tools and guidance to support departments to use findings and recommendations from effectiveness evaluation to support policy and program improvement, as well as expenditure management.

  • Policy on Evaluation, Treasury Board, 2009
  • 2009 Fall Report of the Auditor General of Canada, Chapter 1, Evaluating the Effectiveness of Programs, Recommendation 1.48 and related response of the Secretariat

The Treasury Board of Canada Secretariat, in developing tools and guidance to departments about the use of evaluation findings and recommendations, regularly identifies gaps it needs to act on, develops plans to address these gaps, and acts on these plans.

Management reviewed and accepted the suitability of the criteria we used in the audit.

Period covered by the audit

The audit covered the period between 1 April 2009 and 31 March 2012. Focusing on this period allowed us to examine the first three years of implementation of the new Policy on Evaluation. Our examination of the evaluation of ongoing programs of grants and contributions covered the five-calendar-year period 2007 to 2011 as required by the Financial Administration Act following the introduction of this requirement in December 2006.

This audit period was selected in response to the request from the House of Commons Standing Committee on Public Accounts for a follow-up audit within five years of the 2009 chapter. We completed audit work for this chapter on 5 November 2012.

Audit team

Assistant Auditor General: Neil Maxwell
Principal: Louise Dubé
Lead Director: Colin Meredith
Director: Dawn Campbell

Irene Andayo-Michalowski
Jeff Graham
Anthony Stock
Diana Thibeault

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 1. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Recommendation Response
Meeting evaluation requirements

1.18 Agriculture and Agri-Food Canada and Human Resources and Skills Development Canada should evaluate all of their ongoing grant and contribution programs every five years, as the Financial Administration Act requires. (1.15–1.17)

Agriculture and Agri-Food Canada’s response. Agreed. Agriculture and Agri-Food Canada will increase its efforts to evaluate ongoing grant and contribution programs. The Department’s Five Year Evaluation Plan (2013–14 to 2017–18) will reflect full coverage of the Department’s ongoing grant and contribution programs as required by the Treasury Board Policy on Evaluation and the Financial Administration Act.

The Office of the Auditor General’s calculation of coverage does not include several departmental evaluation reports on the basis that they do not sufficiently demonstrate the relevance and effectiveness of each ongoing grant and contribution program or other departmental program. Going forward, the Department will ensure that all future evaluations sufficiently demonstrate that they address the key issues of relevance and effectiveness, in accordance with the Financial Administration Act and the five core effectiveness issues identified in the 2009 Directive on the Evaluation Function.

The Department will continue to work closely with the Treasury Board of Canada Secretariat’s Centre of Excellence for Evaluation and other federal evaluation units to identify the best approach for determining the level of effort required to ensure that evaluations meet the requirements of the Financial Administration Act and the Treasury Board Policy on Evaluation. The implementation date is June 2013.

Human Resources and Skills Development Canada’s response. Agreed. The Department will work collaboratively with the Treasury Board of Canada Secretariat’s Centre of Excellence for Evaluation and other federal evaluation units in addressing the challenges that departments face in meeting the requirement to evaluate ongoing grant and contribution programs every five years. Further, as part of the Department’s 2013–14 to 2017–18 Evaluation Plan, the Department will include an analysis of all current ongoing grant and contribution programs in the Department, and ensure that the evaluation schedule complies with the requirements of the Financial Administration Act. The level of effort, scope, and focus of evaluations will be determined in accordance with program materiality and risk.

1.35 The Treasury Board of Canada Secretariat should review the tools it uses to monitor evaluation coverage to ensure that they provide an accurate perspective on departmental progress that can be considered in the Management Accountability Framework assessment process. (1.28–1.34)

The Treasury Board of Canada Secretariat’s response. Agreed. The Treasury Board of Canada Secretariat continuously strives to improve the monitoring tools that support its responsibilities under the 2009 Policy on Evaluation for providing functional leadership for evaluation across government, including reporting on the health of the evaluation function across government. While the Management Accountability Framework addresses evaluation coverage, the Secretariat does not use Management Accountability Framework ratings on their own for the purpose of monitoring coverage.

The Secretariat will review the tools it uses for monitoring evaluation coverage to ensure that they provide an accurate perspective on departmental progress that can be considered in the Management Accountability Framework assessment process. The Secretariat will complete this action by the end of March 2013.

1.47 The Treasury Board of Canada Secretariat should provide departments with clear guidance about how to track evaluation coverage of direct program spending, how to assess program efficiency and economy, and how to carry out neutral assessments of their evaluation functions to ensure that departments follow a consistent approach. (1.36–1.46)

The Treasury Board of Canada Secretariat’s response. Agreed. In October 2012, the Secretariat issued draft guidance for departments on how to assess program efficiency and economy when conducting evaluations according to the 2009 Policy on Evaluation. The Secretariat plans to post finalized guidance on its website by the end of March 2013.

The Secretariat has begun developing additional guidance to expand upon its Guide to Developing a Departmental Evaluation Plan (released in June 2011), which advises departments on how to determine which programs should be evaluated so that coverage requirements are met and how to reflect evaluation coverage information in their plans. In particular, the additional guidance will support deputy heads in tracking whether they are meeting their accountabilities under the Financial Administration Act (section 42.1) for evaluating all ongoing programs of grants and contributions every five years. Further, the Secretariat is preparing guidance to departments on how to carry out neutral assessments of their evaluation functions. Both guidance efforts are expected to be completed by 31 March 2014.

1.51 In evaluating the Policy on Evaluation by 2014, the Treasury Board of Canada Secretariat should, in consultation with departments, review the requirements to evaluate all direct program spending over a five-year cycle and to address all five core issues. (1.48–1.50)

The Treasury Board of Canada Secretariat’s response. Agreed. By renewing the Policy on Evaluation in April 2009, the government strengthened its commitment to evaluating the value for money of federal programs. To allow departments time to expand their evaluation capacity, a gradual four-year implementation period for some coverage requirements was built into the policy. To meet a commitment it made at the time the policy was approved in 2009, the Secretariat launched a policy implementation review in March 2012, including extensive consultations with departments, and will complete this review by the end of the gradual implementation period on 31 March 2013.

Building on the knowledge gained through the policy implementation review, the subsequent evaluation of the 2009 Policy, which is required by the policy itself, will be conducted by the end of the 2013–14 fiscal year and will centre on key questions such as the coverage requirements and focus of evaluations.

Generating ongoing performance information

1.63 The audited departments should complete performance measurement strategies for all programs, collect complete performance data, and implement quality assurance processes for ongoing performance information. (1.58–1.62)

Agriculture and Agri-Food Canada’s response. Agreed. Agriculture and Agri-Food Canada’s performance measurement strategies that track and monitor performance have been developed for many departmental programs, including all our new Growing Forward 2 programs. The Department will continue to develop and implement systematic processes to measure the performance of all programs, including data collection and quality assurance processes. Performance measurement will be developed by March 2014 for all departmental programs, using approaches that consider the cost and expected benefits of performance information.

With the launch of the revised Treasury Board Policy on Evaluation in April 2009, the Department now undertakes an annual reporting process on the state of performance measurement to support evaluation.

Since June 2010, the Department has undertaken targeted activities to improve the quality of performance measures, to strengthen accountability for performance measurement, to clarify roles and responsibilities, and to build capacity throughout the Department to support performance monitoring and reporting.

Fisheries and Oceans Canada’s response. Agreed. Regarding the completion of performance measurement strategies, Fisheries and Oceans Canada is nearing completion of its Performance Measurement Action plan concerning the development of performance measurement strategies. Of the total 40 performance measurement strategies for the Department, 17 have been approved by the Chief Financial Officer, 9 are at the final program review and approval stage, and the remaining 14 are nearing completion (note that 7 of these are for internal services and hence not mandatory, but are under development). The implementation date is 30 January 2013.

Regarding the collection of complete performance data, implementation of the approved performance measurement strategies will start in the winter of 2013. The implementation date is 31 March 2014.

Regarding the implementation of quality assurance processes for ongoing performance information, by July 2013 the Evaluation Directorate will develop a risk-based methodology for monitoring the implementation of the performance measurement strategies. By 31 March 2014, the Evaluation Directorate will have started monitoring the implementation of performance measurement strategies, targeting 100 percent of high-risk strategies, 50 percent of medium-risk strategies, and 25 percent of low-risk strategies.

Human Resources and Skills Development Canada’s response. Agreed. Human Resources and Skills Development Canada will undertake an assessment in 2013 to determine the state of performance measurement strategies and data collection systems for all departmental programs, in order to assess the extent and significance of existing gaps and the appropriate corrective measures to be taken.

The Department acknowledges that performance measurement strategies that track and monitor performance have been developed for most departmental programs, but not all. Also, many departmental program areas have developed and used quality assurance processes to monitor and correct problems identified with the quality and completeness of performance measurement data. These practices will be encouraged for adoption by all program areas, where appropriate, based on an assessment of cost of investments in the data systems versus the expected benefit of the performance information.

 


Definitions:

Ongoing programs of grants and contributions—Programs delivered for departments by external organizations for which they are paid by the departments. Payments take the form of either grants (transfer payments made to organizations) or contributions (conditional transfer payments made to organizations). (Return)

Other departmental programs—Programs delivered directly by departments using their own staff and resources. (Return)

Evaluation coverage—The extent to which departments have evaluated all of their ongoing grant and contribution programs as well as other departmental programs for which evaluations are required by the Financial Administration Act and the Policy on Evaluation. (Return)

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: