5.1 We carried out audits of a number of grant and contribution programs in several departments and agencies over the past year. We found that management practices were uneven among these programs. While audit results varied to some extent, all the programs were experiencing problems in one or more areas of management responsibility—program design, performance measurement, project assessment, and project monitoring.

Program management issues

Program design needs attention in most programs, in particular, the specific results expected from the spending of public money.
Management has not yet developed risk management strategies to help deliver programs in a cost-effective way.
Where they have been carried out, program evaluations are often limited in scope and do not provide a clear overview of whether the programs are achieving value for money.
Many programs have not been audited in more than five years.
Parliament, despite changes to the reporting requirements, still receives limited information on program performance.

Project management issues

Decisions to fund projects are often based on partial or perfunctory assessments of project merits and the need for government funding.
Financial control over disbursements of public funds was generally satisfactory, although specific improvements are needed in some programs.
Project monitoring practices ranged from satisfactory in some programs to poor in others.

5.2 We followed up on three audits of grant and contribution programs we had reported previously, including our work in Human Resources Development Canada in 2000. By and large, management has made satisfactory progress in fixing the problems we had found—indicating that good management is achievable, if not always achieved.

Background

5.3 In this chapter, we report the results of our audits of 12 grant and contribution programs in nine departments and agencies and our follow-up of three audits previously reported. We looked for key control elements—rigorous program design, relevant performance measurement and reporting, thorough project assessments, and appropriate project monitoring and financial management.

The departments have responded. The departments we audited have indicated their agreement with our recommendations. The actions they have taken or plan to take are set out in their responses to the recommendations in the respective sections of the chapter.

Introduction

5.4 This chapter discusses our audits of voted grant and contribution programs managed by nine departments and agencies and our follow-up of other audits reported previously. We wanted our work to cover a number of departments and agencies with relatively large grant and contribution programs to determine how well these programs are managed across government. That coverage would also allow us to consider more effectively the important role of the Treasury Board and its Secretariat in encouraging and ensuring good program design and management practices. Our observations on their role are set out in Chapter 4.

5.5 We tried to select programs for audit that would capture as many key features of grants and contributions as possible. These features include the type of recipient, delivery mechanism, duration of funding, policy area, size of department, sharing of cost, and funding mechanism—grant, contribution, or both. We chose the following programs:

Agri-Food Trade Program—Agriculture and Agri-Food Canada
IDEA-SME Program and Regional Strategic Initiatives—Canada Economic Development for Quebec Regions
Operating Grants Program—Canadian Institutes of Health Research
Settlement Contribution Programs—Citizenship and Immigration Canada
Support for Official Language Communities—Department of Canadian Heritage
Contributions made under the Climate Change Action Fund—Environment Canada and Natural Resources Canada
First Nations Policing Program—Solicitor General Canada
Community Futures Program—Western Economic Diversification Canada

5.6 We also followed up on previous audits of grants and contributions:

Department of Canadian Heritage—Grants and Contributions Under the Multiculturalism Program (Chapter 27, 1998).
Industry Portfolio: Investing In Innovation (Chapter 19, 1999)
Human Resources Development Canada—Grants and Contributions (Chapter 11, 2000)

Focus of the audits

5.7 The objective of our detailed audits was to determine whether departments had adequate control over selected grant and contribution programs.

5.8 We looked at whether the programs we examined were managed according to basic rules set by the Treasury Board and Parliament for the proper handling of public money. We used the Financial Administration Act (FAA), the Treasury Board's Policy on Transfer Payments, and the terms and conditions for each program as standards to assess the program's management.

5.9 While the FAA deals with how disbursements in general are to be handled, the transfer payment policy spells out the government's rules for managing grants and contributions properly and for disbursing public funds. The terms and conditions for each program set out the process for assessing project proposals, approving projects and making payments. They are tailored to the specific circumstances of the program and its objectives.

5.10 Insisting on program management according to the rules, however, does not have to mean a system encumbered by red tape. Control over public money should be directly related to a project's significance, including the amount of money and the level of risk involved. More important, perhaps, the government needs to follow its own rules when handling public money.

5.11 The objective in our follow-up of previous recommendations was to determine what departments have done to implement corrective action. Unless otherwise stated, a follow-up is not a second audit of the same issues. It is a report on the corrective action that management tells us or can demonstrate it has taken on the recommendations of our previous audit. We do not exhaustively seek new evidence to support or refute what management tells us, but we do review its claims for reasonableness and report to Parliament accordingly.

5.12 Further details on the audits can be found in About the Audits at the end of the chapter.

Overview of Our Observations and Conclusions

5.13 In this section, we look across our audits of the programs to make observations on the major elements of program management. We do not identify individual programs in this overview; we discuss the key management responsibilities and the practices that are still weak. Our detailed observations on each program are in the rest of the chapter and take into account their specific circumstances. It is important to recognize how the programs differ and how the differences affect the way departments and agencies manage them.

5.14 We looked for rigorous program design, thorough support for decisions to spend public funds, proper control of disbursements, and good knowledge of performance. While the audit results vary among programs, all the programs have problems in one or more of these key areas. Although we found areas in each program that were managed properly, we are concerned that some public funds have been put at risk because management, in one way or another, is not always following the rules established by the government for grant and contribution programs. Where expected results are stated only vaguely, where risks are unassessed, project assessments incomplete, or performance unmeasured, management cannot be confident that the programs are achieving value for money. Moreover, the kinds of problems we found are correctable with proper attention from management, and during our program audits, management began addressing many of the issues we raised.

5.15 Program design needs more attention in many programs. We found that many programs set out the program objectives—the general purposes—reasonably clearly. However, they often do not set out the specific results expected from program spending within a given period of time. Most of the programs lacked a risk management strategy to help deliver the program cost-effectively. While there were various guidelines and procedures for managing specific risks when approving and monitoring projects, risk assessment has not been integrated into program management practices in a formal and systematic way. Moreover, the lack of clearly stated expected results of program spending makes it difficult for management to assess risks thoroughly and therefore to establish appropriate controls.

5.16 No evaluation or incomplete evaluation of programs. Assessing program performance is still a challenge for most departments. Six of the programs we audited have never been evaluated, although they have operated for many years. Two of the six other programs underwent a recent evaluation that looked at all relevant performance issues. The rest have been evaluated and the results were largely positive, but in our view the evaluations did not address all of the key aspects of performance. As a result, the evaluations yielded limited information on program performance and provided an incomplete picture of whether the programs were achieving value for money.

5.17 Insufficient internal audit. Three of the programs we audited for this chapter underwent an internal audit in the last five years; nine did not. As a result, management of those nine programs had little of the kind of independent information internal audit can provide on whether financial and other controls were working properly.

5.18 Limited performance information to Parliament. Given the limited information available on program performance, we were not surprised that Parliament is not well informed about the results of these programs. Reports to Parliament provided limited or fragmented information, or none at all, on the results of many programs we audited.

5.19 Assessment of projects often incomplete. Program terms and conditions set out the process that must be followed in assessing and approving applications for grants and contributions. We often could not assure ourselves in many projects we looked at that assessments had covered all the criteria in the terms and conditions or were thorough enough. Decisions to approve funding were often not supported by a careful assessment of a project's merits or the applicant's need for program funding.

5.20 Financial control can be improved in some programs. Generally, we found that payments meet the requirements of the Financial Administration Act (FAA) and the terms and conditions of the respective programs. However, some programs need to do better at managing payments in advance of need and ensuring that only eligible costs are reimbursed.

5.21 Uneven project monitoring. Project monitoring practices were uneven among these programs. In some programs, management had sufficient ongoing knowledge of project performance, but in others we found that monitoring was inadequate and management thus had only limited opportunity to intervene if there were problems.

5.22 Follow-up shows satisfactory progress. Our follow-up work on previously reported audits of grant and contribution programs found satisfactory progress on almost all the recommendations we had made. For the most part, the recommendations were directed at problems similar to those we found in this year's audits. This suggests to us that management can and does fix its practices when required to do so.

Section 1—Agriculture and Agri-Food Canada—Agri-Food Trade Program

In brief

The Agri-Food Trade Program is a cost-shared contribution program designed to support the Canadian agri-food industry in a wide range of market development, investment, and trade-related activities. We audited about $26 million spent by the program in 1999-2000 and 2000-01.

We found that management has adequate financial control over program spending as well as adequate project monitoring and has taken some initiatives to manage risks. Management informs us that it has begun establishing clear expected results for the program. While management collects information on the results of individual projects, that information is not compiled and used to assess and report on program performance. We found that projects were assessed thoroughly against most of the required criteria. However, we could not assure ourselves that the recipients' capacity to carry out the projects or their need for funding had been adequately considered.

Background

5.23 The objective of the Agri-Food Trade Program (AFTP) is to maintain or increase sales of agriculture, food, and beverage products in the domestic and foreign markets and thereby help maintain or increase investment and employment in the industry in Canada.

5.24 The program targets agriculture and food producers, processors, and exporters who work together through non-profit associations or who organize into alliances and institutions that work with the agri-food industry. The program gives priority to projects that are part of a long-term strategy for industry exports, implemented through agri-food industry associations or alliances and through participation in the Agri-Food Industry Market Strategies, a component of the AFTP.

5.25 The program is administered by the Market and Industry Services Branch of Agriculture and Agri-Food Canada. It is overseen by a small secretariat, and individual projects are managed by officers drawn from the operational directorates of the Branch. The program spent $13 million in each of the two fiscal years we audited, 1999-2000 and 2000-01.

Focus of the audit

5.26 Our audit covered the last fiscal year of the Agri-Food Trade Program 2000 (1999-2000) and the first year of the successor Agri-Food Trade Program (2000-01) that began in April 2000 and expires in March 2005.

5.27 The objective of the audit was to determine whether the Department had adequate control over the program. Further details on the audit can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.28 Program objectives. We expected that the program would be designed to achieve results, manage risks, ensure due diligence in spending, and provide accountability for funds spent. We found that the program's objectives are stated in very general terms: to increase sales of Canadian agriculture, food, and beverage products in domestic and foreign markets. This is consistent with the Department's support for industry's goals of increasing Canada's world-market share of agri-food product exports to 4 percent and reversing the ratio of primary products to value-added products from 60/40 to 40/60. However, the specific contribution expected from the Agri-Food Trade Program in achieving these objectives has not been determined by management. Management informed us that it has started to establish clear expected results and the means to report on them.

5.29 Program management. To be eligible, projects must

have potential to significantly impact the sales of Canadian agriculture, food, and beverage products in national and international markets
be part of the long-term strategies for the industry, implemented through agri-food industry associations or alliances that represent a significant portion of the national production of a given commodity or value-added product
meet federal and/or provincial environmental assessment requirements.

Applicants must have the capability to complete the projects and must demonstrate that they have, or will have, adequate financial resources to discharge their obligations under the contribution agreements.

5.30 We found that program management has developed clear administrative guidelines for project officers in reviewing applications, except with respect to the need for funding.

5.31 There is no formal risk management strategy. Risk management is meant to strengthen management practices, decision making, and priority setting. A risk management strategy helps in identifying, analyzing, assessing, monitoring, and controlling risks in managing contributions. It helps to prevent unsatisfactory or unintended outcomes. Although the program has no formal risk management strategy in its design, we found that management has taken a number of specific initiatives that have helped to manage risks. We also found that there was evidence of co-ordination with other federal export marketing programs.

5.32 In March 2000, management reviewed the program to assess whether it had adequate measures to ensure that funding recipients were accountable for their spending. It also examined the process for submitting, reviewing, and approving applications; the payment process; project monitoring and post-project evaluation; and other key matters. In addition, management has introduced procedures to minimize risks in the processing of claims—for example, a requirement for independent audits before claims are paid and a 10 percent holdback on payment of final claims until final project reports are received.

5.33 Recommendation. The Department should specify clearly the results it expects from the Agri-Food Trade Program. It should also develop a formal risk assessment and management strategy for the program's operations.

Department's response. The Department partially agrees with the first part of the recommendation. The Agri-Food Trade Program objectives are to maintain or increase sales of agriculture, food, and beverage products in the domestic and foreign markets and thereby help maintain or increase investment and employment in the industry in Canada. The five performance measurement indicators currently used are sales, value-added sales, awareness of Canadian products, investment, and the creation of jobs in Canada. The Department acknowledges the need to further articulate the expected program outcomes and intends to clarify these expected results in future program reporting.

The Department agrees with the second part of the recommendation. It currently has a number of provisions in place to manage the level of risk associated with the AFTP. For instance, the program generally reimburses to a maximum of 50 percent of expenses incurred and paid by recipients (according to the terms of a legal agreement); requires external audits on all projects funded; requires a 10 percent "holdback" until the final external audit and performance reports are submitted; and conducts a regular analysis of spending trends to ensure proper management of funds. With the exception of payments of a small number of advances, most other risks have been considered and are covered in the AFTP's legal agreement. However, the Department intends to develop a formal risk assessment and management strategy for the program's operations prior to the renewal of the program's terms and conditions in 2005.

Program performance

5.34 There has been limited program evaluation. We expected that management would know whether the program is meeting its objectives. The Treasury Board Secretariat has established guidelines for periodic program evaluation, normally every five years, to assess program performance. Program evaluation encourages efficiency and effectiveness in program spending, and accountability for funds spent.

5.35 To date, the Department has not evaluated the Agri-Food Trade Program in a comprehensive way. A 1998 evaluation of one program element, company brand promotion, looked at its relevance and success but did not compare it with alternatives to the program. There was a heavy reliance on non-quantitative approaches to evaluating success, such as interviews. Nevertheless, the evaluation concluded that this element of the program was successful in helping Canadian industry deal with competitors in foreign markets. We also noted that the program has not been the subject of an internal audit.

5.36 Performance information to Parliament needs to be improved. Under the revised Treasury Board Policy on Transfer Payments, effective June 2000, the specific results planned for grant and contribution programs over $5 million must be stated clearly in a department's report on plans and priorities. We noted that Agriculture and Agri-Food Canada's 2001-02 report did not mention the specific results expected from the Agri-Food Trade Program.

5.37 There is limited information on the program's performance in the Department's 1999-2000 Performance Report—only the number of projects approved that year. There is a reference at the departmental level to an increase in exports of processed products (from 48 to 52 percent). There is no mention of an increase in exports directly related to this program. Nor is information provided that aggregates the results of individual projects funded under the program.

5.38 Recommendation. The Department should conduct a complete program evaluation of the Agri-Food Trade Program. It should also continue to develop and use appropriate performance indicators to report the results of the program and the projects it funds.

Department's response. The Department agrees. A departmental evaluation of the AFTP is expected to be completed by June 2002. Likewise, an AFTP internal audit is included in the departmental multi-year grants and contributions audit strategy, and is currently scheduled for 2004-05.

Currently, the AFTP has five key performance measurement indicators that are reflected in the objectives of the program. Nonetheless, the planned winter/spring 2002 evaluation of the AFTP will address performance information gaps and include specific recommendations to both revise and report on performance indicators, particularly in the context of the departmental performance report.

Project assessment and approval

5.39 Approval process. Applicants for funding under the Agri-Food Industry Market Strategies (AIMS), a component of the AFTP, submit a project strategy each year, which is assessed by a working group comprising mainly officials of Agriculture and Agri-Food Canada. The working group evaluates the validity and soundness of the strategy and forwards a recommendation to the AIMS Steering Committee, which reviews the project and approves the funding.

5.40 For stand-alone projects that do not come under AIMS, organizations apply directly to the Agri-Food Trade Program for funding. A project officer is assigned to review each of those projects. Before making a recommendation, the project officer forwards the application to two other departmental officials in a different responsibility centre for peer review. All projects and recommendations are reviewed by a management committee, which decides whether to approve the project or not.

5.41 Quality of strategies varies. We expected that the Agri-Food Trade Program would exercise due diligence in approving each project.

5.42 We found that the quality of the project strategies funded under AIMS was generally good. However, 2 of the 23 project strategies we examined were weak, in that they did not identify priority markets, provide a five-year forecast of market demand, explain market priorities, assess the industry's capability to satisfy the demand, or specify achievable five-year goals.

5.43 Problems with assessment of management and financial capability. We expected the assessment of each project to take into account, at a minimum, all the review criteria under the program's terms and conditions. We could not assure ourselves that the management capability and financial capacity of the applicants were assessed in any of the projects we examined. In other words, there was little evidence that the applicant's ability to complete the project had been formally assessed; nor was the applicant's need for government funding explicitly demonstrated.

5.44 Management informed us that the working group leaders assess the financial capacities and management capabilities of applicants based on their day-to-day relations and their knowledge of the associations and industry sectors. However, these assessments are not documented in the project files. In cases where the assessments indicate a potential problem, the working group contacts the applicant and flags the issue for the steering committee and program management. Management has agreed that in the future it will formally document the assessment of an applicant's financial capacity and management capability.

5.45 No move toward self-sufficiency. Most of the AIMS projects are recurring activities over several years. One objective of AIMS is to help trade associations eventually become self-sufficient. We expected to see a strategy for helping applicants achieve self-sufficiency.

5.46 Although the funding approvals provide exit strategies for federal involvement, we saw little effort by project officers to move funding recipients toward self-sufficiency. In none of the projects we audited was it clear that self-sufficiency would be achieved.

5.47 Recommendation. The Department should ensure that the eligibility of each applicant is formally assessed against all the established criteria and that the assessment is properly documented.

Department's response. The Department partially agrees. The Department believes that each applicant is assessed against all established criteria but acknowledges that this information was not properly documented in each project file. Program administration has already implemented a formal checklist to ensure that applicant assessments are properly documented on project files.

Project and financial monitoring

5.48 Adequate project and financial monitoring. We expected management to have reasonable assurance that funding is used for the intended purposes. All contribution agreements require recipients to provide periodic progress or activity reports and a final report at the end of the project. There is a holdback on the payment of the final expense claim until the final report is received. We found that the reports have been submitted more or less on time. Most of the claims submitted by the applicants are audited by an independent auditor before the program reimburses expenses. Where there is no audit, management reviews the claimed expenses. In addition, three recipients are selected each year and audited to determine whether they have respected the terms and conditions of the contribution agreements. In our view, project and financial monitoring is adequate. However, we identified a few areas where improvements could be made.

5.49 Although contribution agreements require that recipients provide progress reports and final reports, we found that there is no system to signal when reports are due and no procedure to record their receipt. Project officers do not always review final reports to determine the extent to which projects achieved the objectives outlined in their strategies. The procedures manual requires that project officers prepare a written evaluation at the end of the project to compare the project results with the objectives. Several of the files we reviewed did not contain these evaluations. Management has indicated that in the future these evaluations will be completed.

Conclusion

5.50 We found that management has adequate financial control over program spending as well as adequate project monitoring and has taken some initiatives to manage risks. Management informs us that it has begun establishing clear expected results for the program. While management collects information on the results of individual projects, these are not compiled and used to assess and report program performance. We found that projects were assessed thoroughly against most of the required criteria, however, we could not assure ourselves that the capacity of recipients to carry out the projects or their need for funding had been adequately considered.

Section 2—Canada Economic Development for Quebec Regions—IDEA-SME Program and Regional Strategic Initiatives

In brief

The IDEA-SME Program (Innovation Development Entrepreneurship and Access) for small and medium-sized enterprises (SMEs) begun in 1995; it is designed to help SMEs increase their business opportunities and participate more actively in improving the development potential of all Quebec regions. The Regional Strategic Initiatives program funds initiatives developed by private or public sector partners and helps create economic development opportunities to stimulate investment and job creation. We examined $90 million in spending by the two programs during the period April 1999 to September 2000.

An evaluation of the IDEA-SME program in 2000 that focussed on the program's effectiveness and relevance demonstrated that it has had some success. We found that generally the assessment of applications under both programs was complete and thorough, but we could not assure ourselves that its assessment of the level of risk and expected results of some projects was adequate. We are generally satisfied with the financial controls of Canada Economic Development for Quebec Regions (the Agency) on project spending, with the exception of advance payments. Further, in December 1999 it put new measures in place that should improve the monitoring of project results.

Background

5.51 The IDEA-SME Program funds projects by SMEs and organizations in three areas of activity: innovation, research and development, and design; market development and export trade; and entrepreneurship—business climate development.

5.52 The Regional Strategic Initiatives (RSI) program, begun in 1997, provides funding for initiatives by private or public sector partners. Its objectives are to contribute to economic development by stimulating investment and job creation and to ensure that the federal government acts consistently for effective horizontal management. The program focusses on four areas of activity: development of the technological capability of the regions; international tourism development; support for the drawing power and international reach of Quebec regions; and support for the adjustment capability of the regions in the new climate of globalization.

5.53 From 1996 to 2001, the IDEA-SME program funded 3,000 projects with an average contribution of $103,000. The RSI program contributed an average of $491,000, to 426 projects. Total annual spending of both programs was $104 million in 1999-2000 and in 2000-01.

5.54 The programs are the responsibility of Canada Economic Development for Quebec Regions. The Operations Branch has 160 employees at headquarters and in 14 regional offices and an annual operating budget of $12 million.

Focus of the audit

5.55 The objective of our audit was to determine whether the Agency had adequate control over the IDEA-SME and RSI programs. We examined IDEA-SME contributions selected from the $76 million in disbursements made by the program between 1 April 1999 and 30 September 2000 and $14 million in contributions by the RSI program in the same period.

5.56 Further details on the audit can be found at the end of this chapter in About the Audits.

Observations and Recommendations

Program design

5.57 When the IDEA-SME and RSI programs began, the Agency developed a detailed management framework. The framework specifies and elaborates on the terms and conditions approved by the Treasury Board. It sets out the programs' objectives; criteria for project assessment; and for each area of activity, the Agency's objectives and priorities, eligibility criteria (client base, activities, and costs), and the type and level of assistance authorized. In our view, this framework is relevant and clear.

5.58 Improvements in program management. Since the programs' inception, and particularly since 1997, the Agency has carried out various surveys, analyses, internal audits, and evaluations of program management and performance. Based on their findings and recommendations, it has made a number of improvements in its management framework and practices. In particular, it defined various elements of the framework, developed reference documents for assessing and monitoring projects, established a performance measurement policy and tools, and provided staff with related training.

Program performance

5.59 We acknowledge the Agency's efforts at ongoing performance measurement and identification of lessons learned throughout the five years of the IDEA-SME program. The Agency compiled the results of the principal assessments, surveys, and audits it has completed and presented them in a "summation" evaluation report.

5.60 The evaluation report notes that the IDEA-SME program has achieved significant results in the areas of employment and profitability of the businesses it has assisted; the general objectives of the program are still relevant; the program meets the needs of small and medium-sized enterprises throughout Quebec; the program must continue to adapt to regional realities; and there is a high level of client satisfaction with the services received. The report suggests some improvements in management practices, service delivery, and the management of partnerships.

5.61 An evaluation of the RSI program is scheduled for 2001-02.

Quality of information to Parliament has improved

5.62 In our December 2000 Report, Chapter 19, Reporting Performance to Parliament: Progress Too Slow, we noted that the information provided by the Agency had improved consistently in the last few years. Our audit this year looked at the information it provided on the IDEA-SME and RSI programs in its 1999-2000 Performance Report.

5.63 In that report, the Agency provides an overview of its clientele and activities and describes its achievements and those of its partner groups. It also describes the lessons learned from its various assessments and surveys and the steps it plans to improve its practices. Furthermore, although the Performance Report emphasizes the results achieved in the preceding 12 months, it indicates the long-term impact of the Agency's activities on employment in small and medium-sized enterprises.

5.64 Based on our review, we believe that certain improvements are still needed. In particular, the Agency needs to describe more clearly the results it expects for each key commitment, explain the limitations of its methods of measuring performance, provide more balanced information, and disclose its share of funding of large-scale projects, particularly in the RSI program. We noted that the Agency responded well to the reporting requirements of the new Policy on Transfer Payments in its 2001-02 Report on Plans and Priorities.

Project assessment and approval

5.65 Under the terms of both programs, the decision to invest in a project and the level of funding to contribute are based on criteria that include the project's fit with the Agency's objectives and priorities as well as those of the region; the potential economic and commercial impact on the enterprise and the region; the strategic, technical, and financial capacities of the applicant; the level of risk involved; the project's impact on sustainable development; and the availability of funds. The RSI program includes additional criteria: the leveraging effect of the contribution on the planned size and schedule of the project; and the project's priority in the community, the community's involvement in its funding, and the capacity to generate funds. An internal directive stipulates that incentive effects of IDEA-SME contributions must also be analyzed.

5.66 We expected that the decision to invest in a project and the level of funding authorized would take into account all of these criteria and be adequately supported. Projects that represent value added for the Agency and the government are those that optimize the use of available public funds and achieve the expected results. The assessment of project applications must therefore consider the advantages to the Agency of investing in a project and the incentive effects of its contribution.

5.67 The projects we looked at were generally consistent with program objectives. The applications were assessed appropriately for the applicant's strategic, financial, and technical capacities and the project's potential impact on sustainable development. In most cases, the assessment showed the incentive effect of the contribution.

5.68 However, we found weaknesses in the assessment of project risk and of the expected economic and commercial results. We audited a total of $90 million in contributions to projects between 1 April 1999 and 30 September 2000; for $50 million of that we could not assure ourselves that the project had been assessed adequately against at least one of these two criteria.

5.69 Lack of consistency in assessing project risk. Of the projects we reviewed, the assessments of 56 percent did not include the level of risk involved. Most of them noted various elements of risk but made no final assessment. In our discussions with staff, we found that their understanding of risk was not consistent. Some view risk as limited to the non-repayment of a repayable contribution. Others may not document the risk assessment clearly, but the special conditions they require in the contribution agreement reflect their assessment. However, the links between the agreement conditions and the elements of risk identified are not always apparent. Some staff told us that where no level of risk is indicated, the risk is low. However, their conclusions were not supported by a formal analysis.

5.70 Management based on risk. We did not find any departmental directive on auditing payments and monitoring projects according to level of risk, nor a formal risk management strategy. The Agency has used a risk rating tool since December 1999 to assess the risk of non-repayment of contributions. This tool permits staff to assign a score, based on a point scale, to various aspects of the project and the organization that is applying, such as financial situation, skills and experience of managers, and the technical and commercial risks of the project. This is a sound practice. However, the Agency could expand the use of this tool to all projects and could add criteria for assessing risk. The risk assessment would be used not only to help decide whether to invest in a project but also to improve the targeting of financial and performance monitoring.

5.71 Expected economic and commercial impacts of projects are not specific enough. The Agency funds organizations that offer services to SMEs in order to benefit from their knowledge and expertise in specific areas and to diversify its service delivery. Given the intermediary role of these organizations between the Agency and the SMEs, we would expect the contribution agreements with them to indicate the expected results of projects in clear, concrete, and measurable terms in order to achieve the intended economic and commercial impacts of the program.

5.72 In 36 percent of the projects we reviewed, the contribution agreements did not outline the expected results clearly. The descriptions and goals of projects were also worded in vague and general terms, such as "allow the continuation of activities, notably . . . " and "ensure the existence of the organization, whose mission consists of . . . consolidat[ing] operations and develop[ing] . . . new activities."

5.73 The Agency recognized this weakness in Lessons Learned From Evaluations Done in 1998-99. At the end of December 1999, it developed a framework for continuous performance measurement of projects carried out by non-profit organizations. Since then, all agreements with such organizations must identify for each project the objectives, the expected short-term outputs and medium-term results, and performance indicators. A performance monitoring tool has also been developed. Furthermore, staff have received training in continuous performance measurement and the role of intermediary groups.

5.74 To date, there has been no review of these tools or structured comparison of their content and quality. Such an exercise would help identify good practices, lessons learned, and improvements needed.

5.75 Need to justify the level of funding provided. The IDEA-SME program's management framework generally limits the level of funding by the program to 50 percent of a project's allowable costs. However, an exception to this rule is possible if the need for a higher percentage is demonstrated. We found that in 5 of the 11 projects whose funding exceeded this limit, the excess was not adequately justified. The limit was exceeded by a total of $1.6 million in these 5 cases.

5.76 Recommendation. The Agency should ensure that the level of risk and the economic and commercial impacts of projects are evaluated thoroughly and should justify adequately the funding it provides in excess of the established limit.

Agency's response. The Agency already automatically assesses the risk of non-repayment of a contribution. However, the Agency is reviewing its assessment model, which will include a specific section on the level of risk taken in financial terms and in terms of the level of expected results.

In the summer of 2001 the Agency conducted a study of intermediaries to determine the best practices in establishing expected results, to identify deficiencies, and to propose solutions. This review will, among other things, improve the assessment of the economic and commercial impact of projects.

Program management frameworks already require that funding in excess of the limits set out in a framework be adequately clarified. To emphasize this point more, we will amend the program review summary to require this adequate clarification when the level of funding exceeds the limit set out in the management framework.

Project and financial monitoring

Financial management and control

5.77 In general, we found that the audit of claims for payment is adequate, and payments made are for allowable costs. However, the Agency needs to be clearer about the extent and nature of audit that it expects in order to ensure consistency and take into account project risks. The new payment procedures introduced in the fall of 1999, for example, allow for the possibility of requiring in the agreement an accountant's report on claimed expenses. The Agency has provided no written guidance to its managers on how to decide whether or not to require such a report.

5.78 Advance payments are not properly supported. The Treasury Board's Policy on Transfer Payments allows for advance payments when they are "based on prudent cash management principles." That is, the amount of each advance payment must be limited to the recipient's immediate cash requirements. Cash requirements are based on the recipient's monthly cash flow forecasts, taking into account any advances outstanding. We expect that the Agency would comply with the policy.

5.79 The Agency made advance payments to 25 of the projects in our sample. It views advances as specifically intended to assist non-profit organizations and private businesses in the start-up phase of projects that will benefit the economic well-being of Canadians. We found that in 21 of the 25 projects, the advances were not supported by the required cash flow forecast. In 9 cases, the Agency received the expense claims anywhere from 6 to 24 months after it had made the advance payment. Moreover, the amounts claimed were not always as large as the amounts paid in advance, which puts into question the level of those advances. We noted, for example, that an organization received an advance of $2.9 million in March 2000, based on an estimate of its expenses for two months (March and April). The forecast represented 80 percent of the total amount allocated to the organization for consulting services during the five-year duration of the project. A claim for the actual costs had not yet been filed at December 2000.

5.80 The internal auditors made similar comments in their October 2000 report. The Agency informed us that it has since brought its policy on advance payments into line with that of the Treasury Board.

5.81 Recommendation. The Agency should comply with the Treasury Board's policy on advance payments.

Agency's response. The Agency has already followed up on the comments of internal auditors on this matter in October 2000. It has since brought its policy on advance payments into line with the Treasury Board's Policy on Transfer Payments.

Project monitoring

5.82 We expected the Agency to be aware of how funded projects were performing, that is, aware of their activities and results. We therefore expected to see appropriate monitoring during the life of each project and at its completion.

5.83 Reasonable project monitoring in most cases. We found that in most cases the Agency monitors and documents the progress of projects adequately. It monitors projects in various ways, such as reviewing activity reports submitted with claims for payment, reviewing the quarterly follow-up reports introduced in July 1999, and participating as an observer on boards of directors or operating boards of several non-profit organizations.

5.84 Performance measurement needs to improve. We also looked at whether the Agency had assessed results at the end of projects. We found that in 15 of 22 completed projects (68 percent) it had measured certain results. However, we saw no comparative analysis or explanation of the significant differences between expected and actual results. Furthermore, in eight projects of non-profit organizations the initial objectives were not clear or measurable. Consequently, although it measured project performance, the Agency could not draw any conclusions about the achievement of results. These are particularly important gaps because the Agency has renewed its contribution agreements with a number of non-profit organizations.

5.85 As already noted, the Agency began continuous measurement of the performance of all projects approved after 1999 and carried out by non-profit organizations. The contribution agreements we examined that were approved after that date contained the required information.

5.86 The Agency has conducted annual surveys of its SME clientele since 1997. These surveys enable it to measure the performance of projects it has authorized throughout the year as well as measuring client satisfaction. It carried out two long-term surveys to determine how projects were performing three years after receiving the Agency's contribution. We did not review the survey methodology or results in depth. Consequently, we cannot comment on the quality of the information the survey generated. However, these are interesting initiatives that may provide valuable information on results within the framework of continuous performance measurement.

5.87 Recommendation. The Agency should

ensure that it identifies the expected results at the outset of projects, describes them in concrete and measurable terms, and establishes a timeframe for their achievement
ensure that the difference between expected and achieved results is assess and take that information into account when renewing agreements.

Agency's response. In December 1999, the Agency implemented performance monitoring for all projects. In addition, it is implementing a training program on performance evaluation, which will be given to all advisors this fiscal year. This training will deal specifically with defining expected results in concrete and measurable terms. Advisors will be better able to state, at the outset of projects, the expected results and the time frames established, based on the nature of the project.

Furthermore, the Agency has developed a scorecard for the results achieved in projects carried out, notably with intermediaries. This scorecard compares the actual results with the expected results, explains the variances, and describes the measures to take to correct the situation, if needed. This analysis of the results will serve as a benchmark in the renewal of agreements.

Conclusion

5.88 An evaluation of the IDEA-SME program in 2000 that focussed on the program's effectiveness and relevance demonstrated that it has had some success. We found that generally the assessment of applications under both programs was complete and thorough, but we could not assure ourselves that its assessment of the level of risk and expected results of some projects was adequate. We are generally satisfied with the Agency's financial controls on project spending, with the exception of advance payments. Further, in December 1999 it put new measures in place that should improve the monitoring of project results.

Section 3—Canadian Institutes of Health Research—Operating Grants Program

In brief

The Canadian Institutes of Health Research (CIHR or the Agency) was established in June 2000. Its programs build upon and expand those of the Medical Research Council, which had existed since 1969. The objectives of the CIHR programs are to excel, according to international standards of scientific excellence, in creating new knowledge and translating it into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system. We audited CIHR's expenditures for research operating grants in 2000-01, a total of $190 million.

We found that CIHR's assessment of applications for operating grants was complete and thorough. It uses a rigorous peer review system to identify projects for funding. However, we found that its monitoring of how quickly researchers spend their grants needs to improve, as does the tracking of research results. CIHR also needs to improve its assessment and reporting of program performance.

Background

5.89 All elements of the former Medical Research Council's mandate are incorporated in that of the Canadian Institutes of Health Research: to promote and assist research in the health sciences by funding research and training in research at Canadian universities, research institutes, and hospitals. However, the Canadian Institutes of Health Research has a much broader mandate that includes intersectoral, interprovincial, and international consultation and partnering. It also has a mandate to translate new knowledge into health innovations. Furthermore, the structure of CIHR, with virtual institutes across Canada focussed on specific areas of health or the health of specific population groups, is much more integrated than its predecessor's.

5.90 We audited the CIHR's operating grants program. As the audit commenced about six months after the launch of CIHR, the program was essentially the same as the Medical Research Council had been delivering at the time that its assets, liabilities, grants, and related funding were transferred to the new agency.

5.91 The total budget for grants and scholarships for the last three years is shown in Exhibit 5.1. Operating grants are made to support projects directed toward defined research objectives. The projects are conducted by teams led by an investigator or several investigators. Grants can be active for six months to five years.

Focus of the audit

5.92 Our audit covered the payments made by the operating grants program in fiscal year 2000-01. Operating grants that year totalled roughly $190 million. We audited a sample of project files; interviewed CIHR officials, selection committee members, and grant administrators; and visited some grant recipients and some other research organizations. We reviewed literature on similar programs in other countries.

5.93 The objective of the audit was to determine whether CIHR had adequate control over operating grants. Further details on the audit can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.94 Program objectives. We expected management to ensure that grant programs were designed to achieve results, manage risks, ensure due diligence in spending, and provide accountability for public funds spent. We found that management has established reasonably clear objectives for its research grants program. The objectives are consistent with the mandate and policy of the organization. The objective of the CIHR is to excel, according to internationally accepted standards of excellence, in creating new knowledge and translating it into improved health for Canadians, more effective health services and products, and a strengthened Canadian health care system.

5.95 There is no formal risk management strategy. Risk management is designed to strengthen management practices, decision making, and priority setting. CIHR does not have a formal strategy for identifying and managing risks in its program operations. A risk management strategy would help it to identify, analyze, assess, monitor, and control risks at each stage of the grant cycle, and would help prevent unsatisfactory or unintended outcomes. Nevertheless, to manage some specific risks it does have in place such processes and procedures as verification of data, oversight committees, policies on conflict of interest and confidentiality, and systems for selecting the most appropriate reviewers of grant applications. CIHR management told us that an enterprise-wide risk assessment and risk management strategy would be completed in fiscal year 2001-02.

5.96 CIHR is developing an internal audit function. Internal audit and evaluation are key elements of internal control and could play a major role in project and financial monitoring. Management had not established an internal audit function and, accordingly, the operating grants program has not been audited. During our audit, CIHR requested additional resources for internal audit and program evaluation functions. It recently developed an internal audit policy that it is ready to implement.

5.97 Recommendation. CIHR should develop a formal risk assessment and management strategy and should establish an internal audit function.

Agency's response. As a new agency, launched in June 2000, CIHR is strengthening key functions in pursuit of its goal of being a model results-based organization. We are currently undergoing an enterprise-wide risk assessment as the initial stage of developing a formal risk management strategy. This will be followed by an appropriately resourced internal audit function.

Program performance

5.98 Little information on program performance. We expected that management would know whether programs are achieving expected results. Government programs are supposed to be evaluated periodically, normally every five years. Program evaluation encourages efficiency, effectiveness, and accountability in program spending.

5.99 We found that management had conducted some studies of the operating grants program and other grant programs. However, it had not specifically addressed the four evaluation issues indicated in the Treasury Board evaluation guidelines: the program's rationale, impact, achievement of objectives, and alternatives.

5.100 CIHR management told us that it would complete an evaluation of the operating grants program in fiscal year 2002-03.

5.101 Reporting performance to Parliament could be improved. We expected that management would report clearly to Parliament on the performance of the operating grants program. Our review of its performance reports for 1998-99 and 1999-2000 found that the quantitative information they contained on performance was limited. Performance measures developed so far for research grants focus more on activities and inputs than on outcomes. The program's impact on health and the economy tends to be reported anecdotally, by specific case, rather than as measured by program-wide indicators. Dollars spent, number of scientists trained, numbers of scientists and projects supported, and number of research publications produced are necessary indicators of program performance but alone are not sufficient. Management needs to assess its performance toward achieving the results expected from the operating grants program. Under the revised Treasury Board Policy on Transfer Payments effective June 2000, the specific objectives and results planned for grant and contribution programs over $5 million must be stated clearly in the entity's report on plans and priorities, together with milestones for achievement. We noted that in CIHR's 2001-02 report, information on its objectives and planned results was provided.

5.102 At the time of our audit, CIHR was developing a strategy for collecting timely and accurate information on the results of its investments in research. It was also developing a set of performance indicators.

5.103 Recommendation. CIHR should conduct an evaluation of its operating grants program that addresses all four evaluation issues stipulated in the Treasury Board guidelines. It should also develop and use appropriate performance indicators to monitor and report the results achieved by research that it funds.

Agency's response. CIHR has already commenced a formal, in-depth evaluation of the operating grants program and expects results by November 2002. A Results-Based Management and Accountability Framework for the program, targeted for February 2002, will identify a best set of indicators for measuring ongoing performance. Activities such as these will deepen and enrich the information base for program management and for CIHR's reporting to Parliament and Canadians.

Project assessment and approval

5.104 Approval process. All applications for research grants go through a competitive process. To make sure that it funds only high-quality research proposals, CIHR uses a peer review system. Applications are screened initially by CIHR staff to ensure that applicants have submitted all the required information. A grant assessment committee then reviews the applications in detail. Committees are made up of experts in various disciplines of the health sciences who are working scientists, mainly at Canadian universities. Many of these peer reviewers are themselves active in related research. Each research proposal is also sent to outside referees who specialize in the applicants' fields; the grant assessment committee considers the referees' comments in making the final assessment.

5.105 Applications are rated on a point scale and ranked in order of merit. The results of the competition and the staff recommendations for funding are scrutinized by an oversight committee. This is a standing committee of the Governing Council, composed of researchers and other stakeholders. It makes final recommendations to the Governing Council for its approval.

5.106 The eligibility criteria used in assessing the research proposals are scientific merit, potential impact of the work, originality, feasibility, relationship with applicant's previous work, and a critical review of the literature. After assessing scientific merit, the review committee examines the budget proposed by the applicant.

5.107 Thorough assessment of applications for funding. In our view, applications for operating grants that totalled $190 million were completely and thoroughly assessed. Grant files were complete and contained all pertinent information on the grants awarded. Standard grant applications, appraisal checklists and forms, approved notices of grant awards, budgets, and financial status reports are used to document and record all decisions by selection committees and outside reviewers and all the related processes.

Project and financial monitoring

5.108 Funds provided by CIHR for approved projects are administered by the business offices of the institutions at which the grant recipients are located—mainly universities, hospitals, and research institutes. The business offices not only maintain common grant accounts but also monitor expenditures charged to each research account. CIHR makes regular monthly or bi-monthly payments to these accounts based on the amounts it has approved.

5.109 Unspent balances increasing. We found that CIHR receives annual detailed statements of actual expenditures, by grant recipient, from the business offices of their institutions. We expected that CIHR would review these statements for large unspent balances to determine whether the pace of expenditures reflects the anticipated progress of the research and the corresponding budget. We found no evidence of systematic follow-up action to reduce unspent balances. CIHR management informed us that its payments to the research institution start two months after the project begins and are bi-monthly thereafter, regardless of the pattern of actual expenditures on the research project. CIHR also allows its researchers to use unspent balances that remain in their accounts for two years after funding ends, in order to wind down their projects or to seek funding from other sources.

5.110 Universities also submit annual summary financial status reports on all the CIHR-funded research projects conducted there. At 31 March 1999 the total unspent balance was $67 million; by 31 March 2000 it had grown to $95 million. At 31 March 2001, unspent balances totalled about $113 million among roughly 3,871 projects administered by 66 research institutions across the country. CIHR's total budget for grants and scholarships that year was about $370 million.

5.111 Although management was aware of this large balance, it did not investigate why it was increasing. CIHR intends to develop a process to identify, investigate, and reduce unusually large year-end balances in the future.

5.112 No systematic tracking of research results. We noted that management does not systematically assess the scientific results achieved for its $190 million support to research.

5.113 Recipients of research grants are not required to submit progress reports unless they have received five-year grants or they apply for a renewal. About 75 percent of recipients apply for new funds and, at that time, the review panel considers their productivity as measured by published research results. We found that management obtains the required progress reports both for five-year projects and for renewals but does not require a final report on research results when projects end. More important, CIHR does not have a framework for capturing the results of completed research as a basis for assessing program performance and providing information to Parliament.

5.114 CIHR management has indicated that it is committed to improving the ongoing measurement of performance where it will yield useful information for Canadians. There are many methods used to evaluate research results. Exhibit 5.2 presents the advantages and disadvantages of using some of these methods.

5.115 There was little on-site monitoring by management. Site visits help ensure that funds entrusted to the research institutions are managed well and spent for the intended purposes. Visits are also useful to assess the progress of projects and collect information on the results of the funded research. Although management conducted some site visits as part of the appraisal process, they were to only a few major projects.

5.116 Management relied to a large extent on the financial and audit monitoring procedures in place at the research institutions. We noted that CIHR had not conducted site visits to examine the appropriateness of these procedures until 2000-01, when it started making monitoring visits jointly with other federal government research funding agencies.

5.117 Continuing eligibility needs to be verified. CIHR is subject to the Financial Administration Act. A person authorizing a payment of public funds must certify under section 34 of the Act that the recipient of the money is eligible for or entitled to payment.

5.118 During the course of our audit, we noted that there was no certification under section 34 for grant payments made by CIHR in 2000-01. Management told us the risk was low that payments were made to ineligible recipients. Its financial systems allow grant payments to be made only by individuals with delegated authority.

5.119 Nevertheless, CIHR acknowledged that explicit certification under section 34 is required and it has resumed this practice. It also indicated that it would further strengthen its assurance of recipients' continued eligibility by having institutions specifically certify through a special annual report that grant recipients at their institutions are still eligible for funding.

5.120 Recommendation. CIHR should do the following:

monitor the financial status of projects and take action when projects accumulate unspent balances higher than are normal; and
develop an appropriate way to assess the results of the research projects it funds.

Agency's response. The Business Office at each grantee's institution provides CIHR management with accurate information on disbursements. As the rate of disbursement for research grants is affected by many unpredictable variables, it is to be expected that year-end accounting will show unspent balances. We appreciate the recommendation that unusually large balances should be individually investigated, and we are putting in place administrative processes so that, when appropriate, payments of grants may be adjusted to more closely reflect the researcher's rate of expenditure.

Our world-class peer review system ensures that CIHR supports good research by Canada's best health scientists. To better capture information on the results of that research, CIHR will have in place a performance measurement framework and strategy before the end of fiscal year 2001-02. We are determined to continually improve our capacity to keep Canadians informed of the important advances in knowledge resulting from their investment in the work of Canadian health researchers.

Conclusion

5.121 We found that CIHR carried out complete and thorough assessments of applications for operating grants. It uses a rigorous peer review system to identify projects for funding. However, we found that its monitoring of how quickly researchers spend their grants needs to improve, as does the tracking of research results. CIHR also needs to improve its assessment and reporting of program performance.

Section 4—Citizenship and Immigration Canada—Settlement Contribution Programs

In brief

The Settlement Contribution Programs of Citizenship and Immigration Canada are in a period of transition between a previous decision to devolve them to the provinces and, later, a decision that the Department would retain and manage them. Following the later decision, the Department began a significant reinvestment of resources in the programs.

We note that financial and management controls over settlement programs are adequate. However, the Department needs to strengthen on-site monitoring and provide its program officers with more financial training.

The Department's normal practice is to enter 12-month contribution agreements with its service providers. We believe it could gain efficiency by moving to a multi-year funding process. However, it should do this only after reaching a common understanding with service providers on its requirements for performance measurement and reporting. The Department does not yet have the data to gauge how well its settlement programs are performing. It has conducted no overall evaluation of its settlement programs since its creation.

Background

5.122 The Settlement Contribution Programs are delivered under two of Citizenship and Immigration Canada's five business lines: Promoting the Integration of Newcomers, and Maintaining Canada's Humanitarian Tradition. We audited the four settlement programs: Language Instruction for Newcomers to Canada (LINC), Immigrant Settlement and Adaptation Program (ISAP), the Host Program, and the Resettlement Assistance Program (RAP).

5.123 The overall objectives of the settlement programs are to help newcomers to Canada adapt, settle, and integrate. While the Department acts as a funding agency, service providers are responsible for actually delivering the programs. The Department enters into contribution agreements with them that specify the services they are to provide.

5.124 The terms and conditions for the language instruction, settlement and adaptation, and host programs state that agreements with service providers will not normally exceed 12 months. The terms and conditions for the Resettlement Assistance Program state that agreements with service providers will not normally exceed 36 months.

5.125 Under a 1991 Canada-Quebec accord, Quebec is solely responsible for settlement and integration services in that province and it receives a federal grant to provide those services. Agreements were also signed with Manitoba and British Columbia to transfer settlement services, effective January 1999 and April 1999 respectively. However, in both those provinces the Department still delivers the Resettlement Assistance Program.

5.126 Language Instruction for Newcomers to Canada. The objective of the language instruction program is to provide adult immigrants with training in one of Canada's official languages to help them integrate into Canada. The Department funds service providers to assess linguistic eligibility and provide language training and course material. Eligible individuals can receive the training whether or not they will enter the labour market. Clients may participate for up to three years from the start of training. The program also offers child-minding services and assists with transportation costs.

5.127 Immigrant Settlement and Adaptation Program. Service providers receive funding to provide direct services, such as the following:

information and orientation on day-to-day activities (banking, shopping, managing a household, and so on);
interpretation or translation services when necessary; and
short-term counselling and employment-related services.

The Department also provides funds for improving the overall delivery of settlement services. This includes funds for research projects and staff training programs.

5.128 Host Program. The Host program funds service providers to recruit, train, match, and support Canadians and permanent residents who volunteer to serve as hosts. Hosts are expected to familiarize newcomers with Canadian ways and educate them about available services and how to use them.

5.129 Resettlement Assistance Program. Every year, Canada admits roughly 24,000 refugees. The government plans for roughly 7,300 of these as government-assisted refugees. Government-assisted refugees are selected at missions abroad and, once admitted, form the client group for the Resettlement Assistance Program. The program provides income support and immediate essential services to refugees who lack the resources to provide for their own basic needs. Income support is paid to government-assisted refugees directly by the Department as part of the Resettlement Assistance Program.

5.130 Income support accounts for over 70 percent of the program's $44 million in planned spending each year. The program provides income support for 12 months, or less if the refugee's income becomes enough to meet his or her own needs. Refugees with physical impairments or emotional problems can get up to 12 additional months of support if needed. Recipients are expected to report any change in their income, as this may reduce the income support provided.

5.131 Exhibit 5.3 presents the actual and forecast expenditures, by business line and by program, for the fiscal years 1999-2002.

Focus of the audit

5.132 The objectives of our audit were to determine the following:

the adequacy of the financial and management controls over the contribution programs of the Department;
how the Department has measured and reported the results achieved by the programs; and
the Department's compliance with financial authorities.

5.133 We examined contribution and income support agreements that made payments between 1 April 1999 and 31 March 2000 under the four programs we audited. The Department also has an Immigrant Loans Program, which provides financial assistance to eligible applicants. Since that program is not one of the Department's grant or contribution programs, our audit did not include it. The Department submits a separate annual report to Parliament on the activities of that program. Further details on the audit can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.134 Contribution accountability framework. The settlement programs are in a period of transition. In 1995 the federal government announced that it wanted to withdraw from federal delivery of the programs. It proposed to finance the services through transfer payments to the provinces. To that end, the Department undertook a settlement renewal initiative from 1995 to 1999.

5.135 The announcement slowed down operations in all of the settlement programs, including new investments and initiatives. Four years of operating in a "wind-up" mode resulted in the closing of some local points of delivery, staff departures, and lack of investment in data systems to gather and report on program performance.

5.136 In 1999 the Minister decided not to actively seek new agreements with the provinces. The Department then began a significant reinvestment of resources in its settlement programs. A major initiative was to develop a contribution accountability framework. The Department's goal is that the framework, once completed, will help it ensure accountability for departmental expenditures, monitor service delivery, and evaluate the effectiveness of contribution programs in meeting the settlement needs of newcomers.

5.137 We reviewed the contribution accountability framework against the Treasury Board's policy on transfer payments. We believe that once all the elements of the framework are in place, the Department will be complying with the key provisions of the policy.

5.138 Allocation of program funds to regions. The Integration Branch at headquarters allocates program funds in a block to each of the Department's regions; the funds do not cover the Resettlement Assistance Program. The total amount provided annually since 1996-97 has been $173.3 million.

5.139 The amount of funding allocated to each region is based on a settlement allocation model. The model incorporates the number of recent immigrants in each province or territory (using a three-year rolling average of past immigrant landings) and the estimated costs of settlement services. Once allocated, the funds are administered by the regions and by the two provinces (British Columbia and Manitoba) that have assumed responsibility for settlement services under federal-provincial agreements.

5.140 The model does not apply to Quebec, which receives a grant determined by a formula in a Canada-Quebec accord as federal payment for settlement services provided by Quebec. The formula includes an escalation clause. As a result, the amount of the grant can either increase or remain the same but can never decrease. The amount payable to Quebec in 2001-02 is $101.7 million.

5.141 The Department provided us with the results of the settlement allocation model for 2001-02 (Exhibit 5.4). For purposes of comparison, it also included payments to Quebec.

5.142 The Refugees Branch at headquarters determines the regional funding levels for the Resettlement Assistance Program (RAP) outside Quebec. Funding for RAP is tied to the annual targets for government-assisted refugees. RAP funding is not included in the allocation arrived at by the settlement allocation model.

5.143 Funding process for contribution agreements. The Department's contribution agreements with service providers are normally for 12 months. Service providers submit an application and a detailed proposal that is analyzed by program officers. The program officers then complete an assessment/recommendation form for the proposal. This procedure is repeated each year in the Department for the several hundred service providers it funds. During our audit we noted that 95 percent of proposals were from service providers who had been funded previously.

5.144 We believe that moving to multi-year funding arrangements with well-performing service providers would be more efficient. However, this can be done only after the Department has put in place key elements of its contribution accountability framework. The Department hopes to have the elements in place by 2005. They must include a common understanding and agreement with service providers on what performance information and reporting the Department requires from them to assess the results of its settlement service programs.

5.145 Recommendation. When the Department finishes implementing its contribution accountability framework, it should consider moving to multi-year funding arrangements with service providers.

Department's response. The Department will consider moving to implement multi-year funding agreements for settlement programs once appropriate management controls and evaluation measures are in place. The Resettlement Assistance Program already allows for multi-year arrangements with service providers.

Program performance

5.146 Since its creation in 1994, the Department of Citizenship and Immigration has conducted no overall evaluation of its settlement programs. However, it has done some limited evaluation of a few pilot projects and other, smaller, regional programs. Evaluation frameworks have now been developed for all settlement programs and the Department plans to have them evaluated by 2004-05. It has indicated that this timing may be subject to change, depending on the availability of proper data for analysis. Meanwhile, in 2002-03, efforts will focus on elements such as service quality, client satisfaction, program design factors, and short-term impacts.

5.147 At present, the Department cannot assess how well its settlement service programs are performing. We noted the following:

contribution agreements for language instruction, the largest program, do not call for a final report summarizing the results attained in the course of the agreement;
service providers have not been given clear direction on reporting requirements and methods; and
information presently available in the regions is not captured for use on a national basis.

5.148 We examined the Department's performance reports for recent years. We noted that the information they present on the settlement programs consists mainly of a general description of the services provided to newcomers rather than the overall results of the programs and the achievement of program objectives.

5.149 Performance measurement is the primary focus of the Department's current and planned work in settlement services for 2000 to 2002. The Department aims to be able to collect national performance data on all programs in electronic form by April 2002. In the meantime, national headquarters has started to collect standard data from all service providers.

Project assessment and approval

5.150 The selection and approval of projects for funding is similar for all service providers under the Language Instruction for Newcomers to Canada, Immigrant Settlement and Adaptation, and Host programs. Service providers submit proposals that the Department assesses, using the program criteria.

5.151 In 98 percent of the agreements we examined, we found that the proposals provided satisfactory details. The only exception was one agreement where the proposal could not be located. We also found a satisfactory level of review by program officers before they recommended approval of funding. We noted that program officers paid particular attention to the cash flow forecasts and budgets submitted by service providers.

5.152 Service providers for the Resettlement Assistance Program are selected and approved on the basis of their ability to satisfy the program officer that they can provide the required services. The program officer reviews the submitted proposal, visits the service provider's premises, and interviews its staff. Rates of payment are negotiated on the basis of costs and the target for government-assisted refugees destined for the area. We found a satisfactory level of review by program officers before they recommended proposals for funding.

Project and financial monitoring

5.153 Language Instruction for Newcomers to Canada, Immigrant Settlement and Adaptation Program, and Host Program. Contribution agreements call for service providers to submit both narrative and statistical reports. We noted that these reports were on file in all the agreements we examined. While we did not determine that every monthly report was on file, we were satisfied with the level of compliance with this requirement.

5.154 The analysis and payment of claims is an ongoing activity in the Department. Service providers submit claims that summarize expenditures, by category, for a given period (normally a month). Before approving payments, the program officer reviews the reasonableness and accuracy of the claims. We noted that program officers closely track the funds that remain available as they approve claims, so that actual costs do not exceed the costs stipulated in the contribution agreements. Overall, we found that claims are properly analyzed and the appropriate approvals obtained before payments are authorized.

5.155 Activity and financial monitoring can include formal site visits. One purpose of these visits is to document key aspects of compliance with the terms and conditions of the contribution agreements. National headquarters has provided guidelines on the number of site visits to be made. However, each region is expected to develop its own monitoring system based on its classifying of agreements according to risk. The number of visits, if any, is usually based on the dollar amount and complexity of the agreement as well as prior experience with the service provider.

5.156 Of the regions that make formal site visits, we found that some follow the guidelines on numbers of site visits while others do not. Where they do not, we noted that program officers keep abreast of service providers' activities by reviewing monthly reports and making informal contacts. From our interviews with program officers and our analyses of files, we were satisfied with the numbers of formal visits made in these regions.

5.157 We also noted that site visits are not made in some regions. Program officers in those regions indicated that they monitor the activities of service providers by informal contacts, review and analysis of claims, and periodic reports. We accept that informal monitoring is a good practice and can substitute for site visits, especially for low-value agreements, that is, those under $25,000. However, we do not believe it can substitute for formal visits where complex or high-value contribution agreements are in effect. For example, informal monitoring provides limited assurance that rates paid to employees comply with those indicated in the contribution agreements.

5.158 Recommendation. For higher-risk contribution agreements, the Department should conduct at least one formal site visit to monitor financial controls and project activities.

Department's response. As part of the contribution accountability framework, Citizenship and Immigration Canada is developing a monitoring framework and management controls to clarify the appropriate level of monitoring required for various types of agreements.

5.159 We found that program officers tended not to visit all locations of service providers who deliver their programs at several locations. There were no monitoring plans outlining the locations to be visited, nor were records kept indicating what locations had been visited. As a result, the Department does not know from year to year which locations of a particular service provider it has visited. Also, when files are transferred to new program officers there is no indication of past monitoring. This could mean that the same locations could be chosen again for monitoring and others not monitored at all.

5.160 Recommendation. The Department should ensure that where service providers are delivering services at several locations, the region develops monitoring plans and reports against them to ensure that all locations are monitored over time.

Department's response. Guidelines and procedures will be put in place to assist regions in developing and implementing a monitoring strategy to ensure the rotation of site visits for organizations with multiple sites.

5.161 Some program officers told us they do not feel appropriately trained to conduct visits for financial monitoring, especially of large service providers. They noted that they lack the necessary financial background and have been offered little training to assist them. The Ontario Region hired two auditors on contract in 1998 and three financial officers in June 2001 to provide support to program officers. Other regions have also identified the need for three financial officers; these positions had not been filled at the time of our visits.

5.162 A compliance audit involves an in-depth examination of the service provider's financial records. Program officers use their discretion to judge whether a service provider is to be audited. A discovery during a monitoring visit that a service provider is not complying with terms of the contribution agreement could lead to a compliance audit. At March 2001, the Ontario Region had completed 22 compliance audits; about a third of them concluded that the service providers had kept proper financial records and the expenses they claimed were allowable under the contribution agreements. However, over half of the audits completed to date have identified overpayments and other compliance issues. The overpayments were due to claims for salaries and expenses that were unrelated to the contribution agreement and expenses claimed without sufficient documentation. During our audit, we noted that the Department follows up on the results of compliance audits. In most cases, it is able to resolve the problems and recover any moneys it is owed. The amounts repaid to the Department ranged from less than $100 to about $112,000. In addition, the Atlantic Region had undertaken audits of all 10 of its language instruction service providers, which were being completed at the time of our fieldwork in March 2001.

5.163 Resettlement Assistance Program. The monitoring of service providers in the program varies among regions. In Ontario, it is similar to monitoring in other settlement programs. Monitoring practices of the other regions vary; some use formal visits and others use informal means of monitoring. Overall, we found that program officers have a good understanding of the operations of service providers.

5.164 Under a departmental guideline, at least 10 percent of income support recipients should be monitored. Monitoring is conducted through mail-in questionnaires, by telephone, or in direct interviews. We noted that monitoring focusses more on the appropriateness and level of services provided than on compliance. Most regions met the 10 percent guideline.

5.165 We examined 42 income support agreements and found that the Department had paid the correct amounts for rent, transportation, food, and incidentals. We also noted that appropriate approvals were obtained before money was disbursed.

5.166 Resettlement Assistance Program, Kosovo. After a request in 1999 from the United Nations High Commissioner for Refugees, Canada committed itself to provide a safe haven for 5,000 Kosovo refugees. The Minister of Citizenship and Immigration expedited the processing of these refugees, and the federal government covered the associated costs. As a group, the Kosovo refugees were considered special needs cases, eligible for income support for up to 24 months. The Kosovo effort started in May 1999 and ended in July 2001. The total cost of the effort was roughly $134.6 million. In total, 7,300 Kosovo refugees arrived in Canada and 2,421 had repatriated by July 2001.

5.167 Before the refugees arrived, the Department's settlement group provided them with basic information about Canada. Once in Canada, the settlement group arranged for identification, clothing, interpretation services, and provision of daily allowances. The Department then arranged for refugees to be matched with sponsors.

5.168 In addition to income support, we examined other payments associated with the Kosovo effort, including payments for clothing and transportation. We found that the need for the payments was well documented and proper steps were followed before the goods were distributed. We noted that the Department made an attempt to monitor 100 percent of the recipients eligible for income support payments. At June 2001, it had monitored roughly 85 percent of recipients.

5.169 The Kosovo effort came to a close at the end of July 2001. The Department's headquarters and its Corporate Review commissioned an extensive lessons-learned study, now being completed. In late 2001 or early 2002, the Department plans a conference to discuss the lessons learned. It plans to bring together all the participants in this effort, including service providers, staff from its regional offices and from other departments, refugees, and other participants. Lessons learned reports have already been prepared by different regions and their results will form the starting point for discussion at the conference.

Conclusion

5.170 In general, we found the Department's financial and management controls over its settlement contribution programs to be adequate. Two areas requiring improvement are the on-site monitoring of service providers and financial training for program officers.

5.171 The Department has not been measuring the results of its settlement contribution programs. It has not carried out any overall evaluation of them since its creation as a department. Information on the programs in its performance report to Parliament is limited to a general description of the services provided to newcomers.

5.172 The Department has started to make progress in capturing data for the purpose of measuring and reporting results. It is developing a contribution accountability framework. It has completed evaluation frameworks and intends to evaluate its programs once the proper data are available. These are all steps in the right direction. The Department needs to sustain these efforts to ensure that it meets its stated goal of implementing its contribution accountability framework by 2005.

Section 5—Department of Canadian Heritage—Support for Official-Language Communities

In brief

Support for Official-Language Communities is a component of the Promotion of Official Languages Program. Its purpose is to enhance the vitality of the English and French minority communities in Canada and to support and assist in their development. Expenditures for 2000-01 totalled $33 million. Some 380 organizations receive funding annually from the program.

We found that the program needs significant improvements in its management framework, performance information, project assessment, and analysis of the results achieved. The program's objectives and expected results are stated in vague and general terms, and the program has not been evaluated since the Official Languages Act was amended in 1988. The Department of Canadian Heritage has not developed performance indicators, and its reports to Parliament mainly discuss program activities rather than the program's results. There were also problems in the Department's assessment of projects that accounted for 33 percent or $9 million of the $27.5 million in program expenditures we audited. Finally, analysis of the results achieved by projects lacked rigour.

In the past year, the Department of Canadian Heritage has introduced several measures to improve the management of the program. Although it is too soon to know how effective these measures will be, they should address several of the problems we identified.

Background

5.173 Parliament amended the Official Languages Act in 1988. The Act sets out the obligation to use the two official languages in delivering government services and in federal institutions. It also sets out the federal government's commitment to promote bilingualism in Canadian society and to support and assist in the development of minority-language communities in Canada.

5.174 The Official Languages Support Program comprises the Promotion of Official Languages Program and the Official Languages in Education Program. As part of the Promotion of Official Languages Program, the Department of Canadian Heritage signed 15 agreements with official language minority communities. The purpose of these agreements is to give the communities greater responsibility for determining their own needs and priorities and deciding who should receive funding and how much.

Focus of the audit

5.175 Our audit focussed on Support for Official-Language Communities (a component of the Promotion of Official Languages Program), which spent a total of $33 million in 2000-01. The objective of the audit was to determine whether the Department had adequate control of the program.

5.176 We examined the projects approved for funding between 1 April and 31 December 2000 for the 2000-01 fiscal year, which represented about $27.5 million. We co-ordinated our audit with that of internal audit and relied on its work to minimize duplication.

5.177 Further details on the audit can be found at the end of this chapter in About the Audits.

Observations and Recommendations

Program design

5.178 The objective of the Support for Official-Languages Communities program is set out in section 41 of the Official Languages Act (1988), namely, "to enhance the vitality of the English and French linguistic minority communities in Canada and assist their development." The Department of Canadian Heritage defines the program's objectives as "to increase the ability of official language minority communities to live in their own language and in their own environment as strong, vibrant and healthy communities and to participate fully in all sectors of activity in Canadian society."

5.179 Expected results need to be defined clearly. Increasing the abilities of an official language minority community is a very broad objective. A recent internal audit report said that such generally stated objectives make it difficult to assess the overall results, impacts, and added value of the program. It is important for the Department that the expected results of the program be clarified, taking into account available resources. Clearly defined expected results are essential to a sound program design and enable the Department to measure the results actually achieved by the program. The Department has not yet finished developing performance measures for use in monitoring and evaluating results achieved. Nor has it developed a formal risk management strategy at the program level.

Program performance

5.180 Limited evaluation. In January 1997, the Department carried out an evaluation of the implementation of its Canada-community agreements. The evaluation found that the agreements allowed communities to take certain responsibilities, establish their own priorities to a certain extent, make better use of the available resources, and take more consistent action at the community level. However, the evaluation was not intended to measure the achievement of the program's overall objective, and in fact the program has not been evaluated since the Official Languages Act was amended in 1988. The Department has told us that an evaluation of official languages support programs is scheduled for 2002-03.

5.181 Reports to Parliament are limited to program activities. In addition to publishing an annual performance report, under section 44 of the Official Languages Act the Minister of Canadian Heritage is required, within a reasonable time after the end of each fiscal year, to submit an annual report to Parliament on the matters related to the Department's mandate for official languages.

5.182 These reports describe mainly the program's activities. Although that information may be relevant, it does not address the program's performance or the results it has achieved. We also noted in the projects that we reviewed that the Department had not always obtained information on their results or had not reviewed the information to better measure the results achieved. In our view, the Department has not specified clearly in its annual reports on plans and priorities the results it expects from this program.

5.183 Recommendation. The Department should clarify the objectives of the Support for Official-Language Communities program by setting clear and concrete expected results, evaluate the program's effectiveness, develop indicators to measure the performance of the program, and report the results to Parliament.

Department's response. The Department accepts this recommendation. It has indeed begun implementing it as part of its plan to improve the management practices of the Official Languages Support Programs.

In December 2000, the Department began developing a results-based management and accountability framework for the Official Languages Support Programs. It is now identifying indicators and tools to gather relevant data for results measurement and will then develop a risk-based audit framework. The entire exercise should be finished in March 2002.

The Department is now completing an integrated results-based management and accountability and risk-based audit framework for the Support for Official-Language Communities program. This framework sets out the issues for the evaluation to be carried out next year, in 2002-03. After this evaluation, the Department will be in a better position to present the results of this program in the annual report on official languages and the departmental performance report, both of which are submitted to Parliament. Moreover, we have already begun reflecting results in the 2000-01 reports. The results of the evaluation will also provide an analytical basis to strengthen program effectiveness at the time of its renewal, scheduled for April 2004.

Project assessment and approval

5.184 Process. Although the terms of the Canada-community agreements may vary from one agreement to another, they all include the requirement to establish a development plan that details the community's priorities, an annual financial commitment for the five years of the agreement, and the appointment of a joint committee of community members and departmental representatives to review applications for funding.

5.185 The Department reviews the recommendations made by the joint committees, and sends each application to the Minister for approval. To date, the Department has provided funds almost exclusively as grants, subject to the receipt and acceptance of a recipient's audited financial statements, activity reports, and annual reports.

5.186 In general, applications for funding come from organizations that the Department funds year after year. According to the Canada-community agreements, a minimum of 20 percent of funding is to go to projects and the balance to financing the community organization's yearly activities.

5.187 Development plans are not all completed. The agreements call for the communities to submit five-year development plans (1999-2000 to 2003-04), including the strategies and general priorities to be considered in funding decisions. However, we found that the Department had not received development plans from 5 of the 15 communities with Canada-community agreements. Without a plan, the Department cannot ensure that its funding decisions support projects and activities that are aligned with community priorities.

5.188 Weaknesses in project assessment. In the Department's assessment of applications that we estimate accounted for $9 million (or 33 percent) of the $27 million in expenditures we audited for 2000-01, we found the following shortcomings:

The general application forms were not filled out and the applications were incomplete.
The applications did not state the expected results of the proposed projects, as required by the terms of the program. Furthermore, applications generally did not include a plan to evaluate the progress of activities toward goals or established objectives.
There was nothing in the file to show that the Department had considered the eligibility criteria in deciding to fund activities. Internal audit raised the same issue in 36 percent of the cases it reviewed in 1999-2000 and 2000-01.

5.189 We also noted in most of the files that the level of funding was not adequately justified. The amounts approved varied from 27 percent to 100 percent of the amounts requested, with no rationale provided in the file.

5.190 The Department has not stipulated how to justify funding decisions, for example, on the basis of the applicant's cash requirements supported by its latest financial statements. In at least three cases, the applicant's cash assets exceeded the amount of funding granted, which puts into question the need for the funding.

5.191 Furthermore, where it granted less than the applicant had requested, the Department did not ask for a revised plan that reflected the lowered funding. It would be useful to the Department to know which activities and results the organizations planned to accomplish with less funding than expected, for better comparison of actual and expected results.

5.192 In most of the projects we reviewed, the application made no mention of attempts to obtain funding from other sources or the results of any attempts, as the program's terms and conditions require. That information could help the Department identify actual or potential duplication of government funding and adjust the amount it approved accordingly.

5.193 The Canada-community agreements contain a conflict-of-interest clause. However, in most of the files examined by internal audit, they found that there was nothing to provide assurance that members of the joint committee were not in a conflict of interest when the committee recommended applications to the Department for funding.

5.194 We also noted that the Department has not yet established service standards for the processing of applications for funding. For example, we noted that funding decisions usually took four to six months. In general, applicant organizations are not told of the funding decisions before June or July, which is a cause for concern and could have an adverse effect on activities planned for the year.

5.195 Recommendation. The Department should ensure that

all applications are complete and meet the program's eligibility criteria
the level of funding granted is adequately justified and, if less than requested, the potential effect on the expected results is documented.

Department's response. The Department accepts this recommendation. It created the National Grants and Contributions Review Committee in December 2000 for official languages support programs to provide ongoing auditing, in order to show greater compliance with the principles of due diligence in the review and approval of grants and contributions. On the basis of a review of all the files by a dedicated team, the Committee makes recommendations on all aspects of the processing of grant and contribution applications in order to ensure consistent interpretation of the program's terms, objectives, and criteria.

This Committee has already issued several administrative and program policy directives (for example, policy on the choice of funding mechanisms with a complexity assessment grid; directives on surpluses, significant cash balances, differed revenues, the organizations' fiscal period; procedures regarding the organization of recommendation files and the preparation of contribution agreements).

As part of an ongoing training strategy, workshops to reinforce the principles of due diligence in all aspects of processing files were offered to officers and managers responsible for official languages (fall 2000 and 2001) and training sessions on how to prepare a results-oriented application were offered to recipients (fall 2001). Officer and applicant guides were prepared for these workshops. Specific modules will be developed in response to other training needs.

Project and financial monitoring

5.196 Lack of rigour in analyses of activity reports and financial statements. Internal audit found in only 39 percent of the files it examined that the Department had reviewed either the activity reports or the annual reports supplied by organizations at the end of their projects. Furthermore, nothing indicated that at the end of the projects the Department had examined accomplishments, results achieved, or value added. This lack of rigour is even more serious in light of the fact that most of these organizations receive funding from the program year after year and that such information is essential for proper evaluation subsequently.

5.197 In many cases, we noted a lack of rigorous analysis of the financial statements submitted by the recipients. It appeared in those cases that the Department did not reconcile the grants declared in the statements with the funding it had provided, nor did it assess the possibility that there had been other sources of funding.

5.198 Recommendation. Where a recipient is required to submit financial statements and reports on results, the Department should compare more rigorously the actual results with the expected results to better evaluate the funding granted and to allow for better decision making in awarding funding in the future.

Department's response. The Department accepts this recommendation. The National Grants and Contributions Review Committee also requires that activity reports and financial statements submitted by client groups be subject to more rigorous analysis, and that this analysis be included in the file and be considered in all subsequent decisions. The 2002-03 applications will show progress in this regard because the applicant's guide used to train recipients in the fall of 2001 provides a sample of a results-oriented report.

5.199 Several measures introduced to improve the program's management. During the past year, the Department introduced several measures and initiatives to strengthen the management of official languages support programs and to emphasize managing for results. We noted that it provided training in the use of due diligence; developed a management and accountability framework and a risk-based audit framework; established a national committee to review all funding recommendations; and used a grid to assess project complexity and risk. Although it is too early to know how effective these measures will be, they should help address several of the problems our audit identified.

Conclusion

5.200 We found that the program needs significant improvements in its management framework, performance information, project assessment, and analysis of the results achieved. The program's objectives and expected results are stated in vague and general terms, and the program has not been evaluated since the Official Languages Act was amended in 1988. The Department of Canadian Heritage has not developed performance indicators, and its reports to Parliament mainly discuss program activities rather than the program's results. There were also problems in the Department's assessment of projects that accounted for 33 percent or $9 million of the $27.5 million in program expenditures we audited. Finally, analysis of the results achieved by projects lacked rigour.

5.201 In the past year, the Department of Canadian Heritage has introduced several measures to improve the management of the program. Although it is too soon to know how effective these measures will be, they should address several of the problems we identified.

Section 6—Environment Canada and Natural Resources Canada—Contributions Made Under the Climate Change Action Fund

In brief

The Climate Change Action Fund (CCAF) is one of several initiatives undertaken by the government to help Canada meet its commitment under the Kyoto Protocol. That commitment was to reduce emissions of certain greenhouse gases in the period 2008 to 2012. For the years 1998 to 2001, funding was allocated to two departments: $40 million to Natural Resources Canada and $10 million to Environment Canada and approximately $30 million was spent in contributions.

A recent mid-term evaluation found that the CCAF was generally well-managed, although it recommended improvements in project monitoring and reporting of performance. We noted that the requirement for the Secretariat of the CCAF to report annually on its major achievements and its use of funds has not been fully respected. We found that in general, CCAF-supported projects were properly assessed and approved. We also found that there was adequate control over disbursements. However, the CCAF does not systematically track and compile information on the ongoing performance of the projects it funds.

Background

5.202 The Climate Change Action Fund (CCAF), announced in the 1998 Budget, is one of several federal initiatives to help Canada meet its Kyoto commitments. Canada, along with 160 other nations, negotiated the Kyoto Protocol in December 1997. In doing so, it agreed to reduce Canada's greenhouse gas emissions to six percent below 1990 levels between 2008 and 2012. The Federal Climate Change Secretariat, established in February 1998 and reporting to the deputy ministers of Natural Resources Canada (NRCan) and Environment Canada, is responsible for co-ordinating the federal government's policy commitments and programming on climate change and for managing the Fund.

5.203 Expenditures approved for the Climate Change Action Fund totalled $50 million a year for three years, beginning in April 1998. The funds were allocated for operating expenses as well as contributions to support private and public sector projects. The annual spending was allocated to NRCan ($40 million) and Environment Canada ($10 million). The funding was extended for another three years beginning in April 2001.

Focus of the audit

5.204 Our audit covered fiscal year 2000-01. The contribution payments made by the Fund in that year totalled about $30 million. The CCAF funds projects through several existing contribution programs in other departments and agencies, so we also looked at its accountability framework.

5.205 The objective of the audit was to determine whether the departments had adequate control over contributions made by the CCAF. Further details on the audit can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.206 Program objectives. We expected NRCan and Environment Canada would ensure that the program was designed to achieve results, manage risks, ensure due diligence in spending, and provide accountability for funds spent.

5.207 We found that in general, the program's objectives are clearly stated. The objectives are to encourage and undertake programs and activities that will contribute to Canada's ability to combat climate change and meet its climate change commitments under the Kyoto Protocol. The principles governing the CCAF are spelled out in an evaluation and accountability framework.

5.208 Roles and responsibilities. We found that the responsibilities under the Fund are reasonably clear, although somewhat complicated. Two departments, Environment Canada and Natural Resources Canada, have the lead responsibility and accountability for the program. They have delegated authority to the Federal Climate Change Secretariat to pay for climate change initiatives from the Fund. The Secretariat has created four separate components, called blocks, to deliver the CCAF program: Public Education and Outreach; Technology Early Action Measures; Foundation Analysis; and Science, Impacts and Adaptation. Most of these blocks, in turn, partner with other programs to deliver projects.

5.209 The Secretariat is represented on all the blocks' management committees. All the blocks have business plans outlining their responsibilities. The Secretariat is supposed to co-ordinate the overall review of the activities that all blocks of the CCAF carry out.

5.210 There is no formal risk management strategy. Risk management is intended to strengthen management practices, decision making, and priority setting. At the time of our audit, the program did not have a formal risk management strategy that would identify risks in relation to the results of the program and lead to improved controls. The Secretariat informed us that it mitigates risk by its involvement in selection committees and core assistant deputy minister (ADM) committees. In addition, two recent audits dealt with aspects of program risk. The Secretariat also recently developed a risk-based audit framework for recipients of CCAF contributions. Nevertheless, we encourage management to conduct a comprehensive risk assessment.

Program performance

5.211 Mid-term evaluation recently completed. We expected that CCAF management would know whether it is achieving expected results and would make reasonable efforts to harmonize and co-ordinate its activities with other organizations delivering similar programs.

5.212 A mid-term evaluation was recently completed of all four blocks of the CCAF program. The evaluation addressed all the required evaluation issues except the program's long-term impacts, since it was only a mid-term evaluation conducted for the first three years of operation. An assessment of the program's long-term impact in reducing greenhouse gas emissions will be undertaken in the future.

5.213 The following are some of the main findings of the evaluations:

Public Education and Outreach has been run and managed well and has successfully funded a variety of projects. The report also suggested that it should adopt an approach that would help successful local projects move to the national level.
Most stakeholders were confident that projects and activities of the Technology Early Action Measures block would likely lead to the expected outcomes and impacts. The report recommended that it target industry sectors that have the greatest potential for reducing greenhouse gas emissions.
Science, Impacts and Adaptation objectives were, and remain, relevant to the federal government's climate change agenda.
The evaluation concluded that overall, the Foundation Analysis block was successful in meeting its objectives and was generally well-managed. However, the report recommended that the Secretariat play an active role in monitoring progress and that performance targets and means to monitor progress be established.

5.214 Last year an internal audit by Natural Resources Canada found that the CCAF's selection and monitoring of projects as well as the payment conditions in the contribution agreements could be stronger. However, it found the payment process adequate.

5.215 In addition, the Secretariat's accountability was audited in 2001 although the audit focussed only on its relationship with NRCan. The audit concluded that the CCAF has many well-managed elements and its decentralized business model makes effective use of existing administrative procedures and expertise from across federal departments. The audit identified a number of needed improvements: in service standards, project monitoring and reporting by different partners, financial and progress reporting to the Secretariat, and measurement of the Secretariat's performance.

5.216 Annual reporting requirements were not fully met. The Secretariat is supposed to provide an annual report to the ministers of the Environment and of Natural Resources on the CCAF's major achievements and its use of appropriated funds. We found that no such reports have been provided. In our view, the annual report is a key accountability document, since the funding is distributed to various programs in several departments and agencies.

5.217 Management informed us that it has presented annual reports to the Conference of the Parties (CoP) of the United Nations Framework Convention on Climate Change. In our view, these reports are primarily detailed descriptions of the projects funded by the CCAF, with some project-level results; they do not fully meet the annual reporting required of the program.

5.218 Performance reporting to Parliament is fragmented, focussed on outputs. We reviewed the departmental performance reports and reports on plans and priorities of both NRCan and Environment Canada for 1999 and 2000, and we found few references to the CCAF. While both departments provide a lot of information on climate change, it is dispersed throughout their reports. The October 2001 Report of the Commissioner of the Environment and Sustainable Development was also concerned about fragmented performance reporting to Parliament. Management states that Treasury Board's requirement to report by business line and now by strategic outcomes limits the ability of both departments to report the Fund's accomplishments in a comprehensive way. In our view, the departments should consider reporting in a consolidated way on the CCAF in the Estimates documents.

5.219 Where the CCAF's achievements are discussed, they are presented primarily as outputs and activities; there is little measurement of results achieved. We recognize that it is too early to expect performance information on the overall impact of a program that, like this one, is mainly designed to help meet commitments between 2008 and 2012. Management informed us that a mechanism to verify results is being developed, as one of the important objectives of Technology Early Action Measures. Management also indicated that very few projects have been completed. In our view, management should continue to work on reporting progress toward its objectives, given that it is too early to report on the program's overall impact.

5.220 Under the revised Treasury Board Policy on Transfer Payments effective June 2000, the specific objectives and results planned for grant and contribution programs over $5 million must be stated clearly in the department's report on plans and priorities together with milestones for achievement. We noted that in both departments' 2001-02 reports, summary information on the CCAF's objectives and planned results was provided.

5.221 Recommendation. Management of the Climate Change Action Fund should do the following:

report annually to the ministers of the Environment and of Natural Resources on the program's major achievements and the use of appropriated funds; and
present clear and consolidated information to Parliament on the performance of the program.

Management's response. CCAF management recognizes the need to present clear and consolidated information to ministers, Parliament, and the public on the performance of the CCAF. In the interests of efficiency, past requirements for annual reporting to ministers had been combined with other annual reporting requirements, such as the compendia prepared for CoP 5 and CoP 6 and the recent report Responding to the Challenge: The Climate Change Action Fund (CCAF) 1998-2001 Report. For 2001-02 and subsequent years, a detailed, integrated, stand-alone annual report will be prepared covering all activities under the CCAF in terms of project spending and results achieved. The CCAF will also be addressed in a more consolidated way in future Estimates documents.

Project assessment and approval

5.222 Departments and agencies that deliver the program generally use their own contribution program terms and conditions. Nevertheless, the CCAF's program authority requires that in verifying eligibility for funding by the CCAF, delivery agents must consider the Fund's overall objectives. They must also look for opportunities to leverage funds and share costs wherever possible with the provinces and the private sector, set concrete milestones and expected results, and ensure that the approval process is transparent.

More diligence needed in approving projects

5.223 We found that for the most part, the Secretariat satisfies itself that each program component exercises due diligence in approving projects. We found some problems in areas of project management, such as assessing project proposals, determining eligibility, documenting key project assessment decisions, and managing repayable contributions. (Our sample did not cover contributions managed by the Foundation Analysis block because total spending in 2000-01 was not significant.)

5.224 Some problems with verifying eligibility and assessing project proposals. At the time of our audit, to be eligible for funding through Technology Early Action Measures, project proponents had to be members of a climate change association. Membership in such an association demonstrates the applicant's commitment to climate change issues. In 14 of 20 cases we looked at, Technology Early Action Measures projects were approved although the applicants did not have the required membership.

5.225 Two projects, one under Public Education and Outreach and one under Science, Impacts and Adaptation, were approved without assessing the proponents' capabilities to carry out the projects. In both cases, management informed us that the project proponent had carried out another project for them and was known to management. In our opinion, adequate documentation should have been included in the project files.

5.226 During the period under review, some commercial projects were approved by Technology Early Action Measures without a formal marketing assessment. Although this is not a criterion for eligibility, a more structured approach to evaluating commercial projects that includes review of a marketing or business plan would be appropriate. The mid-term evaluation of the CCAF raised the same point. Management has indicated that delivery agents who did not have market assessment tools later received training to deal with commercial projects.

5.227 Key documentation not found in many files. We found that project documentation varied greatly among the different components of the CCAF. Files on Public Education and Outreach projects and Science, Impacts and Adaptation projects contained all key documents. The Technology Early Action Measures project files contained the standard project recommendation form, documenting the project assessment. However, we found that each project file we examined from Technology Early Action Measures lacked one or more pertinent documents such as applications, contribution agreements, approval decisions, progress reports, and final reports. Technology Early Action Measures management told us that this was because the projects are managed by delivery agents from participating departments and all the relevant documents are in the delivery agents' project files. They also informed us that in the future it will attempt to keep complete documentation on file, especially the contribution agreements. Having all the relevant documents at the block level is critical for the Secretariat to determine whether the projects are meeting CCAF objectives and to assess their progress.

5.228 Problems in determining repayment terms for contributions. The Policy on Transfer Payments requires departments to ensure that where a contribution to a business is intended to allow the business to generate profits or increase its value, the business must repay the contribution or share the resulting financial benefits with the government, commensurate with its sharing of the risks.

5.229 According to eligibility criteria for funding by Technology Early Action Measures, contributions to all commercial projects—those where profit is expected within three years—are repayable. We found that some of these projects were approved by Technology Early Action Measures before the terms, timing, and basis of repayment were determined.

5.230 Both Technology Early Action Measures management and delivery agents are supposed to ensure that the recipients of a repayable contribution declare the status of any previous debts to the federal government before approving the contribution. We found that this was not done for some recipients.

5.231 Contracts were used where contribution agreements were appropriate. We found that four projects were handled as contracts instead of contributions by two delivery agents. The projects in question had all the characteristics of contributions: they were subject to the same eligibility criteria and went through the same project assessment and selection process; their costs were shared with private firms; the government was not getting any goods or services in return; and above all, some of the funds were repayable. In our view, these projects should have been handled as contributions and not contracts. In deciding whether contracts or contribution agreements should be used, delivery agents should ensure that they characterize appropriately the nature of the contractual arrangement and comply with applicable Treasury Board policies on contracting and on transfer payments.

5.232 Recommendation. The Climate Change Action Fund management should ensure the following:

that each funded project meets all program criteria;
that project files at the block level contain all required documentation;
that the terms, basis, and timing of repayments are specified and the status of any outstanding debts to the Crown is known before repayable contributions are approved; and
that delivery agents use contracts and contributions appropriately.

Management's response. To minimize overhead costs, the CCAF builds on existing federal program infrastructure. CCAF management recognizes the need to ensure that its delivery agents fully respect program criteria, use appropriate financial instruments, and maintain complete files. Departments and project selection committees will be directed to take greater care in making sure that project proposals meet all program criteria, repayment considerations are in order, and appropriate financial instruments are used. Effective January 1, 2002 all CCAF blocks will maintain centrally a complete set of project files, each containing all required documentation.

Project and financial monitoring

5.233 Need for more systematic and meaningful monitoring of results. The Secretariat is supposed to receive quarterly progress reports on projects administered by the Fund's blocks and delivery agents. While a variety of reporting mechanisms exist, we found no formal or systematic reporting of project results to the Secretariat. The required progress reports are to include projects undertaken, deliverables and milestones achieved, and funds leveraged, so the Secretariat can know whether project performance is on track. We found that the Secretariat does not systematically track and compile results and milestones achieved.

5.234 However, last year's internal audit of the CCAF found that delivery agents monitor the projects they administer. The internal audit also noted that progress reports on 6 of the 26 projects it reviewed were late. The audit report recommended that the CCAF ensure that delivery agents provide the program blocks with regular progress reports to demonstrate the progress of CCAF-funded projects and to promote greater accountability.

5.235 Financial control of disbursements. We found that CCAF management had properly approved all disbursements transferred to delivery agents. An internal audit conducted by Natural Resources Canada also concluded that there were appropriate controls on payments. It also found there was evidence on file to demonstrate that claims were reviewed before payment, no payments were made before agreements were signed, and signing authorities were respected.

5.236 Financial monitoring by the Secretariat can be improved. We found that the program blocks do not provide the Secretariat with the required quarterly financial reports. The Secretariat has received some reports from them as needed. To be useful, financial reporting must be not only accurate, as we found it, but also regular and systematic.

5.237 Recommendation. CCAF management and the program components should track the results of funded projects in a formal and systematic way to ensure that project goals are met.

Management's response. To minimize the administrative burden on CCAF delivery agents, CCAF management in the past has issued calls for financial and progress reports on an as-needed basis. CCAF management agrees that a more formal and systematic approach should be implemented to ensure that program goals are met. Quarterly financial and progress reports will be formally requested from each CCAF block, beginning with a report due January 31, 2002 covering the third quarter of 2001-02.

Conclusion

5.238 A recent mid-term evaluation found that the CCAF was generally well-managed, although it recommended improvements in monitoring and reporting performance. We noted that the requirement for the Secretariat of the CCAF to report annually on its major achievements and its use of funds has not been fully respected. We found that in general, CCAF-supported projects were properly assessed and approved. We also found that there was adequate control over disbursements. However, the CCAF does not systematically track and compile information on the ongoing performance of the projects it funds.

Section 7—Solicitor General Canada—First Nations Policing Program

In brief

The purpose of the First Nations Policing Policy is to help improve social order, public security, and personal safety in First Nations and Inuit communities by establishing and maintaining police services that are professional, effective, and responsive to the communities' needs. The federal government contributes 52 percent of the costs of these police services, through contribution agreements negotiated among it, provincial and territorial governments, and the First Nations communities. Our audit covers the federal expenditures on the program in 1999-2000, a total of $56 million.

At 31 March 2000, the program was providing funds for more than 800 police officers in 317 First Nations communities, covering about 62 percent of the eligible First Nations population. Since 1995, the number of signed agreements has more than doubled, to 123. We noted that the program objectives are clear. However, one of the objectives—to provide 80 percent of the on-reserve population with local policing by 1995—has not yet been achieved and the Department needs to estimate the additional funding required to reach this coverage. While we noted that the design of the program is sound, we found that significant improvement is required in assessing the police services that First Nations communities need. We found some failures to comply with program authorities. We estimate that about $3 million of spending was not properly supported. We found that of advance payments that were unspent by year-end and carried forward to the next fiscal year, an estimated $1.8 million exceeded the authorized limit. Where Royal Canadian Mounted Police (RCMP) services were contracted, the agreed numbers of RCMP officers were not always provided.

Background

5.239 The First Nations Policing Policy was approved in 1991. In 1992 the Department of Indian Affairs and Northern Development transferred its responsibilities for the First Nations Policing Program to the Department of the Solicitor General. The program provides funding for police services on reserves and certain federal Crown lands, through contribution agreements among the federal government, provincial or territorial governments, and First Nations and Inuit communities. The federal government provides 52 percent of the funding, and the province or territory 48 percent, for police services that are to be professional, effective, culturally appropriate, and accountable to the First Nations and Inuit communities they serve.

5.240 The objectives of the First Nations Policing Program, originally stated in the 1991 terms and conditions approved for the program, are the following:

to provide 80 percent of the on-reserve First Nations population with access to First Nations policing services by 1995-96 (revised to 2000-01);
to provide on-reserve First Nations communities with access to a level of service equal to that enjoyed by similar non-First Nations communities in the region; and
to increase the satisfaction of on-reserve First Nations communities with the policing services provided to them.

Focus of the audit

5.241 Our objective was to determine whether the Department had adequate control over the First Nations Policing Program. Our audit covered the fiscal year 1999-2000, when total funding for the program was $56 million. That amount represented 123 agreements (including 53 self-administered police services that cost $35 million) and covered about 62 percent of the onreserve population.

5.242 Further details can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.243 Program objectives are clear. The defined need for the program is to expand and improve the level and quality of First Nations policing services. In our view, the program objectives and expected results that we have already described are relatively clear. The federal share of funding is set at 52 percent.

5.244 Program management. The mechanisms to ensure accountability for the use of federal contribution funds include the agreements themselves; the requirement for audited financial statements; regular compliance audits; and monitoring and follow-up of action to address audit observations. These mechanisms constitute a relatively sound management framework for a contribution program used to fund ongoing police services.

5.245 The program has a total staff of 28, including regional staff, and an annual budget of $3.3 million for operations and maintenance. We noted that the staff complement has remained relatively constant since 1995, while the number of agreements has more than doubled. Based on our observations of the financial control over this program, this suggests that its resource levels and financial management capacity will need to be reviewed to ensure effective control over funding.

5.246 There is no formal risk management strategy. Risk management is designed to strengthen management practices, decision making, and priority setting. A risk management strategy helps in identifying, analyzing, assessing, monitoring, and controlling risks in managing contributions. It also helps to prevent unsatisfactory or unintended outcomes. During our audit, management documented its management practices. However, it has not developed a formal risk management strategy to ensure that it manages risks properly.

Program performance

5.247 A program evaluation in 1995 assessed the program's relevance and partially assessed its success and cost effectiveness. It also assessed to what extent the program was being implemented as planned. Among other things, the evaluation found the following:

The program was being implemented in accordance with the First Nations Policing Policy.
Anecdotal evidence suggested that communities were either more satisfied than before or saw no change.
While there was substantial progress toward achieving the objectives of both the policy and the program, the objective of 80 percent coverage by First Nations police had not been achieved.
The evaluation raised some concerns, for example, that the agreements seemed to "yield an enhanced policing service" compared with the service available to similar communities that were not First Nations; and not all items were funded, or funded the same way, across agreements.

The Department advised us that a program evaluation is presently scheduled for 2004-05.

5.248 In 1996, the Department estimated that it would need $56 million a year by 2000-01 to provide First Nations policing to 80 percent of the total on-reserve population. By 1999-2000 the cost of the program had already reached $56 million, but the program covered only about 62 percent of the eligible population. At the time of our audit, the Department was reviewing the estimated cost of extending coverage to 80 percent of the eligible First Nations population. In our view, it will be important to know why the funding provided to achieve 80 percent coverage achieved only 62 percent coverage. The Department advised us that it is assessing the trade-offs between enhancing police services funded under existing agreements and negotiating new agreements. This assessment should contribute to the planned evaluation of the program and to a more accurate estimate of the program's eventual cost.

5.249 Reporting to Parliament. We reviewed the Department's 1999-2000 Report on Plans and Priorities and its Performance Report. We found that the Department's Performance Report presented its accomplishments for each key results commitment in its Report on Plans and Priorities. However, it was not easy to make a clear comparison between what was accomplished and what had been planned. For example, the Department reported as accomplishments the number of new policing agreements being negotiated, but did not compare it with its expected result of expanding First Nations policing coverage to between 75 and 80 percent of the eligible First Nations population. We noted that in its 2001-02 Report on Plans and Priorities, the Department responded well to the requirements of the new Policy on Transfer Payments.

5.250 Recommendation. The Department should develop a reliable cost estimate of the eventual cost of achieving all of the program's objectives.

Department's response. The Department agrees with this recommendation.

The Department has initiated discussions with First Nations organizations and with provinces on the need to balance the twin objectives of achieving greater coverage under the First Nations Policing Program (FNPP) and of ensuring the ongoing adequacy and effectiveness of existing FNPP police services over time. These discussions will be helpful in estimating the desired level of service and related costs.

The Department will develop options that will address the Program's coverage and capacity goals and related resources requirements.

Project assessment and approval

5.251 Under the terms and conditions of the program, funding is to be consistent with the costs of policing in communities in the region with similar conditions. Levels of policing service are to be determined on the basis of the following:

demographic characteristics of the population to be served;
size and nature of the geographical areas to be covered and their degree of isolation;
the police workload as determined by reactive and proactive measures; and
a clear statement of the goals of the agreement and a plan for evaluating their achievement.

5.252 What is unique about this program is that while it is funded through contribution agreements, it is based on negotiated tripartite agreements among the federal government, the province or territory, and the First Nations community. The level of services negotiated by the parties has a major impact on the funding provided through the contribution agreement. Generally, agreements are for three to five years and set the funding levels for each year. The agreements can take more than a year to negotiate and sometimes five to six additional months to approve, which often results in several short-term extensions.

5.253 Lack of adequate support for funding decisions. We found that the analysis supporting the level of service and the related level of funding was incomplete in 10 of the 16 self-administered agreements we audited. This includes agreements renewed with funding increases that were not clearly or properly justified, as the program's terms and conditions require. For example, the Department had no information on comparisons with similar non-First Nations communities or no documented rationale for why funding levels exceeded those in comparable communities. While the level of crime is often cited to explain increases in funding, in most cases we did not find analyses of caseloads, by police officer, to justify the numbers of police officers funded. Of the $35 million paid by the federal government through self-administered contribution agreements in 1999-2000, we estimate that about $3 million (or 8.6 percent) had not been properly justified under the program's criteria.

5.254 Unclear agreement terms. Funding is provided through contribution agreements negotiated between three parties—two funders and the recipient. While we recognize that no two agreements will be exactly the same because they are negotiated, we expected to see in all of them basic requirements to conform with program authorities and to ensure accountability for performance. However, we found that the following elements were missing from the 16 self-administered contribution agreements we audited:

Number of police officers. Five agreements did not include a clause specifying the minimum number of police officers that must constitute the police force and for which funding was provided.
Recovery of overpayments or unused advances. We found nine agreements that did not have clauses on recovery of overpayments if total funding provided should exceed eligible actual expenditures. This contravenes the Treasury Board's policy on transfer payments.
Eligible costs. Almost all agreements lacked clauses that defined eligible policing costs. For example, under the program's terms and conditions, minor capital expenditures are eligible. Yet the agreements do not refer to minor capital expenditures, define them, or say which are eligible and not eligible.

5.255 Agreements approved without proper authority. One important control over the spending of public funds is the formal approval of contribution agreements. The Solicitor General has delegated to specific officials of the Department the signing authority for contributions below $1 million. We found three agreements representing total federal funding of $2.6 million that were signed by someone to whom authority had not been formally delegated.

5.256 Recommendation. The Department should properly support the level of funding provided under each contribution agreement and ensure that all contribution agreements are properly approved and include all required elements.

Department's response. The Department agrees with this recommendation.

There are many individual factors that enter into the tripartite negotiation process. For example, some police services are located close to large urban centres, while others involve remote communities policed under a regional police service model. The degree of criminal activity in a community, geographic location, and the costs associated with policing in a remote area can all influence the level of funding provided. The Department has undertaken analysis at the national and regional level of the factors related to effective First Nations policing. It will ensure that the justification to support the level of funding provided in individual agreements is properly documented, and other requirements are met.

Unlike many contribution programs that fund one-time projects or initiatives, the FNPP provides cost-shared resources for essential, and ongoing, police services, with the bulk of the program dollars actually flowing to police salaries. Certain flexibilities thus are required for the effective management of these ongoing police services. The Department will renew the terms and conditions of the First Nations Policing Program and seek applicable exemptions to the Transfer Payment Policy by December 31, 2003.

Project and financial monitoring

5.257 The cash management provisions of the government's policy on transfer payments allows contributions to be paid in advance as the recipient's cash flow requirements dictate. The program's terms and conditions require a 10 percent holdback, to be released on receipt of the final claim provided that the recipient has met the terms and conditions of the contribution agreement. Agreements require that recipients submit audited financial statements within five months after year-end. Any funding unspent at year-end can be carried over to the next year for police service costs, as the individual agreements may provide. Some agreements require written approval by the federal and the provincial governments for any carry-over above a certain percentage of annual funding (generally 5 percent).

5.258 Failures to comply with authorities. We found instances where the Department acted outside the authority of the program's terms and conditions and outside the cash management requirements of the Treasury Board's transfer payment policy. The Department tells us it will ask the Treasury Board to approve revised terms and conditions that will better reflect changing realities and that will incorporate the flexibility needed for optimal administration. Nonetheless, the following are examples we found where the Department did not comply with existing authorities:

Quarterly advance payments instead of monthly. The transfer payment policy allows for quarterly or monthly advance payments, according to the amount of the contribution. In this program, the Department made advance payments in equal quarterly instalments. In 15 of the 16 self-administered agreements we audited, the amount of the contribution dictated that only monthly advance payments were allowed under the transfer payment policy.
Advances not based on cash requirements. The policy on transfer payments requires that the amount of each advance payment be based on a cash flow forecast from the recipient and that it take into account any advance payments issued and interest earned on them. Also, the policy requires that balances carried over to the next fiscal year be limited to the cash requirement for the month of April. The Department complied with neither of these requirements in most cases. We estimate that about $1.8 million of advances unspent at year-end exceeded the authorized levels for carry-overs and were not recovered.

5.259 Funding provided for ineligible costs. Of the $35 million paid by the federal government under the self-administered contribution agreements in 1999-2000, we estimate that about $0.8 million (or 2.3 percent) was for expenses that were not eligible. They included items such as major capital expenditures (specifically, funding for police stations), the previous year's deficits, and other ineligible expenses identified by audits the Department conducted of some agreements that year. We found that the Department has paid for capital infrastructure through a number of agreements, in various ways. For example, it has reimbursed loans and interest to finance construction of police stations or major repairs to them, provided fixed amounts for capital purposes each year, and allowed surpluses from the operating budgets to be used for capital spending. Spending on infrastructure such as buildings (police stations), houses, and roads is not an eligible expenditure under this program. As mentioned earlier, the agreements do not stipulate which costs are eligible and which are not; they mention only that funding is to be used for policing costs.

5.260 While the Department told us that the mandate to fund capital infrastructure for policing services had not been transferred from Indian and Northern Affairs Canada (INAC), departmental correspondence indicates that INAC does not have the same understanding. This needs to be clarified and the Department has told us it is addressing the issue at present. There is also a risk that some rents are reimbursed for the use of police stations whose construction INAC might have paid for already.

5.261 Retroactive payments made for which no cost had been incurred. Payments are made retroactively when a new agreement that provides increased funding could not be negotiated before the term of the existing agreement ended. For example, in one case, an interim agreement was approved with the same terms until a new agreement could be reached. When a new agreement was signed it increased the number of police officers, retroactive to six months before. The Department had not verified that the additional police officers had actually been hired and the costs incurred when it paid the recipient the increased funding. We noted that at 31 March 2000 the recipient had a surplus that it was allowed to keep.

5.262 Project evaluation and audit. Ten of the agreements we examined stated that an evaluation could be conducted in the last year of the agreement. Six of the agreements had been evaluated. These evaluations provided feedback on the effectiveness of the police services. We noted that program management took action where required to ensure that any necessary improvements were made.

5.263 All but one of the agreements we examined had an audit clause. Audits are conducted regularly, at least once every five years for significant agreements, and management follows up on the audit recommendations. In the one exception, an audit had been scheduled for 2000 but the community refused the auditors access. At 30 April 2001 the issue was still unresolved, but the Department has since advised us that it plans to conduct an audit in 2001-02. We also noted that there has been no internal audit of the program's management.

5.264 Monitoring. The program's regional managers are responsible for ensuring that communities receive the police services the Department has funded. This is done in many ways: visiting the communities; participating in meetings; obtaining activity reports; and holding discussions with band council, police board members, and chiefs of police and officers. We did not assess how well this is done. However, we noted that the Department does not always request and obtain performance or activity reports for all agreements. As a consequence, it does not have complete information on the actual levels of service, that is, the number of police officers in service and the service they provide.

5.265 At the time of our audit, the Department had neither the consistent and reliable data nor the key indicators it needs to measure the program's performance. It has recognized the difficulty of getting consistent and reliable data to measure the program's performance. At the time of our audit, the Department had undertaken a number of initiatives to assess both the efficiency and the effectiveness of First Nations policing in order to ensure that funds are administered appropriately and the policy objectives met. It was too early, however, to assess the results of these initiatives.

5.266 Where the RCMP continues to provide police services under community tripartite agreements (CTA), the three parties agree each year on the number of police officers needed in the community. For 1999-2000, a total of 171 RCMP officers were contracted for but only 148 (14 percent fewer) were actually provided and paid for. This means that the agreed level of service was not always provided in some communities. For example, in one province there were 85 RCMP police officers, 16 percent fewer than the 101 contracted for. According to RCMP officials, the situation improved in 2000-01.

5.267 Recommendation. The Department should seek the Treasury Board's clarification on who is responsible for federal funding of infrastructure for police services and should co-ordinate with Indian and Northern Affairs Canada, as appropriate, to ensure that any funding for major capital expenditures is provided in accordance with proper authorities. The Department should also improve the monitoring of projects' performance and its financial control over projects.

Department's response. The Department agrees that the mandate, authority and funding for infrastructure for police services needs to be clarified. The Department has already initiated discussions with the Treasury Board Secretariat and other central agencies, with PWGSC, INAC, the RCMP, and the provinces, and plans to continue these discussions.

In terms of improving project performance monitoring, the Department recognizes the importance of performance indicators, notwithstanding that such indicators are preliminary at best even in "mainstream" policing contexts. A number of initiatives are currently under way, including research on effectiveness, and are expected to provide a basis for developing some preliminary performance indicators for First Nations policing services.

The Department acknowledges that the rapid growth in the number of agreements—a reflection of strong community interest in enhanced public safety—has put pressures on departmental program management resources.

Discussions are under way with provinces on the potential for co-ordinating both financial and operational audits. The five-year audit plan was recently expanded to include audits of the RCMP-delivered community tripartite agreements. The in-house capacity for analyzing and monitoring financial statements will be strengthened through the establishment of an additional financial resource. As well, generic financial clauses are being developed that will be included in the self-administered policing agreements, where feasible, as they are renewed.

Conclusion

5.268 At 31 March 2000, the program was providing funds for more than 800 police officers in 317 First Nations communities, covering about 62 percent of the eligible First Nations population. Since 1995, the number of signed agreements has more than doubled, to 123. We noted that the program's objectives are clear. However, one of the objectives—to provide 80 percent of the on-reserve population with local policing by 1995—had not yet been achieved. The Department needs to estimate the additional funding it requires to reach this coverage. While we noted that the design of the management framework is sound, we found that significant improvement is required in assessing the police services that First Nations communities need. We found some failures to comply with program authorities. We estimate that about $3 million of spending was not properly supported. We found that of advance payments that remained unspent by year-end and were carried forward to the next fiscal year, an estimated $1.8 million exceeded the authorized limit. Where RCMP services were contracted, the agreed numbers of RCMP officers were not always provided.

Section 8—Western Economic Diversification Canada—Community Futures Program

In brief

The Community Futures program was established by Human Resources Development Canada in 1986 under the Canadian Jobs Strategy. Its purpose was to create Community Futures Development Corporations (CFDCs) that would diversify the Canadian rural economy by creating or maintaining jobs in new or existing businesses. They would do this by delivering a range of services in the areas of community economic development, business assistance, and access to capital.

Community Futures was transferred to Western Economic Diversification Canada (WD) in 1995 and that Department entered new contribution agreements with the CFDCs to reflect the change. WD provides the 90 CFDCs with a total of roughly $21 million in operating funds each year, which they use for accommodation, salaries, materiel, and legal obligations. Since taking over the program, WD has provided $72 million to CFDCs in investment funds, which are used to make loans.

We found that the program terms and conditions are well designed, setting a clear framework within which CFDCs are responsible for their operations and accountable to WD. The Department's monitoring of the performance of CFDCs relies heavily on information whose accuracy it does not systematically assess. Although WD is making improvements, its monitoring procedures do not yet enable it to assess adequately whether CFDCs are complying with the program's terms and conditions. In our view, Parliament does not receive enough information on the results of the Community Futures program.

Background

5.269 The federal government created Western Economic Diversification (WD) in 1987 to develop and diversify the economy of Western Canada, to co-ordinate federal economic activities in the West, and to reflect Western interests in national decision making. The Department offers services to every community in Western Canada through a variety of programs, such as the Western Canada Business Service Network, Women's Enterprise Initiatives, Canada Business Service Centres, and the Community Futures Development Corporations (CFDCs).

5.270 The Community Futures program was started by Human Resources Development Canada in 1986 as part of the Canadian Jobs Strategy but was transferred to WD in 1995. The program provides the CFDCs with roughly $20.7 million a year in operating funds. Since taking over the program, WD has provided $72 million in investment funds to the CFDCs. WD contributions to CFDC investment funds are conditionally repayable; operating funds are non-repayable. Upon dissolution of a CFDC, once outstanding liabilities are settled, all remaining investment funds must be returned to WD.

5.271 The CFDCs are non-profit corporations responsible for a variety of community-focussed activities that support economic development in designated rural areas. They help communities to develop and diversify through the following activities:

Strategic community planning—working with communities to assess local problems; establish objectives; plan and implement strategies to develop human, institutional, and physical infrastructures; and support entrepreneurism, employment, and the economy.
Business services—delivering a range of business, counselling, and information services to small and medium-sized enterprises.
Access to capital—providing capital to assist existing businesses or to help entrepreneurs create new businesses.

5.272 The CFDCs use their operating funds from WD for community economic development, business services marketing, and administration. They use their investment funds to make loans at no less than two percent interest plus prime, to guarantee loans, and to provide share capital to eligible businesses. Each CFDC operates with a small core staff of officers and a volunteer board of directors drawn from the local community.

Focus of the audit

5.273 Our audit covered the operations of the Community Futures program during 2000-01. Our objective was to determine whether WD had adequate control over this program.

5.274 Further details can be found in About the Audits at the end of the chapter.

Observations and Recommendations

Program design

5.275 The terms and conditions of the program provide a good framework to hold CFDCs accountable for their use of public funds. They require WD to ensure that no CFDC receives more than $300,000 each year for operating costs (with an additional $20,000 to CFDCs in remote areas for travel costs). No CFDC is to receive more than $6 million in investment funds over its life.

5.276 CFDCs are to develop and diversify the economy by loaning investment funds to help create new businesses and stabilize or expand existing businesses, primarily outside metropolitan areas. The program's terms and conditions set a number of rules for CFDC lending practices, in particular the following:

CFDCs are not allowed to make any loan of over $125,000 to any business.
They may provide loans only to businesses that have a reasonable expectation of economic viability and whose principals have an appropriate level of financial involvement.
CFDCs may not use investment funds from WD to make grants, contributions, or forgivable loans and they must avoid conflicts of interest.
They cannot make a loan until the applicant provides evidence that it has explored all other reasonably available avenues of financial assistance.

5.277 CFDCs must deposit and maintain contribution funds in two separate accounts: operating costs and investment fund capitalization costs. CFDCs can collaborate with each other or with private sector lending institutions to establish and administer investment funds. They must also complete annual business plans, periodic activity reports, and annual audits of their operations.

5.278 Risk assessment. WD has started acting on the results of a quality assurance review that identified risks in grant and contribution programs and related best practices. It has incorporated risk assessment in its funding procedures. The risk assessment considers factors such as the amount of funding, complexity of the project, number of funders, public profile, previous history, financial statements, and duration of the project.

Program performance

5.279 Most of the available information on the performance of the Community Futures program comes directly from the individual CFDCs. The Department has not conducted an evaluation of the program, nor has it carried out an internal audit.

5.280 When Community Futures was transferred to it, WD entered new contribution agreements with the CFDCs to emphasize its own program objectives. Its contribution agreements with the CFDCs set out performance targets and performance reporting requirements. However, our review of CFDC performance reports showed that they usually do not include all the performance information stipulated in the agreements, particularly on CFDC assistance to Aboriginal clients and disabled clients.

5.281 The agreements require CFDCs to provide business plans, activity reports, and audited financial statements as a basis for WD's funding decisions. Performance reports by CFDCs usually contain management assertions on their performance. But WD has no procedures for assessing the accuracy of these assertions or auditing the results claimed, for example, numbers of jobs created or maintained by CFDC activities. One CFDC indicated in its annual report that its lending activities were responsible for creating or maintaining hundreds of jobs with a payroll of tens of millions of dollars over 10 years. Another CFDC reported dozens of jobs created and maintained through a relatively small loan and even less invested by the borrower. WD should have some assurance that such results can be fully attributed to its funding.

5.282 Reporting to Parliament. The department's performance report includes information on the leading activities of the CFDCs—number and value of loans—as well as job creation and maintenance numbers reported by CFDCs to WD. However, WD provides little information that links the results of these lending activities to its objectives or strategic initiatives. The Department's 2001-02 Report on Plans and Priorities indicates two planned results for the community futures program. We encourage the Department to add and report on outcomes and milestones in future reports.

5.283 Recommendation. The Department should do the following:

establish procedures to assess the accuracy and validity of information that CFDCs provide on their performance;
provide a complete overview of the program's performance in its reports to Parliament; and
consider conducting a program evaluation and an internal audit of the program.

Department's response. The Department agrees.

WD will strengthen procedures to assess the accuracy and validity of CFDC performance information provided.
The observations reflect the inherent difficulty of performance measurement in economic development programs where results are achieved over a long period of time. Efforts to improve performance reporting are ongoing. WD will expand its reporting on the Community Futures program in its 2002-03 Performance Report.
The terms and conditions of the Community Futures program call for an evaluation by May 2003. WD will initiate the evaluation in April 2002, based on a results-based management and accountability framework that is under development. An audit of the management control framework of both WD and the CFDCs for operational and loan funding will be undertaken concurrent with the evaluation.

Contribution assessment and approval

5.284 Contribution agreements with CFDCs. Following the transfer of the Community Futures program to WD in 1995 and a subsequent increase in its contribution limits, the Department entered new contribution agreements with CFDCs to more fully reflect its program objectives and reporting obligations.

5.285 We found that WD had signed new contribution agreements with all CFDCs in our sample. The agreements state the financial limits on both operating and investment funds for each CFDC. Contributions are identified as repayable, non-repayable, or conditionally repayable, depending on the terms and conditions of agreement. We found that amounts stated in the agreements and the funds subsequently provided were within the financial limits authorized by the Treasury Board.

5.286 However, it was not clear to us how WD established the amount of funding it would provide to individual CFDCs each year. Its contribution agreements require CFDCs to indicate any additional sources of funding. Many CFDCs are involved in a range of social and economic development activities, often in a fee-for-service arrangement with another government department. Management at WD told us that it takes these additional sources of income into account in its funding decisions. However, the documentation supporting the contribution agreements we audited gave no indication of how WD had established amounts of annual funding.

5.287 Recommendation. The Department should ensure that its decisions on how much funding CFDCs receive are properly supported and documented.

Department's response. The Department agrees. WD's decisions on funding for CFDCs are supported by an analysis conducted in 1998 of annual funding levels required by CFDCs to deliver core services. Many factors are considered to determine individual CFDC funding, such as anticipated interest revenue from investment funds, cost of serving remote communities and/or large geographic areas, economic hardship of communities served, and high numbers of targeted clients (Aboriginal, Francophone, persons with disabilities).

Each CFDC submits an annual operating plan providing in-depth information related to the funding request, including sources of funding and expenditures for all activities, as well as information on the previous year's performance.

WD will review the current annual processes and level of documentation to determine what further action may be required.

Financial and performance monitoring

5.288 CFDC lending needs closer monitoring. The program terms and conditions set out a number of clear expectations for the conduct of CFDCs. In particular, they give CFDCs the difficult task of lending to businesses that are viable but that cannot get financing elsewhere. Accordingly, WD considers CFDCs to be high-risk lenders. As a result, we expected that WD would actively monitor the operations and lending activities of each CFDC.

5.289 We found that WD has begun to implement an adequate system for monitoring the operations and lending activities of CFDCs. Until recently, however, its monitoring consisted primarily of reviewing CFDC business plans, annual reports, and audited financial statements. While these are useful sources of information, they do not cover many of the performance expectations set out in the program's terms and conditions . Monitoring progress toward those expectations requires a good knowledge of CFDC lending activities.

5.290 WD officials also have informal ways of checking on CFDCs. In our view these practices are insufficient. WD officials maintain telephone contacts with CFDC officers to stay informed of general developments and activities in the region. They also make ad hoc visits to CFDCs. However, these contacts do not provide a systematic and objective overview of CFDCs' lending activities to ensure that they are meeting the program's terms and conditions. Management in the Department told us its procedures reflect a desire to remain at "arm's length" from the lending decisions of CFDCs. We recognize that the program's terms and conditions clearly do not require WD to be involved in individual lending decisions by CFDCs. At the same time, they do require the Department to know whether CFDCs are complying with the contribution agreements.

5.291 Starting with the fiscal year ending 31 March 2001, WD asked each CFDC to have its auditors provide an opinion on its compliance with the contribution agreement along with the annual opinion on its financial statements. In our view, this should provide WD with good information on the overall lending activities of each CFDC. However, we found these opinions on compliance in only half of the files we examined. Management informs us that the financial statements for the year ending 31 March 2001 were due at the end of July. The Department is following up with CFDCs who submitted statements that did not meet its reporting standard.

5.292 Monitoring means more than simply getting information; it also means acting on it. From time to time, WD has carried out limited reviews of specific CFDCs and found weaknesses in their loan procedures, including a high proportion of loans that were in arrears. We found no indication that WD had acted on these problems. Although it informed us that it is addressing these deficiencies, we found no evidence that its monitoring or follow-up of the CFDCs in question has become more active. We also found that management was aware of non-compliance with the terms and conditions of the program in specific cases but had not acted to correct the problems.

5.293 Recommendation. The Department should ensure that it has accurate and timely information on the activities of CFDCs and should address problems appropriately as they arise.

Department's response. The Department agrees. WD implemented an effective monitoring system through a quality assurance review process that became effective July 1, 2001. The Department is committed to implementation of this system, including improved file documentation and taking appropriate action on issues identified as a result of the monitoring procedures.

5.294 Financial management and control. We found that WD exercised adequate control over disbursements to CFDCs for operating costs. As required by the contribution agreements, CFDCs provide quarterly and annual reports on their expenses and submit audited financial statements.

Conclusion

5.295 Western Economic Diversification Canada provides contribution funds to CFDCs on a five-year cycle. Any contribution program emphasizes the continuing responsibility of the funding organization to monitor and report on the use of the funds and the obligation of the recipient to meet certain conditions throughout the life of the contribution agreement. WD's monitoring of the performance of CFDCs relies heavily on information whose accuracy it does not systematically assess. Although it is making improvements, WD's monitoring procedures do not yet enable it to adequately assess whether CFDCs are complying with the program's terms and conditions. In our view, Parliament does not receive enough information on the results of the Community Futures program.

Section 9—Follow-Up: Department of Canadian Heritage—Grants and Contributions Under the Multiculturalism Program (1998, Chapter 27)

Background

5.296 In 1998, we audited the Multiculturalism program of the Department of Canadian Heritage. We reported that the results expected from the program's grants and contributions were not linked clearly with its objectives. In 30 percent of the cases we reviewed, we noted a lack of due diligence in assessing and approving projects for funding. We also found inadequate follow-up on the performance of projects. We recommended that the Department further clarify the objectives of the Multiculturalism program by defining clear, attainable goals and expected annual results; ensure that due diligence is exercised in reviewing and approving grants and contributions under the program; and ensure that recipients of funding provide the required information on their projects' performance.

5.297 A later internal audit of projects funded under the program between October 1998 and 31 March 1999 identified a continued lack of due diligence in a number of areas. A second internal audit of projects approved for funding between January and May 2000 found that 19 percent of the audited files did not meet the minimum standard of due diligence, and in another 37 percent it was only "borderline acceptable."

5.298 In our 2000 Report, Chapter 33, Follow-Up of Recommendations in Previous Reports, we noted that the Department had not made satisfactory progress on any of our 1998 recommendations.

Focus of the follow-up

5.299 In our follow-up this year, we looked for progress toward correcting these problems. We reviewed tools and processes that the Department had developed and put in place in response to our observations and recommendations. We also interviewed officials and examined a sample of files.

5.300 Our sample was selected at random from files on grant and contribution projects approved by the Department after 1 April 2001. We thought that if the Department had resolved the problems, these more recent files would be the most likely to demonstrate the improvement.

Conclusion

5.301 Our follow-up found that since our last report in December 2000, the Department of Canadian Heritage has made satisfactory progress in addressing our 1998 recommendations.

Observations

5.302 We are pleased to note that the Department of Canadian Heritage has finalized a performance framework for the Multiculturalism program. This framework sets out goals, expected results and performance indicators. We encourage management to use the framework fully in the future.

5.303 We noted as well that the Department has made major changes in the way it processes grants and contributions. Developing and using tools is a step in the right direction to improve the management of grants and contributions. But to ensure due diligence, it is important that the Department follow the established process and obtain the information it needs to assess the merits of proposed projects.

5.304 We found that the Department has made good progress in acting on the problems we reported in 1998. For example, each project file lists the program objectives, spells out the need that the project will meet, links the expected results to that need, and gives other sources of funding. Although it has not yet fully applied it, the Department is implementing a management framework that is proceeding as expected. The framework is an important part of ongoing changes in management and staff practices.

5.305 Among the tools the program officers use in assessing a project proposal is a complexity assessment that incorporates various potential risks. For one of these risks, the sensitivity of the project, the tool attempts to anticipate unexpected social or political outcomes. However, we note that these unwanted potential outcomes are not recorded in the files, making later management review difficult. We encourage management to continue using and refining this tool.

5.306 In 1998 and last year we expressed concern about final reports missing on completed projects. Management has informed us that it no longer requires final reports for projects funded through grants, although it does request them. Our original recommendation is therefore no longer relevant to grants since the program's rules have changed. Nevertheless, it is unclear how management will track the results of these projects in the future for program performance and accountability purposes. Contribution recipients are expected to provide a report at the end of their projects. It is too early to tell whether final reports are routinely received for projects funded through contributions.

Section 10—Follow-Up: Industry Portfolio, Investing in Innovation (1999, Chapter 19)

Background

5.307 Industry Canada, the National Research Council, and Natural Sciences and Engineering Research Council (NSERC) promote innovation in Canadian industry through various research and development programs. Our 1999 audit sought to determine whether four grant and contribution programs were designed well to help improve Canada's innovation performance. Our follow-up this year reviewed the actions taken in response to our 1999 recommendations.

Conclusion

5.308 Industry Canada, NSERC and the NRC have made reasonable progress in addressing all of our 1999 recommendations. While Technology Partnerships Canada has fully implemented one of our recommendations and has put in place some monitoring practices, it has not yet established a comprehensive portfolio monitoring system.

Observations

Industry Portfolio

5.309 The Industry Portfolio is a diverse collection of departments and agencies, all with responsibilities touching on at least one aspect of Canada's system for innovation. In developing policy and delivering programs, improving innovation is one objective but not the only one. In 1999 we reported that individual programs we audited were providing some information on how they contributed to innovation. However, we noted that the Industry Portfolio as a whole had not set out fully its co-ordinated strategy for linking the spending decisions of its member organizations to improving innovation or addressing innovation gaps in the economy. Management recognized an important opportunity for Industry Portfolio programs to work in concert. Program managers needed to agree on what innovation performance issues could best be addressed by the Portfolio and what results would be expected from each of its grant and contribution programs.

5.310 Industry Canada has developed an approach to innovation that focusses on four interrelated elements: knowledge infrastructure, commercialization of knowledge, human resources, and business environment. The organizations in the Portfolio are addressing each of these elements. Industry Canada has also set a number of targets for the results expected from the Portfolio's wide-ranging programs and activities. These expected results provide a reference framework for each organization in the portfolio to target the results of its own programs and initiatives.

5.311 The January 2001 Speech From the Throne laid out an agenda for the Government of Canada that had, as its first statement, a focus on "building a world-leading economy driven by innovation, ideas and talent." The federal government, under the leadership of Industry Canada, is currently drafting an innovation paper that will include both an analysis of Canada's current innovation performance and an indication of the future directions it intends to take to strengthen that performance. The paper is intended to stimulate a broad national debate on innovation issues.

National Research Council

5.312 Industrial Research Assistance Program. The National Research Council delivers the Industrial Research Assistance Program (IRAP), which helps small and medium-sized enterprises develop and exploit technologies.

5.313 In 1999 we found that in only 15 percent of the projects we audited had the program based its funding decisions on an assessment of all the project criteria. The need for program funding was not explained in 84 percent of the projects we audited. While the assessments of technical feasibility were reasonably complete and thorough, it was less clear to what extent projects would increase the technological competence or the innovation capability of the funding recipients. We also found little information on the success and results of funded projects. Therefore, we recommended that the program establish its expected results and report the actual results; ensure that it assessed the merit of projects and their need for program funds; gather information on results of individual projects; establish a results-based plan and performance measures for its advisory services; and conduct an evaluation of the program.

5.314 Since 1999, IRAP has redefined its design and become more client-focussed rather than project-focussed. As part of this redesign exercise, it restated its expected results and redefined its program processes, emphasizing more complete assessment of the merits of projects and their need for IRAP funding. It also began using tools to manage and monitor key aspects of its client relationships.

5.315 The program also conducted studies to determine its impact on client companies. It used that information to develop a performance framework that applies to IRAP's advisory services as well as the activities it funds.

5.316 IRAP has designed and developed processes and automated tools to assist in the collection and compilation of information on results. The automated system, in place since April 2001, helps to structure the collection and reporting of the information.

5.317 IRAP's performance report provides general information on the program's performance and includes selected information on the results achieved by specific program initiatives. IRAP indicates that its new automated process for collecting and compiling client information will allow for more complete aggregated reporting on the program's contribution to Canada's innovation.

5.318 The National Research Council indicated that it will begin a comprehensive evaluation of IRAP in late 2001. The evaluation design includes questions on the program's success in achieving expected results.

5.319 IRAP'S comprehensive approach to re-examining its contribution to Canadian innovation seems to include all the items we identified in 1999. Because it has taken a comprehensive approach, its action on our recommendations is taking longer than it might have otherwise. IRAP should continue to focus on changing its processes to ensure that it can demonstrate its contribution to Canada's innovation and report that contribution fully to Parliament.

Natural Sciences and Engineering Research Council

5.320 Research Partnerships Programs. The Research Partnerships Programs foster interaction and partnership between university researchers and researchers in other sectors. The purpose is to generate new knowledge, develop new expertise, and transfer this new knowledge and expertise to Canadian-based organizations.

5.321 Our recommendations to NSERC in 1999 focussed on its need to clarify the results expected of the Research Partnerships Programs (RPP) and to report progress toward achieving them; to ensure that project assessments examine the nature and significance of project benefits and the need for NSERC funding; to gather information on the results of projects; and to conduct an evaluation of the Research Partnerships Programs.

5.322 NSERC has clarified the expected results of the Research Partnerships Programs and has developed performance indicators for the programs. NSERC has rewritten guidance for applicants on the results expected of its programs. File documentation indicates that NSERC is examining applications to ensure that expected results are clearly stated, including the nature and significance of the project's expected benefits. Further, NSERC conducted an environmental scan that will assist in further refining its programs.

5.323 In March 2001, the Treasury Board approved new terms and conditions for NSERC. In particular, the Treasury Board provided NSERC with direction on how to deal with applicants' needs for funding. Applicants are required to provide enough information for a review committee to assess other sources of funding and recommend the appropriate level of funding by NSERC.

5.324 NSERC conducted a pilot project to obtain information on the longer-term impacts of projects (that is, after two years) under one element of the RPP, the Collaborative Research and Development program. It plans to expand this initiative to other elements of RPP and to some of its other programs. It also evaluated two components of the Research Partnerships Programs, addressing important questions about the success and performance of the components. These are all useful activities that provide information on overall performance and help gather information on project results.

Industry Canada

5.325 Technology Partnerships Canada. Established in 1996, Technology Partnerships Canada was a relatively new program at the time of our audit. The program promotes the development and commercialization of innovative technologies that help increase economic growth and create jobs and wealth. We found that during the first years, management established practices for due diligence in assessing project proposals. Our 1999 report recommended that TPC ensure that it appropriately justified the amounts it contributed, monitor funded projects, and improve specific aspects of reporting to Parliament.

5.326 In its 1999 response to our recommendation to justify amounts contributed to specific projects, TPC indicated that it based its funding decisions on information that varied from project to project and, on occasion, it would continue to use alternative evaluation strategies to support project assessment. TPC continues to maintain flexibility in negotiating the amounts of certain transactions. It documents those investment decisions as part of the assessment process.

5.327 In response to our recommendation on project monitoring, TPC committed to fully implement a comprehensive portfolio monitoring system before 31 March 2000. However, additional work is still needed to complete this monitoring system.

5.328 Nevertheless, since 1999, TPC has taken steps on many fronts to improve its monitoring of projects. Our follow-up found that it is more systematic in monitoring repayments, benefits, and compliance with contribution agreements for individual projects. It has developed a formal Program Assessment Guideline for Due Diligence, and project files indicate that the guideline is being applied. TPC has entered into an agreement with Industry Canada's Co-ordination and Management Services to monitor its contribution agreements during the repayment phase. It has established a contribution audit plan to verify the compliance of specific projects, based on defined risk-related criteria.

5.329 Finally, TPC's annual report to Parliament now explains more completely its leveraged funds, the sharing of risk and return, the main factors it uses to determine risk and return, and the extent to which contributions are fully repayable.

Section 11—Follow-Up: Human Resources Development Canada—Grants and Contributions (2000, Chapter 11)

Background

5.330 In 1999-2000, Human Resources Development Canada (HRDC) spent about $3 billion on non-statutory or voted grant and contribution programs and those under Part II of the Employment Insurance Act. In 1999, its Internal Audit Bureau audited project files that represented about $234 million in grants and contributions. The internal audit revealed serious and widespread problems in the Department's systems and practices for managing grants and contributions. In particular, it found inadequate processes for selecting and approving projects, inadequate monitoring of projects, and unacceptable financial management practices. The Department accepted the findings of the internal audit.

5.331 In 2000, we examined four of the grant and contribution programs the 1999 internal audit covered. Our work extended to areas the internal audit had not reviewed, such as program design, management of individual projects, and the measurement of project and program results. We also assessed corrective action HRDC had taken in response to its internal audit findings, including the progress it had made and had reported.

5.332 We concluded in 2000 that there were widespread deficiencies in the management control frameworks of all four programs. We reported that our observations confirmed and extended those of the Department's 1999 internal audit. We also indicated that corrective actions and plans HRDC was implementing would address the deficiencies our audit had found. In particular, we reported that HRDC had made good progress toward meeting the commitments in its Six-Point Action Plan, and work was proceeding on related initiatives aimed at expanding or complementing the action plan. The Department responded positively to our recommendations and agreed to sustain its progress in meeting the commitments of the action plan and related initiatives.

Focus of the follow-up

5.333 In this follow-up, we reviewed HRDC's progress in responding to our October 2000 recommendations and the recommendations made by the Standing Committee on Human Resources Development and the Status of Persons with Disabilities. However, we did not review the Department's progress in responding to the recommendations made by the Public Accounts Committee on 17 May 2001; most of those recommendations concerned performance reporting for 2001-02, and the Department's performance report had not been released at the time of our audit.

5.334 We examined various documents, including the Department's progress reports on its Six-Point Action Plan and other initiatives. We assessed the reliability of an internal audit of the National Grants and Contributions Performance Tracking Directorate. We also interviewed staff and managers responsible for administering and managing grant and contribution programs, and we visited two regional offices and two human resources centres. This follow-up provides audit-level assurance, unless otherwise indicated.

Conclusion

5.335 We found that Human Resources Development Canada has made good progress in acting on our recommendations and on those of the Standing Committee on Human Resources Development and the Status of Persons with Disabilities. We encourage the Department to maintain its momentum in fully implementing recommendations. In particular, HRDC set up a system for tracking the performance of several grant and contribution programs, developed guidelines and tools, trained program delivery staff and managers, and reported its progress to the public and Parliament. The new integrated information system it is developing is on schedule and should be ready by December 2002. HRDC recently started developing comprehensive risk management frameworks to manage contribution projects more efficiently and effectively. It is also streamlining its practices and tools to reflect risk management principles. We encourage the Department to sustain those efforts.

5.336 We found that the Department has developed results-based accountability frameworks and improved equality of access to all 26 of its voted grant and contribution programs as well as those under Part II of the Employment Insurance Act. It strengthened its internal audit function and established a follow-up process to monitor the progress of corrective action on internal audit findings.

5.337 As part of the Six-Point Action Plan, the Department committed to review files of projects that were likely to have received overpayments and that were active in 1998-99 and 1999-2000 but have ended since then (dormant files). The Department completed its review of a sample of dormant files from both years to identify overpayments attributable to sponsors' actions. We cannot conclude that the reviewed files are representative of all high-risk dormant files because of weaknesses in how files were selected. We recommend that the Department consider reviewing additional dormant files for these two years.

5.338 Ongoing tracking of performance is essential to assessing whether the balance between controls and services is appropriate. The Performance Tracking Directorate has confirmed that in the programs where performance is monitored, financial controls have been strengthened. These programs represent about 63 percent of the total value of all voted grant and contribution programs, as well as those under Part II of the Employment Insurance Act, that are delivered by the Department. At the end of our follow-up, management was completing a review of Aboriginal programs that should further extend the coverage of the Directorate. We did not examine that review. HRDC is developing plans to monitor the performance of the remaining programs. Whether it has reached the right balance between controls and services is difficult to say at present, because service performance information is currently being developed.

Observations

Quality assurance measures

5.339 Significant improvements in administering several grant and contribution programs. In our 2000 Report, we recommended that the Department put in place quality control measures to review corrective action, and ongoing monitoring to ensure that staff understand the fundamentals of financial control. To support the Six-Point Action Plan, Human Resources Development Canada developed a quality assurance framework. It had begun to implement it at the time of our audit last year and this will continue for some time.

5.340 Overall, the Department is making good progress with quality assurance measures that provide for ongoing monitoring and ensuring that staff understand the basics of financial control. New quality assurance measures include quality control at the local level, post-audit and compliance functions at the regional level, and the Performance Tracking Directorate at the national level.

5.341 Local measures. Program operations consultants provide technical expertise, monitor 15 percent of files to ensure program integrity, and report regularly to management on the findings of file reviews. They also provide feedback, coaching, and training to project officers. Most of these new positions were staffed only recently and training is still under way or has just finished. It is too early to assess the effectiveness of this new quality control function.

5.342 Regional measures. Post-audit financial review ensures compliance with HRDC policy, the Treasury Board's transfer payments policy, and relevant sections of the Financial Administration Act. Regional program compliance officers confirm that procedures and guidelines are being followed and that projects conform to program terms and conditions. The objective is to review a sample of 5 percent of projects managed by responsibility centres. A risk assessment identifies the responsibility centres subject to post-audit and compliance review. We noted on our field visits that review methodologies were in place and a first cycle of reviews had begun. Again, it is too early to assess the effectiveness of these new control mechanisms.

5.343 National measures. The National Grant and Contribution Performance Tracking Directorate provides for quality control in measuring performance and improvement in the overall administration of grant and contribution programs. It has the following mandate:

assess the overall program integrity of grant and contribution activities in HRDC by performing file reviews and financial verifications;
highlight key areas of risk and provide guidance on approaches to mitigate those risks;
contribute to knowledge transfer throughout the Department; and
co-ordinate activities with other monitoring and post-audit functions in HRDC and by outside agencies to ensure adequate coverage without duplication.

5.344 The Department's Internal Audit and Risk Management Services audited the work of the Performance Tracking Directorate. The audit focussed on the Directorate's activities related to the first three quarterly progress reports. These reports account for the reviews of projects that started between 1 February and 30 June 2000. The internal audit concluded, among other things, that project files were selected properly and the review methodology was applied consistently. We found these two conclusions reliable, based on our review of the internal audit work.

5.345 The reviews by the Performance Tracking Directorate show that after the 1999 internal audit there were significant improvements in administering several grant and contribution programs during the period reviewed. The Directorate's reviews outlined in Exhibit 5.5 touched on some major deficiencies that we also identified in our 2000 audit. Review reports also identify areas for improvements. We encourage the Department to set benchmarks for determining what constitutes appropriate performance and systematically identifying areas that need improvement.

5.346 Given that the Directorate's mandate is to assess the overall integrity of grant and contribution programs, we would have expected the reviews to provide assurance that all voted grant and contribution programs delivered by HRDC are well managed. In fact, the Directorate's reviews covered about 63 percent of HRDC's voted grant and contribution programs as well as those under Part II of the Employment Insurance Act (Exhibit 5.6). The Directorate is taking steps to conduct special reviews of the programs that have not been covered. It was completing a special review of Aboriginal Human Resources Development agreements in the Aboriginal Relations Office as we were writing this report which should further extend the coverage of the Directorate. We did not review it. It also plans to conduct special reviews of the remaining programs. Management informed us that programs involving transfer payments to provinces and/or territories are excluded from the Directorate's reviews because external or legislative auditors audit these expenditures.

5.347 We found that the Directorate's performance reports do not clearly indicate the dollar value and number of projects in each grant and contribution program that its reviews included and excluded. The review results reported in Exhibit 5.5 apply only to the grant and contribution programs it examined.

5.348 The Performance Tracking Directorate is responsible for reviewing the management of all stages in the project life cycle, including the closeout stage. Management informed us that the Directorate has not yet reviewed a statistically valid sample of files closed in 2000 because only few of the projects it selected for review had reached the closeout stage. It plans to conduct a special review of closed files.

5.349 The Directorate also uses ongoing review of its quality control framework and reporting processes to identify areas for improvement. Coordination of its activities with those of other monitoring and post-audit functions is expected to evolve as those other functions develop. This will help to avoid duplication and share best practices.

5.350 Recommendation. The Department should ensure that in its regular performance reports, the Performance Tracking Directorate clearly indicates the scope of its reviews by specifying for each grant and contribution program the dollar value and number of project files that its reviews covered and did not cover. It should ensure that the Directorate extends its review to the programs not yet examined, in order to provide assurance that the Department is managing adequately all voted grant and contribution programs, as well as those under Part II of the Employment Insurance Act, that it delivers.

Department's response. The Auditor General recognized in his October 2000 Report and in subsequent appearances before the Public Accounts Committee the steps the Department had taken to strengthen the administration of its grant and contribution programs. HRDC is pleased that the Auditor General continues to recognize the good progress the Department is making to strengthen the management of these programs.

The Auditor General previously described the Performance Tracking Directorate (PTD) established by HRDC as "an innovative system of tracking performance — one that allows for tracking improvements in the management and administration of grants and contributions" that will "give management an ongoing measure of performance in the management and control of grants and contributions."

HRDC agrees with the specific recommendation in this chapter to improve this management tool. The Department has in fact already taken action in this regard. Consistent with the established methodology and mandate for the PTD, HRDC senior management, in August 2001, expanded its coverage to include grant and contribution programs excluded from the initial reports. In addition, the PTD reports will include information on the dollar value of the project files included and excluded from the review of each program, as recommended by the Auditor General.

Recent audits and reviews detected few overpayments

5.351 Problem files. In 2000, we recommended that the Department use quality control measures and ongoing monitoring to ensure that all overpayments in problem files are properly identified and recovered.

5.352 We indicated in the Auditor General's 2000 Report (Chapter 11, Human Resources Development Canada—Grants and Contributions, paragraph 11.79) that overpayments could be established only when the Department could show that the project sponsor did not fulfil the obligations stipulated in the contribution agreement. These overpayments are then recorded as account receivables for recovery purposes. When performing financial reviews of project files, the Performance Tracking Directorate identifies and reports on the nature and extent of incorrect payments attributable to inappropriate practices by the Department and by the sponsor. Examples of inappropriate practices by the Department include making payments for ineligible expenses or outside the funding period. In the four programs we examined in 2000, we found that incorrect payments due to inappropriate practices by the Department were significant. Results of reviews by the Directorate (Exhibit 5.5) show significant improvements in reducing inappropriate practices. It is important that the Department continue to gather comprehensive information on incorrect payments due to inappropriate practices to identify their significance, their causes and appropriate corrective actions. This information is necessary for assessing the effectiveness of the grant and contribution payment process.

5.353 That said, we found that in identifying problem file overpayments attributable to sponsors, the Department has made good progress. In 2000, we questioned the appropriateness of some payments in 81 of the 309 contribution agreements we examined. We asked the Department to review and correct these 81 problem files. We found in our follow-up that the Department had completed its review and had identified the overpayments attributable to sponsors. Exhibit 5.7 shows the overpayments that were identified in that review as well as those in other reviews and audits since 1999. They amounted to less than 0.3 percent of the total value of the projects examined in the four reviews indicated in the exhibit.

5.354 Dormant files. We had also recommended that the Department identify and recover all overpayments from high-risk dormant files, that is, files that were likely to contain overpayments and that were active in 1998-99 and 1999-2000 but later closed.

5.355 As part of the Six-Point Action Plan, the Department committed to review high-risk dormant files, and it selected 516 for review. Management informed us that the review of the 516 high-risk files required considerable effort. In this follow-up we examined how, in selecting these files, the Department ensured that the sample was representative of all high-risk dormant files. We would expect it to do the following:

assign to programs a low or high risk of overpayments attributable to sponsors using a well-defined risk assessment model, focus the review on high-risk programs, and identify all high-risk projects using sound criteria applied consistently to all projects of the same program; and
using a well-defined sampling methodology, select a representative sample from the population of high-risk dormant files to assess the significance of overpayments and determine the need for additional review of dormant files.

5.356 We found weaknesses in the process the Department used to identify high-risk files and select a sample. We expected HRDC to focus the review of dormant files on programs with a high risk of overpayments. We found that it did not establish the risk profile of each grant and contribution program. The risk assessment model the Department developed for the review of dormant files uses nine attributes to identify high-risk files. We found that the Department used only two of the nine attributes: dollar value greater than $100,000, and characteristics of the sponsor (that is, number of projects or contracts and the adequacy of the sponsor's management control framework). Management informed us that its methodology had relied on judgment in attempting to select the highest-risk files. We noted also that the Department did not apply consistently among programs the two attributes it did use. Nor did it properly document and support the rationale for its selection of files to review. Because of these weaknesses in the way the files were selected, we cannot conclude that the files selected for review are representative of the population of high-risk dormant files or are the highest-risk files.

5.357 The Department completed the review of dormant files in January 2001. It identified 35 overpayments attributable to sponsors that totalled about $430,000. We found that 83 percent of that amount represented overpayments in only one program: Labour Market Partnerships. This suggests that the program had a high risk of overpayments. The objective of the program is "to work with partners to help communities, employers, and workers improve their capacity to address human resource requirements and to implement labour force adjustments." It is similar in design to the Sectoral Partnership Initiative that we audited last year and in which we found serious deficiencies. Given that a high percentage of the overpayments detected were from the Labour Market Partnerships program, we think the Department should consider reviewing additional high-risk files from that program.

5.358 At the same time, because the results of its review are not representative of all high-risk files, the Department needs to develop a risk profile for its other programs as a basis for deciding whether a review of additional dormant files is necessary.

5.359 Recommendation. The Department should consider reviewing additional high-risk files that were active in 1998-99 and 1999-2000 but later closed.

Department's response. The Department agrees with this recommendation and has already begun the process of reviewing its methodology for identifying additional high-risk files that were active in 1998-99 and 1999-2000 but later closed. This review will assist the Department in assessing the cost effectiveness of reviewing additional files.

Strengthening key controls

5.360 Good progress in meeting the major commitments of the Six-Point Action Plan. The core purposes of the Six-Point Action Plan the Department released in February 2000 were to strengthen the administration of grants and contributions and to correct the problems identified in the 1999 internal audit of grant and contribution programs. In 2000, we recommended that the Department strengthen key controls as it had committed to do in the action plan.

5.361 The Department has reported its achievements under the action plan in four progress reports; the latest was issued in April 2001. The results of two third-party assessment reports on the implementation of the action plan were also made public. The most recent third-party assessment, at 15 January 2001, revealed that HRDC had fulfilled, or would fulfil on schedule, all of its commitments. In our view, at 31 August 2001 the Department had made good progress toward meeting the major commitments of the action plan. It continues to dedicate the necessary effort and resources. We found that the Department has generally met the target dates for completing its planned activities.

5.362 The Department continues to track performance through regular monitoring by the Performance Tracking Directorate. Management is also making good progress in important long-term and supporting activities such as enhancing tools, delivering training, developing a new integrated information system, and reporting progress to the public and Parliament.

5.363 The Department strengthened its internal audit function. In 2000, we recommended that key controls be supported by a strong internal audit function that meets the highest professional standards. We also recommended that the Department put in place a well-established system for taking corrective action on internal audit findings and following up on that action.

5.364 We found that the Department has made good progress in strengthening its internal audit function, Internal Audit and Risk Management Services (IARMS). In 2000-01, the number of staff in the internal audit function increased from 33 to about 50. IARMS is developing a departmental internal audit policy and a follow-up audit policy. In addition, it recently developed a draft audit manual outlining audit policies, standards, and procedures as well as a quality assurance review process to ensure that all internal audits meet professional standards.

5.365 The Audit and Evaluation Committee, chaired by the Associate Deputy Minister, is responsible for tracking management responses to audit recommendations to ensure that the Department takes timely and effective corrective action. In May 2001 the Committee approved the process for following up on corrective action. At the end of September 2001, the Audit and Evaluation Committee had not received any follow-up report on corrective action. We were told that a follow-up report would be forwarded to the Committee soon.

Program management study

5.366 Study of grant and contribution programs. In 2000, we noted that the Department needed to state the expected results of programs in clear and measurable terms and use project results and program evaluations to make necessary changes and modifications (manage for results). We recommended that the Department complete the program management study it had initiated to ensure that the desired results of all grant and contribution programs were clearly defined and that performance measures were in place to demonstrate the achievement of desired results. We also recommended that the Department ensure, in consultation with the Treasury Board Secretariat, that it would meet its proposed timeframe for reviewing the objectives of all its grant and contribution programs and making appropriate changes, including measurement of results. The Treasury Board's new transfer payments policy, effective June 2000, requires that all new and renewed grant and contribution programs have results-based accountability frameworks in place.

5.367 Our follow-up found that the Department has made good progress. In May 2000, HRDC had initiated a comprehensive program management study with two objectives: to ensure that all programs have clear objectives and results-based accountability frameworks in place; and to renew the terms and conditions of these programs by the end of 2002. The study concluded that 12 of the 26 active voted grant and contribution programs and those under Part II of the Employment Insurance Act already had adequate results-based accountability frameworks in formal partnership agreements with the provinces and Aboriginal communities. Results-based accountability frameworks were developed for the remaining 14 programs. As at the end of our follow-up, the Department was awaiting the Treasury Board's final comments on the frameworks.

5.368 The Department recognizes that results-based management is a long-term process and it takes time to develop indicators and align management systems, even before collecting performance data. Some programs have identified performance indicators for which data are not yet available. We noted that some frameworks rely heavily on evaluation cycles to indicate results. At present there is only one indicator of service quality, speed of service, for only some benefit transactions under the employment program. At the end of our follow-up, the Department was assessing the quality, availability, and cost effectiveness of indicators for all its grant and contribution programs with a view to identifying a final set of indicators. We encourage the Department to complete this work for all programs as soon as possible.

5.369 The Treasury Board's revised transfer payment policy requires departments to review and resubmit for approval the terms and conditions of all grant and contribution programs by 31 March 2005. HRDC has chosen to meet this requirement by the end of 2002. It has begun reviewing the effectiveness and relevance of all its grant and contribution programs to ensure that they address the government's priorities. It has also identified priorities for renewal of program terms and conditions and has begun to review them.

Equality of access

5.370 Improved equality of access to the Department's grant and contribution programs. In 2000, we noted that the methods used to promote grant and contribution programs did not always ensure that all potential applicants had equal access to the program. We recommended that the Department assess whether the information it provides to Canadians on grant and contribution programs adequately promotes equality of access and that it take corrective action as required.

5.371 We have found that the Department has made progress in improving equality of access to all grant and contribution programs. Since December 2000, its public Web site has provided detailed information on objectives, eligibility criteria, and applying for all its voted grant and contribution programs as well as those under Part II of the Employment Insurance Act. Information kits and information pamphlets for sponsors are available through local offices and these also contribute to equality of access to programs. The Department also examined equality of access as part of the program management study. The study found that some programs have adopted measures such as on-line application, guidelines for applicants, regional consultations, and national and local advertisements to ensure and promote equality of access.

Equip and support staff

5.372 Continued support for change. We recommended in 2000 that management sustain its current strong support for change to ensure that staff are fully equipped and supported. We also recommended that the Department conduct ongoing monitoring of the impacts of new tools, training, concepts, and approaches.

5.373 In our follow-up, we found that senior management has continued to provide staff with the necessary ongoing support for change. First, we noted in our review of some performance agreements that departmental executives and managers responsible for grants and contributions are explicitly accountable for implementing the Six-Point Action Plan. The Treasury Board approved a reallocation of $50 million from existing budgets in 2000-01 for implementing the action plan and other related initiatives. We believe that this reallocation has helped the Department provide the necessary tools, training, and systems support for making the required changes. In addition, regular reporting of progress to staff and the public has helped maintain momentum.

5.374 We found that the Department has made good progress in several initiatives to equip and support staff in their work. In particular, it developed operational policies and model files, adopted a new grant and contribution manual, and improved the departmental information systems. The Department developed and provided mandatory training courses to managers and staff. Management informed us that by the end of March 2001, about 1,600 program delivery staff from all provinces and territories had attended mandatory training in program delivery. The Department also indicated that managers attended relevant courses on various topics such as modern comptrollership. It is currently developing a course on program integrity and public accountability.

5.375 The Department has monitored some of the impacts of the new tools and training. For instance, the Performance Tracking Directorate found that new tools and training and other initiatives outlined in the Six-Point Action Plan had significantly contributed to the strengthening of financial controls. However, the impacts on the quality of services and the efficiency and effectiveness of grant and contribution activities have not been assessed formally.

5.376 New integrated information system is expected to be completed by December 2002. Under the Six-Point Action Plan, the Department committed to develop better information systems over the next three years that would simplify the management and monitoring of projects. We recommended that the new information system be fully integrated with the Department's financial system. In 2000 we reported deficiencies in the capacity of current information systems to provide timely and useful performance information, such as information on sponsors' current and past projects with the Department. Our follow-up found that the Department has upgraded its current systems to make financial information on grants and contributions available from one system. However, this interim measure does not meet all the information needs of project officers and managers. On visits to regional and local offices we found that client, contract, and performance information on individual projects is difficult to obtain; it must be retrieved—when the information is available electronically—from various systems that are not integrated.

5.377 An integrated information system is critical to ensuring sound and effective administration and monitoring of grant and contribution programs. We found that the Department has made progress in developing a new integrated information system, the Common System for Grants and Contributions. Since May 2001, the Department's Intranet site has provided project officers and managers with access to electronic tools such as sample model files, project checklists, and on-line training. The integration of client information with the financial systems and the monitoring of the client's progress through each intervention of the Common System are expected to be completed by March 2002. The Department expects that by December 2002 at the latest, it will have completed the remaining components of the Common System, such as the administration of all business elements of contracts and the design and development of various performance reports. We encourage the Department to take all the necessary steps to meet that target completion date.

Comprehensive analysis of workload

5.378 Workload analysis identified the need for supplementary funding. In 2000, we recommended that the Department, in collaboration with the Treasury Board Secretariat, complete a workload analysis of the resources needed to deliver different types of grant and contribution programs. HRDC committed to conduct a study of its operational capacity by December 2000. Its purpose was to determine the incremental resources needed to implement the Six-Point Action Plan.

5.379 The Department completed its capacity study in May 2001. The study concluded that to manage $1.6 billion of grant and contribution program funds under the new guidelines would require incremental funds of roughly $75 million. The Department developed various costing methodologies to estimate that amount because it did not have activity-based costing systems and a streamlined financial coding structure for all its grant and contribution programs. We have not assessed the reasonableness of the estimate.

5.380 The Treasury Board authorized $50 million in 1999-2000; it did not approve funding for future years. However, it asked HRDC to provide a justification for permanent funding levels, taking into account the results of two studies: a third-party study of possible delivery models for the Department's grant and contribution programs; and the Department's program management study. The third-party study team was asked to assess whether a more optimal delivery model was feasible, given that in the workload analysis HRDC had identified the need for more funds to improve the administration of its grant and contribution programs and implement the Six-Point Action Plan.

5.381 The third-party study, completed in July 2001, concluded that the Department's current delivery model needed significant changes to increase the efficiency of operations, the quality of service, and the consistency of controls, as well as reducing administrative complexity. The report recommended the automation of the grant and contribution delivery process; the concentration of administrative processes at one or several delivery points; and the specialization of specific steps in the processes. It added that it could not compare costs with those of other government departments because costing information was not available in enough detail.

5.382 The Department has also considered several initiatives to streamline the current delivery model and thereby reduce costs. It is currently evaluating the savings that could result from these initiatives, which include implementing risk management frameworks. At the end of our follow-up, the Department was integrating the results of the review of delivery models, the capacity study, and the program management study. The integrated results would serve as a basis for determining the amount of permanent funding it would request from the Treasury Board.

Balancing control and service

5.383 Commitment to achieve the right balance between controls and services. In 2000, the Standing Committee on Human Resources Development and the Status of Persons with Disabilities asked the Department to report by 1 June 2001 on how it had struck a balance between decentralization, flexibility, and control. In its June 2001 response to the Committee, the Department reported that striking the right balance between responsive and flexible service to Canadians and sound financial controls is difficult to achieve and requires constant effort to maintain. It also noted the need to stay in touch with its stakeholders and carefully apply risk management principles to maintain the right balance.

5.384 We have found that the Department remains committed to achieving the right balance between controls and services. Ongoing tracking of performance is essential to assess whether the balance is achieved and to determine areas that need corrective action. The Department's performance tracking system has confirmed that in the programs where performance is monitored, financial controls have been strengthened. These programs represent about 63 percent of the total value of all voted grant and contribution programs, as well as those under Part II of the Employment Insurance Act, that are delivered by the Department. HRDC is developing plans to monitor the performance of the remaining programs.

5.385 On our visits to regional and local offices, we found that managers and project officers have very little performance information on services in most grant and contribution programs. Our review of the program management study revealed that the majority of the 26 grant and contribution programs have no service-related performance indicators and standards. The quality of service is monitored informally; no formal assessments are carried out. The Department is developing client service indicators as part of its program management initiative. Given the current lack of service performance information, it is difficult to conclude whether the Department has reached the right balance between controls and services. Management informed us that by June 2002 it would complete the final policy and framework for service standards in all its voted grant and contribution programs as well as those under Part II of the Employment Insurance Act.

5.386 We found that the administrative work needed to deliver grant and contribution programs has increased due to several factors, including the new requirements of the file model. In addition, our interviews with management and project officers at the regional and local levels indicate that staff have become more risk-averse. In other words, work proceeds more slowly because staff may be worried about making mistakes. This may explain to some extent why the Department lapsed about $140 million of grant and contribution funds in 2000-01, an increase of about $70 million over the previous year.

5.387 The Department has recognized the need to constantly examine and reduce paperburden while ensuring that controls are sound. It has taken some steps recently to ensure that administrative and financial requirements are commensurate with the risk profile of programs and projects. For instance, it developed a detailed administrative streamlining workplan for July to December 2001. It has also started to streamline its practices and tools in order to reflect risk management principles. In particular, in late June 2001 it introduced a streamlined, generic contribution project file and grant model file in one program. The Department is developing streamlined model files for all its voted grant and contribution programs.

5.388 In 2000, the Standing Committee on Human Resources Development and the Status of Persons with Disabilities asked the Auditor General to provide guidance "on ways to balance efficiency and flexibility in terms of program delivery and the need for sound financial management." In our October 2000 Report we noted, "Not all grant and contribution programs are of equal risk, nor are all projects in a program." We also acknowledged that "there is no simple answer to what the ideal balance should be." However, we believe that the use of comprehensive risk management frameworks is critical to reducing undue paperburden and improving the quality of services to Canadians while maintaining sound financial control. Management informed us that the Department recently started to develop such frameworks in collaboration with the Treasury Board Secretariat, with a view to managing grant and contribution projects more efficiently and effectively. We encourage the Department to sustain those efforts.

5.389 The Committee's report contained about 30 recommendations, 20 of which were addressed to HRDC. Exhibit 5.8 summarizes the Department's progress in acting on those recommendations. We note that HRDC has made good progress or has fully addressed the Committee's recommendations relating to the Department.

About the Audits

Objectives

The objective of our detailed audits was to determine whether departments had adequate control over selected grant and contribution programs.

The objective in our follow-up of previous recommendations was to determine what departments had done to implement corrective action.

Scope and approach

We attempted to select programs for audit that would capture as many of the key features of grants and contributions as possible, including kind of recipient, delivery mechanism, duration of project funding, policy area, size of department, burden of cost. To that end, we selected the following programs for audit at these departments:

Agriculture and Agri-Food Canada—Agri-Food Trade Program
Canada Economic Development for Quebec Regions—IDEA-SME Program and Regional Strategic Initiatives
Canadian Institutes for Health Research—Operating Grants Program
Citizenship and Immigration Canada—Settlement Contribution Programs
Department of Canadian Heritage—Support for Official-Language Communities
Environment Canada and Natural Resources Canada—Contributions made under the Climate Change Action Fund
Solicitor General Canada—First Nations Policing Program
Western Economic Diversification Canada—Community Futures program

We also followed up on the following previous audits of grant and contribution programs:

Department of Canadian Heritage—Grants and Contributions Under the Multiculturalism Program (Chapter 27, 1998)
Industry Portfolio: Investing in Innovation (Chapter 19, 1999)
Human Resources Development Canada—Grants and Contributions (Chapter 11, 2000)

In addition to examining the management of individual projects under each program, we assessed the design of the programs and the measurement of project and program performance. We chose the projects that we audited under each program so the results of our audit could be extended to the program's expenditures during the sampled period. The specific period is indicated with each description of the programs audited.

In all programs, we reviewed project files and documents for our audit criteria and interviewed project officers and others involved in managing the projects and the programs. We also visited selected regions, projects, and funding recipients in some programs.

Criteria

Where we carried out detailed audits, we expected that departments would do the following:

comply with authorities;
ensure that grant and contribution programs are designed to achieve expected results, manage risks, ensure due diligence in spending, and provide accountability for public funds spent;
exercise due diligence in approving individual grants and contributions;
have reasonable assurance that the funding is used for the purposes intended;
know whether programs are achieving expected results;
report clearly to Parliament on program performance;
reinforce public sector values and value for money attitudes among program staff; and
make reasonable efforts to harmonize and co-ordinate their activities with other organizations delivering similar programs.

Audit team

Assistant Auditor General: Richard Flageole
Principals: Peter Simeoni; John Hitchinson (Citizenship and Immigration, Section 4)
Directors: Jean Pierre Plouffe, Amipal Manchanda, Rafid Warsalee, and Yvon Roy

Patrick Bédard
Monique Charron
Cheryl Churcher
Johane Garneau
Susan Gomez
Louise Grand'Maison
Geneviève Hivon
Debbie Jacob
Denis Jobin
Nicole Lauzon
Anthony Levita
Chatherine Livingstone
Suzanne Moorhead
Barry Neilson
Nicolas Paré
Sylvie Paré
Paul-Henri Rhéaume
Sylvie Soucy
Annie Tremblay
Shawn Vincent

For information, please contact Peter Simeoni.