2012 Fall Report of the Auditor General of Canada

 

Performance audit reports

This report presents the results of a performance audit conducted by the Office of the Auditor General of Canada under the authority of the Auditor General Act.

A performance audit is an independent, objective, and systematic assessment of how well government is managing its activities, responsibilities, and resources. Audit topics are selected based on their significance. While the Office may comment on policy implementation in a performance audit, it does not comment on the merits of a policy.

Performance audits are planned, performed, and reported in accordance with professional auditing standards and Office policies. They are conducted by qualified auditors who

  • establish audit objectives and criteria for the assessment of performance;
  • gather the evidence necessary to assess performance against the criteria;
  • report both positive and negative findings;
  • conclude against the established audit objectives; and
  • make recommendations for improvement when there are significant differences between criteria and assessed performance.

Performance audits contribute to a public service that is ethical and effective and a government that is accountable to Parliament and Canadians.

Main Points

What we examined

Since 2007, Industry Canada has authorized assistance of almost $1.2 billion through two transfer payment programs to the aerospace sector: the Strategic Aerospace and Defence Initiative (SADI) and the Bombardier CSeries Program. SADI is the federal government’s second largest program of direct spending on research and development, created to support private sector industrial research and pre-competitive development in Canada’s aerospace, defence, security, and space industries through repayable project contributions. The Department has 25 individual SADI projects with recipients.

Industry Canada also manages two repayable contribution agreements for the Bombardier CSeries Program. These contributions are intended to encourage research and development that will result in the development of new commercial aircraft technologies.

We examined whether Industry Canada had sufficient information to determine if the transfer payments were meeting the programs’ objectives. We also looked at whether the Department managed these programs according to the key requirements of the Treasury Board’s Policy on Transfer Payments and the terms and conditions of the programs. In addition, we examined whether Industry Canada collected repayments from recipients for contributions that are repayable under two previous transfer payment programs: the Defence Industry Productivity Program and Technology Partnerships Canada.

Audit work for this chapter was completed on 17 July 2012. More details on the conduct of the audit are in About the Audit at the end of this chapter.

Why it’s important

The federal government regards the aerospace sector as crucial to Canada’s economic development, sovereignty, national security, and public safety. Aerospace companies are important contributors to the Canadian economy. National Research Council Canada reported that Canada’s aerospace sector is the fifth largest in the world, as of 2008, with total aerospace sector revenues reaching $23 billion.

What we found

  • Before 2010, Industry Canada had inadequate performance information to determine progress being made to achieve the Strategic Aerospace and Defence Initiative’s objectives. Since 2010, the Department has made improvements and now collects and consolidates sufficient information to allow it to determine progress against the program’s objectives. However, the Department has not followed through on its commitments to report publicly on the program’s results. As well, the Department will need to follow through on commitments to collect additional performance information so that it can complete its planned evaluation of the program in 2016–17. For the Bombardier CSeries Program, the Department has not collected all documents required by the contribution agreements to determine progress toward the program’s objectives, so it has a more limited picture of the program’s performance.
  • Industry Canada has managed most aspects of these transfer payment programs appropriately, using a control framework that is reasonable—for example, it reviews recipients’ claims and progress reports before issuing payments. As well, the Department funded only recipients that met program eligibility requirements. It also undertook a detailed review of proposed projects before signing contribution agreements with recipients. However, the Department did not define the appropriate level of monitoring and reporting for the different levels of risk assigned to the projects and for the amount of funding provided to individual recipients.
  • In cases where contributions under the Defence Industry Productivity Program and Technology Partnerships Canada were repayable, the majority of repayments we examined were obtained by Industry Canada on time.

The Department has responded. The Department agrees with our recommendations. Its detailed responses follow the recommendations throughout the chapter.

Introduction

6.1 The aerospace sector in Canada includes commercial aerospace; military aerospace; aircraft maintenance, repair, and overhaul services; and the development, deployment, and operation of space assets. Aerospace companies are important contributors to the Canadian economy. The National Research Council Canada reported that Canada’s aerospace sector ranks as the fifth largest in the world as of 2008, with total revenues of $23 billion.

6.2 Many countries provide financial support to their own aerospace sector. The Government of Canada, which sees the aerospace sector as crucial to Canada’s economic development, sovereignty, national security, and public safety, has provided financial support to the sector continuously since the 1950s. The government credits this support with contributing to a viable Canadian aerospace industry that is internationally competitive in areas such as aircraft engines, landing gear, and aircraft production.

6.3 The aerospace sector is research and development intensive, and often there is a long period of time between when a new technology is conceptualized to when it is produced. Providing financial support is not without risks—the government recognizes that not all projects will succeed, technically and commercially, and therefore not all projects will generate the original expected outcomes. Economic change, global competitiveness, and changing market conditions all add an element of risk. The federal government’s approach—sharing risks and rewards—is intended to encourage private sector investment.

Industry Canada uses transfer payments to support the aerospace sector

6.4 Industry Canada is the lead department for science within the federal government. Its mandate is to encourage the fullest, most efficient, and most effective development and use of science and technology. Industry Canada also works on making Canadian industry more productive and competitive in the global economy.

6.5 Industry Canada delivers the Strategic Aerospace and Defence Initiative (SADI), which is the federal government’s second-largest research and development direct spending program. SADI was launched in 2007 as an ongoing program to support private sector industrial research and pre-competitive development in Canada’s aerospace, defence, security, and space industries through repayable contributions to support projects. SADI has three objectives:

  • to encourage strategic research and development that will result in innovation and excellence in new products and services;
  • to enhance the competitiveness of Canadian aerospace and defence; and
  • to foster collaboration between research institutes, universities, colleges, and the private sector.

6.6 Examples of supported industrial research include the development of innovative aircraft simulator systems and research to develop a safe and effective alternative to the use of cadmium in the electroplating process for aerospace applications. Examples of supported pre-competitive development include projects to improve the reliability and performance of landing gear systems and projects to develop lighter aircraft engines with more power, better fuel consumption, and improved durability.

6.7 As of 1 April 2012, a total of 22 companies signed 25 unconditionally repayable or conditionally repayable contributions (Exhibit 6.1) for strategic research and development projects. Under SADI contribution agreements, Industry Canada disburses funds to recipients over multiple years; SADI contributes up to 40 percent of eligible costs of funded projects. Funds are typically repayable over 15 years after the end of the project. The Department authorized just under $825 million in contributions (funds disbursed as of 1 April 2012 were almost $325 million). Within Industry Canada, the Industrial Technologies Office manages this program.

Exhibit 6.1—What is a contribution?

A contribution is a transfer payment subject to performance conditions specified in a funding agreement. A contribution is to be accounted for and is subject to audit.

A repayable contribution is a contribution whereby the recipient is expected to repay all or part of the amount, or the government expects to receive a financial return. The terms may specify a date or dates for repayment, or may prescribe the particular time(s) or circumstance(s) that will determine repayment.

Repayable contributions are further classified as unconditionally repayable and conditionally repayable. Unconditionally repayable contributions must be repaid without qualification. The contribution agreement contains specific repayable terms. Conditionally repayable contributions are contributions, all or part of which are repayable, if conditions specified in the contribution agreement come into being.

Source: Treasury Board of Canada Secretariat

6.8 The Department also manages two conditionally repayable contribution agreements with Bombardier Inc. in support of research and development projects associated with the Bombardier CSeries Program. The contributions are intended to encourage research and development that will result in the development of new commercial aircraft technologies, including the new CSeries commercial jet. Expected results include encouraging and leveraging private sector investment, as well as maintaining and growing the technology base and technological capabilities of Canadian aerospace firms. A total of $350 million in funding was authorized for the fiscal years 2008–09 to 2014–15 (funds disbursed as of 31 March 2012 were just under $203 million). The Aerospace, Defence and Marine Branch of Industry Canada manages this program.

6.9 Finally, Industry Canada manages the Defence Industry Productivity Program (DIPP) and Technology Partnerships Canada (TPC). Both programs are in the repayment phase, which is the final phase for these programs. The objective of DIPP, which ran from 1959 to 1995, was to develop and maintain defence-related industries capable of competing in domestic and export markets. Recipients received $2 billion from 1985 onwards. Fifty-five percent of these DIPP contribution agreements were repayable.

6.10 DIPP was replaced by TPC, which ran from 1996 to 2006. Its mandate was to invest in projects with firms to support technological development that fostered innovation, commercialization, sustainable development, and increased investment. Industry Canada has disbursed $3.1 billion in repayable contributions to recipients. All TPC contribution agreements were repayable.

Focus of the audit

6.11 We examined whether Industry Canada had sufficient information to determine if the Strategic Aerospace and Defence Initiative (SADI) and the Bombardier CSeries Program were meeting their stated objectives, and whether these programs were managed according to the key requirements of the Treasury Board Policy on Transfer Payments and the terms and conditions of the programs. Our audit included the 24 SADI agreements signed prior to 1 March 2012 and the 2 Bombardier CSeries Program agreements. We also examined whether the Department collected amounts that recipients were to repay under two legacy programs—the Defence Industry Productivity Program and Technology Partnerships Canada. Our audit examined Industry Canada’s performance; we did not audit recipients of government funding. A separate audit on transfer payment program reforms appears as Chapter 2 of this report, Grant and Contribution Program Reforms.

6.12 Audit work for this chapter covers the period from 1 January 2007 to 1 June 2012. More details on the audit objectives, scope, approach, and criteria are in About the Audit at the end of this chapter.

Observations and Recommendations

Measuring progress on objectives

6.13 The Policy on Transfer Payments requires that departments establish a performance measurement strategy at the time of program design and that the strategy be maintained and updated to support the evaluation or review of the program’s relevance and effectiveness. A performance measurement strategy guides program and corporate decision making. It also helps ensure there is adequate and relevant information to support continuous monitoring. Industry Canada committed to developing performance measurement strategies for both the Strategic Aerospace and Defence Initiative (SADI) and the Bombardier CSeries Program to measure whether these programs are meeting their objectives.

6.14 Measuring whether overall program objectives are met requires that performance information is collected from recipients at the individual project level. The terms and conditions for both SADI and the Bombardier CSeries Program require that contribution agreements with individual recipients contain provisions for them to submit periodic reports to Industry Canada so the Department can assess the progress of projects. These reports should include information on the achievement of project objectives, progress against performance milestones, and how the funds provided were used.

6.15 With the end of Technology Partnerships Canada in 2006, Industry Canada had an opportunity to incorporate lessons learned with respect to performance measurement and reporting into the design and delivery of its successor, SADI. The Department also committed to undertake multiple evaluations of SADI and to evaluate the Bombardier CSeries Program. Finally, for SADI, the Department committed to reporting publicly each year on contribution recipients, on program results and accomplishments, and on how approved projects met the assessment criteria—responding to its commitment to set new standards for transparency.

6.16 We sought to determine whether the Department

  • put in place performance measurement strategies for SADI and the Bombardier CSeries Program to collect information;
  • collected performance information on individual projects;
  • evaluated Technology Partnerships Canada and fulfilled early evaluation commitments for SADI; and
  • publicly reported each year on SADI contribution recipients, program results, and accomplishments.
Performance measurement strategies are in place

6.17 Industry Canada established performance measurement strategies for SADI and the Bombardier CSeries Program. These strategies are consistent with the main elements of a performance measurement strategy outlined in Treasury Board of Canada Secretariat guidance.

6.18 The SADI performance measurement strategy sets out an approach to measure SADI objectives (innovation, competitiveness, and collaboration) and includes performance indicators associated with program objectives and outcomes. For example, under the objective of innovation, indicators include research and development dollars invested leveraged per SADI dollars invested. It also includes project-specific outcomes, linked to program objectives, that may result from individual SADI projects. For example, new patent applications filed or granted are project-specific benefits under the innovation objective.

6.19 The Bombardier CSeries Program performance measurement strategy includes performance indicators such as the number and type of research and development positions maintained and created under the agreements, as well as the number and type of collaborations with research institutions, universities, and other companies.

6.20 These performance measurement strategies provide the Department with a plan for continuously monitoring and assessing program results; a means to collect the information to make informed decisions and take appropriate, timely action with respect to programs; a means to provide effective and relevant departmental reporting on programs; and a way to ensure that credible and reliable performance data is collected to support evaluations of the programs.

The Department is now collecting more performance information on individual projects

6.21 A contribution is a transfer payment subject to performance conditions specified in a funding agreement. The Treasury Board Directive on Transfer Payments specifies that agreements should include a description of the purpose of the contribution and the activities, project, or initiative being funded and the results that the recipient is to achieve. The results expected should be within the reasonable control of the recipient and should be stated at a level of detail that will support accountability and performance measurement. Agreements should also include a description of the information that will be requested from recipients for financial and performance reporting, including information on results achieved to support the program’s performance measurement strategy and departmental reporting. The contribution agreement constitutes the entire agreement between the parties.

6.22 Program and project objectives. We examined whether contribution agreements clearly set out project objectives and whether these objectives corresponded to overall program objectives. We also examined Industry Canada’s monitoring of recipients through reporting requirements specified in the contribution agreements—progress reports, site visits or progress meetings, and annual performance reports. We wanted to determine if information was captured on the progress of projects, their results, and how the projects contributed to achieving overall program objectives.

6.23 For SADI, we did not find a clear link between the project-level objectives contained in many of the 24 contribution agreements with recipients and SADI’s overall program objectives. In most cases, the contribution agreements did not include a clear description of project-specific objectives that were linked to overall program objectives. In our opinion, this makes it more difficult for Industry Canada to determine and demonstrate the extent to which individual projects are contributing to achieving the overall SADI objectives.

6.24 Anticipated outcomes and benefits. We noted that the Department made improvements in later SADI agreements to ensure a clearer link between the project-level objectives contained in the individual agreements and the overall SADI program objectives. In addition, six contribution agreements signed from April 2010 onwards contained a specific section on the projects’ anticipated outcomes and benefits, which corresponded to overall program objectives. We were encouraged by this important improvement. Contribution agreements document requirements the recipients must meet in return for receiving funds from the federal government.

6.25 Including anticipated outcomes and benefits in contribution agreements should allow for easier reporting on progress and results achieved by the projects. It may also help the Department collect the information it needs to report on the achievement of program objectives. We noted that in response to an internal review on collaboration in 2011, the Department agreed to make collaboration expectations and reporting requirements contractually explicit in the contribution agreement template. Industry Canada’s officials informed us that the Department will continue the practice started in 2010 of including the anticipated outcomes and benefits for individual projects in the contribution agreements.

6.26 Monitoring and reporting. Industry Canada included several monitoring and reporting requirements in the SADI contribution agreements. The agreements require that project recipients prepare quarterly or monthly progress reports to demonstrate whether projects are progressing according to the work plans outlined in the contribution agreements. The agreements also specify that annual site visits or progress meetings must be carried out to monitor what is happening “on the ground.” All of the contribution agreements include a clause that requires recipients to provide information on the project’s outcomes if the Department requests it. As well, the first 17 contribution agreements that were signed contained a specific requirement for companies to prepare an annual performance report that included information on projects’ progress toward meeting anticipated project outcomes and benefits. The last 7 contribution agreements instead contained a requirement for recipients to participate annually in preparing and updating benefits reports to document the outcomes of the project and to assist in communicating the value and performance of SADI.

6.27 We found that recipients provided all of the progress reports that the contribution agreements required. Some reports were detailed and provided specific information not only on technical progress but also on the projects’ outcomes and benefits, while other reports did not clearly state the outcomes and benefits that were achieved. This was particularly the case for progress reports that were prepared before 2010. We saw that there was generally more information on project outcomes and benefits in the progress reports after the Department specifically asked recipients in contribution agreements for this type of information. We noted that Industry Canada officials reviewed the progress reports and contacted recipients for more details and clarification when needed.

6.28 We saw evidence that almost all of the SADI site visits and progress meetings that the contribution agreements required took place. However, they were not always well documented in the project files, which makes project monitoring more difficult over time. We noted that the more recent agreements require preparing meeting minutes with recipients, which could lead to better documentation of meetings in the future.

6.29 We verified whether Industry Canada obtained annual performance reports as required by the contribution agreements and whether these reports included information on project outcomes and benefits. We found that Industry Canada did not request these reports before 2010. The Department therefore lost an opportunity to collect some of the information it needed on how the projects were meeting SADI’s program objectives. This information was needed to meet its requirement to report publicly each year on contribution recipients, and on overall SADI program results and accomplishments. The Department stated that it collected some of this information through other mechanisms, including progress reports and site visits.

6.30 For the 2010–11 fiscal year, departmental officials prepared annual project benefits reports for each project, outlining progress and achievements related to key performance indicators. The reports indicated that some projects were not at a stage where all of the indicators for assessing progress on outcomes could be measured.

6.31 For the Bombardier CSeries Program, the project progress reports contained limited information on outcomes and benefits, such as collaboration with research institutions or universities. As a result, the Department cannot rely solely on progress reports as a source of performance information. We also found that although the Department conducted site visits, it did not document them well. Annual progress reports were required by the contribution agreements, but Industry Canada did not request them. The Department told us that while these required reports were not received, it did use other sources of information to monitor progress. However, without the documentation required under the contribution agreement, the Department has a more limited picture of the program’s performance.

6.32 Progress reports, annual site visits or progress meetings, and annual performance reports are important tools for Industry Canada to monitor whether recipients are meeting project commitments and whether the projects are meeting program-level objectives. Since Industry Canada did not consistently collect comprehensive performance information through these monitoring mechanisms before 2010, determining whether projects are meeting program-level objectives and undertaking program-level evaluations may be more difficult than if the Department had complete performance information on each project. With respect to SADI, since 2010, the Department has made improvements and now collects and consolidates sufficient information to allow it to determine progress against objectives.

6.33 Recommendation. Industry Canada should ensure that monitoring and reporting requirements set out in its contribution agreements are fully met and documented.

The Department’s response. Agreed. The Department will ensure that the monitoring and reporting requirements of contribution agreements are fully met and documented.

6.34 Recommendation. Industry Canada should ensure that future contribution agreements it signs with recipients specify project-specific objectives and identify anticipated outcomes and benefits that are aligned with the Strategic Aerospace and Defence Initiative program objectives.

The Department’s response. Agreed. The Department will continue the practice, started in 2010, to include in its contribution agreements a section specifying project-specific objectives and identifying the anticipated outcomes and benefits of the project in order to demonstrate clearly their alignment with the Strategic Aerospace and Defence Initiative’s objectives.

The Department has been slow to measure progress against program objectives

6.35 We examined the Department’s efforts to evaluate Technology Partnerships Canada (TPC) to help inform performance measurement for SADI. We found that in 2003, Industry Canada conducted an evaluation of TPC that focused on program design and delivery, which provided the Department with important information about operational issues. The evaluation assessed performance over the first six years of TPC’s operations, from 1996 to 2002.

6.36 We found that a final evaluation of TPC’s program effectiveness—including of the program’s longer-term technological, economic, and societal outcomes—has not been done. The Department had committed to completing this evaluation in 2011. The Department’s multi-year evaluation plan, dated May 2012, commits the Department to complete an evaluation of TPC in 2015–16. However, this means the Department is not yet in a position to integrate lessons learned from a final evaluation of TPC to potentially improve performance measurement for SADI.

6.37 For SADI, we found that Industry Canada has delayed and cancelled some of its evaluation commitments, potentially missing out on early improvement opportunities. The initial implementation review, due in the 2008–09 fiscal year, was intended to assess program implementation and the performance measurement strategy. It was not completed until October 2010, at which point the program was already well established. In addition, it addressed only the development of the performance measurement strategy; program implementation was not addressed.

6.38 Departmental officials cancelled the interim evaluation of the achievement of SADI’s intended immediate outcomes, due in 2009–10. This decision, which the Departmental Evaluation Committee approved, was made because the Department felt there would have been little value in undertaking such an evaluation at that time, given the number of projects in their early stages.

6.39 In October 2011, the Department reported a review of the status of collaboration between project recipients and post-secondary institutions (one of SADI’s objectives). As well, an evaluation of SADI was reported in January 2012. The evaluation assessed program relevance and aspects of performance to date, including investments in research and development, collaboration, and the development of products, processes, and services. We noted that Industry Canada’s multi-year evaluation plan has committed the Department to completing another evaluation of SADI in 2016–17. This additional evaluation effort is required for a full assessment of whether program objectives are being met, including whether longer-term outcomes are being realized.

6.40 For the Bombardier CSeries Program, no implementation review or evaluation has been carried out. An implementation review was recommended in the performance measurement strategy to be completed in 2009–10. The Department’s May 2012 multi-year evaluation plan commits to an evaluation of the Bombardier CSeries Program to be reported in 2015–16 as per the direction provided to Industry Canada when the program was approved. However, this means that no formal assessment of the program is planned before all funding has been disbursed. As a result, the Department is missing an opportunity to make potential timely improvements to the program.

6.41 Since 2007, Industry Canada has authorized assistance of almost $1.2 billion in contributions through SADI and the Bombardier CSeries Program (almost $528 million has been disbursed as of April 2012). Implementation reviews and evaluations are intended to establish whether the programs are on track to meet program objectives. The Department has been slow to meet its evaluation commitments. For SADI, the Department decided to delay or cancel its commitments because it determined that it was too early to measure results. However, without the benefit of this performance information, it is more difficult for the Department to demonstrate progress against program objectives.

6.42 Recommendation. Industry Canada should ensure that its planned evaluations of Technology Partnerships Canada, the Strategic Aerospace and Defence Initiative, and the Bombardier CSeries Program are carried out to allow the Department to measure progress against program objectives. These evaluations should be reported within the time frames the Department has committed to in its multi-year evaluation plan.

The Department’s response. Agreed. The Department will evaluate its transfer payment programs in accordance with its multi-year evaluation plan.

The Department’s public reporting on SADI achievements has been limited

6.43 The Department committed to report publicly each year on contribution recipients, program results and accomplishments, and how approved projects met the assessment criteria. This was to ensure program transparency. As well, the SADI performance measurement strategy requires that, once a year, the Department consolidate performance information that is collected on each project into an Annual Program Benefits Report. The report’s aim is to communicate clearly the results the program has achieved relative to the established objectives and outcomes.

6.44 Although this requirement has been in place in one form or another since 2007, we found limited reporting to meet this commitment. In 2009, Industry Canada publicly released Program Highlights 2007–09, a document that contained limited information on program achievements. In 2012, the Department completed an internal report covering the 2010–11 fiscal year, which provided some information on how selected projects are meeting SADI objectives. However, this report was not released publicly. Our review of Industry Canada’s departmental performance reports and the Department’s website found few references to SADI results and accomplishments.

6.45 Recommendation. Industry Canada should ensure that it meets its commitment to report publicly each year on how approved projects met the Strategic Aerospace and Defence Initiative assessment criteria on contribution recipients and on program results and accomplishments.

The Department’s response. Agreed. Industry Canada will augment the information already available on its website to include additional information in these areas, although the commercially sensitive nature of the projects will continue to limit the amount of information that can be lawfully released to the public. The Department will report this information by the end of December 2012 for all projects approved by 2011–12, and annually thereafter.

Managing aerospace transfer payments

6.46 Funds that Industry Canada transfers to recipients under the Strategic Aerospace and Defence Initiative (SADI) and the Bombardier CSeries Program entail a significant commitment of federal government resources. The government has committed to ensuring that transfer payment programs are delivered and managed fairly, accessibly, and effectively.

6.47 The Treasury Board Policy on Transfer Payments and its associated Directive and Guidelines specify the requirements for departments to follow in managing transfer payments. These documents require, among other things, that Industry Canada

  • identify eligible recipients along with the nature and type of initiatives that will be considered eligible under the transfer payment program;
  • ensure that monitoring and reporting requirements reflect the risks specific to the amount of funding and the risk profile of the recipient; and
  • ensure that cost-effective oversight, internal control, performance measurement, and reporting systems are in place.

6.48 For SADI and the Bombardier CSeries Program, we sought to determine whether Industry Canada

  • ensured that only recipients eligible for funding under program terms and conditions of programs received funding (we did not examine the application process itself),
  • ensured that recipient monitoring and reporting requirements were proportionate to the risk profile of individual recipients and to the level of funding,
  • assessed recipients’ claims and progress reports before issuing payments, and
  • established a management framework for the control and oversight of transfer payments.
The Department funded recipients that met program eligibility requirements

6.49 The program terms and conditions for both SADI and the Bombardier CSeries Program define eligible recipients, activities, and costs. We examined whether Industry Canada conducted sufficient analysis to determine the eligibility of each of the 24 SADI projects and the 2 Bombardier CSeries Program projects before signing agreements with recipients.

6.50 For the 24 SADI projects and the 2 Bombardier CSeries Program projects, we found that before approving the projects, the Department conducted eligibility reviews, reviews of project feasibility, and reviews of companies’ ability to deliver on proposed technological, social, and economic benefits. Overall, we found that these reviews were consistent with the Department’s guidelines.

Monitoring and reporting requirements in contribution agreements do not reflect the level of risk or amount of funding

6.51 The Policy on Transfer Payments and its associated Directive emphasize a risk-based approach to recipient monitoring and reporting. Departmental officials are responsible for ensuring that the requirements placed on recipients are proportionate to the risk level of the project. In particular, monitoring and reporting requirements in the agreements should reflect the level of funding and the risk profile of the recipient. For the 24 SADI projects and the 2 Bombardier CSeries Program projects, we reviewed whether the Department assessed risks to determine the level of monitoring and reporting requirements to be included in the individual contribution agreements.

6.52 We found that a risk analysis was completed for all 24 SADI projects. The analysis for each project included an assessment of technological capability and feasibility risks as well as managerial, financial, and market risks. The projects were then assigned an overall risk level. The analyses were detailed, incorporating the views of departmental officials and, on occasion, external experts. The Department identified only one project as low risk and one as medium-low risk. Conversely, for the Bombardier CSeries Program projects, although a risk analysis was completed that included a description of various risks, an overall rating was not assigned.

6.53 As well, we found that the Department has not defined the appropriate level of monitoring and reporting for the different risk ratings and amount of funding. The Department uses a standard template to draft contribution agreements, which means that all agreements include the same monitoring and reporting requirements regardless of the assessed risks. Departmental officials informed us that the standard frequency for recipients to submit claims and progress reports is quarterly, but that if recipients ask to submit them monthly, the Department will accept them. In our view, defining such levels would help the Department better target its monitoring and reporting efforts.

6.54 We found that monitoring and reporting requirements specified in the contribution agreements with recipients were not proportionate to the levels of risk assigned to the SADI projects. For example, the monitoring and reporting requirements in the contribution agreements for three projects identified as high risk were mostly the same as for the two projects identified as low risk and as medium-low risk. Departmental officials told us that the wording of the contribution agreements provides them with the flexibility to request more information from recipients if risks change.

6.55 As well, we found that there was no correlation between the level of funding and the level of monitoring and reporting requirements outlined in the contribution agreements. This is important, given that the funding provided through the agreements ranged from just under $300,000 to $300 million. For example, for claims monitoring, monthly claims and monthly progress reports were required for 3 of the 24 SADI projects. The projects were for significantly different amounts, indicating that the level of funding was not a factor that determined the level of monitoring and reporting. The other 21 agreements required quarterly claims and progress reports, even though their respective risk ratings ranged from low to high.

6.56 We observed that the Department has recently been more proactive in revisiting its risk ratings for the SADI projects. It reviews ratings on a quarterly basis and has revised the overall risk levels for projects as required. In 2009, the Department established the Industrial Technologies Office (ITO) Risk Management Committee, which is responsible for overseeing ITO’s efforts to manage project and program risks. We found that the Committee discussed risks for individual projects and proposed mitigation measures. However, we did not find actions taken in response to be well documented.

6.57 Having monitoring and reporting requirements in the contribution agreements that are proportionate to risks is important, because these mechanisms could allow Industry Canada to identify potential problems in higher-risk projects earlier on. This may enable the Department to take corrective action more quickly than would otherwise be possible. Since we found that monitoring and reporting requirements are not proportionate to project risks, efficiencies could be missed, unnecessary burdens may be placed on recipients, and departmental resources may not be well targeted.

6.58 Recommendation. The Department should document its approach to risk management, specifying the appropriate levels of monitoring and reporting to be used, proportionate to the risks of the project, the value of funding in relation to administrative costs, and the risk profile of the recipient.

The Department’s response. Agreed. The Department will document its risk management approach for the Strategic Aerospace and Defence Initiative to demonstrate more clearly the rationale behind the specific approaches taken to manage the various risks inherent in funding aerospace research and development projects. The Department will document its overall practices in this regard by the end of December 2012.

The Department reviewed recipient claims and progress reports before issuing payments

6.59 Section 34 of the Financial Administration Act requires that the responsible departmental officer must certify that the recipient is entitled to the payment before a payment is made. The contribution agreements for SADI and the Bombardier CSeries Program projects require that a progress report be submitted with claims. We verified whether the Department reviewed progress reports and claims before issuing payments.

6.60 We found that the Department has a process in place to verify claims that recipients submitted. This process is set out in its Verification of Claims Policy, which defines the roles and responsibilities of officials.

6.61 We examined selected claims made under four SADI agreements and both Bombardier agreements and found that Industry Canada officials reviewed them and the progress reports before the Department released payments to recipients. We found that through their detailed verification of claims, officials determined that some costs were not eligible. For SADI, assessments of progress reports were documented. However, for the Bombardier CSeries Program, assessments of the progress reports were not documented. We observed that an authorized departmental officer certifies that the recipient is entitled to payment before funds are released.

6.62 Industry Canada has a multi-year recipient audit plan for financial audits of SADI recipients. The Department can also initiate other audits of recipients when it determines they are necessary.

A management control framework is in place to support transfer payments

6.63 The Treasury Board Policy on Transfer Payments requires departments to ensure that cost-effective oversight, internal control, performance measurement, and reporting systems are in place to support the management of transfer payments. The Policy further requires departments to establish reasonable and practical service standards for transfer payment programs. As Industry Canada administers close to 30 transfer payment programs of various sizes, ensuring that its management framework is current and relevant is important.

6.64 We examined three elements of the Department’s management framework for the control and oversight of SADI and Bombardier transfer payments:

  • departmental policies and procedures setting out requirements for control and oversight,
  • information and reporting systems to manage and track agreements, and
  • service standards for transfer payment programs.

6.65 Policies and procedures manuals. The Department has a Programs Policies Manual and a Programs Procedures Manual to provide guidance to departmental employees delivering programs. Both manuals predate the 2008 Policy on Transfer Payments. We found that Industry Canada has not finished reviewing the manuals to determine if they adequately reflect the current Treasury Board requirements for managing transfer payments.

6.66 In 2008, the Treasury Board of Canada Secretariat encouraged departments to establish Departmental Action Plans to improve the delivery and efficiency of transfer payments. Industry Canada told us it did not develop a formal action plan since it did not consider this to be a requirement. Starting in 2010, the Department prepared progress reports on its transfer payment reform activities. Updating internal policies and procedures would have been a first step toward aligning programs with the revised Policy on Transfer Payments. We found that the Department has not yet completed its update to internal policies and procedures manuals and that this activity was not part of the progress reports. Having a work plan in place to guide actions that needed to be taken to make changes required by the 2008 Policy on Transfer Payments could have helped the Department to ensure that internal policies and procedures manuals were updated on a timely basis. Our recommendation at paragraph 70 addresses this finding.

6.67 Information and reporting systems. The Contribution Management Information System (CMIS) is the Department’s primary system for monitoring and managing agreements. Additional systems were created because the Department found that CMIS was not meeting all of its needs. For example, the Industrial Technologies Office created a Data Warehouse to capture performance information, since this function is not available through CMIS.

6.68 In 2008, an internal audit noted that limitations to CMIS diminished its usefulness as a tool to support program staff in managing contribution agreements. The audit recommended that the Department articulate a vision and transition strategy for a new management system for departmental transfer payments and ensure that any replacement system include the ability to track and report on program benefits. The Department agreed with the recommendation, saying that the design of a new replacement system would include the ability to track program benefits and to produce reports on these benefits.

6.69 However, the Department has not yet identified a replacement system for CMIS. The Department obtained an additional assessment of CMIS in March 2012. The assessment recommended taking steps to sustain the existing CMIS platform for the next six years and proposed an approach to phase in new components to extend the existing CMIS system capability.

6.70 Recommendation. Industry Canada should establish a plan with deadlines to ensure that its management control framework for transfer payments, including its policies, procedures, and electronic systems, is consistent with requirements set out in the Policy on Transfer Payments and is meeting the needs of the Department.

The Department’s response. Agreed. The Department will develop a plan with deadlines by November 2012.

6.71 Service standards. Industry Canada established service standards for managing SADI as the Policy on Transfer Payments required. The Department has assessed its performance in managing SADI through three service standards (Exhibit 6.2). We noted that while the Department did not establish formal service standards for the Bombardier CSeries Program, it stated that it had an informal standard of 30 days for the processing of claims. Because this was not a formal standard, we did not review the Department’s performance against it.

Exhibit 6.2—Strategic Aerospace and Defence Initiative (SADI) has created service standards to assess its performance

Application processing time

  • Complete the review of a proposal and provide recipients with a funding decision within six months of receiving a complete project application

Claims processing time

  • Process completed claims and release payments for contribution agreements in good standing within 45 calendar days

Application assistance response time

  • Respond to requests for assistance with a project application within one business day

Source: Industry Canada

6.72 As of April 2010, the Department reported publicly on whether it met the service standards for SADI. The latest report, for the fourth quarter of 2011–12, states that 100 percent of applications were processed within six months, that 100 percent of application assistance requests were answered within one business day, and that 100 percent of completed claims were released for payment within 45 calendar days.

6.73 We compared the Department’s reporting against the 45-calendar-day standard for processing claims for payment as part of our audit. In our review, which encompassed several quarters, we found that close to 70 percent of completed claims for SADI were processed within 45 calendar days. The Department explained this discrepancy by telling us that the service standard does not apply when the recipient takes more than 10 days to provide missing information; when an amendment is being negotiated to resolve a possible breach of the agreement; and when there is no payment due, such as in cases where funds are being held back pending project completion. However, taking this approach means the performance standard is misleading.

6.74 Recommendation. Industry Canada should revisit its methodology to measure and report the achievement of service standards for the Strategic Aerospace and Defence Initiative, in particular the claims processing time standard, to ensure that the results reflect actual performance.

The Department’s response. Agreed. Industry Canada will review the claims service standard for the Strategic Aerospace and Defence Initiative by the end of December 2012.

Obtaining repayments from recipients

6.75 The Treasury Board Directive on Transfer Payments requires that contributions made to for-profit businesses that are intended to allow the businesses to generate profits or increase the value of their businesses are to be repayable. Departmental managers are responsible for ensuring that amounts repayable by or recoverable from recipients are recognized as debts due to the Crown, and that appropriate action is taken to recover these amounts.

6.76 Since 1996, Industry Canada has received $767 million in repayments for Technology Partnerships Canada (TPC) and since 1985, it has received $607 million in repayments for the Defence Industry Productivity Program (DIPP). As of May 2012, Industry Canada forecasted a total of $1.5 billion in repayments from the TPC program and $299 million in repayments from the DIPP over the next 10 years, if repayment conditions are triggered.

6.77 Repayments received from companies under TPC flow back into payments to companies under the Strategic and Aerospace Defence Initiative (SADI) and the Bombardier CSeries Program. The Department’s ability to fund these projects depends partly on funds obtained through TPC repayments. If Industry Canada does not receive repayments from recipients, then it may not be able to pay claims to recipients under the current contribution agreements. This risk was noted in Industry Canada’s Corporate Risk profile.

6.78 Within the Department, the Comptrollership and Administration Sector is responsible for contribution agreements when they enter the repayment phase. Repayment conditions and schedules are specified in the contribution agreements. However, due dates and amounts due can change as market realities change. For example, if a repayment amount is based on royalties from sales, it could be different from what had been forecasted. If the company does not meet the repayment conditions—for example, if it does not meet a sales revenue target—it does not have to repay the amount due.

6.79 We examined whether Industry Canada has policies and procedures in place to manage repayments and whether it ensures that funds due under TPC and DIPP are collected from recipients. This work included drawing a sample of agreements in the repayment phase to determine the actions the Department has taken to collect repayments due. We did not examine the process used by the Department to determine whether the conditions triggering repayments were met.

The Department took appropriate steps to obtain repayments due

6.80 We found that, of the repayments we reviewed as part of our audit, the Department obtained most repayments on time. Our findings are summarized in Exhibit 6.3. As well, Industry Canada has a Repayable Contribution Policy, adopted in 1996. The Department has documented procedures to manage repayable contributions. We noted that Industry Canada has recently started updating the Repayable Contribution Policy and its procedures.

Exhibit 6.3—In most cases examined, repayments were received on time

Summary of repayments review Technology Partnerships Canada projects Defence Industry Productivity Program projects
Projects
Number of projects examined 41 10
Number of projects in which conditions were met to trigger repayments 32 10
Repayments
Number of repayments due 132 55
Number of repayments received late 52 11
Repayment amounts
Funds repaid as of May 2012 (millions) $64.1 $21.0
Forecasted repayments over next 10 years (millions) $275.8 $5.7
Source: Industry Canada

6.81 We reviewed a representative sample of 41 TPC repayable project contributions, comprising a total of about $554 million in government funding, and a random selection of 10 DIPP repayable project contributions, comprising a total of about $40 million. We conducted these reviews to determine if the Department monitored the repayment process and obtained repayments on time from 2007 to 2012.

6.82 For TPC, we found that for 32 of the 41 projects, conditions that triggered repayments were met for at least one payment period during 2007 to 2012. A total of 61 percent of repayments were obtained on time. For DIPP, all 10 projects in our selection had at least one payment period where the conditions triggering repayment were met. A total of 80 percent of the repayments were obtained on time.

6.83 For repayments that were late, the Department took appropriate steps to obtain these payments. Actions taken included sending emails and letters to recipients reminding them that repayments were due. Officials told us that in cases where this communication does not result in repayments, files can be transferred to the Department’s Loan Insurance and Recoveries Directorate for additional action.

6.84 In summary, Industry Canada has policies and procedures in place to manage repayments, and it ensures that funds due under DIPP and TPC are collected from recipients. We found that of the repayments we reviewed as part of our audit, the Department obtained most on time. For repayments that were late, the Department took appropriate steps to obtain these payments.

Conclusion

6.85 Before 2010, Industry Canada had inadequate performance information to determine progress being made to achieve the Strategic Aerospace and Defence Initiative’s (SADI) objectives. Since 2010, the Department has made improvements and now collects and consolidates sufficient information to allow it to determine progress against objectives. For the Bombardier CSeries Program, progress reports contained limited information on outcomes and benefits. The Department did not request annual progress reports, required under the two contribution agreements, meaning it had a more limited picture of the recipient’s performance.

6.86 We conclude that the Department has made improvements since 2010, but it needs to follow through on commitments to collect additional performance information. This is important so that the Department can report on progress against objectives as the projects mature. Planned evaluations of SADI, to be reported in the 2016–17 fiscal year, and of the Bombardier CSeries Program, to be reported in 2015–16, will rely on such information.

6.87 When it introduced SADI, the Department committed to increasing its level of transparency. However, the Department has not followed through on its commitment to report publicly each year on the program’s results. As well, our review of Industry Canada’s departmental performance reports and its website found few references to SADI results and accomplishments.

6.88 Industry Canada has a management control framework in place to guide the way it manages these transfer payments. The Department carried out a detailed review of proposed projects before signing contribution agreements with recipients and funded only recipients that met program eligibility requirements. It also reviewed recipient claims and progress reports before issuing payments. However, the Department needs to make some improvements, including better documenting of its approach to risk management.

6.89 In cases where contributions under the Defence Industry Productivity Program and Technology Partnerships Canada were repayable, the majority of repayments we examined were obtained by the Department on time. The Department has documented procedures to manage repayable contributions, and has recently started updating these procedures.

About the Audit

All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by The Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, we also draw upon the standards and practices of other disciplines.

Objective

The objective of our audit was to determine whether Industry Canada has sufficient information to assess whether selected science and technology transfer payments to the aerospace sector were meeting their stated objectives and whether the Department managed these transfer payments in accordance with key requirements of the Policy on Transfer Payments.

Scope and approach

We looked at the Strategic Aerospace and Defence Initiative (SADI) and the Bombardier CSeries Program. We also examined whether Industry Canada collects repayments from recipients for contributions that are repayable. We looked at two legacy programs—Technology Partnerships Canada (TPC) and the Defence Industry Productivity Program (DIPP).

We reviewed the 24 contribution agreements under the SADI program signed as of 1 March 2012, as well as the 2 contribution agreements for the Bombardier CSeries Program. We looked to see if the Department had reviewed selected claims submitted under these agreements, but we did not audit the eligibility of the claims.

Separate selections of agreements were drawn for the TPC and DIPP programs. To complete the work on repayable contributions, for the TPC program, a representative sample of 41 agreements was selected from the total population of 188 active agreements. This sample size is sufficient to conclude on the population with a confidence level of 90 percent and a confidence interval of no more than 10 percent. For DIPP, a random selection of 10 agreements signed after 1985 was selected from a total population of 950 active agreements. We did not examine the process used by the Department to determine whether the conditions triggering repayments were met.

We analyzed contribution agreements, files, records, and reports. We interviewed Industry Canada officials who are responsible for the design and delivery of these programs. We also met with a number of companies that received contributions to obtain their views and opinions. Our audit examined Industry Canada’s performance; we did not audit recipients of government funding.

Criteria

Criteria Sources
To determine whether Industry Canada had sufficient information to assess whether selected science and technology transfer payments to the aerospace sector were meeting their stated objectives, and whether it managed these transfer payments in accordance with key requirements of the Policy on Transfer Payments, we used the following criteria:

Industry Canada monitors recipients in accordance with program requirements, funding agreements, and the Transfer Payment Policy.

  • Terms and Conditions of the selected programs
  • 24 Strategic Aerospace Defence Initiative and 2 Bombardier CSeries Program funding agreements
  • Policy on Transfer Payments, Treasury Board, 2008 and 2000
  • Directive on Transfer Payments, Treasury Board
  • Financial Administration Act
  • Guideline on Performance Measurement Strategy under the Policy on Transfer Payments, Treasury Board of Canada Secretariat

Industry Canada has a performance measurement strategy or system that provides for appropriate collection of performance information at the level of individual recipients and at the transfer payment program level.

  • Policy on Transfer Payments, Treasury Board, 2008 and 2000
  • Directive on Transfer Payments, Treasury Board
  • Guideline on Performance Measurement Strategy under the Policy on Transfer Payments, Treasury Board of Canada Secretariat
  • Policy on Management, Resources and Results Structures, Treasury Board, 2010 and 2005
  • Management Accountability Framework, Treasury Board of Canada Secretariat
  • Financial Administration Act
  • Policy on Evaluation, Treasury Board, 2009 and 2001

Industry Canada uses performance information to modify programming as necessary and to demonstrate whether program results meet objectives.

Industry Canada collects amounts repayable by, or recoverable from, recipients under agreements.

  • Policy on Transfer Payments, Treasury Board, 2008 and 2000
  • Directive on Transfer Payments, Treasury Board
  • Guideline on the Directive on Transfer Payments, Treasury Board of Canada Secretariat
  • Accounting Standard 3.2—Treasury Board—Transfer Payments (Grants and Contributions)
  • Interest and Administrative Charges Regulations under the Financial Administration Act
  • Policy on Receivables Management, Treasury Board, 2002

Management reviewed and accepted the suitability of the criteria used in the audit.

Period covered by the audit

Audit work for this chapter covers the period from 1 January 2007 to 1 June 2012. Audit work for this chapter was completed on 17 July 2012.

Audit team

Assistant Auditor General: Ronnie Campbell
Principal: Glenn Wheeler
Director: Nadine Cormier

Laurent Bergeron Collin
Maxine Leduc
Katherine Ludwig

For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free).

Appendix—List of recommendations

The following is a list of recommendations found in Chapter 6. The number in front of the recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses indicate the paragraphs where the topic is discussed.

Recommendation Response
Measuring progress on objectives

6.33 Industry Canada should ensure that monitoring and reporting requirements set out in its contribution agreements are fully met and documented. (6.26–6.32)

Agreed. The Department will ensure that the monitoring and reporting requirements of contribution agreements are fully met and documented.

6.34 Industry Canada should ensure that future contribution agreements it signs with recipients specify project-specific objectives and identify anticipated outcomes and benefits that are aligned with the Strategic Aerospace and Defence Initiative program objectives. (6.22–6.25)

Agreed. The Department will continue the practice, started in 2010, to include in its contribution agreements a section specifying project-specific objectives and identifying the anticipated outcomes and benefits of the project in order to demonstrate clearly their alignment with the Strategic Aerospace and Defence Initiative’s objectives.

6.42 Industry Canada should ensure that its planned evaluations of Technology Partnerships Canada, the Strategic Aerospace and Defence Initiative, and the Bombardier CSeries Program are carried out to allow the Department to measure progress against program objectives. These evaluations should be reported within the time frames the Department has committed to in its multi-year evaluation plan. (6.35–6.41)

Agreed. The Department will evaluate its transfer payment programs in accordance with its multi-year evaluation plan.

6.45 Industry Canada should ensure that it meets its commitment to report publicly each year on how approved projects met the Strategic Aerospace and Defence Initiative assessment criteria on contribution recipients and on program results and accomplishments. (6.43–6.44)

Agreed. Industry Canada will augment the information already available on its website to include additional information in these areas, although the commercially sensitive nature of the projects will continue to limit the amount of information that can be lawfully released to the public. The Department will report this information by the end of December 2012 for all projects approved by 2011–12, and annually thereafter.

Managing aerospace transfer payments

6.58 The Department should document its approach to risk management, specifying the appropriate levels of monitoring and reporting to be used, proportionate to the risks of the project, the value of funding in relation to administrative costs, and the risk profile of the recipient. (6.51–6.57)

Agreed. The Department will document its risk management approach for the Strategic Aerospace and Defence Initiative to demonstrate more clearly the rationale behind the specific approaches taken to manage the various risks inherent in funding aerospace research and development projects. The Department will document its overall practices in this regard by the end of December 2012.

6.70 Industry Canada should establish a plan with deadlines to ensure that its management control framework for transfer payments, including its policies, procedures, and electronic systems, is consistent with requirements set out in the Policy on Transfer Payments and is meeting the needs of the Department. (6.63–6.69)

Agreed. The Department will develop a plan with deadlines by November 2012.

6.74 Industry Canada should revisit its methodology to measure and report the achievement of service standards for the Strategic Aerospace and Defence Initiative, in particular the claims processing time standard, to ensure that the results reflect actual performance. (6.71–6.73)

Agreed. Industry Canada will review the claims service standard for the Strategic Aerospace and Defence Initiative by the end of December 2012.

 

PDF Versions

To access the Portable Document Format (PDF) version you must have a PDF reader installed. If you do not already have such a reader, there are numerous PDF readers available for free download or for purchase on the Internet: