This Web page has been archived on the Web.

1999 September Report of the Auditor General of Canada

Main Points

• Industrial Research Assistance Program - National Research Council (NRC)
• Research Partnerships Program - Natural Sciences and Engineering Research Council (NSERC)
• Networks of Centres of Excellence - NSERC
• Technology Partnerships Canada - Industry Canada

19.3 We also sought to determine if these programs were well managed and if management knew whether value for money was being achieved. We could not assure ourselves that many of the contributions under the Industrial Research Assistance Program and many of the grants under the Research Partnerships Program were properly supported. While the technological merit of the projects we examined had been well documented, there was often little explanation of the commercial or pre-commercial benefits expected from the projects, and of the need for government support. There are also important performance issues for which management has little information - in particular, on the commercial or pre-commercial results of funded projects.

19.4 We concluded that due diligence had been exercised in the grants we audited under the Networks of Centres of Excellence program. We also concluded that the management of Technology Partnerships Canada (TPC) had exercised due diligence in making the contributions that we audited, with specific exceptions. TPC could make improvements in monitoring the progress and results of funded projects, and in reporting to Parliament on how it shares risks and returns with funding recipients.

Background and other observations

19.6 However, our review of the issues suggests that the causes and effects of this gap are not straightforward. While innovation is undoubtedly an important factor in economic growth, assessing the actual innovation performance of the economy is a multifaceted challenge. Although a comprehensive assessment is still difficult to make, there is growing evidence that Canada's performance lags behind that of its major competitors in a number of specific ways. It is reasonably clear that better innovation performance depends on more than just increased spending on research and development; it involves supporting research and development with the activities needed to embed new technologies in the economy. It seems equally clear that spending on research and development is not the only determinant of the Canadian economy's rate of productivity growth, and may not be the most important one.

19.7 Promoting innovation in the economy is one of the principal objectives of the Industry Portfolio, which is made up of the organizations for which the Minister of Industry is responsible. The programs we audited focus on supporting research and development and account for the bulk of the grants and contributions made by the Portfolio toward that objective:

• The National Research Council delivers the Industrial Research Assistance Program, which helps small- and medium-sized enterprises develop and exploit technologies ($120 million in 1998-99).
• The Natural Sciences and Engineering Research Council delivers the Research Partnerships Program with the objective of fostering interactions and partnerships between university researchers and other sectors in order to generate new knowledge and develop new expertise, and to transfer this new knowledge and expertise to Canadian-based organizations ($95 million in 1998-99). NSERC also delivers the Networks of Centres of Excellence program to improve Canada's performance in science and technology, and to facilitate transfer of knowledge to those who can use it to advance Canada's social and economic development ($47 million in 1998-99).
• Technology Partnerships Canada is a special operating agency within Industry Canada. It is intended to promote the development and commercialization of innovative technologies that contribute to increasing economic growth and creating jobs and wealth ($250 million in 1998-99).

Introduction

• a review of the issues of innovation performance in the economy; and
• the results of our audit of these organizations to determine whether management is exercising due diligence and achieving value for money in their investments in innovation projects.

19.10 What does innovation mean? There is probably no single definition of innovation that would meet with universal approval. Nevertheless, the newness of a product or process is clearly a necessary criterion; and innovation is a broader idea than invention alone, for it also includes the development, application and commercialization of the invention. Innovation, then, is not only the initial flash of inspiration or the scientific breakthrough, but the process of rendering the idea practically or commercially viable. The United States General Accounting Office, in a 1996 report to Congress, defined innovation as both invention and commercialization. While this definition limits the idea to mainly business settings, the implicit point seems to be that innovation, wherever and however it occurs, means knowledge put to new use. Beyond this broad meaning, there are a number of views on how to further define or to categorize different models of innovation (see Exhibit 19.1).

19.11 National and regional innovation systems. There is more to innovation in the economy than just new products or processes. In recent years, an international policy discussion has centred on the idea of national and regional innovation systems as the key to understanding and improving innovation performance. The Organization for Economic Co-operation and Development (OECD) has played a major role in promoting this view, which stresses that flows of technology and information among people and institutions are key to the innovative process in national and regional economies. Innovation results from a complex set of relationships among participants in the system, which is made up of businesses, universities and government research institutes (see Exhibit 19.2).

19.12 As a result, the rate of technological change in a country and the international competitiveness of companies do not depend simply on the scale of research and development. They also depend on how available resources are managed and organized, at both the enterprise and the national level; in other words, they depend on the structure that encourages and exploits innovations.

19.13 For policy-makers, an understanding of national and regional innovation systems can help identify leverage points for enhancing innovative performance. The recommended policies tend to be those that improve the system itself by building networks of institutions and that aim at improving the innovative capacity of firms, particularly their ability to identify and absorb technologies. An important result of this policy discussion is that innovation performance is now regarded as more than simply the result of research and development spending. How well technology is diffused and adopted throughout the economy is as important as, if not more important than, its creation in any one particular "high-tech" sector.

19.14 Innovation as public policy. As a matter of public policy, Canada's innovation performance has aroused concern for many reasons. First, many observers argue that Canada's relatively weak economic growth over the last two decades is attributable to a marked decline in the rate of productivity increase. It is widely held that innovation and its diffusion in the economy plays a role in determining this rate. For example, in summarizing the factors underlying the relatively poor productivity performance in Canada, a 1996 report by the government's policy research initiative concludes: "In simple terms...[it] could be linked to a lack of adjustment and innovation." In a similar vein, the Conference Board of Canada's Performance and Potential (1997) maintains that technology diffusion has a decisive impact on the growth of productivity.

19.15 A second concern about innovation as public policy relates to the changing global economy. Many observers contend that the global economy is undergoing a fundamental restructuring that is ushering in a new economic paradigm known as the knowledge-based economy. According to this view, to survive and prosper in the knowledge-based economy, firms, industries and economies must adopt a strategy of continuous innovation and must engage in innovation competition. The application of new knowledge and ideas and the creation of new products and processes is the essence of the knowledge-based economy.

19.16 Concern about Canada's capacity for innovation leads to a number of public policy questions. Are we innovating widely enough and fast enough? Are we doing as well as other countries? The last question is frequently framed in terms of an "innovation gap" between Canada and its major economic rivals.

19.17 The innovation gap. The idea of a gap is a popular one in the context of studies of relative economic performance. For example, numerous empirical studies have attempted to account for the productivity gap between Canada and the United States. Recently the notion of an innovation gap, coined by the OECD, has gained acceptance.

19.18 The term is commonly attributed to the OECD's 1995 Economic Survey of Canada that stated: "The view is widely held that Canadian industry suffers from an innovation gap compared with other developed countries." The OECD attributed the gap to insufficient efforts in research and development, as well as to private and public sector attitudes that have contributed to an inflexible and non-innovative industrial structure. Exhibit 19.3 provides information on different aspects of Canada's innovation performance.

19.19 Closing the gap. In the fall of 1994, the government set out its economic program in Agenda: Jobs and Growth. One of the four papers released, Building a More Innovative Economy, focussed on four policy areas - marketplace climate, trade, infrastructure and technology. The 1996 Budget built upon the priorities of the jobs and growth agenda and, in particular, announced the creation of Technology Partnerships Canada as part of the government's intention to make Canada one of the most innovative countries.

19.20 Also in 1996, the federal government's strategy Science and Technology for the New Century asserted: "it is well documented that, by international standards, Canada's S&T effort has considerable room for improvement, particularly in the development, adoption and commercialization of technology." It goes on to link the innovation gap to poor productivity performance, and the poor productivity performance to depressed real incomes and high government deficits.

19.21 Industry Canada's 1998-99 Report to Parliament on Plans and Priorities points to the innovation gap and states that failure to capitalize on innovation has been a major reason for Canada's relatively slow productivity growth over the past two decades. The Department's 1997-98 Performance Report warns that without innovation, real income growth will not meet Canadians' expectations. It goes on to state: "While a sizable overall innovation gap remains, it has been reduced over time. The government has supported innovation through a number of key initiatives, such as the Canada Foundation for Innovation, Technology Partnerships Canada, Networks of Centres of Excellence and the Industrial Research Assistance Program." However, as noted in exhibits 19.1, 19.2 and 19.3, innovation is a multifaceted idea. Most of the existing information is on inputs to innovation, but not on innovation performance directly.

19.22 Exhibit 19.4 explains some of the challenges involved in measuring innovation. What this information means in aggregate is not at all clear. For example, the 1996 report by the government's policy research initiative argues that our understanding of what is happening in the knowledge-based economy is constrained by the extent and quality of current indicators. The report states that new indicators are needed that capture the innovation process and the distribution of knowledge among key organizations.

19.23 Toward that end, Statistics Canada is currently developing indicators of science and technology activity and a framework to tie them together into a coherent picture. Several key areas are covered by this initiative, such as innovation, technology diffusion, human resources in science and technology and interrelations among different institutions.

19.24 Innovation and productivity growth. Innovation and productivity growth go hand in hand. Labour productivity is a measure of the quantity of goods and services produced per unit of labour input. When output per unit of input rises, it is possible for some people to consume more without others consuming less, or even for everyone to consume more. In other words, higher productivity means a higher standard of living. Innovation, in turn, entails changing the goods and services we produce and how we produce them. However, the rate of labour productivity growth in Canada and other industrialized countries has declined dramatically since 1973. Thus, while this phenomenon is not unique to Canada, a number of recent reports have concluded that a productivity gap exists between Canada and the United States.

19.25 Recent studies identify a number of factors accounting for our relatively poor productivity performance in relation to the United States: slower and weaker adjustment to the two energy price shocks, a slower rate of capital accumulation, a slower rate of adoption of best practice technologies, a slower rate of growth in research and development spending, weaker adjustment to the knowledge-based economy, and weaker competition in both product and factor markets. Some commentators have also cited relative tax rates as a key reason why American productivity growth has outstripped our own. Others cite smaller scales of operation, and a low Canadian dollar shielding less productive firms. Although spending on research and development has some role to play, there is clearly no one factor that explains this productivity gap.

19.26 Summing up. It is difficult to do justice to these broad economic issues within the context of this chapter; however, a number of observations can be offered:

• Assessing the innovation performance of the economy is a multifaceted challenge for which little comprehensive information yet exists.
• It is reasonably clear that better innovation performance depends on more than just spending on research and development; it involves supporting research and development with the activities needed to embed new technologies in the economy.
• Spending on research and development is not the only determinant of the Canadian economy's rate of productivity growth, and may not be the most important one.

Focus of the audit

• Industrial Research Assistance Program - National Research Council
• Research Partnerships Program - Natural Sciences and Engineering Research Council
• Networks of Centres of Excellence - NSERC
• Technology Partnerships Canada - Industry Canada

19.28 With the previously noted issues in mind, we examined whether management:

• could show that these programs are designed to improve Canada's innovation performance;
• exercises due diligence in investing in innovation projects; and
• knows if the programs are achieving value for money.

Observations and Recommendations

Addressing the Innovation Gap

19.31 This message is repeated in various ways in many recent reports to Parliament by Industry Canada, the Natural Sciences and Engineering Research Council, and the National Research Council. For example, the government's 1998 report on federal science and technology, Building Momentum, claims that Technology Partnerships Canada has become an effective tool for closing the innovation and productivity gaps; the 1999-2000 Report to Parliament on Plans and Priorities of the Natural Sciences and Engineering Research Council states that its financial support for university-industry research projects leads to new products, processes and services; and the National Research Council (NRC) reports that the Industrial Research Assistance Program (IRAP) activities stimulate the innovative capacities of Canadian small- and medium-sized enterprises.

19.32 Accordingly, we looked at whether management had a strategy that linked the funding of individual projects carried out by or in partnership with business to specific aspects of the broader government goal of improving innovation performance in the economy. In particular, we wanted to know how these programs were supposed to address specific gaps in innovation performance and to what extent.

19.33 In supporting innovation, a number of overlapping roles are possible, including:

• encouraging the invention and introduction of new or improved technology;
• making firms aware of existing technologies;
• adapting existing technology to new uses; and
• helping firms improve their innovation capacity - that is, helping them improve the assets, processes and tools that keep them competitive.

19.35 Each program supports innovation. Each of the programs we audited makes grants or contributions (IRAP provides advice as well) to carry out one or more of these roles. We found that management of each program is developing goals and measures for assessing performance in relation to these roles. However, these goals and measures are not yet used to manage program activities. In the case of the Networks of Centres of Excellence program, efforts have been made to establish broad goals that capture the program's intended impact on innovation performance. Nevertheless, the expected results for these programs are not expressed in innovation performance terms - in particular, in terms that capture the intended impact on innovation systems. Clear statements of expected results are prerequisites to sound program design and management, and to measuring and interpreting the actual results achieved.

19.36 While each program undoubtedly contributes to improving innovation performance, there is little information on the extent of that contribution or its significance. We found that no strategy or management framework (with goals and targets) existed for any of these grant and contribution programs, linking spending decisions to an innovation gap or gaps. Nevertheless, program management has taken steps to develop broad approaches to supporting innovation. Technology Partnerships Canada, for example, has developed a plan explaining what kind of projects it wants to support whereas IRAP takes a decentralized, community-based approach that is intended to be reactive and industry-driven. However, these initial steps are general in nature, and are not yet associated with clear results and performance measures. Before adequate management frameworks can be developed, certain basic questions have to be answered. What improvement in innovation performance, and in which aspects, is expected from the money spent either on a program or on a project-by-project basis? How has innovation performance actually improved as a result of program spending? Until management is able to answer these questions, it cannot be confident that it is achieving the most value for money in funding particular projects.

19.37 Toward a coherent portfolio approach. The government's 1996 S&T strategy, and subsequent reports to Parliament, imply a focussed, co-ordinated approach by the Industry Portfolio to addressing innovation performance problems in the Canadian economy and thereby improving the rate of productivity growth. While the Portfolio is working on such an approach, we believe that it needs further development. Moreover, management recognizes that there is an important opportunity for Industry Portfolio programs to work in concert. To do this, management needs to agree on what innovation performance issues can be best addressed by the Portfolio and what results are expected from each of its grant or contribution programs.

19.38 Good program design is at least partly based on good information and on an analysis of the issues being addressed - why they exist, how important they are, and what can be done. As discussed in the introduction to this chapter, there are a number of fundamental issues concerning innovation performance for which good information does not yet exist, although Statistics Canada is working on gathering this information. By working collaboratively, management in the Industry Portfolio will be able to improve the available information as a basis for good program design.

19.39 Industry Canada, the National Research Council and the Natural Sciences and Engineering Research Council should provide Parliament with information on the following issues:

• What does innovation performance of the economy mean?
• What specific innovation performance problems are Industry Portfolio programs supposed to address?
• What specific results are expected from Industry Portfolio programs toward addressing these problems?
• Are these expected results being achieved?

• documenting the role innovation plays in Canadians' standard of living and quality of life and promoting its understanding, throughout Canadian society;
• increasing the creation of knowledge in Canada and improving access to and diffusion of knowledge created in Canada and abroad;
• encouraging the early consideration of applications of knowledge and speeding its application to processes, products and services; and
• ensuring that Canada's market frameworks encourage the development of knowledge and innovation.

Statistics Canada is involved in developing a set of indicators of innovation performance, and private sector organizations, for example, the Conference Board of Canada, are developing complementary approaches to monitoring Canada's innovation performance. These are being studied and adapted in designing and refining appropriate methodologies for performance measurement in the Industry Portfolio.

Industrial Research Assistance Program

19.41 IRAP has a field staff of about 260 industrial technology advisors, located in 90 communities across Canada, who assist SMEs in resolving their technology-related problems, or in exploiting opportunities. About 30 percent of the technology advisors are National Research Council employees. The rest work in over 130 public and private sector organizations that participate as IRAP network members. IRAP has contribution agreements with each network member for the salaries and other expenses of network technology advisors.

19.42 IRAP also co-ordinates the Canadian Technology Network, which comprises 1,000 members nationally and is an informal affiliation of organizations that provide assistance to SMEs. We did not examine IRAP's role in the Network.

19.43 We selected a sample of 120 contributions to firms from 1994 to June 1999. The amount of funding provided ranged from $1,000 to $998,000. We also examined selected contribution agreements with network members.

19.44 In 1996, IRAP went through an extensive strategic planning exercise. At the same time, a separate review of the program concluded that although IRAP was successful in helping SMEs become more innovative, management practices needed significant improvement. The objectives and direction set out in IRAP's new strategic plan led to specific changes to the terms and conditions of the program in 1998 for its contributions to firms and to network members. Although IRAP had focussed for many years on building the innovation capability of SMEs through improving their technological competence, this role was made explicit in its new objective and in the project assessment criteria (see Exhibit 19.5). As a result of these recent changes, IRAP was in a period of transition during our audit as it developed and applied the tools its technology advisors need to assess and help firms make improvements in innovation capacity.

Concerns about support for funding decisions

19.46 We expected that IRAP's funding decisions would take into account all the criteria established for assessing potential projects and that these decisions would be based on reliable information. We recognize that many contributions under IRAP are for relatively small amounts, and an exhaustive analysis should not be required. For example, 43 of the projects we audited received less than $15,000 in funding. Nevertheless, we expected that to meet the requirements of due diligence, there would be at least minimal information supporting the assessment of each of the criteria for all contributions.

19.47 We found that only about 15 percent of the funding decisions were based on an assessment of all of the project criteria, mainly because the need for IRAP support had not been considered. We also found a range of practices in assessing the other criteria - in particular, how projects would improve the technological competence of funding recipients. In our view, the assessment of individual funding criteria for many projects was not sufficiently thorough.

19.48 Little support for need for funding. We expected that the project files would provide an explanation of why IRAP funding was necessary for the project to proceed, since this is a key test of the need for government financial support. This test is commonly referred to as incrementality, because it means funding worthwhile projects that would not have proceeded otherwise with similar results. However, we found that the need for IRAP funding was not explained for 84 percent of the projects. In these cases, we could find no evidence that incrementality was considered in IRAP's funding decision. We found examples where the companies receiving assistance have annual sales ranging from $6 million to $31 million and yet have received funding ranging from $1,000 to $225,000. Although these firms might have been new to research and development, it was not explained why they needed IRAP assistance for their projects.

19.49 Assessments often limited to technical feasibility. As discussed previously, IRAP changed the assessment criteria in 1998 to explicitly include improving the innovation capability of SMEs. Nevertheless, for some time before that, IRAP management had formally stated that it wanted to support projects that had an enduring impact on the technical competence of the firm rather than a one-time process or product innovation. Under the previous assessment criteria, for example, the assessments of applications for funding were supposed to judge how much improvement a project would make in the technology base of the firm.

19.50 For most of the projects we audited, the assessments of technical feasibility were reasonably complete and thorough. What was less clear was how much projects would increase the technological competence or innovation capability of the funding recipients. We found a wide range of practices in how technological competence or innovation capability were defined and considered in project assessments. We also found that technology advisors have not yet been provided with the tools they need to assess and help with improving the innovation capacity of client firms.

19.51 Expected business benefits not always considered. In the project assessments, the expected benefits from the proposed projects are expressed as increased sales, cost reductions or jobs. These projections are used as indications of increased competitiveness, which is one of the funding criteria for the program. We appreciate that detailed projections of sales or cost reductions are too speculative to provide reliable support to research and development (R&D) proposals. However, we expected that, at the very least, the plausibility of the projected sales and market potential for the end products would be challenged. Instead, we found a range of practices in assessing benefits to firms. For about 10 percent of the projects, we could not satisfy ourselves that these benefits had been properly considered in the funding decisions.

19.52 As mentioned previously, the depth of analysis underlying a funding recommendation varies according to the significance of IRAP's financial involvement. For larger projects, expected benefits are normally better explained and substantiated. However, regardless of the amount of funding provided, we believe that IRAP has not properly assessed the merit of a project unless it examines the rationale for expected business performance improvements.

19.53 IRAP asks applicants to identify socio-economic benefits to Canada from their projects. Applicants normally identify desirable business impacts, such as increases in their firms' revenues and work force or at least the maintenance of existing jobs. Jobs created or maintained are considered a socio-economic benefit. We found that expectations of job creation or maintenance were not adequately supported, even taking into account the uncertainties in R&D outcomes.

Mixed program performance information

19.55 Incomplete information on project results. We expected that IRAP would know if funded projects were achieving the expected technical and commercial results. However, there was often little information on file concerning the success of projects. Although final reports on the technical performance of projects are required, about one third of the completed projects we audited did not have one.

19.56 In addition, we noted that some regional offices require technology advisors to report on results an average of six months after project completion. These reports were available for 60 of the 90 completed projects. While these reports provide some information on project results, such as the number of jobs created, there is no explanation of what produced these results or how the innovation capacity of the firms was improved. Nevertheless, we also found other information on file suggesting that the majority of projects had at least met their technical goals of developing technology.

19.57 Results of advisory service unclear. As discussed previously, IRAP has about 260 technology advisors, 70 percent of whom are not National Research Council employees. Although technology advisors spend a considerable portion of their time providing advice to clients, IRAP has not set expected results for its advisory services, nor does it track the results. We expected that IRAP would at least be aware of the extent to which the information needs of its clients were being properly identified and met.

19.58 Under the agreements signed with member organizations, network technology advisors are expected to set annual objectives and report performance toward those objectives monthly and annually. However, we found that expected and actual performance are described almost entirely as activities (number of clients visited, telephone calls, networking activity, professional development) and not as results. In other words, it is unclear what results the advisory services expect to or actually do achieve. IRAP management is not certain what help was requested by its clients and what its advisors did for them.

19.59 Incrementality needs clarification. IRAP regularly surveys the companies it funds and asks what difference its contributions and advice made. In our view, the results of these surveys suggest that the majority of the projects funded by IRAP may have gone ahead anyway without IRAP funding or advice. It appears that IRAP's financial support facilitates projects rather than making the difference between whether they proceed or not.

19.60 We examined the results of these client surveys from 1996 to 1998. In each of these years, about 40 percent of respondents indicated that their project had needed IRAP funding to proceed. The remaining 60 percent reported that they would have proceeded anyway without IRAP funding, although perhaps more slowly or with added difficulty. A separate survey in 1996 found that the majority of respondents considered IRAP support helpful for their projects. However, 40 percent indicated that they would have gone ahead if IRAP funding had been smaller; and 17 percent did not know.

19.61 What is unclear from the results of these surveys is whether the projects would have proceeded with the same scope and achieved similar results. Undoubtedly, IRAP funding mitigates some of the risk for these firms by reducing their direct investment and therefore potential losses if the project fails. However, mitigating the risk of projects that would have proceeded anyway is not a formal objective of the program.

19.62 In addition, IRAP asked firms that had received funding for their projects about the usefulness of its advisory services. Only 13 percent of survey respondents reported that the project would not have proceeded if technical assistance had not been provided; 30 percent of recipients indicated that IRAP advisory services had no effect and the remainder reported that they would have gone ahead with the project, although with added difficulty.

19.63 This survey information on performance, while insufficient to support a definitive conclusion, suggests that many projects might not have needed IRAP's funding or advice to proceed. The true incrementality of IRAP support needs further investigation.

19.64 Job creation may be overstated. IRAP contributes to job creation in two ways. First, direct jobs may be created when technical staff are hired by the client to work on an IRAP-supported project. Second, new jobs may be created from the commercialization of a product or process resulting from a project to which IRAP was one of the contributors.

19.65 In recent public reports, IRAP has claimed that it creates between 9,000 and 10,000 jobs a year. This figure is based on a 1995 study that IRAP did on jobs created, saved or lost through all projects it had supported that terminated in fiscal year 1991-92. Many were multi-year projects, and this was taken into account in the study.

19.66 Our review of the methodology followed in this study suggests that the job creation numbers may be unreliable for the following reasons. First, the elimination or loss of jobs in other firms is not considered. Second, incrementality (whether IRAP funding was needed for the project to proceed) and attribution (the extent to which the jobs can be attributed to IRAP's support for a project) may not have been adequately considered, leading to an overestimation of IRAP's role in job creation. In our view, any job creation and maintenance numbers reported by IRAP need careful explanation and qualification.

19.67 Limited program evaluation. We examined two evaluation reports, respectively completed in 1990 and 1996, to determine whether IRAP had adequately assessed its performance. Both reports concluded that there is clear evidence that IRAP is successful in helping SMEs become technically capable. Management has informed us that IRAP's role in supporting Canada's innovation system was left aside in the 1996 assessment to concentrate on other issues. In our view, these evaluations provide only a limited perspective on the relevance, success and cost-effectiveness of IRAP.

19.68 The National Research Council should:

• establish clear expected results for the Industrial Research Assistance Program (IRAP) and report performance against them;
• ensure that the decision to fund a project is based on an appropriate assessment of the project's merits and the firm's need for IRAP support;
• ensure that reliable information is gathered on the results of individual projects;
• establish results-based plans and performance measures for its advisory services; and
• conduct a program evaluation of IRAP that addresses all of the key evaluative questions.

We agree with the recommendations to improve the measurement of IRAP's performance, and share the opinion that we should provide industrial technology advisors and managers with the tools and information needed to make quality decisions, report on performance and manage the program effectively. We are already acting on the recommendations, and indeed some of the changes were in the process of development at the time of the audit, as outlined below:

• A program performance framework was approved in 1998-99. Under active development are: a model to understand IRAP's role in the innovation process, data collection instruments, assessment of incrementality, and plans to capture performance information, including a follow-up system for IRAP projects to record the downstream benefits against IRAP objectives.
• An upgraded quality assurance process is being implemented across the country and is providing information on due diligence in the project decision-making process, including the application of selection criteria. One result, initially for funded projects, will be a more visible record of the need for and anticipated benefits of IRAP assistance.
• A major business process improvement effort was initiated in April 1998 to modernize the IRAP client business process. This will enable the best use of information to support decision making close to clients, enhance collaboration, encourage sharing and learning, as well as measure impact and results. This process will continue to be the focus of our efforts.
• An information management plan and strategy is being prepared. This will guide the refinement of existing and development of new IRAP systems to be compatible with the recent adoption by NRC of a new large-scale, integrated enterprise business information system. The plan and strategy cover a comprehensive electronic database for project approval processes, project management, network interactions and project results.
• As part of its normal assessment schedule, NRC is planning to undertake an evaluation of IRAP, which will include addressing questions in the Auditor General's Report.

Research Partnerships Program

19.70 Our audit focussed on the Research Partnerships Program, which accounted for $95 million of NSERC's expenditures in 1998-99, and the Networks of Centres of Excellence program, which had expenditures of $47 million in 1998-99.

19.71 The objective of the Research Partnerships Program is to foster interactions and partnerships between university researchers and other sectors in order to generate new knowledge and develop new expertise, and to transfer this new knowledge and expertise to Canadian-based organizations.

19.72 In recent reports to Parliament, NSERC explains that one of its main goals is to facilitate the transfer of knowledge from universities to other sectors, and the commercialization of university-derived technologies. NSERC does this because university "partnerships with industry connect researchers with those who can use the new knowledge productively and thereby enhance Canada's capacity for innovation." In Part III of its 1997-98 Estimates, NSERC reported that it facilitates knowledge transfer from universities to other sectors, and the commercialization of university-derived technologies. Its financial support for industry-university partnerships leads to new products, processes and services.

19.73 Putting this program rationale into practice, NSERC set the following criteria for deciding on which research proposals to fund through the Research Partnerships Program:

• merit of the research proposal (originality and quality of the research, quality of the research team, benefits of the proposed research and its potential impact on Canada's economy, industry, society and/or environment);
• interactions and partnerships (nature and extent of contributions from participants, ability of the partners to exploit the research results to the benefit of Canada);
• training (extent to which all participants are involved in the training of highly qualified personnel); and
• management and budgeting ( funds requested from NSERC and contributions from other sources and its management).

19.74 For our audit, we selected a sample of 100 grants from the various categories of grants made from 1994 to 1997 under the Research Partnerships Program as well as seven Networks of Centres of Excellence grants made in the years 1994 to 1998.

Concerns identified in support for funding decisions

19.76 We found that the scientific merit of the projects we examined had been well established, as had the potential for training. However, scientific merit deals with the originality and quality of the proposed research, not with its potential significance to industry or other partners, or with innovation capacity building. There was often little information on file explaining the significance of the expected benefits to the partner company, or the need for government support. In our view, proper examination of all of these issues is essential to determining whether a prospective project is likely to enhance industry's capacity for innovation.

19.77 Scientific merit well evaluated. The merit of the research proposals and the quality of the researchers were well evaluated. NSERC uses a system of project review committees consisting of researchers and industry representatives. In addition, projects are submitted to external referees who provide a written evaluation of each proposal. The use of external referees greatly expands the expertise available within the panel and thus provides a vital source of information in the decision-making process.

19.78 Little information on significance of expected results. Given NSERC's criteria for funding and its statements to Parliament on the role of the program in developing partnerships that lead to new products, processes and services, we expected that funding decisions would take into account these kinds of expected results. Although the extent of this analysis would vary with the type of grant, we expected that the justification for funding a project would explain the usefulness and potential significance of the research results in pre-commercial and commercial terms.

19.79 For about a third of the projects we audited, we could not assure ourselves that NSERC had adequately considered the significance of the research for the partners. While the link between the research and the company's operations was explained, the significance or extent of the potential benefit was not evident. For these grants, there was little explanation of how the projects might lead to significant new or improved products or processes or how much the firm's knowledge and skills would be improved. For the remainder of the grants we audited, these issues were adequately considered.

19.80 NSERC has informed us that its project selection committees consider the expected benefits of the project to the partners, but this is not always documented. In our view, due diligence requires that the assessment of funding criteria be consistently documented so as to provide proper justification for decisions to spend public money and so that subsequent management review and performance measurement are possible.

19.81 Little explanation of potential socio-economic benefits of proposed projects. Beyond their impact on individual firms, research projects should have the potential to generate socio-economic benefits such as enhancing the country's economic strength and developing new industries. In general, little information was provided on the potential socio-economic benefits of the projects other than the training of qualified personnel and the creation of direct jobs. While this assessment is obviously difficult for certain types of research grants, NSERC has not developed specific indicators or other decision aids to help program officers and project selection committees to reasonably assess whether proposed projects are likely to achieve the desired results. Where on file, the descriptions of possible economic benefits were very broad in nature and not substantiated.

19.82 No evidence that the need for funding was considered. We expected that the project files would provide an explanation of why program funding was needed for the project to proceed. We found that the need for funding was not addressed in almost 90 percent of the projects.

19.83 Few agreements on intellectual property rights. One of the funding criteria for these grants is that the partners or other potential users must have the capacity to apply the research results in Canada. To avoid potential conflicts, NSERC recommends that the partner firms and the universities involved in the research proposals negotiate a research agreement before starting a research project. Usually, the university and/or the researchers employed by the university own the rights to the intellectual property arising from NSERC grants.

19.84 Intellectual property agreements would not be appropriate for all types of grants. NSERC recommends, but does not insist, that these agreements be negotiated before starting a research project. Nevertheless, we found that NSERC was often unaware of whether an intellectual property agreement had been reached. This finding points to a lack of attention to how projects will enhance Canada's innovation capacity.

Limited performance information

19.86 We noted the following limitations in measuring the results achieved by the Research Partnerships Program.

19.87 Little information on impact of projects. NSERC regularly receives progress reports and final reports from the research partners. These reports indicate whether the project's research and development goals are met; however, there is little information in the project files to indicate how the results of the research and development might be used to achieve commercial benefits such as patents, new processes or services or licences. Management informed us that in many of these cases, it is premature to assess possible benefits at the end of the project. However, in our view, project results need to at least be compared against the results expected when the project was approved and the likely pre-commercial and commercial benefits.

19.88 In the sample of 100 files we examined, 11 projects had failed for various reasons while 31 were still ongoing and the remaining 58 had more or less achieved their objectives. However, we were unable to determine from NSERC's records whether the research results were used by industry partners following the completion of the projects.

19.89 One formal evaluation done. NSERC recently evaluated the performance of one type of grant under the Research Partnerships Program - its support for strategic projects. This evaluation found that, in order for the program to remain relevant, management should choose a clear role for it. As well, funded applicants indicated that many projects would not have proceeded without program funding. The evaluation also noted that little information on the results at either the program or the project level was available to management. The contribution of program funding to industrial, economic, social and environmental impacts was difficult to isolate from other effects. The evaluation recommended that the program objectives be more clearly defined and that performance indicators be used.

19.90 A pilot study was conducted in 1998 to measure some of the outcomes of the Collaborative Research and Development grants. This was not a full evaluation, as it did not address all the required evaluation issues. The study was done through a survey and indicated that while some industrial collaborators created new processes, products, standards or services, others updated their knowledge and had access to new ideas as a result of the funded research projects. The outcomes of many other projects were still unknown at the time of the survey.

19.91 The Natural Sciences and Engineering Research Council should:

• establish clear expected results for the Research Partnerships Program and report performance against them;
• ensure that the decision to fund a project is based on an appropriate assessment of the nature and significance of the project benefits and the need for NSERC support and is appropriately documented;
• ensure that reliable information is gathered on the results of projects, including the benefits to industrial partners; and
• conduct a program evaluation of the Research Partnerships Program that addresses all of the key evaluative questions.

NSERC shares the Auditor General's goals of improving accountability and performance measurement. In that spirit, we accept the recommendations of this chapter. However, we feel obliged to raise a point about the context of the chapter. It must be remembered that university research in partnership with industry is very different from industrial research, where clearly defined commercial benefits can be identified. In the Research Partnerships Program, NSERC invites industry to share in the risk of conducting university research that is long-term and often without quantifiable expected results. The expectations of results, documentation and follow-up must take this into consideration, and they must be commensurate with the small size of the investment. For example, a typical project in the Research Partnerships Program is funded at less than $100,000/year and can be as low as $5,000, a very small fraction of the funding of a Network of Centres of Excellence. These awards go only to universities, and in response to applications that succeed in a peer-reviewed competition, where applications are assessed against criteria that include the potential for impact of the research results on Canada's economy, industry, society and/or environment. Many applications are turned down because they do not meet a high enough standard on these criteria. The chapter acknowledges that innovation, how it works, and how its performance is measured are very complex issues that are still not fully understood. Within this context, NSERC agrees that greater efforts are required to track the longer-term impacts of our investments, and therefore we are continuously improving performance measurement activities as our knowledge and understanding improve.

Networks of Centres of Excellence

19.93 In addition to our audit criteria, we assessed the NCE grants using the five selection and evaluation criteria for the program, namely: excellence of the research program, highly qualified personnel, networking and partnerships, knowledge exchange and technology exploitation, and network management.

19.94 We concluded that due diligence had been exercised in the grants we audited under the Networks of Centres of Excellence. We found that for each proposal submitted, there was a complete description of the network and its planned research activities, and the files included all relevant information. Each proposal was assessed by an expert panel and examined by the selection committee for final approval by the steering committee.

19.95 All the networks provide regular reports on the progress of the projects and site visits are done. Every agreement defines intellectual property ownership.

19.96 Finally, a 1997 evaluation concluded that the Networks of Centres of Excellence program had succeeded in all of its objectives. While the report did not consider all of the basic evaluation issues, it suggested that the program will provide substantial net economic benefits.

Technology Partnerships Canada

19.98 TPC has been directed to take an investment approach, sharing risks as well as returns, with all repayments being used to fund future investment opportunities. Repayments are normally based on royalties on sales. On average, the TPC sharing ratio is not expected to exceed 33 percent of eligible project costs. Some highly successful projects may return much more than the original investments to help offset less successful projects. TPC's annual budget is $300 million.

19.99 To be eligible for funding, projects must be in a qualifying technology sector: environmental technology, enabling technology (that is, advanced manufacturing and processing technology, advanced materials, biotechnology and selected information technology) and aerospace and defence industries. TPC has been mandated to target one third of its funding to environmental and enabling technology.

19.100 Between 1996 and the end of June 1999, TPC had made 81 contributions for a total of $827 million. We audited 30 contributions, representing $580 million and ranging from $161,000 to $100 million.

19.101 We expected TPC to ensure that the projects represented value for money. To do this, funding decisions need to be based on and respect all the assessment criteria set for the program by the Treasury Board Secretariat, and should achieve a reasonable balance in meeting all of them.

Due diligence in assessing business case issues

• the significance of the proposed innovation to the firm and to industry;
• the expected project benefits to the firm;
• the recipient's ability to carry out the project and exploit the results commercially; and
• the likelihood of success.

Support for funding decisions

• applicant eligibility;
• the need for TPC funding;
• the expected leverage of new investment; and
• the net expected economic and other benefits.

19.105 Little support for amount of contribution to three projects. The amount of TPC's contribution is supposed to be based on an assessment of the minimum funding required by the applicant to proceed with the project and on the forecasted benefits for Canada.

19.106 We found that TPC had assessed the minimum amount required by applicants for all the projects we audited, except in the cases of three large contributions. For these contributions, management claims that the justification for the amount of government funding included a challenge to the cost of the project, analysis of the project's socio-economic benefits, and historical knowledge of the industry. Based on this analysis, TPC concluded that the amount was the minimum required to make sure the project proceeded in Canada.

19.107 Nevertheless, we expected that the specific amount of the contribution would be supported by an analysis of the company's ability to carry out the project without support, and the minimum TPC investment needed to make the project successful. Because TPC had not carried out this analysis for these agreements, we are concerned that it did not assure itself that the specific amounts of these contributions were the minimum needed for the projects to proceed.

Performance monitoring and reporting

19.109 Project and results monitoring could be improved. We expected that TPC would monitor the progress of funded projects and ensure that the applicant is meeting all its obligations under the funding agreement.

19.110 TPC assures itself prior to payment of a claim that funds are being used for the purposes agreed. We noted that TPC monitored the progress of projects through some site visits and that it had conducted a few audits to date. However, TPC does not yet systematically monitor project results. For example, recipients are required to send semi-annual and annual reports. Those reports are supposed to contain information on progress and the project's contribution to jobs and economic growth. TPC does not ensure that all the reports are received on time and there is little information on actual jobs created. Although we found audited financial statements on file, there was no evidence that such information was reviewed on an ongoing basis to help assess the continued viability of the companies or the projects.

Reporting to Parliament

19.112 Leveraged funds need to be explained. TPC has reported that during the first two years of its operations, it invested $563 million of public funds that would leverage $2.4 billion of new private sector investments. For about $400 million of the new private sector investment, we found that TPC had also counted other government funding, federal and provincial research and development income tax credits, investments prior to project funding and possible future investments in related projects, as well as money spent outside the country. TPC ought to disclose the sources and uses of leveraged funds to avoid misinterpretation.

19.113 Reporting to Parliament needs clarification. We examined how the program's approach to making contributions is described in reports to Parliament. Under its terms and conditions, one of TPC's objectives is to manage its contributions by taking an investment approach through sharing in returns as well as risks.

19.114 Reports to Parliament have repeated this performance expectation. For example, Industry Canada's 1999-2000 Report to Parliament on Plans and Priorities states, "The assistance is fully repayable, with TPC sharing in both the risks and rewards of the projects with its partners." TPC's most recent annual report states that "the program supports the private sector through investment rather than subsidy, sharing in both risks and rewards."

19.115 However, we found that the reports do not explain how management interprets these aspects of TPC's mandate. In particular, there is no explanation of the basis for or extent of sharing risks and returns with firms. In practice, how TPC shares risks and returns varies significantly among agreements. Under many agreements, TPC will recover its contribution through royalties on all sales of the supported product or technology, starting with the first sale. Under other agreements, TPC assumes more risk by starting to receive royalty payments only after several years of sales or after a certain number of products are sold. For still other agreements, royalty payments are structured in such a way that TPC's contributions are unlikely to be fully repaid.

19.116 Management has indicated that specific repayment terms are negotiated on a case-by-case basis and are arrived at by assessing applicant need, firm and project risk and the magnitude of the strategic benefits that would result if the project is undertaken in Canada. Management also informed us that it must balance the objective of "taking an investment approach through sharing in returns as well as risks" with other TPC program objectives such as increasing economic growth and wealth creation.

19.117 Nevertheless, in our view, proper reporting to Parliament requires that TPC explain how it interprets and acts on key parts of its mandate. In particular, the meaning of "taking an investment approach through sharing in returns as well as risks" and the extent to which contributions are "fully repayable" need to be better explained in any report to Parliament.

19.118 Industry Canada should ensure that:

• there is appropriate justification for the specific amount of all contributions; and
• the performance of projects funded by Technology Partnerships Canada is appropriately monitored.

• explain the sources and uses of funds leveraged by its contributions; and
• provide information to clarify:
  - the extent to which TPC shares risks and returns;
  - the basis on which TPC shares risks and returns, including the main factors taken into consideration when establishing royalty payments; and
  - the extent to which contributions are "fully repayable".

Industry Canada's response: The Department notes that the Auditor General concluded that Technology Partnerships Canada exercises due diligence in assessing projects. One key aspect in this regard is the assessment of the minimum amount of assistance required to make sure that projects beneficial to Canada proceed. The information set available to make such a determination varies from project to project and consistently requires the application of considerable judgment. As explained in paragraph 19.106, management is of the opinion that it does take all reasonable measures to ensure that there is appropriate justification for the specific amount of all contributions. The few specific projects that gave rise to the Auditor General's observation do not follow the traditional nature of product development assistance. TPC, therefore, developed alternative evaluation strategies in support of their assessment. The Department believes this approach was and is appropriate for these projects.

As indicated in the chapter, TPC is a relatively new program and management has appropriately concentrated on, first, establishing rigorous due diligence processes for assessing proposed projects and, second, the effective management of the processing of claims. Management will now focus on ensuring that funded projects are appropriately monitored. To this end, all investments will be reviewed, client reporting requirements brought up to date, and a comprehensive portfolio monitoring system implemented before 31 March 2000.

In reporting leverage, TPC's intent is to communicate additional levels of innovation spending generated as a result of TPC investments. Therefore, TPC leverage data focus on the use rather than the source of spending. In the future, TPC will more fully describe the basis on which it reports leverage so as to ensure a more comprehensive understanding.

TPC's objective is to invest to maximize the number of projects at the minimum level of support required, and hence also to maximize the private sector investment leveraged. In some circumstances, this results in accepting greater repayment risk in order to ensure that projects proceed. The framework within which TPC operates is complex and requires judgment and trade-offs to arrive at the best mix of risks as well as rewards. Given the public policy mandate of the program, its risk and reward sharing is not restricted to merely the financial aspects of the investments. The Department will expand future reporting to be included in TPC's Annual Report, which is tabled in Parliament.

Conclusion

• could show that these programs are designed to improve Canada's innovation performance;
• exercised due diligence in investing in innovation projects; and
• knew if the programs were achieving value for money.

19.121 We concluded that there are significant opportunities for management to improve the exercise of due diligence in approving contributions under the Industrial Research Assistance Program (IRAP) and grants under the Research Partnerships Program.

• We found that only about 15 percent of the funding decisions under IRAP were based on an assessment of all of the project criteria, mainly because the need for IRAP support had not been considered. We also found a range of practices in how the other criteria were assessed - in particular, how projects would improve the technological capability of recipients. In our view, the assessment of individual funding criteria for many projects was not sufficiently thorough.
• For the Research Partnerships Program, we found that the scientific merit of the projects we examined had been well established, as had the potential for training. However, scientific merit deals with the originality and quality of the proposed research, not with its potential significance to industry or other partners, or with building innovation capacity. We found that there was often little information on file explaining the significance of the expected benefits to the partner company, or the need for government support.

• We noted that various efforts have been made to assess the performance of IRAP services. Nevertheless, management needs to gather reliable information on the results of funded projects and its advisory services, and to investigate the extent to which its funding supports projects that would not have proceeded otherwise. In our view, any job creation numbers reported for the program need to be carefully explained and qualified.
• We also noted that the information gathered on the results of projects funded under the Research Partnerships Program is limited to success in relation to research goals. There is little information on the success of the projects relative to the commercial and pre-commercial benefits expected when they were approved. As well, only one part of the program had been recently subjected to a program evaluation.

19.124 We also concluded that management of Technology Partnerships Canada had exercised due diligence in making the contributions that we audited, with some specific exceptions. Nevertheless, project and results monitoring and reporting to Parliament on expected performance need to be improved.

About the Audit

Objectives and Scope

Specifically, we examined whether management of the programs we audited:

• could show that these programs are designed to improve Canada's innovation performance;
• exercised due diligence in investing in business innovation; and
• knew if the programs were achieving value for money.

Approach

Criteria

• the significance of the innovation;
• the expected project benefits;
• the recipient's capability to carry out the project and exploit the results successfully; and
• the likelihood of success.

• eligibility;
• compatibility with program objectives and existing portfolio of grants or contributions;
• need for assistance/expected leverage;
• net expected economic and other benefits; and
• likelihood of repayment (TPC).

We expected that management would assure itself that the program is achieving expected results by assessing whether:

• project goals are met;
• expected project benefits are realized;
• expected net economic benefits are realized;
• conditions of funding are respected;
• repayments owed the Crown are made (TPC); and
• advice meets clients' needs (IRAP).

Audit Team

Isabelle Barolet
Louise Bertrand
Ian Campbell
Veronique Dallaire
Bryan De Pape
Geneviève Hivon
Julie Pelletier
Michelle Salvail

For information, please contact Peter Simeoni.