This Web page has been archived on the Web.

1981 Report of the Auditor General of Canada

Introduction

2.1 The federal government spends money on two kinds of science and technology activity -- Research and Development (R&D) and Related Science Activities (RSA). Research and development is defined as "creative work under- taken on a systematic basis to increase the stock of scientific knowledge and technology". Expenditures on research and development, excluding certain Crown agencies, support the effort of a reported 15,000 persons in the federal government. Forecast expenditure for 1980-81 on R&D in government laboratories was about $621 million out of a total federal government R&D forecast of $1,211 million. The remaining $590 million was spent in non-government facilities or institutions either on contract work or as grants and contributions. The majority of this amount was spent on grants and contributions.

2.2 Related Science Activities include mapping by the Geological Survey of Canada, hydrography by the Oceans Science and Surveys Division of the Department of Fisheries and Oceans, and all the work conducted by Statistics Canada in characterizing the Canadian economy and population. In 1980-81, forecast expenditures for RSA were $691 million. About 86 per cent of RSA is performed by 14,000 federal government employees; the balance is largely carried out by the private sector on a contract basis.

2.3 This report covers R&D only; our examination did not extend to Related Science Activities.

2.4 For many years, there have been discussions about a Canadian Science Policy. In 1962, the Glassco Commission reviewed federal science activities and recommended establishing mechanisms for developing a national science policy. It also recommended introducing central co-ordination processes to pursue the objectives of that policy. From 1968 to 1972, the Lamontagne Committee reviewed national science activities and the federal role in science and made several recommendations related to the central co-ordination of science and technology. A strong central co-ordination function for Research and Development does not exist in the federal government. Money is voted for programs in different departments and agencies and, in some cases, is spent on R&D as one of the activities supporting a given program. Further, these expenditures are not always specifically identified in the Estimates as being for R&D activities. Some organizations, like the National Research Council, do identify expenditures for research programs and, until this year, the Research Branch in the Department of Agriculture did so as well. Decisions establishing science priorities and resource commitments between and among departments are ultimately a Cabinet responsibility. We audited the management controls over 73 per cent of the funds allocated to R&D as a result of such decisions.

2.5 In our opinion, R&D management controls in departments and agencies to ensure due regard to economy, efficiency and effectiveness were generally satisfactory, although in some organizations we found significant deficiencies. Where we found deficiencies, department managers were working to correct the problems or indicated their intention to do so.

2.6 Published information about R&D expenditures has been inadequate. Information for Parliament on R&D expenditures, prepared annually by the Ministry of State for Science and Technology, omitted significant elements of such expenditures. The Statistics Canada publication on national R&D expenditures did not adequately describe limitations on the use and reliability of the data.

2.7 In our opinion, three organizations audited did not have proper authority to undertake contracts with non-government groups for the paid use of government R&D facilities and services.

Audit Objective and Scope

2.8 The objective of this government-wide audit of R&D was to assess and report on the adequacy of the controls used by management to ensure that R&D was being conducted or fostered throughout government with due regard for economy, efficiency and effectiveness. In doing so, we also developed methodology and criteria for auditing R&D in future departmental comprehensive audits.

2.9 We reviewed management controls in a number of areas, including:

    - those for managing R&D in government laboratories, conducted in support of department or agency missions;
    - those for managing grants and contributions in support of R&D; and
    - those for administering the Treasury Board guidelines, "Science and Technology -- Contracting-out."
2.10 In addition, we reviewed the practices and procedures related to the contracted use of government R&D facilities and services by the public. We also reviewed the reporting of R&D information to Parliament and the public.

2.11 In 1977, we reported on a government-wide audit of the administration of grants and contributions. In the present audit, we reviewed the progress made in implementing the recommendations made in 1977, in so far as they affected programs in support of R&D.

2.12 We assessed the management controls over R&D against criteria that we consider represent appropriate standards. The criteria are presented in Annexes 1 and 2 to this chapter. When using these criteria to audit the management controls in each of the organizations, we took into account a number of considerations. First, certain aspects of the R&D management process are more important in some organizations than in others. Accordingly, we determined which were most critical in a given organization, and focused our audit on these aspects. Second, we recognized that different types of research call for different types of controls; different organizations can manage in different ways and still demonstrate due regard for economy, efficiency and effectiveness. Thus, all steps in the management process may not always occur in the same sequence in all organizations, and small scale research projects may not need the same formality of control as that required for large scale projects. Furthermore, strengths in one step of the management process may compensate for weaknesses in another.

2.13 We did not assess the technical quality of research carried out or whether the work being done was the most appropriate in terms of meeting an organization's objectives. We assessed the appropriateness of management procedures designed to control research efforts and direct them to productive areas.

2.14 The problems associated with R&D personnel management (for example, lack of human resource planning) are the same as those found elsewhere in the public service and are covered by the payroll costs management segments of departmental comprehensive audits. Accordingly, we did not address these areas in this audit.

2.15 This chapter reports only on those aspects of R&D management that, in our view, are of significance to Parliament. We have discussed details concerning each audited entity with the responsible managers and have made recommendations for correcting identified weaknesses.

2.16 Exhibit 2.1 displays the 1980-81 forecast R&D expenditures of federal departments and agencies and the expenditures within our audit scope. Our audit did not cover all the R&D expenditures in every organization or every organization responsible for R&D.

(This exhibit is not available)

Summary of Audit Observations

R&D in Government Laboratories

2.17 We found the systems and procedures for managing R&D in government laboratories varied in quality and completeness. In some organizations, the systems were satisfactory, and any weaknesses were relatively minor. For example, in one organization, the process for selecting R&D projects was satisfactory, although there were no department-wide standards for guiding research establishment managers in selecting R&D projects; each had his own standards. This could lead to inappropriate allocation of funds within department-wide programs. In other organizations, some systems were satisfactory, others were not. Some of the weaknesses were significant. For example, in one organization, planning weaknesses resulted in a failure to communicate clear objectives to the research workers. In others, operational control of projects was weak, resulting in non-productive use of time and unnecessary delays. Where we found weaknesses, managers were working to correct them or indicated their intention to take corrective action.

2.18 Executives responsible for the major R&D operations in government have formed a committee to work with the Office of the Comptroller General in developing guidelines for operational control of R&D projects. We have been informed that they intend to develop guidelines on other aspects of the R&D management process, including planning and project selection. We support these efforts to develop standards of management practice for R&D in government.

Grants and Contributions

2.19 Management systems and procedures for programs involving grants and contributions were generally satisfactory, except for deficiencies in the procedures used to assess program effectiveness in some granting areas. In every case where we identified any weaknesses, management had undertaken corrective action.

2.20 In following up our 1977 recommendations, we found that they had been implemented in so far as they affected grants and contributions in support of R&D.

Administration of policies Relating to R&D

2.21 Contracting out. In our audit, we found that the Treasury Board Secretariat had not monitored the implementation of the 1977 guidelines on contracting out R&D to ensure that departments and agencies, including MOSST, were discharging their assigned responsibilities.

2.22 One stated objective of the contracting-out policy is to achieve "a more even balance between scientific activities" performed by industry and governments. The contracting-out guidelines request departments to submit annual science and technology plans for review by the Treasury Board in consultation with the Ministry of State for Science and Technology. These plans are intended to provide a basis for achieving the "more even balance" by identifying ongoing activities that could be contracted out to the private sector. In 1978, nine departments submitted plans; only one did in 1979. No science and technology plans were submitted in 1980 or 1981.

2.23 In 1972, when the government's contracting-out policies were announced, the proportion of natural science R&D contracted out to industry was about five per cent. This proportion rose to about 17 per cent by 1977 and has remained relatively unchanged since then. This levelling may be partially explained by resource constraints in the public service since 1978.

2.24 Contracting in. Several departments provide R&D services on a contract basis to the public. The services provided include the use of helicopter test facilities, NRC's large wind tunnel, and the National Defence instrumentation barge for undersea exploration. Our legal counsel advises us that two distinct authorities are required by a department or agency before it can provide and charge for services to the public. First, the responsible minister must have explicit legislative authority to provide specific services to the public. Second, Order in Council authority is required under section 13 of the Financial Administration Act before a department can specify the level of fees or charges.

2.25 Three of the five organizations we audited lacked required authority. The Departments of Energy, Mines and Resources and Fisheries and Oceans had Orders in Council but lacked explicit legislative authority. NRC has clear legislative authority but did not have the required Order in Council to set rates.

Information for Parliament

2.26 Information about federal budgetary expenditures on R&D supplied by the Ministry of State for Science and Technology (MOSST) for consideration with the Main Estimates would be more useful if it included information on the total federal involvement in R&D. In our opinion, the scope of information presented annually in Federal Science Activities, commonly called the "Red Book", was too narrow. First, certain classes of R&D expenditures, which represent about $340 million in federal R&D involvement, were not included in the Red Book. For example, there are income tax reduction programs designed to provide incentives for industry to carry out R&D work. R&D expenditures by commercially oriented Crown corporations, such as Air Canada and Canadian National Railways, were not recorded. Also omitted were administrative costs, including program management and accommodation directly attributable to R&D. Second, forecast R&D expenditures were insufficiently related to intended results or possible benefits; however, MOSST informed us that the original purpose of the Red Book had not required this. We believe it would be appropriate for MOSST and the Office of the Comptroller General to review the purpose, format and content of the Red Book as part of the program to revise the form and content of the Estimates.

2.27 The information on national R&D expenditures in Statistics Canada's Annual Review of Science Statistics was inadequately presented. There was no clear, comprehensive and consolidated statement of the limitations on the reliability and use of the information. For example, one important figure presented in the Annual Review is the GERD -- gross domestic expenditure on R&D in the natural sciences. This is determined by summing the estimated R&D expenditures by the various sectors of the economy. The uncertainty of the estimates is stated only for each sector; there is no indication of the uncertainty when all sectors are summed to give the GERD estimate. Further, the GERD has limitations when used to compare R&D performance of several countries. For example, in the Canadian economy, the non-manufacturing sectors are relatively large; these are traditionally much less R&D intensive than the manufacturing sector. Such comparison limitations are not indicated in the Annual Review.

Background to Government Involvement in R&D

2.28 In recent years, about $2.5 to $3 billion has been spent annually in Canada on all types of R&D. The federal government's involvement, both inside and outside its own laboratories, is extensive and varied. The following table displays the total 1979 expenditures on R&D in Canada by sector of performance and source of funds. The table is taken from the 1980 Annual Review of Science Statistics .

Research and Development in Canada - 1979

Source of Funds Sector of Performance
Governments Business
Enterprises
Universities Private
Non-profit
Total

(millions of dollars)

Governments

$ 784

$ 160

$ 308

$ 14

$ 1,266

Business Enterprises

8

886

2

896

Universities

543

543

Private Non-profit

34

9

43

Foreign

1

60

6

67

$ 793

$ 1,106

$ 893

$ 23

$ 2,815

R&D in Government Departments

2.29 To give some idea of the breadth and diversity of R&D conducted in government departments, the following examples are extracted from the 1981-82 edition of the Red Book.

2.30 Energy. A recently stated government objective is the attainment of Canadian self-sufficiency in energy, specifically liquid fuels, by 1990. The role of energy-related R&D is to explore alternative means of meeting this objective, such as finding substitutes for oil, using energy more efficiently and developing new energy sources. A panel for co-ordinating government R&D in energy has been set up in the Department of Energy, Mines and Resources. Other organizations carrying out energy-related R&D include the Departments of Agriculture, Environment, Fisheries and Oceans, Public Works, and Transport, and the National Research Council.

2.31 Food. The term "food science" encompasses scientific and technical activities concerned with food production, processing, distribution and retailing. Food science also includes the nutrition, safety and quality of food. The major departments active in this area are Agriculture and Fisheries and Oceans. The Department of National Health and Welfare is active in food-related health protection regulatory activities.

2.32 Natural resources. Mineral, forestry and water resources are included under this category. The Department of Energy, Mines and Resources has the major responsibility for mineral resources. Most of the research is done at the Canada Centre for Mineral and Energy Technology (CANMET) which performs, contracts for and co-ordinates research on mining, extracting and utilizing minerals and metals. The Department of the Environment is responsible for R&D in forestry and water resources, with much of the research directed toward ensuring the long-term supply and quality of these resources.

2.33 Oceans. Canada has a longer coastline than any other country in the world, and oceans-related R&D in this country has been directed toward gaining more scientific knowledge of the oceans and how the marine environment relates to human activities.

2.34 The Department of Fisheries and Oceans spends about half of the funds that the government allocates to all oceans-related R&D. It also contracts out research on the properties and movement of sea ice to determine ways to make it possible for tankers and liquified gas carriers to travel throughout the year to and from the Arctic. The Department of Transport carries out research in this area as well. Planning for the National Research Council's Arctic Vessel and Marine Research Institute is in its final stages. The Departments of Fisheries and Oceans, Communications and Energy, Mines and Resources are collaborating on remote sensing of ocean properties via satellite. Other departments, including National Defence, are also involved in various aspects of oceans R&D. Because so many departments participate, various co-ordinating mechanisms have been established, such as the Panel on Oceans Management, chaired by the Deputy Minister of Fisheries and Oceans.

2.35 Contracting in. The government's contracting-in practice allows private industry to use government facilities and laboratories to do testing and other development work. An example is the private sector's use of a National Research Council wind tunnel, the largest and most sophisticated in Canada.

R&D in Other Sectors

2.36 Over the years, the Government has made declarations of policy in certain areas involving R&D. One recurring theme has been to increase R&D capacity in the private sector, universities and other organizations. Several methods are used to accomplish this objective, including:

    - grants and contributions in direct support of industrial and university R&D;
    - contracting-out of science and technology requirements;
    - various tax incentives; and
    - the use of government procurement to further national R&D objectives.
2.37 R&D in industry. The total expenditures on R&D in the business enterprise sector were estimated to be $1,106 million in 1979. In 1980-81, federal expenditures on industrial R&D were forecast at over $229 million in two categories of expenditure: contracted research and development ($100 million) and grants and contributions designed to foster industrial R&D ($129 million).

2.38 The government's contracting-out policy is designed to increase the amount of R&D performed by private industry and to foster the industrial capacity to carry it out. To encourage contracting out, the government assists departments in financing unsolicited mission-related R&D proposals, if they have scientific merit, are sufficiently unique and contribute to the attainment of a departmental mission. In 1980-81, this fund had about $15 million available to it.

2.39 The government's largest industrial R&D expenditure is made through grants and contributions programs. For example, in 1980-81, the Enterprise Development and Defence Industry Productivity programs of the Department of Industry, Trade and Commerce contributed almost $90 million toward fostering industrial R&D. In addition, the National Research Council had authorized expenditures of $21 million in 1980-81 for its Industrial Research Assistance Program (IRAP).

2.40 A number of income tax incentive programs have been created to foster industrial and other R&D activity. The Department of Finance estimates the annual income tax revenue forgone at approximately $142 million. In addition, there are remissions of customs duties payable on importing scientific equipment destined for R&D applications.

2.41 Finally, the government is implementing procurement practices of buying Canadian goods and services to achieve certain socio-economic objectives, such as fostering industrial R&D activity in Canada.

2.42 R&D in universities. Universities play an important role in providing trained manpower and in carrying out basic research in Canada. In 1980-81, the federal government gave grants, mostly to universities, amounting to $256 million in support of R&D. In addition, it spent $17 million for R&D contracts and $12 million for scholarships and fellowships. The Medical Research Council, the Social Sciences and Humanities Research Council, and the Natural Sciences and Engineering Research Council distributed most of these funds. Several departments also gave various universities small amounts to support R&D work. For example, in 1980-81, the Department of Agriculture had a grants and contributions program of $1.3 million to foster agricultural R&D and the development of agricultural scientists.

2.43 R&D in Crown corporations. Expenditures under this category include those made by Crown corporations such as Atomic Energy of Canada Limited (AECL) and Air Canada. The amounts involved are large. For example, AECL had annual R&D expenditures in its own laboratories of $72 million, and it contracted out a further $11 million to other agencies. AECL also had contracts from the federal and provincial governments and private enterprise to carry out about $17 million of R&D work.

2.44 Other R&D funding. The provinces and various non-profit organizations received more than $84 million from the federal government in 1980-81 for research and development activities. The International Development Research Centre and the Canadian International Development Agency spend about $40 million each year in support of R&D in developing countries.

The Role of Central Agencies

2.45 The following central agencies have major responsibilities in connection with R&D:

    - Ministry of State for Science and Technology;
    - Treasury Board, including the Treasury Board Secretariat and the Office of the Comptroller General; and
    - Department of Supply and Services.
2.46 MOSST acts as a policy and program adviser to government on science and technology and is also responsible for compiling and publishing information on R&D expenditures in two books. The first, Federal Science Activities, (Red Book), is a narrative description of many of the science activities carried out by government departments and agencies. The Red Book is a major source of information on R&D for Members of Parliament in their consideration of the Estimates. The second publication, Federal Science Expenditures and Personnel (Grey Book), is more statistically oriented. Grey Book data can be used to determine whether R&D money is spent on research inside or outside government.

2.47 Statistics Canada publishes national expenditure statistics in the Annual Review of Science Statistics . One important figure in the Annual Review is the GERD, which estimates the gross expenditure on R&D in the natural sciences in Canada. The GERD is looked at in relation to the GNP to compare Canada's R&D activities with those of other nations. The government has also designated the GERD as the indicator for a national target of increased expenditure on R&D in the natural sciences.

2.48 The Treasury Board Secretariat is responsible for promulgating administrative policies for R&D. The Office of the Comptroller General carries out functions that are related to those of the Secretariat and is responsible for advising departments and agencies on management controls and on accounting and other systems used to record and control revenues and expenditures.

2.49 The Department of Supply and Services acts as the government's agent for contracting out a large part of R&D work to outside agencies, including the private sector and universities. DSS also acquires the goods and services needed to carry out R&D in government laboratories and is responsible for administering the Unsolicited R&D Proposal program.

The R&D Process and its Management

2.50 A key activity or phase of R&D work is idea-generation -- creative thinking. Idea generation involves linking a perceived need (new knowledge, new or improved processes or products) with the capacity to meet the need through scientific and technical research activities.

2.51 Studies on R&D show that ideas are plentiful; but focusing R&D efforts on those few that are worth examining extensively is critical if research efforts are to be economic and effective. In other words, good research and development concentrates on important problems that can be solved and solves them at reasonable cost. Yet there is a practical difficulty in maintaining this focus: the importance of a problem, whether it can be solved, and how much it will cost cannot always be determined before the research work begins. Although experienced R&D personnel can often intuitively decide a problem's importance and whether a solution is possible, technical, economic and market studies can also be very important. The primary purpose of R&D management procedures is to focus research effort to produce solutions to important problems.

2.52 Five features distinguishing R&D from other processes are set out below and an effective R&D management process would be designed around them. A description of an R&D management process that does this is given in Annex 1.

2.53 Source of ideas and use of outputs. Ideas for sound R&D work can come from many sources, and the results may often be unpredictable. Moreover, unexpected applications for R&D results frequently arise. Managing the interaction between the research worker and the client for research poses a challenge for managers. The client must be kept up to date on what is happening in the laboratory, and the researcher must be kept informed of what output will satisfy the client.

2.54 Risk. In both basic and applied research, there is always the possibility that the suggested approach will not be feasible and the work will lead to a dead end. In development, there is a possibility that the market will not accept the product or that manufacturing costs will be too high. Being aware of such risk is a major challenge for R&D managers. Therefore, good procedures for selecting and reviewing projects are needed to reduce the risks of failure.

2.55 Organization of R&D work. Usually, R&D work is divided into self- contained projects which are often interrelated. The results of one project could serve to identify areas needing further R&D activity. Some projects can continue indefinitely because of the need to maintain scientific expertise in a given area. Therefore, a procedure for managing groups of projects is needed.

2.56 Uncertainty of project duration. The length of time needed to produce research results is often unpredictable and can be quite short or very long. Moreover, the route between research results and successful application can be long, expensive and uncertain. Accordingly, management procedures must provide "controlled flexibility."

2.57 Nature of R&D organizations. A productive R&D organization is built and maintained on the creative capacity, knowledge and skills of scientists, engineers and technologists. A critical function of research managers is to focus the creative energies of research workers. Further, retaining people with these skills requires long-term funding commitments to maintain both the momentum of on-going research and the morale and creativity of the research staff.

R&D in Government Laboratories

Organizations Subject to Audit

2.58 We audited the systems for managing laboratory R&D in the Departments of Agriculture; Energy, Mines and Resources; Fisheries and Oceans; and National Defence; and in the National Research Council. These entities are responsible for about 75 per cent of federal government expenditures on R&D conducted in government laboratories.

2.59 Agriculture. The objectives of the Research Branch of the Department of Agriculture are to develop new knowledge and improve technology that will ensure the efficient production of an adequate supply of food and agricultural products and assist in maintaining a stable and profitable agricultural industry. Expenditures on R&D in 1980-81 were estimated at $141 million, with about 3,700 authorized person-years. The Department has its headquarters in Ottawa and operates 47 research establishments and institutes across the country.

2.60 Energy, Mines and Resources. We reviewed two organizations carrying out research in the Department -- the Canada Centre for Minerals and Energy Technology (CANMET) and the Canada Centre for Remote Sensing (CCRS). Research carried out in CANMET relates to energy and mineral resources. The energy-related component principally addresses recovery, processing and effective use of fossil and nuclear energy; mineral research is concerned with resource determination and exploitation. CANMET had 687 authorized person-years in 1980-81 and a budget of $29.5 million. In CCRS, we audited selected aspects of R&D management. CCRS had a budget of $13 million and 106 authorized person-years in 1980-81. Its role is to further the development of remote sensing technology and related data analysis systems for use in resource and environmental management. For the sensing, it employs satellites and specially equipped aircraft.

2.61 Fisheries and Oceans. In this Department there is one research program for fisheries and another for oceans. Fisheries research assesses fish stocks and promotes more and better quality fish, fish habitat and nutrition. Oceans research works toward understanding precautions and remedies required to restore and protect the aquatic and marine environment. The total R&D expenditure forecast for 1980-81 was $58 million. There are 13 research establishments across the country, employing about 1,500 people. The comprehensive audit of the Department is reported in Chapter 8.

2.62 National Defence. The Department of National Defence spends about $102 million on R&D annually. We reviewed the operations of the Research and Development Branch, which spends $97 million and employs about 1, and civilian personnel who carry out roughly equal amounts of two kinds of R&D activities associated with:

    - acquiring equipment, called technology application; and
    - maintaining and developing Canada's capability in defence technology, called technology base.
2.63 The Research Branch is organized with a headquarters in Ottawa and six Defence Research Establishments located in Dartmouth, Valcartier, Ottawa, Toronto, Suffield, and Victoria that are involved largely in the technology base activities; most of the technology application work is contracted out.

2.64 National Research Council. NRC is a departmental Crown corporation with a mandate to undertake, assist or promote scientific and industrial research. In its Research Program, NRC has eight Ottawa-based laboratory divisions and three regional laboratories as well as a number of national research and test facilities. It has a total budget of $246 million, of which $192 million is designated as R&D. Mere are 3,125 employees working in a range of research activities, from work in the Hertzberg Institute of Astrophysics to construction engineering in the Division of Building Research. The comprehensive audit of NRC is reported in Chapter 12.

Audit Criteria and Scope

2.65 The criteria against which we assessed departmental procedures are given in Annex 1. The criteria were developed by this Office and are based on experience in auditing R&D organizations in government, on an extensive review of the literature on managing R&D, and in consultation with experienced R&D managers. They were discussed with, and accepted as valid by, managers of R&D in government laboratories. They were also discussed with the committee of senior executives responsible for major R&D operations in government.

2.66 We identified five steps or aspects of the R&D management process that we considered the most important to successful management of most R&D organizations. This chapter reports on only these five; Annex 1 sets out criteria for the whole process. The five critical aspects are:

    - planning, including stating objectives;
    - providing for client-researcher interaction;
    - selecting projects;
    - operational control of projects; and
    - evaluation.
2.67 In interpreting our observations, it should be noted that we did not assess the technical quality of the research carried out or whether the work being done was the most appropriate in terms of meeting established objectives. We assessed the appropriateness of management procedures designed to control research efforts and direct them to productive areas.

Observations

2.68 In this section, we report our observations for the five organizations audited. First, we present our observations on each of the critical management aspects of the four departmental R&D organizations. Because of the unique nature of NRC's operations, we have reported our findings on it separately at the end of the section.

2.69 Planning. This aspect of the management process is designed to link the R&D mandate, assessments of needs or opportunities, organizational capacity and resources. It consists of strategic planning, operational planning and project planning. Our criteria call for this process to be formalized in large organizations and appropriately related to other aspects of the management process. In small organizations, the function should be identifiable but may be less formal.

2.70 In the Department of Agriculture, CANMET and CCRS of the Department of Energy Mines and Resources, and the Department of National Defence, we found that the strategic planning process was satisfactory. In the Department of Fisheries and Oceans, a strategic planning process had been under development for about a year, and progress has been satisfactory.

2.71 In our opinion, operational planning was satisfactory in the Departments of Agriculture and National Defence, as well as in CANMET and CCRS of the Department of Energy, Mines and Resources. In the Department of Fisheries and Oceans, an operational planning system was being developed.

2.72 Stating objectives. One purpose of a planning process is to ensure that statements of objectives are clear enough to guide lower levels of management in specifying such things as:

    - the clients or users of R&D outputs;
    - how to select projects; and
    - how to evaluate progress and results.
2.73 We found that this was done satisfactorily in the Departments of Agriculture, Fisheries and Oceans, National Defence and in CANMET and CCRS.

2.74 Client-researcher interaction. The generation of good proposals for scrutiny by the selection process is more likely if an appropriate relationship is established among the client for research service, research managers and the research worker. The appropriate kind of interaction will vary among R&D organizations. Our criteria require that there be provision for interaction at all stages of the R&D process. This usually requires communication between the client and the research organization during strategic and operational planning, idea-generation, project selection, and project and program evaluation, as well as during technology or results transfer.

2.75 Procedures for managing client-researcher interaction were well-defined and operating satisfactorily in the Departments of Agriculture, Fisheries and Oceans, and National Defence and in CANMET; we did not review this aspect in CCRS.

2.76 Selecting projects. Project selection is the process by which strategic and operational plans are translated into decisions on specific research activities. The selection process should ensure that the projects chosen are those considered - most likely to further organizational objectives, given existing organizational capacities. Selection represents approval in principle for a project. Our criteria state that responsibility for operating the selection process should be clearly assigned, and selection criteria and procedures should be documented, where appropriate.

2.77 The process in the Department of Agriculture was generally satisfactory, but there were no departmental guidelines specifying minimum standards for preparing proposals and selecting projects. The Department informed us that it was improving its project management system and that department-wide standards would be developed as part of this work.

2.78 In our opinion, the project selection process was satisfactory in both CANMET and CCRS in the Department of Energy, Mines and Resources.

2.79 In the Department of Fisheries and Oceans, there were satisfactory procedures for controlling the established research programs that form the bulk of the Department's R&D activities. There were, however, no department-wide guidelines or procedures for reviewing or selecting project proposals in new areas of research or for allocating resources to these areas. Mechanisms were not used for ensuring that the various research establishments were selecting such projects on a consistent basis and in a manner that would further the Department's long-term goals as established by senior management.

2.80 The systems and procedures used in the research branch of the Department of National Defence to select and manage both technology application and technology base projects were satisfactory.

2.81 Operational control of projects. Operational control is a process of monitoring and taking corrective action to ensure that resources are used economically and efficiently in pursuit of research objectives. It includes:

    - preparing a detailed project plan for proposals approved in principle;
    - granting final approval based on the detailed plan;
    - implementing the plan with appropriate project monitoring; and
    - taking corrective action as necessary.
2.82 There should be monitoring of scientific and technical aspects as well as the budget and time schedule. Operational control ensures that research work continues to focus on initially approved goals, but it should also allow the direction of research and development efforts to be altered to respond to scientific discoveries and changing priorities. At the time of this audit, guidelines for operational control were being developed by a committee of senior officials from the public service science community in co-operation with personnel from the Office of the Comptroller General.

2.83 Operational control of projects was generally satisfactory in the Department of Agriculture. In the Department of Fisheries and Oceans, project controls were informal, except for a few of the large and complex projects. We found that departmental requirements did not call for detailed project plans, and several large projects had no such plans. In the Department of National Defence, operational control was generally satisfactory.

2.84 In CANMET, managers had a project control system, but it was inadequate for complete control of their projects. Specifically, project plans were not detailed enough to provide a satisfactory basis for controlling either progress or budget. The Department was developing methods for providing managers with the information they need for monitoring and controlling their project costs. CANMET management stated that this system will be used after it has been tested. In CCRS, operational control of projects was satisfactory.

2.85 Evaluation. Traditionally, evaluation in R&D occurs in different ways at a number of stages in the R&D process -- in strategic planning, when a proposal is selected, as research is being carried out, when a project is reviewed for renewal and when a project or program is finished. Under this approach, outside experts are frequently involved, but program managers have the primary responsibility for evaluation. Recently, the Office of the Comptroller General issued the Guide on the Program Evaluation Function which supports the policy that departments will periodically review their programs to evaluate their effectiveness in meeting objectives and the efficiency with which they are being administered. One important feature of the program evaluation function is that it be independent of program management. All the implications for the traditional approach to evaluating R&D that arise from the new policy have not yet been clarified. In our opinion, it would be desirable to develop guidelines specifically for evaluating R&D.

2.86 The Research Branch of the Department of Agriculture has been satisfactorily evaluating its R&D activities in the traditional manner for a number of years. We noted that this work had not been fully co-ordinated with departmental efforts to establish the independent program evaluation function. We have been informed by the Department that steps are under way to co-ordinate the two approaches.

2.87 The Department of Fisheries and Oceans has also been evaluating R&D satisfactorily in the traditional way for a number of years. One feature of its process is an annual review and evaluation of each project by program management. Outside experts often assist management in assessing the scientific merit of the research in these annual reviews. We noted that these evaluation efforts were not yet fully co-ordinated with the departmental program evaluation function.

2.88 The Research Branch of the Department of National Defence was also satisfactorily evaluating its research activities in the traditional way. As we completed our audit, an evaluation assessment of the departmental research function was beginning. This assessment is the first step in conducting a program evaluation and deals with the feasibility of doing such an evaluation.

2.89 The Program Evaluation Branch in the Department of Energy, Mines and Resources has been carrying out evaluations of CANMET research programs at three to five-year intervals. Minerals and Energy research programs were evaluated in 1978 and 1980 respectively. At the time of our audit, the Branch was carrying out an evaluation assessment of CCRS. Except for contracted-out work, there were no formal systems within CANMET for evaluating projects or parts of projects that had been completed. For the contracted-out work, DSS, as the government's contracting agency, requires an evaluation of the contractor and CANMET requires an evaluation of the contract results. Project evaluation systems within CCRS were found to be satisfactory.

2.90 National Research Council. The observations set out below should be considered in conjunction with the NRC comprehensive audit report in Chapter 12. NRC is the only audited organization that is not carrying out mission-oriented R&D in support of a department of government. Its role is broader. NRC is also much more heterogeneous than other organizations because of the activities it carries out. About 26 per cent of the budget is spent on a broad range of non-targeted basic research in such widely varying areas as biology and astrophysics; other organizations are usually involved in only one or two areas of targeted basic research. Another 37 per cent is spent on support of Canadian industry, including manufacturing, electronics, construction, mining and so on. Accordingly, NRC's R&D activities, taken as a whole, are more complex than those of other audited organizations.

2.91 In late 1980, NRC produced a five-year corporate plan entitled The Urgent Investment . Notwithstanding this recent effort, planning at NRC needed improvement. For example, the statements of objectives at NRC did not appear to have been translated into operational terms, and priorities had not been defined in all units. However, some units, such as the Solar Energy and Computer Technology Research Programs, did have appropriate operational planning. Only some units, for example, the Division of Building Research, managed client-research interaction appropriately. NRC had a satisfactory procedure for selecting projects for which new or additional resource commitments were required; however, in research areas to which resources had been committed in previous years, the method that some units were using for selecting projects was inadequate. Only some units had satisfactory operational control over projects. NRC has assessed program effectiveness in basic research using a peer review of individual research efforts. However, many units, in our judgement, did not have adequate procedures for assessing program effectiveness in the area of applied research.

2.92 During our audit, we noted that managers at NRC were addressing many of the weaknesses in procedures. These activities are described in Chapter 12.

Grants and Contributed in Support of R&D

Introduction

2.93 A recurring theme of government policy has been to increase the R&D capacity of universities and industry. One approach is to make payments to individuals and institutions in these sectors. Many programs have therefore been set up to administer grants and contributions to do this. An R&D grant is a payment made for a specified purpose but without an audit requirement. A contribution is a payment made for a specific purpose but subject to audit to ensure that funds have been used in accordance with contribution arrangements arrived at between the government and recipient.

Organizations Subject to Audit

2.94 We examined the management controls over approximately $275 million in grants programs aimed largely at supporting university research. These programs are administered by three granting councils, the Medical Research Council, the Social Sciences and Humanities Research Council, and the Natural Sciences and Engineering Research Council. We also audited two National Research Council programs that contributed approximately $22 million in 1980-81 to support industrial research and development.

2.95 We reviewed the internal audit of the grants and contributions programs in the Department of Energy, Mines and Resources. In 1980-81, expenditures for R&D projects under these programs were estimated at $48 million. In the Department of Industry, Trade and Commerce, we reviewed an internal audit that the Department was conducting of its Enterprise Development Program. Expenditures under this contribution program were authorized at $66 million.

2.96 We estimate that the programs we audited and those covered by departmental auditors account for about three-quarters of all federal expenditures on grants and contributions in support of research and development.

2.97 Medical Research Council. The objective of this Council is to help attain the quality and scale of research in the health sciences essential to main-training and improving health services. In 1980-81, the Council's authorized expenditures in the Main Estimates for its Grants and Scholarship Program were $78 million, with a total of 39 authorized person-years to administer the Program. The Council provides grants in the health sciences to help meet operating and equipment requirements for research projects and to provide direct support for investigators and research trainees. Grants are made to help carry out research on topics of national concern as well as to promote research and the provision of research facilities in regions in Canada where these have not been adequately developed. Scientific merit as judged by a peer review process is the major criterion on which approval of such grants is based. Grants also support symposia and international scientific activities and exchange programs.

2.98 Social Sciences and Humanities Research Council. This Council began operation on 1 April 1978, taking over programs previously administered by the Humanities and Social Sciences Division of the Canada Council. The objective of the Council is to promote and assist research and scholarship in the social sciences and humanities. The Council operates over 60 granting programs, with most funds awarded on the basis of excellence as established by peer assessment. The programs provide fellowships for research training and grants supporting independent research, international scholarly exchanges, research resources, publishing and conferences. In 1980-81 the Grants and Scholarships Program budget provided 105 authorized person-years and $38 million, $25 million of which was for R&D.

2.99 Natural Sciences and Engineering Research Council. This Council took over the Grants and Scholarships program of the National Research Council in 1978. Its objective is to promote and support research in the natural sciences and engineering and to ensure a supply of highly qualified people in these areas. In 1980-81 the Council was authorized to have 75 person-years and expenditures of $163 million. Almost 98 per cent, or $159 million, was paid out to university researchers and students in the form of research grants and scholarships. Of this amount, $145 million was designated for R&D. The Council operates about 30 different granting programs, with most funds awarded on the basis of excellence of the proposed research or of the researcher. The merit is judged by specialists in the same area from university, industry and government.

2.100 National Research Council. NRC has two contributions programs that directly support industrial innovation and development. In 1980-81, the Industrial Research Assistance Program (IRAP) had expenditures of $21 million; the Science and Engineering Student Program (SESP) had expenditures of $1 million. IRAP partially funds industrial R&D projects conducted by industry. SESP funds the hiring by small firms of students to work on development projects.

Audit Criteria and Scope

2.101 The audit criteria against which we assessed the management procedures in the grants and contributions programs are given in Annex 2. These criteria were developed by this Office and are based on previous experience in auditing grants and contribution programs. They were discussed and agreed with program managers during the audits.

2.102 Our audit addressed what we consider to be the four most important aspects of the management of grants and contributions programs:

    - stating objectives;
    - reviewing and approving applications;
    - monitoring funded proposals; and
    - assessing program effectiveness.

Observations

2.103 The management systems and procedures were generally satisfactory in all the programs audited. In particular, the systems in place for planning grants and contributions programs, choosing research proposals for funding, and monitoring approved proposals were satisfactory. However, in a number of the program areas, procedures for assessing effectiveness were either unsatisfactory or absent.

2.104 Stating objectives. Departments and agencies have considerable latitude in interpreting their legislated mandates as they apply to supporting research and development, whether the R&D activity occurs in the university or the industrial sector. This latitude extends to establishing both the target groups for support and the guidelines covering who is eligible for funding.

2.105 We observed that management had translated the broad statements of roles or program objectives into terms specific enough to provide direction to those managing grants and contributions programs.

2.106 Reviewing and approving applications. Developing procedures for reviewing applications and approving those that will result in the best proposals being chosen for financial support is critical to managing grants and contributions. In our view, program managers had taken care to ensure that this function was carried out properly. Although we found some instances of weak administrative procedures supporting the decision-making process, these were not significant, and the organizations involved were taking steps to improve them.

2.107 Monitoring funded proposals. Systems for monitoring grants and contributions after they had been disbursed varied in quality. For grants administered by the granting councils, systems for monitoring the progress of funded proposals and expenditures went beyond central agency requirements. Even though grants are not subject to audit, program managers had developed procedures that generally allowed them to monitor funded proposals and determine whether recipients were using funds for intended purposes. These procedures were particularly effective in the cases of multi-year grants which have to be reviewed annually.

2.108 In the case of contributions, appropriate procedures for monitoring approved proposals were in place, but were not always followed.

2.109 Assessing program effectiveness. The most general weakness that we found was in the assessment of program results achieved through grants and contributions programs. Either little attempt had been made to measure program results, or the measurement methods used were deficient.

2.110 The Social Sciences and Humanities Research Council had satisfactory procedures for assessing program effectiveness.

2.111 The Medical Research Council and the Natural Sciences and Engineering Research Council did not attempt systematically to assess the effectiveness of their various programs, but both Councils had measured the effects of some programs. It is important to recognize that the absence of systematic assessment procedures does not mean that managers were unconcerned with issues of program effectiveness. However, in our opinion, both these Councils needed to conduct assessments designed to determine those aspects of their programs for which cost-justified and satisfactory effectiveness measurements could be made.

2.112 The National Research Council has attempted to assess the economic benefits resulting from the IRAP and SESP contributions programs. However, our examination revealed a number of deficiencies in evaluation methodology. For a discussion of these, see Chapter 12.

Reliance on Other Auditors

2.113 In the Department of Industry, Trade and Commerce, an internal audit of the Enterprise Development Program was under way at the time of our audit. We shall monitor developments resulting from this work as part of the comprehensive audit of the Department.

2.114 During our audit, the internal audit group of the Department of Energy, Mines and Resources was starting to examine the grants and contributions made in 1979-80, excluding those made under the Oil Import Compensation Program. Their audit did not include new programs launched since that year. We reviewed the terms of reference, the internal audit plans and criteria and suggested some changes that were accepted. When the audit had been completed, we reviewed the working papers and reports and concluded that the findings of the internal audit were appropriate. They indicated that management and financial control over the grants and contributions covered by the audit were generally satisfactory. The internal audit team noted some minor deficiencies and recommended corrective action.

2.115 In this case, we relied on the findings of the internal audit group without further audit work on our part.

Follow-up of Previous Audit Work

2.116 We restricted our follow-up of the government-wide issues raised in our 1977 grants and contributions audit to those that apply to R&D. Our review indicated that central agency directives and guidelines issued since our 1977 audit had adequately addressed the matters raised, and the organizations that we audited were generally complying with them.

Administrative Processes Relating to R&D

2.117 The Treasury Board and the Ministry of State for Science and Technology are responsible for the policy, "Science and Technology -- Contracting Out" and for collecting and presenting R&D information. In addition, the Treasury Board and several departments have joint responsibility for administering the contracting-in practices.

Contracting Out

2.118 In 1972, the Government announced a "make or buy" policy for R&D. At that time, the proportion of R&D activity contracted out to Canadian industry was about five per cent. Departments were expected to make an explicit decision whether to "make or buy" in meeting any new natural science R&D requirements. The immediate purpose of the policy was to increase the proportion of mission-related R&D work done by Canadian industry. The resulting increase in industrial R&D capability was expected to improve the innovative capacity of the private sector.

2.119 The policy was strengthened in 1974 with the "Unsolicited Proposal" program designed to encourage industry-initiated, mission-related R&D. A fund was set up to help departments finance those unsolicited proposals that appeared likely to contribute to the economic attainment of a departmental mission.

2.120 In 1975, a major review of the make or buy policy was conducted by MOSST, and policy changes were introduced over the next two years. These culminated in the Treasury Board guidelines, "Science and Technology -- Contracting Out" that took effect from 1 April 1977. By this time, the proportion of R&D activities contracted out had grown to about 17 per cent. Our review of R&D information published by MOSST indicated that there has been no significant change in this figure since 1977. This levelling may be partially explained by the resource constraints in the public service since 1978.

2.121 The new guidelines increased the scope of the policy in two major ways. First, related scientific activities and R&D in some fields of human science were to be contracted out, in addition to R&D in the natural sciences. Second, all science and technology activities were to be considered eligible for contracting out, not just new activities. Included in the guidelines were criteria to be used in judging whether an existing science and technology activity was to be retained in-house or contracted out.

2.122 One stated objective of the 1977 guidelines was to increase the share of mission-oriented science and technology activities carried out in the private sector. Specifically, the objective was" to achieve a more even balance between scientific activities carried out by industry and governments."

2.123 Audit scope. We reviewed the administration of the 1977 guidelines on contracting out in the Departments of Agriculture; Energy, Mines and Resources; Fisheries and Oceans; National Defence; and the National Research Council. We also reviewed the roles of the Treasury Board Secretariat and the Ministry of State for Science and Technology in implementing and monitoring the contracting-out policy and the mechanisms for updating and revising the policy and related guidelines. In addition, we reviewed the DSS Science Centre which is responsible for various aspects of contract administration.

2.124 The standards against which we assessed the administration of the policy are contained in the Treasury Board Guidelines on Contracting-out. As an example, the guidelines state that "each department is responsible for ensuring that the contracting-out policy is implemented as effectively as possible...". MOSST "is responsible for reporting to Cabinet on the implementation of the policy, the evaluation thereof, and recommendations for changes therein." The first evaluation was to begin in May 1979. Further, MOSST ""will report annually to the Treasury Board the effects of contracting-out policy on the distribution of activity between government and industry, and on intramural research and development competence and manpower."

2.125 In addition, we included a general criterion that central agencies that issue policies and guidelines should monitor their implementation. In general, they should also evaluate their effects and revise such policies and guidelines, as necessary. Even in cases like contracting out where these latter responsibilities are delegated, the Treasury Board, in our view, retains a responsibility to monitor to ensure that these delegated responsibilities are carried out. The Treasury Board Secretariat needs to have an appropriate process for doing this, although departmental internal audit groups are responsible for assessing compliance with policy directives.

2.126 Observations. The contracting-out procedures used in the DSS Science Centre were satisfactory, and the organizations we audited complied with the guidelines to a limited extent. The contracting-out policy guidelines call for departments to submit science and technology plans annually as part of the budgetary process. In 1978, nine departments submitted plans but, in 1979, only one did. No science and technology plans have been submitted since, and there have been no further requests for them. In 1979, MOSST reported to Treasury Board Secretariat that departmental science and technology plans had been "inadequate" and that, in their plans, departments had not provided fully for contracting out. MOSST made several proposals to encourage departments to follow the contracting-out policy but, although some negotiations took place among MOSST, Treasury Board Secretariat and the departments, no further action resulted. This inaction could be partially attributed to changes in the budgeting process with the introduction of the expenditure management system in 1979. MOSST did not make the annual reports on the effects of the policy to the Treasury Board. However, it should be noted that the absence of departmental science and technology plans undermined the basis of such reports. An evaluation of the policy was begun in September 1980 although planning for it had begun much earlier. In March 1981, the Treasury Board Secretariat followed up by asking MOSST for the status of the annual reports and of the evaluation. A draft evaluation was given to the Treasury Board Secretariat.

2.127 MOSST and Treasury Board Secretariat should review the contracting-out policy and guidelines and make revisions as needed.

Contracting In

2.128 Contracting in is the process whereby outside organizations gain access to government R&D services and laboratory facilities. The operating departments establish and collect fees for providing such services. The annual revenue generated amounts to about $10 million.

2.129 In the early 1970s, Treasury Board recommended to the Administrator in Council that the ministers of several departments be authorized to prescribe the fees to be paid for scientific services provided by the departments to the public. Departments interpreted the related Orders in Council as the required authority to provide and charge for services.

2.130 Our legal counsel advises us that two distinct authorities are required by a department or agency before it can charge for services it provides to the public. First, the responsible minister must have explicit authority to provide specific services to the public. One purpose of this requirement could be to ensure that departments do not engage in unauthorized competition with the private sector. Second, Order in Council authority is required under section 13 of the Financial Administration Act before a department can specify the level of fees or charges. One purpose of this is to ensure that unauthorized subsidies are not provided to individuals or groups. Chapter 10 of the Guide on Financial Administration sets out the policy and guidelines to be followed by the Treasury Board Secretariat and departments in establishing rate structures for contracting in.

2.131 Audit scope. We reviewed the contracting-in process in the Departments of Agriculture; Energy, Mines and Resources; Fisheries and Oceans; National Defence; and the National Research Council to determine whether proper authority had been obtained and whether contracting in was being conducted with due regard for economy, efficiency and effectiveness. We also reviewed the role of Treasury Board Secretariat in the contracting-in process.

2.132 Observations. In our opinion, three of the five organizations we audited did not have proper authority to charge for services. The Department of Agriculture informed us that it does not contract in. The Department of National Defence had the required legislative authority in Vote 1 of its appropriation act.

2.133 The Department of Fisheries and Oceans has legislation enabling it to carry out many activities related to fisheries, recreational harbours, hydrography and marine services. This legislation enables the Department to establish research facilities, conduct research and provide for the use of some of these facilities by non-government groups. However, the Department had provided some facilities and charged for their use without explicit legislative authority, although it had Order in Council authority to set rates. The control over some of the revenues thus generated is discussed in Chapter 8, paragraph 8.157 of this Report.

2.134 The Department of Energy, Mines and Resources has explicit legislative authority to sell publications, maps and other documents issued by the Department. However, it did not have explicit legislative authority to provide R&D services to the public, although Order in Council authority to set rates for such services had been obtained.

2.135 The National Research Council has clear legislative authority to charge for any R&D services to the public, but it had not obtained Order in Council authority to set rates. In September 1981, we were informed by NRC that steps were being taken to obtain the required Order in Council.

2.136 In our review of administrative procedures for contracting in, we found that some organizational units provided research facilities free rather than go through the legislative and administrative difficulties of seeking authority and then setting and collecting fees. In others, because of internal administrative delays, non-government groups were unable to contract for the timely use of research facilities. In some cases, the moneys received and dispersed were not clearly disclosed in the Estimates and Public Accounts.

2.137 The Treasury Board should ensure that the problem of departmental authority for contracting in is resolved.

2.138 Departments should speed up their administrative procedures to facilitate contracting in.

Information for Parliament

Introduction

2.139 We examined the quality of R&D information that the Ministry of State for Science and Technology and the other organizations we audited supply to the House of Commons. As a supplement to the Main Estimates, MOSST collects, analyses and reports information on science expenditures in Federal Science Activities (Red Book). Statistics Canada publishes the Annual Review of Science Statistics , a compilation of national expenditure statistics on Research and Development. The Annual Review includes the widely used science statistic, the GERD, referred to previously. We assessed controls over the preparation and presentation of this information.

Observation

2.140 Federal expenditures. The Red Book is provided as a source of budgetary information on R&D for Members of Parliament. Its usefulness could be increased by expanding the scope to include information about other federal involvement in R&D. For example, the Red Book does not mention the cost and purposes of tax expenditures or duty remissions related to R&D; the Department of Finance estimates the annual income tax forgone at about $142 million. A number of significant elements, such as the expenditures on R&D by some commercially-oriented Crown corporations, estimated at about $ 95 million, are not included. Also omitted are overhead costs such as accommodation, services provided by other departments, including ships and aircraft, and portions of administrative costs attributable to R&D activities, estimated by Statistics Canada to be about $50 million. Recoupment costs, which are payments made by the federal government to cover a portion of a firm's R&D expenditures, are not reported. These costs may take the form of a payment, identified as being for R&D, on capital item procurements, such as aircraft or military ships. They may also be bonuses added to R&D contracts to encourage a firm to do its own R&D work or to reimburse a firm for hiring additional R&D staff. For 1981-82, recoupment costs are estimated to be over $50 million. These omissions result in federal involvement in R&D being understated by about $340 million. Because of compensating factors, these omissions do not significantly influence the estimate of the GERD.

2.141 In addition, the usefulness of the information in the Red Book would be improved by relating forecast science expenditures to intended results and possible benefits and by ensuring that it is available when Members of Parliament begin to review the Main Estimates.

2.142 Any improvements should be undertaken in concert with the government program to revise the form of the Estimates. This program of revision is being co-ordinated by the Office of the Comptroller General.

2.143 We observed deficiencies in quality control over the data collection process. For instance, lack of preciseness in the definitions of different kinds of science activities can lead to different interpretations of what constitutes an R&D expenditure; this leads to inconsistencies. In the five organizations audited, we found under-reporting of R&D spending by at least $20 million, out of a total of about $540 million. Some organizations reported procurement contracts for goods and services associated with R&D in government laboratories as expenditures supporting R&D in other sectors of the economy.

2.144 MOSST has informed us that it has begun a review of the purpose, scope and content of the Red Book, as well as the controls over data quality.

2.145 National expenditures. The information on national R&D expenditures in the Statistics Canada publication, Annual Review of Science Statistics , was inadequately presented. As noted earlier in this chapter, limitations on the reliability and use of the information are not described clearly, comprehensively or in a consolidated form.

2.146 Statistics Canada calculates the GERD figure by summing the estimated expenditures on R&D by the various sectors of the economy. The uncertainty in the estimates is stated only for each sector; there is no indication of the uncertainty when the estimates for each sector are summed to give the GERD estimate. The uncertainties represent large amounts of money and, since they are not expressed in dollar terms and consolidated, their significance could escape the reader. We consider that the range of uncertainty for estimates of the GERD is about plus or minus $200 million out of total expenditures of $2.5 billion. It should be noted that this represents an uncertainty range of plus or minus 8 per cent; a range that is not abnormal for this type of statistic.

2.147 In addition to the reliability problem, there are issues of usefulness and comparability. To make comparisons among its members, the Organization for Economic Co-operation and Development (OECD) computes a Canadian GERD using data supplied by Statistics Canada. The OECD-calculated GERD includes expenditures on R&D in both the natural sciences and the humanities and social sciences. The Canadian GERD includes expenditures on R&D in the natural sciences only. Statistics Canada informed us that the narrow basis of calculation had been chosen as the most appropriate for the Canadian situation. The excluded Canadian humanities and social sciences component amounts to about $315 million. Both statistics appear in the Annual Review, and there is a risk of erroneous comparison because the differences are not clearly indicated.

2.148 Although Statistics Canada cannot be held responsible for misuses of statistics it presents, it should, in its publications, warn readers of possible limitations to the use of these data. It is well known that the GERD is used as a comparative indicator of Canadian performance in R&D and, in this regard, it has significant limitations. For example, the Canadian economy depends heavily on its non-manufacturing sectors. These sectors are traditionally much less R&D intensive than the manufacturing sector. Therefore, the Review should indicate that the GERD/GNP ratio can be used only with caution when comparing Canada to other countries.

2.149 The 1980 edition of the Annual Review was published while our audit was in progress. Among the improvements in that edition were the inclusion of data and discussions on a number of indicators of science and technology performance and a table displaying the total expenditure on R&D in Canada. We encourage continued efforts to develop a broad range of science activity indicators.

2.150 Our audit also revealed that Statistics Canada had identified and was working toward resolving significant reporting problems. However, quality control procedures over R&D expenditure data collection were often informal and undocumented. Management informed us of its intention to review the collection and presentation of the national expenditure information on R&D.

2.151 As part of the Treasury Board's program to revise the form and content of the Estimates, the Ministry of State for Science and Technology, with the Office of the Comptroller General, should review the reporting of R&D information to Parliament .

2.152 Statistics Canada should improve the presentation of the data in its Annual Review of Science Statistics. In particular, it should clearly describe significant limitations on reliability and use of publicly reported data.

2.153 Both the Ministry of State for Science and Technology and Statistics Canada should improve their controls over the quality of information on R&D for public use.


Annex 1

Audit Criteria for the Management of Research and Development in Government Laboratories

The audit criteria presented here are summary statements of more detailed criteria used in our audits and represent standards that we consider appropriate for R&D generally. The key audit question was whether appropriate management controls for the specific circumstance were in place and working.

We took into account a number of considerations when using these criteria. First, certain aspects of the R&D management process are more important in some organizations than in others. Accordingly, we determined which were most critical in each organization and concentrated our work on these aspects. Second, different areas of research call for different types of controls; different organizations can manage in different ways and still demonstrate due regard for economy, efficiency and effectiveness. The steps in the management process, therefore, may not always occur in the same sequence in all organizations and small projects may not need the formality of control required for large projects. Furthermore, strengths in one step of the management process may compensate for weaknesses in another.

The structure of the audit criteria for the control of R&D activities in government laboratories is based on a general management process as outlined in Diagram 2.1. The reader should bear in mind that there is considerable feedback and interaction among the several functions shown in the diagram -- the typical R&D management process generates much greater exchange of information than the model might suggest.

(This diagram is not available)

Planning

This aspect of the management process is designed to link together the R&D mandate, assessments of needs or opportunities, organizational capacity, and resources. It consists of strategic planning, operational planning and project planning, the latter two being successively more specific. One major purpose of a planning process is to ensure that statements of objectives are translated into sub-objectives that direct research efforts to approved areas and provide adequate guides to action.

Strategic planning. This level of planning defines the role and objectives of the R&D function and should result in identification of the clients for R&D services and types of service to be provided.

    Strategic objectives for R&D should reflect the legislative mandate of the department or agency and be stated clearly enough to be guides to operational planning.
Statements of strategic objectives should:

    - reflect the longer-term perceived needs and opportunities for R&D;
    - indicate the type of clients to be served and the type of R&D services to be provided; and
    - reflect assessments of the technical feasibility of the R&D work and the organizational capacities to do R&D.
    Estimates of the needed resources should be set out for each R&D objective.
    Responsibility for strategic planning should be clearly assigned.
Operational Planning. Whereas strategic planning provides a basis for policy approval, operational planning provides the basis for committing resources to specific short, medium or long-term goals. An operational plan can take the form of a portfolio of resources allocated to particular research areas and clients and, in many research organizations, operational planning includes the selection of specific R&D projects and programs within the resources allocated to each area.

    Objectives set by operational planning should be clear enough to guide R&D managers in specifying the R&D clients, selecting projects and evaluating progress.
Operational plans for R&D should specify:

    - the R&D needs to be met;
    - who will prepare proposals;
    - what resources are available for each research area;
    - the program or project duration; and
    - the procedures for monitoring and controlling against plan.
    Operational plans should be appropriately documented.
    Responsibility for operational planning should be clearly assigned.

Providing for Client-researcher Interaction

    R&D management should provide for appropriate client-researcher interaction at all stages of the R&D process -- strategic and operational planning, generation of ideas, preparation of proposals and their selection, transfer of R&D results and evaluation.

Preparing Proposals

    Management should province guidelines for the preparation of proposals.
    Proposals should be appropriately documented.
Where appropriate, project proposals should:

    - indicate how the project will help meet the objectives of the department or agency;
    - identify the client for the project;
    - state expected results and outputs and, wherever appropriate, assess the estimated costs and the chance of success against estimated pay-off;
    - explain and justify the proposed research approach and the specific scientific and technological methodology to be used and, where applicable, name any outside participants;
    - indicate how the project will be controlled and progress assessed;
    - include a project budget, giving cost estimates of major items;
    - state the mix of personnel and qualifications required to carry out the work and how these resources are to be obtained;
    - state the research facilities and support services required, and when and for how long they will be needed; and
    - state to whom the results of the project will be reported and how.

Selecting Projects

This is the process through which strategic and operational plans are translated into decisions on specific research projects. Selection of a proposal for a project or program represents its approval in principle. Frequently outside experts and the client for the research service are involved in the review process.

    Proposals should be formally reviewed and approved or rejected using criteria developed by management.
    The criteria and decisions should be appropriately documented.
    Responsibility for the selection process should be clearly assigned.

Operational Control of Projects

Operational control is the process designed to ensure economic and efficient performance of the research work. It includes the development of a detailed project plan, the granting of final approval based on this detailed plan, implementing the plan, and monitoring progress and taking corrective action. Final approval can be delegated to the appropriate level so long as the project objectives and resource commitments in the detailed plan are consistent with the proposal approved in principle.

In September 1981, we received from the committee of senior executives of the science community draft guidelines entitled "Operational Control of Research and Development in Departments and Agencies of the Federal Government of Canada." Future audits will be conducted with reference to these guidelines.

Detailed project planning. After a proposal has been selected, project planning should be carried out at a level of detail appropriate to the project.

    Management should province guidelines for preparing detailed project plans.
    Project plans should be appropriately documented.
Where appropriate, project plans should:

    - state project objectives clearly enough to guide research work;
    - describe the proposed R&D methodology;
    - specify activities and tasks to be performed and assign responsibility for them;
    - list resource requirements, particularly for shared facilities, and provide a schedule for their use;
    - describe project organization, including delegation of authority;
    - define the project monitoring procedures, including qualitative and quantitative indicators, for assessing progress and use of resources; and
    - identify decision points for continuation or termination.
Final approval. The approval of the detailed plan represents final authority to begin the research work. In most cases where the duration of the project falls within normal financial commitment periods, approval to begin would constitute approval for the whole project. However, where the project is expected to continue for an extended period, or if completion depends on a critical success point, approval to begin should not represent "final" approval for the entire project.

    Authority and criteria for final approval of projects should be clearly defined.
Implementation and operational control of projects. Implementation is the term describing actual laboratory or development work. Project control involves monitoring technical progress and expenditures and taking appropriate corrective action when deviations from plan are detected. Control procedures should be appropriate for the project. In particular, some control procedures may be informal for smaller projects.

    Responsibility and authority for project management should be clearly assigned.
    Operations should be conducted with due regard for economy, efficiency and effectiveness and for the safety and security of research workers and the public.
    There should be accurate, relevant and timely information that indicates to management the actual status of each project.
    Major deviations from an approved project plan should be analysed and appropriate action taken.
    The appropriate level of management should review, approve and document significant changes to project plans; for example, redefinition of objectives, increase of resource needs, extending schedule or terminating.

Transferring R&D Results

Throughout the R&D process, particularly at the proposal preparation and selection stages, consideration should be given to the transfer of research results to users.

    Managers should develop specific means to facilitate use of research results.
    Procedures should exist for ensuring, that, where appropriate, scientific results are published and, whenever appropriate, discoveries and developments are patented and the patents licensed.

Evaluation

Traditionally, evaluation in R&D occurs in different ways at a number of stages in the R&D process -- in strategic planning, when a proposal is selected, as research is being carried out, when a project is reviewed for renewal and when a project or program is finished. Outside experts are frequently involved. The Office of the Comptroller General has issued the Guide on the Program Evaluation Function which supports the policy that departments will provide independent, periodic review of their programs to evaluate their effectiveness in meeting their objectives and the efficiency with which they are being administered. All the implications for the traditional approach to evaluating R&D that arise from this policy have not yet been clarified. In our opinion, it would be desirable to develop guidelines specifically for evaluating R&D.

    The evaluation of R&D programs should meet standards appropriate for R&D.

Management Information and Control

An appropriate management information system is required for overall control of R&D programs and projects. Informal systems are often suitable for one or two-man projects. However, for controlling larger projects and for comparing actual results with operational and strategic plans, a formal information system is required. The information will be a mixture of quantitative data and narrative description.

    There should be a system for providing managers with reliable, relevant and timely information on program and project resource consumption and progress against strategic and operational plans.

Support Systems

    To the extent possible, R&D support systems, such as finance and administration, internal audit and materiel should be designed for R&D.

Annex 2

Audit Criteria for the Management of R&D Grants and Contributions Programs

A number of processes are common to the management of R&D grants and contributions in all departments and agencies. For each of these processes, we developed audit criteria, which can be regarded as statements of what constitutes good practice in management control of grants and contributions. The emphasis or applicability of specific criteria may, of course, vary depending upon the nature of a program.

We have listed our criteria according to the eight management processes as shown in Diagram 2.2.

(This diagram is not available)

Stating Objectives

    The objectives of R&D grants and contributions programs should be defined and translated into operational terms that will be clear enough to:
    - specify the group(s) eligible for funding;
    - indicate appropriate guidelines for reviewing and approving applications; and
    - identify measures for determining the effectiveness of the programs.

Establishing Terms and Conditions

Terms and conditions translate legislative intent into operational terms that must be approved by the Treasury Board. This approval provides a control over purposes and recipient eligibility.

    Terms and conditions for grants and contributions programs should reflect the legislative mandate and requirements of central agencies.

Informing Potential Applicants

    Information on R&D grants and contributions programs, and particularly information on the rules governing eligibility, guidelines for applicants and the preparation of proposals, should be distributed to potential applicants.

Reviewing and Approving Applications

    Procedures should be established for ensuring that the applications approved are those most likely to achieve program objectives. These procedures should be documented.
    - Applications should be assessed against documented selection criteria.
    - Decisions should be adequately documented.
    - The review procedures should require applicants to disclose all other sources of funding for a particular proposal.
    - Any delegation of authority for the approval or extension of a grant or contribution should be accompanied by appropriate controls.

Making Payments

    Grants and contributions payments should be consistent with the approved level of funding.
    - Applicants should indicate in writing their acceptance of all conditions attached to a grant or contribution before funds are disbursed.
    - Payments should not exceed the approved level of funding.

Monitoring

    Approved R&D grants and contributions should be monitored.
    - For R&D grants, there should be reasonable assurance that funds are being used for the purpose intended.
    - For contributions, payments should be in accordance with contribution arrangements.
    - Audit requirements in the contribution arrangements should be met.

Management Information Systems

    The management information system should provide to the appropriate managers reliable, relevant and timely information on program operations and expenditures.

Assessing Program Effectiveness

    The objectives and effects of grants and contributions programs should be specified as precisely as possible.
    Program objectives and effects that can be measured should be identified.
    Procedures for measuring program effectiveness should reflect the state of the art and be cost-justified.
    The results of effectiveness measurement should be reported.
    Evaluations should be used to increase program effectiveness.