This Web page has been archived on the Web.

1986 Report of the Auditor General of Canada

Chapter 15—Follow-Up of Recommendations in Previous Reports

Introduction

Scope

Current Status

Review of Cash Management—1984, Chapter 4

Scope of Follow-up

Developments to Date

Overall Conclusion

Management of the Function

Cost of Existing Banking Arrangements/Renegotiation of Banking Arrangements with Financial Institutions

Billing, Collection, Deposit and Payment Practices

Management of Travel—1984, Chapter 7

Management of Job Classification—1984, Chapter 8

Introduction

Developments Since 1984

Scope of Follow-up

Quality Controls/Awareness of Cost Implications

Central Agency Monitoring and Control

Accountability for Job Classification

Conclusion

Canadian International Development Agency—1984, Chapter 9

Food Aid

Bilateral Program Delivery

Agency Management

National Energy Board—1984, Chapter 10

Pipeline Regulatory Activities

Export Charge Administration

Overall Conclusion

Department of National Defence—1984, Chapter 12

Background

Developments to Date

Overall Assessment

Summary by Line of Enquiry

Department of Public Works—1984, Chapter 13

Introduction

Developments to Date

Overall Conclusion

Accommodation Program

Marine Program

Services Program

Revenue Dependency

Secretary of State—1984, Chapter l4

The Translation Bureau

The Canada Student Loans Program

Program Evaluation—1983, Chapter 3

Purpose

Developments Since 1983

Scope

Criteria

Sample

Quality of Program Evaluations

Use to Outside Readers

The Comptroller General's Role

National Museums of Canada—1981, Chapter 11

Overall Conclusion

General Management

Museums Functions

The Canadian Heritage Information Network

Introduction

15.1 In Chapter 14 of our 1985 annual Report we included for the first time information on the status of action taken by departments and agencies in response to observations and recommendations of the Office and, where applicable, of the Public Accounts Committee. This initiative was well received.

Scope

15.2 This year we completed follow-up audits or reviews on important recommendations contained in eight chapters from the 1984 Report, additional follow-up on a 1983 chapter partially reported in last year's follow-up chapter, and follow-up of one chapter from our 1981 annual Report. In all cases, the scope of the audit was designed to permit us to arrive at an overall conclusion on the effectiveness of the actions taken in response to our recommendations.

Current Status

15.3 The status of actions taken in response to our recommendations varies considerably. In most instances, positive actions have been taken or are under way in response to the major recommendations; however, in a few instances little has been done.

Review of Cash Management - 1984, Chapter 4

15.4 Our 1984 Review of Cash Management focused on the management of the function, the cost of existing banking arrangements with financial institutions, and deposit and payment practices. We identified problems in all these areas. In our opinion, most of the problems and missed opportunities for savings identified in the chapter related directly to the lack of clarity of roles, responsibilities and relationships with respect to the cash management function. We made eight recommendations.

Scope of Follow-up

15.5 This year's follow-up consisted primarily of a series of interviews with officials from the Office of the Comptroller General, the Receiver General for Canada and the Department of Finance. We also reviewed material prepared since our 1984 Report such as Treasury Board directives, the new banking arrangements settlement, a report on the roles, responsibilities and relationships within the cash management framework, produced by an independent consultant, and the March 1986 progress report of the Office of the Comptroller General to the Public Accounts Committee. We did not perform a detailed audit of this information.

Developments to Date

15.6 In early 1985, the Public Accounts Committee held two hearings on cash management. The second hearing was held at the request of the Canadian Bankers' Association on the renegotiation of banking arrangements with the financial institutions. The Committee issued a report in May 1985 recommending that the Office of the Comptroller General be given the responsibility for all aspects of cash management. The Committee also expressed concern over the lack of progress in renegotiating banking arrangements and urged settlement by 30 August 1985. There were five recommendations in the Committee's report. The following are some of the actions taken by the government:

    - In January 1985, the government approved a strategy for cash management. Approval in principle was also given to proceed with developing a number of specific initiatives such as strengthening current policies and practices for accounts receivable and scheduling payments to suppliers to meet due dates.
    - In March 1985, a Senior Cash Management Review Committee was established to ensure good cash management co-ordination. The Committee is chaired by a senior official from the Office of the Comptroller General and includes representatives from the Department of Finance, the Department of Supply and Services, Treasury Board Secretariat and the Bank of Canada. Representatives from other departments participate as appropriate.
    - In June 1985, the Deputy Minister of Finance provided the Public Accounts Committee with an update on the status of banking negotiations. He indicated that considerable progress had been made since the hearings, high priority had been placed on the banking arrangements, and the two parties were co-operating to ensure that a settlement was reached quickly.
    - In July 1985, Treasury Board approved the establishment of a Cash Management Directorate in the Office of the Comptroller General. The directorate is now operating with a staff of 12.
    - In January 1986, an independent consulting group completed a review of the roles, responsibilities and relationships for cash management in the Government of Canada. This review agreed with the assignment of responsibility for cash management to the Office of the Comptroller General and supported the establishment of a Senior Cash Management Review Committee.
    - In March 1986, a progress report on cash management was prepared by the Office of the Comptroller General and submitted to the Public Accounts Committee, outlining the savings achieved through improved cash management practices and information on improvements in cash management policies and practices.
    - In April 1986, a revised banking arrangements settlement between the government and the financial institutions went into effect.
15.7 Over the past two years, the Office of the Comptroller General has issued a number of directives on cash management. These include:

    - a governmental strategy for cash management;
    - policy and procedures for the collection of overdue accounts;
    - debt write-off regulations;
    - requirements regarding release of pay cheques;
    - policy and procedures for the payment of suppliers' accounts;
    - guidelines for monitoring and controlling obligations of Crown corporations to Canada;
    - guidelines for strengthening current policies and practices for managing revenue and collecting debts owed to the government; and
    - a drawdown policy for Crown corporation funding.
15.8 The Receiver General has also issued several directives and information bulletins. These are in respect of advance notice to the Receiver General of requisitions for large payments, electronic deposit of public money to the credit of the Receiver General for Canada, and the implementation of payment of suppliers' accounts policy - payment on due date.

Overall Conclusion

15.9 The government has now given cash management a high priority and has addressed all the recommendations we made in this area in 1984. Progress has been made in establishing good cash management practices. The full potential of savings from sound cash management practices will only be known once current and planned initiatives have been operational for a sufficient period of time to calculate savings reasonably.

15.10 One concern we have is that good cash management is still not always taken into account by many financial officers and program managers at the departmental level. The Office of the Comptroller General is aware of this and has informed us that it is developing training programs to aid in the implementation of good cash management practices. Initially, these training programs will be directed to financial officers and later to program managers.

Management of the Function

15.11 The Office of the Comptroller General has been assigned responsibility for cash management in areas in which Treasury Board has a mandate. The roles of other central agencies and departments have been clarified as a result of a study done by an independent consulting group and by consultation among the central agencies. The Senior Cash Management Review Committee, mentioned previously, has met on several occasions over the past year and a half. Therefore, there is now a leader in areas in which Treasury Board has a mandate for cash management and a means whereby central agencies and departments can get together and solve cash management problems. These changes, which meet our recommendations, were also fully supported by the independent consulting group's study.

Cost of Existing Banking Arrangements/Renegotiation of Banking Arrangements with Financial Institutions

15.12 The government and financial institutions have now completed negotiations for new banking arrangements. The new arrangements cover the period 1 April 1986 to 31 March 1989. The Office of the Comptroller General will be monitoring annually the cost-effectiveness of these new arrangements.

Billing, Collection, Deposit and Payment Practices

15.13 Billing and collection. The Office of the Comptroller General chose the area of accounts receivable for intensive cash management work in 1985-86. During the last two years, it issued two directives on the managing of accounts receivable dealing with the collection of overdue accounts and the strengthening of current policies and practices for managing revenue and collecting debts owed to the government. The Comptroller General's monitoring of this area indicates significantly improved collections of accounts receivable in 1985-86.

15.14 Deposit. A system for the electronic deposit of public money to the Receiver General account has been established and was scheduled for implementation in the summer of 1986. It applies to all departments, agencies and Crown corporations that operate within the Consolidated Revenue Fund, except for Revenue Canada and Canada Post Corporation where other arrangements have been made. It will replace the current Bank Settlement Voucher system.

15.15 Revenue Canada deposit facilities have been tendered to the financial institutions. The government's intention is to eliminate the use of Bank Settlement Vouchers for these deposits also.

15.16 Payment. The Office of the Comptroller General issued a directive in August 1985 requiring that payments to suppliers be made as close as possible to, but no later than, the due date. This policy was implemented in August 1985. The Office of the Comptroller General has informed us that its preliminary analysis indicates that savings are being realized and could be significant. The Office of the Comptroller General will continue to monitor the savings.

15.17 Advance payments to Crown corporations. In our 1984 Report, we noted that departments were remitting funds to Crown corporations and agencies before the money was actually required. The Office of the Comptroller General issued a directive in March 1986 outlining a policy that would closely tie advances to Crown corporations to their actual cash needs. This should result in interest savings to the government. The Office of the Comptroller General is monitoring adherence to this policy.

15.18 Cash recoveries from Crown corporations. A number of Crown corporations had accumulated large amounts of cash that were surplus to their requirements. During 1985-86, $364 million was recovered. A further $74 million is to be recovered in 1986-87. Based on the Office of the Comptroller General's calculations, annual interest savings of more than $30 million will be realized from the recovery of these amounts.

15.19 Other savings and initiatives. The Receiver General's Office has informed us that it has reduced the number of departmental bank accounts over the past year from l,000 to 22, increasing the level of cash on deposit in interest bearing accounts by an estimated $20 million. A number of other cash management initiatives have occurred that could result in savings to the government. For example, the Department of Finance negotiated a reduced commitment fee on the unused portion of Canada's U.S. line of credit and reduced commissions paid to distributors of marketable bonds. The Office of the Comptroller General, as a pilot project, is testing the use of travellers' cheques in departments and agencies instead of travel advances for government travel. The government is also considering the use of credit cards for travel and payment of suppliers' accounts. In 1986 the Receiver General will be expanding, beyond the current pilot project, a voluntary program of direct deposit to their bank accounts of employees' pay, remuneration of judges, Members and staff of the House of Commons, and public service and military pensions.

Management of Travel - 1984, Chapter 7

15.20 In 1984, our tests of management practices and controls over travel in a number of departments indicated significant weaknesses in several areas. We commented on the need for certain improvements to the Travel Policy that we believed would lead to more efficient and economical travel management.

15.21 The Office of the Comptroller General and the Treasury Board Secretariat have not yet finalized changes to the Travel Policy and its provisions. These are intended to simplify pre-authorization procedures and to reduce administrative costs.

15.22 In the four departments selected for follow-up, our reviews of practices and procedures now in place for managing and controlling public service travel revealed no significant deficiencies in three of them. In the case of the Department of External Affairs, while there has been considerable progress in most areas, there remained weaknesses relating to the verification of Central Travel Service billings and a lack of follow-up on credits due from unused airline tickets.

15.23 On 30 April 1986 the Minister of Supply and Services announced that a private credit card organization had been selected to provide a corporate travel card system. As well, a commercial travel group was chosen to provide a reservation and ticket service for public servants travelling on government business. The travel card system and the new travel service are to be phased in over a period of from 18 to 24 months beginning in December 1986.

15.24 The new system and service are designed to:

    - promote the use of credit cards by travellers, costing the government less than the former system;
    - save money by replacing the present Central Travel Service Payment and Billing system, giving responsibility to departments and individual credit card holders to settle their accounts directly instead of through the former intermediary system; and
    - improve accountability over the purchase of transportation tickets and financial administration of travel claims.
15.25 We will examine the implementation of the new Travel Policy and the operations of the new travel card and service systems when they have become fully operational.

Management of Job Classification - 1984, Chapter 8

Introduction

15.26 In 1984 the Office carried out a government-wide audit of the management of job classification in the federal public service. Job Classification refers to the processes used to determine the relative worth of jobs within the federal public service (actual rates of pay for most public service jobs are established through collective bargaining).

15.27 We observed that the job classification system was under considerable upward pressure and that countervailing forces, including management practices, were inadequate to offset these upward pressures. There was no clear accountability for the integrity of the job classification system or for making individual classification decisions, most of which were made under authority delegated to departments. A Treasury Board Secretariat audit of the quality of classification decisions, conducted in 1983, had revealed a serious rate of misclassification - one in every four public servants was misclassified - most of them over-classified - entailing a net annual payroll cost of $125 million in 1983.

15.28 Our audit report contained five recommendations aimed at achieving improvements in the following three areas:

    - controls over quality and awareness of the cost implications of classification decisions;
    - central agency monitoring and control of the job classification system; and
    - accountability of Treasury Board Secretariat and departments for job classification.

Developments Since 1984

15.29 The Public Accounts Committee held two hearings in May 1985 on job classification. The Committee issued a report to the House with a recommendation that the Treasury Board Secretariat report back to the Committee in April 1986 on its progress in implementing corrective measures. This was done.

Scope of Follow-up

15.30 Our follow-up report is based on a review of the reports submitted to the Public Accounts Committee by Treasury Board Secretariat (TBS), an updated report, provided by TBS, of progress against each recommendation and a review of selected documents and discussions with TBS officials. Because several of the TBS initiatives have been undertaken since May 1985, and in some cases are not fully implemented, we have not conducted an on-site verification of the efficacy of the corrective measures.

Quality Controls/Awareness of Cost Implications

15.31 We recommended that TBS and departments ensure that classification decisions are subjected to rigorous and effective quality control measures before they are implemented. We also recommended that TBS and departments develop measures to ensure that consideration is given to the cost implications of organization changes and related classification decisions. These recommendations were intended to ensure that classification decisions are correct when they are made and that the cost implications of classification decisions, particularly upward reclassification, were known and accounted for. Corrective action would require, among other things, that TBS ensure that classification standards - the basis for accurate classification decisions - are current and that departments improve controls over making classification decisions to ensure their accuracy relative to the classification standards and the work being performed or to be performed in the positions being classified.

15.32 Treasury Board Secretariat reports that the following corrective measures have been implemented:

    - A special task force has been established to review the 50 classification standards that have not undergone a significant review in the last 10 years. Six reviews have been completed - the remaining 44 standards will have been reviewed and updated by 1 April 1987. A five-year cyclical review of classification standards will begin in 1987-88. By 1988-89, all 69 classification standards will have been reviewed; none will be more than five years old.
    - TBS is implementing a "key job system". This involves identifying actual positions in departments to be used as references in the classification of other positions in the same department. To date, some 24 positions in 9 major departments have been identified as key jobs in several major occupational groups. When fully implemented, the key job system will provide references for approximately 100,000 positions.
    - In departments, classification decisions must now be made by a committee that includes at least two line managers knowledgeable in classification. Further, prior to reclassifying a position upward and promoting the incumbent, departments are required to ensure that the duties actually performed coincide with those set out in the job description. TBS reports that since August 1985, this review has resulted in departments rejecting 16.9 per cent of upward reclassification requests.
    - The automatic funding of upward reclassifications has been eliminated. Managers are required to attest that the cost implications of a classification decision are known and that funds are available. TBS will be examining the capability of a new information system, implemented on l April 1986, to estimate the cost effects on payroll attributable to departmental classification activities.

Central Agency Monitoring and Control

15.33 Our recommendations in this area were that TBS have good information on the performance of the classification system on a service-wide basis as well as on a departmental basis; that this information be collected in a systematic and co-ordinated fashion so as to avoid duplication of effort; and that classification problems be corrected effectively and promptly, through the application of sanctions where necessary.

15.34 TBS reports that the following corrective actions have been taken:

    - Departments are required to submit annual classification monitoring and audit plans for review by TBS.
    - Departments are required to conduct biennial audits of the quality of their classification decisions using standardized TBS audit methodology and subject to TBS verification. Results are reported to TBS for compilation on a service-wide basis. The first audit was conducted in 1985-86 and indicated a current misclassification rate for the public service as a whole of 11.4 per cent. This audit examined the classification of 3,568 positions and is accurate within 2 per cent for the public service as a whole, 95 per cent of the time. The statistical methodology used was similar to that used in the 1983 TBS audit. The audit work was carried out by departmental personnel in accordance with guidelines published by TBS and was subject to TBS verification. Of the 11.4 per cent misclassified positions, 7.5 per cent were overclassified. Although a direct comparison with the 1983 TBS audit is not possible, the comparable figures from that audit were 23.8 per cent misclassified of which 19.4 per cent were overclassified.
    - In conjunction with the biennial classification audits scheduled for 1988 and 1990, individual departments have been required to establish goals for job classification based on their misclassification rate identified in the 1985-86 audit. These performance targets have been set by all departments with more than 400 person-years; they have also been reported to the Public Accounts Committee.
    - Departments are required to submit monthly reports to TBS on actions undertaken to correct misclassifications identified through the 1985-86 audit as well as misclassifications identified through their own monitoring activities.
    - TBS will be renewing an agreement with the Public Service Commission whereby the Commission will audit certain aspects of the classification process during its audits of departmental staffing. Where there is evidence of a thorough internal audit of a department's classification processes, TBS will exempt the department from the Commission review of classification in order to avoid duplication of effort.
    - Effective l April 1986, TBS introduced a computerized information system to monitor departmental classification activity. This system will provide for comparison of individual departmental activity with service-wide norms.
    - Through TBS directives, responsibilities for job classification have been clarified with respect to the roles of personnel specialists and departmental managers. Personnel specialists attest to the correctness of the classification decision; departmental managers attest to the cost implications and availability of funds. TBS has also directed that proposals for reorganization must be approved in departments by managers who are at least two levels above the positions affected by any proposed classification action. Departments have been informed that the classification performance of managers is an integral part of their performance appraisal. Where a department's classification program is unsatisfactory, the deputy head will be called before the Treasury Board to explain plans for corrective action.
    - TBS is closely monitoring the action undertaken by departments to correct misclassification.

Accountability for Job Classification

15.35 Our recommendation in this area called for Treasury Board to report periodically to Parliament outlining the activities and performance of the job classification system. In the course of testimony before the Public Accounts Committee, the Secretary of the Treasury Board indicated that the President of Treasury Board would abide by the wishes of the Committee with respect to a report to Parliament on job classification. The Public Accounts Committee did not decide formally on this recommendation. The members who spoke on the issue were prepared to wait for the April 1986 report of TBS to the Committee to decide on a final disposition of the recommendation. A recommendation for a report to Parliament has not been made.

Conclusion

15.36 In our view, Treasury Board and its Secretariat have taken strong and direct action to address the problems identified in our 1984 audit of job classification. We are concerned, however, about the pace of the TBS review of outdated classification standards. TBS undertook to review and update 50 classification standards by 1 April 1987. TBS informs us that all the standards are now in the process of review and that six have been completed. We appreciate that the task is difficult and the time frames tight; nevertheless, in view of the TBS commitment to the Public Accounts Committee, we suggest that the Treasury Board Secretariat monitor the situation closely.

15.37 The results of TBS's 1985-86 service-wide audit of job classification indicates that TBS measures are having a positive effect. In view of these results, and the TBS estimate that the rate of misclassification will drop to 6.4 per cent service-wide in 1990, we believe it is appropriate that a report to Parliament on job classification be put on hold until the results of the 1988 and 1990 audits are known and have been subjected to verification by this Office.

Canadian International Development Agency - 1984, Chapter 9

15.38 Our 1984 comprehensive audit of the Canadian International Development Agency (CIDA) reviewed four main program components: food aid, bilateral aid, multilateral assistance, and special programs. The report concentrated primarily on food aid and bilateral aid. We made several recommendations with respect to accountability for results, roles and responsibilities, planning, monitoring, technical support, procurement, the use of evaluation findings, human resource planning, and Agency management.

15.39 The Public Accounts Committee held four hearings on CIDA. In its Eighth Report to the House, tabled in October 1985, the Committee made eight recommendations to the Government and CIDA.

15.40 In our follow-up of the 1984 audit, we reviewed the actions taken by the Agency to ensure that changes had been made in line with the recommendations. We also reviewed the reports prepared for the Public Accounts Committee. We discussed system changes with management and reviewed relevant documentation. Our follow-up procedures did not include detailed project file reviews or on-site visits to projects, and therefore we cannot conclude on the results of the system changes that have been introduced. Our review addressed food aid, bilateral aid, human resource planning, evaluation, and Agency management issues.

15.41 CIDA has made a serious effort to address the recommendations resulting from the 1984 comprehensive audit and has made significant progress in most cases.

Food Aid

15.42 Our 1984 Report stressed the need to re-examine the organization, administration and accountability of the food aid program. We observed weaknesses with respect to information on food needs, monitoring of delivery and distribution, evaluation, and the procurement of skim milk powder.

15.43 We also expressed concern over the diffusion of accountability for the program. The Public Accounts Committee recommended that the Government establish CIDA as the clear, overall program manager for the food aid program.

15.44 Accountability. Since 1984, CIDA and Supply and Services Canada have executed a memorandum of understanding on procurement that covers their respective responsibilities pertaining to the procurement of food and transport. The Government reaffirmed to the House of Commons, Agriculture Canada's lead role for the World Food Program and stressed the value of maintaining a close linkage and flow of information between the food aid program and the Canadian agricultural sector. CIDA, however, is accountable for the overall management of the food aid program and seeks the advice of other departments when developing food aid policy.

15.45 While no formal agreement, covering dealings with the World Food Program, has been established between CIDA and Agriculture Canada, CIDA's current operating relationships with Agriculture Canada are working well. Through the implementation of an improved communication system involving the World Food Program and Agriculture Canada, regular reports are now available containing up-to-date information on food aid commitments, shipments and their destinations. This computerized system is still under development.

15.46 CIDA has improved its internal co-ordination of the procurement process and the delivery of bilateral food aid. The Food Aid Co-ordination and Evaluation Centre (FACE) and the Procurement Division have implemented new operating procedures. FACE now provides a Project Resource Officer for all food aid projects.

15.47 Information on food aid needs. Since the 1984 audit, the existing information networks have been strengthened. This was in partial response to the crisis in Africa. In addition to meeting regularly with representatives from the World Food Program and the Food and Agriculture Organization, FACE frequently meets informally with the European Economic Communities and USAID. CIDA has contributed funds to strengthen the World Food Program monitoring and delivery systems for multilateral and bilateral aid to Africa. FACE now disseminates more information on food aid needs to the non-governmental organization community.

15.48 Monitoring. The Agency has changed the instructions to officers in the field. Guidelines are now in place that clearly specify the work to be done. Monitors are under contract to be present at ports when food shipments arrive and are required to report their findings to the Post. Co-ordination among the various agencies located in the field is also carried out as a result of CIDA's reinforced message on the importance of the monitoring function.

15.49 Skim milk powder. The method of vitamin fortification of skim milk powder has been reviewed by the Agency, Agriculture Canada and the Canadian Dairy Commission. The cost of fortification will be brought down to a level comparable with other donor countries. The automated process, which meets Canadian and international standards, is expected to reduce the cost of fortification since it eliminates the need for manual debagging and rebagging of the powder.

15.50 The Agency has recognized the problems associated with using woven polypropelene bags. As of April 1986, the Agency will use the regular 5-ply paper bags that reveal puncture marks and are easier to stack. Regular damage reports are to be filed in the field to record shipping problems.

15.51 Skim milk powder bags are now product dated to determine if the storage period has exceeded the shelf life of the fortified product. The recipient country will now be able to determine other uses, recall, or destroy any stale dated powder to reduce any possibility of harming food aid recipients.

15.52 Although the pricing formula for skim milk powder is fully satisfactory; there is no provision for periodic review by the Agency of the calculation of the price per tonne. CIDA informally maintains an awareness of price movements within the industry as well as the prices paid by other aid agencies. However, no formal record of these other prices is kept by the Agency. Regular review of the application of the pricing formula by CIDA would confirm adherence to Treasury Board requirements.

Bilateral Program Delivery

15.53 Project planning. CIDA has taken steps to improve project planning by emphasizing the need to assess factors critical to a project's operational viability and to consider lessons learned from previous projects or phases. It has also introduced measures to reduce the length of time taken between project identification and mobilization.

15.54 The assessment of operational viability has improved through:

    - ensuring adequate challenge by bilateral branch project review committees;
    - emphasizing the importance of providing for the unforeseen by establishing specific contingency allowances; and
    - establishing a "Bilateral Information Feedback System" designed to provide access to evaluation information derived from CIDA's prior projects.
15.55 CIDA projects often extend into second or third phases. In 1984 we found that the lessons learned from previous phases were not regularly considered in subsequent phases. The project approval system has now been modified. Proposals for the second or third phases of projects must use the results of the evaluations of prior phases and similar projects, and show how these will be incorporated into subsequent phases. However, planning and approval documents that we reviewed had no specific references to the evaluation results of earlier phases. How the problems identified in earlier phases would be alleviated or resolved was not evident. The resolution of problems may be implicit in the plan, but explicit reference would provide better information to the approving authority - senior management, the Minister or Treasury Board. The Agency now requires that the Evaluation Division prepare a critical review of the plans for projects that require a CIDA contribution greater than $2 million.

15.56 To reduce the time from project identification to mobilization, CIDA has obtained an increased delegation of authority for project approval, encouraged concurrent planning activities, and reduced duplicate documentation. CIDA also plans to formalize contract planning as part of the project planning framework.

15.57 Project management structure. In 1984 we observed weaknesses with respect to roles and responsibilities of project team members and field representatives, monitoring, the provision of adequate technical and administrative support and the use of project evaluations. The Agency has made progress in addressing these weaknesses.

15.58 In view of government policy to constrain the number of person-years while increasing the volume of development assistance, CIDA is considering alternative program delivery approaches. During the past two years, CIDA has increased the level of contracting for project planning and execution to the private sector, non-governmental organizations and institutions. Decision-making authority has been decentralized by giving authority to the Posts to engage local consultants and to manage co-operants. Together with the Treasury Board, CIDA undertook a contract practices review that dealt with contractor accountability, contract definition, monitoring and administration of contracts. Other policy changes may occur as a result of the joint House/Senate review of foreign policy. A report of the Standing Committee on External Affairs and International Trade is expected in 1987. Formal resolution of problems, including achieving a balance between domestic and international objectives, will await consideration of the reports of these Committees.

15.59 In addition to the study on contracting policy, CIDA has taken steps to increase the number of standing offers for technical assistance, to increase the amount spent on monitoring contracts, to establish field support units under contract, and to issue a manual on the establishment of such administrative and technical units. The Agency is examining different possibilities for increased monitoring and decentralization.

15.60 Standing offers, project monitors and field support units provide increased availability of specialist advice in planning, monitoring and evaluating projects. CIDA has increased the number of standing offers with consultants thereby providing the project team in Headquarters with extensive and ready access to specialist expertise in a number of sub-sectors. Major contracts for project monitors and the existence of technical and administrative support units should ensure better on-site monitoring and reporting to the project team in Headquarters.

15.61 The Agency is seeking Treasury Board approval for additional person-years to ensure that adequate in-house resources are available to manage increased technical responsibilities and to maintain effective contractor accountability.

15.62 CIDA has re-examined the roles and responsibilities of key project team members such as the project team leader, the resource officers, the CIDA representative in the Post, the executing agents and project monitors. The Agency will soon issue instructions clarifying their respective roles and responsibilities for project delivery.

15.63 Procurement. In 1984 we found that CIDA needed to strengthen its procurement planning and monitoring. This included the need to improve the Agency's communication of procurement policies, procedures and guidelines to the project teams and consultants. A recently completed internal audit of the Procurement Division indicated that there is now an improved accountability framework. The improvement is due to a new mandate and organizational structure, memoranda of understanding between CIDA and Supply and Services Canada and between the Procurement Division and FACE, written procedures and guidelines, and the preparation of a divisional training plan.

15.64 However, the draft internal audit report indicated that Procurement Division staff "were not included in project planning and monitoring activities to the extent which is dictated by their mandate, and to the extent required to ensure an effective procurement function in CIDA." Although representatives of the Division are usually assigned to the projects that have significant procurement components, these assignments are often made after the procurement mode has been chosen and the detailed plans of operation have been prepared. The internal audit also noted that procurement mode decisions were not substantiated by documented information and consistent analysis.

15.65 A detailed action plan to resolve these issues is being prepared to respond to the recommendations arising from the internal audit.

Agency Management

15.66 Comprehensive Agency Management Plan. Progress on completing remaining Comprehensive Agency Management Plan (CAMP) projects has been satisfactory. Of the remaining active CAMP projects, the four major ones, representing 80 per cent of the budget, are now under suitable budgetary control. Scheduled reviews of CAMP projects have been completed and appropriate action has been taken to address outstanding issues. The Comptroller General's Office is validating completed projects.

15.67 Corporate memory. In 1984 we examined minutes of the President's Committee dealing with operational issues and noted that no mechanisms were in place to ensure structured and systematic follow-up of Committee decisions. As a result, there were delays in implementing corrective action.

15.68 Since 1984, CIDA has initiated a Corporate Management Initiative Program. Major reports with policy and program implications are channelled through the President's Committee where priorities are set. Action plans are developed by the appropriate branch and incorporated into workplans. Their status is reviewed semi-annually by the President's Committee.

15.69 Recommendations from audit and evaluation reports are considered and approved or rejected by the Audit and Evaluation Committee. In July 1984 the President's Committee delegated the responsibility to the Audit and Evaluation Committee for monitoring the implementation of approved audit and evaluation recommendations. Internal audit recommendations are systematically followed up and status reports are prepared. However, there are no similar procedures for monitoring the implementation of actions taken to address the approved recommendations arising out of corporate and program evaluation studies considered by the Committee.

15.70 Evaluation planning. In 1984 evaluation activity was not being carried out in an integrated and co-ordinated way. Procedures to co-ordinate evaluation activity in the various program branches and to ensure the overall quality of corporate, program and project evaluations needed to be developed. The Agency's Evaluation Division now emphasizes the integration of project and corporate evaluation plans. The Division prepares an annual report on evaluation activities. In addition, the Division issued standard guidelines for planning and conducting project evaluations and developed an evaluation training course for project managers.

15.71 Human resource planning. CIDA and its staff work in a changing environment. Externally, the relationships with its client countries are often being redefined. Internally, the Agency undergoes frequent structural, program or administrative changes. These factors have a major impact on the type and level of expertise required, the specific location where these skills are needed to meet operational objectives and when they are needed. The Agency must have mechanisms and information that can permit it to make timely adjustments to meet new demands or orientations.

15.72 In 1984 CIDA's Human Resource Plan was not linked into the operational planning process and there was no multi-year human resource planning. Since then, CIDA has implemented the key elements of a human resource planning system. It provides managers with an integrated planning framework, an adequate data base, a co-ordinated staff redeployment cycle and an articulated career progression process. A Multi-year Human Resource Plan (MYHRP 1986-1990) has been approved by the Treasury Board. The next MYHRP is meant to be included in the Agency planning cycle. The reintegration of staff posted overseas and the redeployment of personnel are now co-ordinated through an annual exercise administered by Personnel and involving every branch in the Agency.

National Energy Board - 1984, Chapter 10

15.73 In our 1984 audit of the National Energy Board, we examined the procedures leading to and supporting the Board's regulatory decisions, particularly the management controls over the analysis supporting these decisions and the technical completeness of the analysis. We also examined the procedures used by the Board to monitor compliance with its decisions and to determine the effects of the decisions. Also included was the Board's administration of the Export Charge program. We identified some weaknesses and made three recommendations.

15.74 We followed up on the three recommendations, bearing in mind the changing environment in which the Board must now regulate. Since the audit in 1984, the Government has been moving toward less regulation of oil and gas exports and pricing. In some cases, this has altered the impact of our recommendations. Our follow-up consisted of discussions with Board staff in the functional branches and a review of supporting documentation. We did not perform a detailed examination.

Pipeline Regulatory Activities

15.75 We recommended that a long-term program should be undertaken to strengthen the Board's efforts to encourage economy and efficiency on the part of the regulated pipeline companies. The Board took three initiatives in its response to this recommendation.

15.76 First, the Board has been considering a concept of "focused management audits", or a type of value for money audit. Progress had been made in this area when the "Agreement on Natural Gas Markets and Prices" was signed in 1985. In this agreement, the federal government agreed to establish an Independent Pipeline Inquiry Panel to conduct an all-encompassing review of the role and operations of interprovincial and international pipelines. Concurrently, the Board was conducting toll hearings respecting two main carriers. Because of these happenings, and recognizing that it was in a transitional period, the Board decided that the time was inappropriate to begin management audits. These audits may begin in the latter part of 1986 or early 1987.

15.77 The second initiative concerns strengthening the Board's cost control over pipeline construction. During initial hearings, the Board has been conscious of the design, routing and cost estimates of major construction projects. Cost estimating and control are now more prominent in these hearings. A standard condition has been included in certificates that requires companies to file monthly cost progress reports and Board staff are examining approved cost estimates and forecast schedules as part of a "cost monitoring program". Any variances between original estimates and progress reports are followed up.

15.78 A third initiative involves improvements to the audit organization at the Board.

Export Charge Administration

15.79 Our other recommendations concerned control of oil exports. Previously, all exports of oil were required to be licensed, and companies paid an export charge on each cubic metre of oil exported. Effective 1 June 1985, the licensing requirement was terminated, as was the Export Charge. These changes had a significant impact on Board operations.

15.80 Our second recommendation was that the Board should determine the amount of oil and bulk products being exported illegally, for example, by truck. We agree with the Board's decision not to implement this recommendation; with the de-regulation of oil exports and the Export Charge set at zero, the need no longer exists.

15.81 Our third recommendation, that the Board should establish whether its current enforcement practices fully meet the intent of the legislation, has taken on a new meaning. Under de-regulation, the Board's audit function as it relates to Export Charges will be terminated when planned audits are completed. Our 1984 audit found that the Board had not adequately evaluated the scope and extent of testing done of oil companies to improve the effectiveness of the Export Charge audit function. We reported that the Board had not assumed responsibility for ensuring that all exports are reported to the Board. Until recently, there has been a lack of information on previous assessments and audit findings on which to base audit planning. The Board now plans to re-examine the extent of audit testing to be performed as results of current audits become known. We agree with this approach.

Overall Conclusion

15.82 The Board's efforts to address our recommendations have been tangible and realistic. However, because each of the initiatives has been taken relatively recently and action is still under way, we will consider further follow-up.

Department of National Defence - 1984, Chapter 12

Background

15.83 Our second comprehensive audit of the Department of National Defence (DND) was reported in Chapter 12 of our 1984 Report. This audit comprised six lines of enquiry: planning, control and evaluation; financial management and control systems; major equipment acquisitions; facility construction projects; automated data processing (ADP) systems development; and human resource management. There were 29 recommendations in the chapter that spanned these six lines of audit enquiry. The Public Accounts Committee devoted three sessions during February and March of 1985 to the chapter and, on 30 April 1985, it put forward a number of recommendations in its Third Report to the House of Commons.

15.84 Our follow-up activities were designed to assess the progress that has been made by the Department in carrying out the recommendations put forward by the Office and the PAC. We have divided the activities into two stages. For the purposes of this report, we have provided an initial assessment of progress. In 1987 we will undertake a more detailed examination.

Developments to Date

15.85 The Department agreed with all our recommendations. In some cases, the Department indicated that steps had already been taken in the direction desired. In a few cases, it was pointed out that factors external to the Department constrained its ability to effect the needed changes. In all areas, however, the Department assigned responsibilities for addressing each recommendation, established action plans, and installed a monitoring system to report internally on progress on a quarterly basis.

15.86 The Third Report of the PAC contained eight recommendations directed toward identification, measurement and reporting of non-military objectives associated with major capital projects; the determination and disclosure of life-cycle costs associated with major capital projects; dual-sourcing of spare parts for the CF-18; and the designation of a senior official to be chief financial officer and comptroller. The Committee asked DND to report back to it with respect to these recommendations by 3l July 1985. It further requested that the Department prepare progress reports on automated data processing (ADP) and internal audit, and submit them to the Auditor General for review on behalf of the Committee by 30 August 1985.

15.87 On 23 July 1985, the Department submitted its report to the Committee and to us. On 30 August, we reported to the Committee that progress in the areas of ADP and internal audit, while preliminary, appeared to be satisfactory.

15.88 On 6 February 1986, the Deputy Minister and the Chief of the Defence Staff were briefed on the scope and direction of the comprehensive audit of the Department that is scheduled for reporting in 1987. During this briefing, they were advised that follow-up on the 1984 Report and the Public Accounts Committee's Third Report recommendations would be carried out in the two stages described above. On 26 March 1986 we requested a status report on progress which the Department provided on 18 April.

Overall Assessment

15.89 The Department has moved aggressively in addressing the recommendations. Of the 37 recommendations (8 from the Public Accounts Committee and 29 from our Office) requiring a change in management systems or practices, action is reported as complete on 15; another 2 are awaiting final approval. Eighteen are being addressed, while another two are inactive until other decisions are made.

15.90 Our preliminary review of progress has led us to conclude that, in most cases, the response by DND has been thorough and properly directed to addressing the factors that led to a need for change. In many areas the issues involved are very complex, and it will be some time before the full effects of the Department's actions are known.

Summary by Line of Enquiry

15.91 Planning, control and evaluation. As we indicated in our 1984 Report, an approved peacetime force structure for the longer term, with an appropriate link to wartime requirements, is needed to guide the planning and management of virtually every aspect of the Department's activities. During the audit, the Department initiated a Normative Planning Project to determine peace and war requirements. Since our 1984 Report, activities in this area have been broadened and deepened into what is now called the Defence Policy Implementation Process.

15.92 In the latter part of 1985, the Minister of National Defence issued guidelines for a defence policy review which resulted in the formulation of defence policy options including " illustrative, costed, military force structures". An action plan has been established and is ready to be implemented once an option has been selected. If events proceed as planned within the next year, the Department will have defined the longer-term force structure plan called for in our 1984 Report. We will continue to monitor progress in this very important area.

15.93 The Department has moved quickly in response to our recommendations concerning feedback mechanisms. A Chief of Review Services (CRS) position has been established, reporting to the Deputy Minister and to the Chief of the Defence Staff. CRS has the task of, among other things, co-ordinating feedback mechanisms. A new operational readiness reporting system has been put in place. As well, steps have been taken to ensure that comprehensive Canadian exercises are regularly scheduled by the Chief of the Defence Staff. However, many challenges still face the Department in this area. For instance, the relationship between the responsibilities of the CRS and those in the line chain-of-command has yet to be resolved. We will continue to monitor developments in this area.

15.94 Financial management and control systems. A satisfactory report on progress in the area of internal audit was provided to the Office for review on behalf of the Public Accounts Committee. Responses to the recommendations dealing with establishing an internal audit policy, plan and quality assurance function have also been satisfactory. Steps have also been taken to expand the scope of audit activity. Work is under way to put forward a departmental position on the comptrollership function.

15.95 Major equipment acquisitions. The Department has issued guidelines relating to concepts of operating new equipment items, reporting changes to project scope or cost, and project management procedures. Our future audits will examine compliance with these guidelines. Work continues on developing new information systems to improve capacity definition, efficiency measurement, work standards, cost information and performance measurement for maintenance facilities.

15.96 The Department undertook a review of alternatives to fixed-ceiling funding for major equipment items still in the development stage, and concluded that, given fiscal constraints, a funding ceiling provides necessary management discipline. This recommendation is linked, however, to the recommendation that changes in project scope or cost should be reported to the appropriate authorities. If ceilings must remain inflexible despite unit cost increases resulting from technological or design changes, then care must be taken to ensure that the necessary adjustments to the number of units or necessary support items acquired are understood and reported.

15.97 DND has devoted considerable attention to addressing the need for consistently applied life-cycle costing of major capital projects. Discussions on this are under way with officials from Treasury Board and the Office of the Comptroller General, and a decision is expected soon on the categories of costs to be reported.

15.98 Non-military objectives (e.g., industrial development, regional growth, research and development, employment) associated with major defence purchases are often difficult to quantify, cost or, indeed, separate from defence objectives. The Public Accounts Committee noted, however, that these non-military objectives play a role in justifying extremely large expenditures. In many instances, their attainment can significantly increase project costs. The Committee, therefore, felt that it is necessary to identify, measure and report the attainment of non-military objectives.

15.99 The Department is now finalizing the procedures needed for identifying and reporting non-military objectives. Part III of the 1987-88 Estimates will contain a description of non-military objectives anticipated for all major Crown projects, along with a report on the achievements of a selection of these projects.

15.100 Facility construction. The Department has established procedures for performing post-completion evaluations of construction projects.

15.101 Human resource management. Action related to recommendations dealing with balancing training commitments and resources, identifying training needs emerging from major capital projects, investigating the use of new training technology and identifying civilian training needs has been completed in a satisfactory manner. Progress is under way to integrate the various personnel functions and to improve the personnel management information system. Once again, progress in these areas appears to be satisfactory. Although little progress has yet been made on the redesign of the classification and trade structure and on the identification of civilian resource requirements we anticipate that work will proceed in these areas as major components of the long-term force structure plan.

15.102 Automated Data Processing (ADP). Progress is satisfactory in all the areas of role definition, planning, costing, performance monitoring and survivability planning. A departmental ADP strategy has been approved, long-and short-term national level plans have been produced, with Command and National Defence Headquarters Group plans having been produced or in process. Performance monitoring and ADP resource accounting have been enhanced, improving managerial control and providing feedback to planning. Planned equipment upgrades have occurred and major systems development is progressing. A survivability plan for the national-level facilities has been produced implementing the departmental ADP Survivability Policy. Disaster Response Plans are being drafted.

Department of Public Works - 1984, Chapter 13

Introduction

15.103 Our 1984 comprehensive audit of the Department of Public Works (DPW) focused on the Accommodation Program, but extended to other key operations and activities including IMPAC/revenue dependency, the Marine Program and the Professional and Technical Services Program. We identified many serious problems in the management of accommodation and the lack of cost-consciousness in the expenditure of public funds. We made 15 recommendations.

15.104 At the time of our follow-up, a review of progress with respect to the Department's Management Improvement Plan was being carried out by the Office of the Comptroller General and DPW's Management Improvement Branch. We decided to rely on their work to the extent we considered practicable. Our review consisted of discussions with representatives from functional branches at DPW's headquarters and three regional offices, as well as a review of pertinent documents. We did not perform a detailed audit.

Developments to Date

15.105 The Public Accounts Committee held hearings in February 1985 to discuss the audit report on DPW. The Committee subsequently issued a report on 1 April 1985, criticizing, among other things, "Treasury Board's failure to enforce its rules, and the inept and ineffective reporting practices between the Department and Treasury Board". The PAC report also made six recommendations. In response to them, the following events took place:

    - In May 1985, the Minister of Public Works and the President of the Treasury Board provided written responses to the Committee indicating that specific actions had been taken.
    - On 30 July 1985, the Department provided our Office with a progress report concerning management of accommodation. The Department stated that it had decided to address in one comprehensive Management Improvement Plan (MIP) the observations from our 1979 and 1984 Reports, the PAC Report, the IMPAC/revenue dependency plan and the Department's internal audit reports.
    - In August 1985, we advised the PAC that all the recommendations contained in its Report and those contained in our Reports had received serious study by the Department. However, although implementation of the MIP was under way in certain areas, it was too early to assess its success. Consequently, we did not carry out tests on the outputs resulting from the new systems, policies and procedures.
    - In October 1985, the Department provided us with the first six-month progress report on the implementation of revenue dependency and the establishment of an effective cost accounting system. DPW indicated that it was proceeding, incrementally, to put in place the infrastructure necessary to support revenue dependency, pending a final direction by the government.
    - In November 1985, in view of an impending policy statement as a result of consideration of a report by the Nielsen Task Force on Program Review that would have a major impact on the Department in terms of its mandate and particularly the area of revenue dependency, we suggested to the PAC that we would provide an assessment of the above progress report, together with a second six-month report, as of 31 March 1986.
15.106 On 21 November 1985, the Department appeared again before the Public Accounts Committee and presented a detailed progress report on the actions taken with respect to the Auditor General's and PAC's observations. The Committee said it was pleased with the efforts of the Department.

15.107 As a result of the Nielsen Task Force study on real property management, the government, in December 1985, announced new measures to improve management of the federal government's real property holdings. The new approach will also streamline DPW's custody role in the management of general purpose properties, and augment its common service role in providing operations and maintenance, architectural, engineering, real estate and contracting services.

15.108 In May 1986 the Department provided us with the second six-month progress report, effective to 3l March 1986, on its progress in implementing the infrastructure necessary for revenue dependency. We subsequently provided the PAC with an assessment of the report, described later in this section.

Overall Conclusion

15.109 We found that all our recommendations had received serious study, and a total of 64 action plans had been developed by the Department to address the Auditor General's and PAC's observations and recommendations. Despite cuts in resources and adjustments to programs, and a freeze on discretionary spending in February and March 1986, improvement has been made in many areas. Thirty of the 64 action plans are fully complete. A number of action plans were behind schedule, but most of them were substantially in progress.

15.110 Although essential elements of key management information systems have been introduced and the necessary infrastructure for revenue dependency is gradually being put in place, these information systems are neither fully operational nor satisfying user needs.

Accommodation Program

15.111 Leasing. To encourage compliance with existing Treasury Board policies, DPW has sent letters to all its regional offices and client departments stressing adherence to Treasury Board policy on lease tendering. The Department indicated that 18.4 per cent of its leasing transactions were publicly tendered in 1985-86, compared with 6 per cent in 1984-85. The rest were carried out in accordance with section 8 of the Government Contracts Regulations.

15.112 Lease-purchase arrangements. In response to our recommendation, DPW has disclosed the particulars of lease-purchase arrangements in Part III of the Estimates since 1984-85.

15.113 The authority to enter into a ground lease with a developer has now been clarified; amendments have been made to the relevant statutes.

15.114 In 1984 we noted that there were conflicting legal opinions given to DPW regarding the borrowing done to finance the lease-purchase transactions. DPW legal services, in collaboration with the Department of Justice, is of the opinion that lease-purchase transactions to date do not constitute a borrowing under section 36 of the Financial Administration Act.

15.115 Managing space. A national program to dispose of surplus properties was announced in April 1985. Phases l and 2 of the program have resulted in the sale of over 200 properties, for which proceeds totalled some $50 million. The Department has provided information on the current status of the properties that the PAC expressed specific interest in, including Block 56 in Vancouver; the Daly Building, the Teacher's College, the Mulligan Building and Canlands in Ottawa; the parcel of land in Downsview; and the Dominion Public Building in Toronto.

15.116 The first phase of the Facilities Inventory System (FIS) has been implemented to provide basic inventory data and information on space occupancies. The Occupancy Instrument (a formal, written agreement concerning accommodation provided) is the prime data input to FIS. Slippage has continued to occur in signing these documents by client departments; the Department expects completion by the end of 1986-87.

15.117 A new classification of vacant space has been developed in an effort to manage the inventory better. This new classification has indicated that some 374,465 square metres of vacant space exist in an inventory totalling 6,775,000 square metres. DPW reports that only 131,302 square metres of vacant space are marketable. That represents l.9 per cent of total space. For 1985-86 the Department targeted 120,930 square metres of vacant space to be occupied or disposed. They report 124,480 square metres achieved. Initial efforts to do a complete reconciliation of the data have been delayed because the definition of "vacant space" and interpretation of data are not yet consistent in all regions. The Department is also working toward space optimization. In this respect, it has experienced problems in defining space optimization and interpreting data. The Department reports some 26,100 square metres achieved against the 1985-86 target of 27,750 square metres.

15.118 Phase two of FIS, which is designed to provide planning data for input to the Department's management processes and the Multi-year Operational Plan, is partially operational in some regions. Some enhancement to the system is required; it is being planned for 1986-87.

15.119 The budgetary process. A new work planning and budgetary control process was introduced in 1985-86, and targeted for automation in 1986-87, to ensure that plans are linked with budgets and that variances are reported and acted on promptly.

15.120 DPW regional offices have completed the 1985-86 program of facilities inspection to ensure that work plans include all the work necessary to achieve a pre-determined level or quality of maintenance.

Marine Program

15.121 As directed by Cabinet, DPW has pursued various options to dispose of the dry docks. The principal dry dock users were invited to submit proposals, but no satisfactory bids were received. The Department has recently suggested other courses of action to Treasury Board and is waiting for direction.

Services Program

15.122 Time reporting systems for realty management services and real estate services and phase one of the Time Reporting/Project Monitoring System for architectural and engineering services have been implemented. However, the time reporting systems for realty services do not capture operational planning and project control data. The second phase of the Time Reporting/Project Monitoring System is being developed to enhance project management control.

15.123 The Department advised us that a Treasury Board submission had been drafted to present to the Board the results of its studies respecting norms for resource allocation and evaluation, fees for services, and an A-Base Review related to architectural and engineering services.

15.124 The Department also indicated that the role and mandate of the Technology Branch had been clarified and specific initiatives had been planned by DPW to ensure that its regional offices are aware of the Branch's role and mandate.

Revenue Dependency

15.125 As at 1 April 1985, DPW began charging client departments for the direct costs of its services; that is, project disbursements and payroll costs of DPW staff directly performing the work. In December 1985, Treasury Board directed the Department to initiate market rates in the Services Program as soon as possible and to report by 31 March 1986 on the internal mechanisms necessary to introduce the new regime. As of l May 1986, no report had been submitted. The Department advises that this is tied to a variety of complex issues related to the government's decisions on real property management. The Department's comprehensive response to the Treasury Board in this regard will include proposals for the implementation of revenue dependency.

15.126 The Financial Management System is one of the key management information systems being implemented by DPW. The first release (FMS I) deals with the provision of the necessary financial accounting information to management and to government while the second release (FMS II) intends to provide the necessary management accounting information. FMS I was implemented in 1985-86 and is close to fully operational. However, we noted a number of problems with this system. DPW advises us that a private sector study to assess the current status of the system has been carried out and recommendations are now being implemented.

15.127 FMS II addresses the cost accounting requirements of the Department and provides other financial information for management. The target completion date is April 1987.

15.128 In our opinion, progress has been made by the Department in implementing revenue dependency for its Services Program. However, the Department has not indicated a firm time frame for introducing market rates for the Services Program. In compliance with Treasury Board direction, the Department, jointly with the Treasury Board Secretariat and the Office of the Comptroller General, is immediately to assess the viability of the systems and internal mechanisms necessary for its operation and its readiness (including impact on its clients) regarding the implementation of revenue dependency based on market rates. The Department advises us that it will be making a submission to Treasury Board by 31 October 1986 concerning the introduction of revenue dependency for both its Services and Accommodation Program.

Secretary of State - 1984, Chapter l4

15.129 Our 1984 audit covered all the major operational sectors of the Department of the Secretary of State: education support, translation and citizenship registration.

15.130 At that time, we identified weaknesses in the management of Translation Bureau operations as well as in the control over the development of a new loans system for the Canada Student Loans Program.

15.131 This follow-up report is limited to those two areas and is based on interviews with managers from the four branches of the Translation Bureau as well as officials of the Canada Student Loans program, which is part of the Education Support Program. We were able to obtain and examine the available information that was relevant to our task. However, we did not conduct a detailed audit of these programs.

15.132 We observed that the Secretary of State has adopted corrective measures in response to each of our recommendations. At the time of our audit, the changes resulting from these measures had been implemented only recently or were just being implemented.

The Translation Bureau

15.133 Strategy for deploying resources. As a first step in developing a strategy for an optimal balance between the use of in-house resources on one hand and the use of translation firms and freelance translators on the other hand, the Bureau established an acceptable, accurate methodology for determining its in-house costs. The methodology was then assessed by outside consultants that concluded that it was a valid management tool.

15.134 The data from this new source of information will provide one of the significant elements on which to base a make-or-buy policy. This second step should be achieved during the first half of fiscal 1986-87.

15.135 Monitoring productivity. Following recommendations from the task forces assigned to look into the issue, senior management decided to adopt a new philosophy of management. It is oriented toward the results that are to be achieved, recognizes the value of professional contributions to translation services, and emphasizes the management abilities of translation service directors.

15.136 Instead of adopting a Bureau-wide quantitative standard to be used as a performance benchmark or as a criterion in promoting translators, the achievement of results related to objectives approved by senior management will become the basic element in evaluating both management and employee performance.

15.137 The principal steps in introducing this new management approach are as follows:

    - simplifying the administrative structure and increasing the authority and responsibilities of translation service directors;
    - improving the overall existing planning system for translation services, allowing management to identify all translation activities to be undertaken and to set production objectives for each activity; and
    - developing an operational information system to give managers the information they need to assess the achievement of objectives.
Detailed work plans were produced for the first time for fiscal 1986-87, and the operational information system should be in place by August 1986.

15.138 Control of workload. The production control unit set up to control the workload in July 1984 has not produced the desired results. After a trial period of several months, the system was abandoned because delays in producing the necessary information meant that control could not be exercised effectively.

15.139 Senior management hopes that the monitoring methods and rapid access to information offered by its new management approach will solve the problem of how to apportion the workload. As well, the Operational Planning and Co-ordination Directorate will be responsible for comparing forecast with actual production at regular intervals. The Directorate will analyse any discrepancies and report to the Translation Operations Branch, which should be in a position to identify problems and take corrective action.

15.140 Service to clients: Standards of quality. In 1985 and 1986, 300 translators attended seminars designed to increase their awareness of the importance of maintaining high standards of quality.

15.141 In April 1986, senior management approved a less rigorous standard of quality regarding the acceptability of the texts. However, it set up an improved quality control system.

15.142 One of management's objectives for 1986-87 is that at least 90 per cent of texts be of sufficient quality that they can be sent to clients as is or with only minor revision. To achieve this objective, a plan is to be completed during the current fiscal year.

15.143 Meeting deadlines. With the establishment of the operational information system in August 1986, translation services directors should be in a position to monitor progress on texts being translated. They will be responsible for following up to determine the reasons for missed deadlines and for taking steps to eliminate them.

15.144 The objective set by the Translation Operations Branch is that at least 80 per cent of texts be produced within the time specified; the operational information system will keep the Branch informed about the extent to which this is being achieved.

15.145 Human resource management: Resource planning. During 1986-87, the Bureau expects to complete implementation of a data bank containing the information needed for human resource planning for the whole organization.

15.146 As well, a multi-year human resources plan was prepared for the first time in 1985-86. It took account of the Department's operational objectives in a very general way.

15.147 The detailed work plans, introduced in April 1986 as part of the new management approach, also allow identification of human resource needs within the various translation services directorates. However, there is no means of collecting and putting this information to use as the basis of action plans for the Bureau as a whole.

15.148 Management roles and responsibilities. The Translation Bureau has clarified management roles and responsibilities. A new organization chart was drawn up in March 1986, and the mandate of each part of the Bureau was defined, along with its functional authority and relationship, if any, with other parts of the Bureau.

15.149 Promotion criteria for translators. The Bureau approved a new promotion policy for the TR group in 1985-86. The new guidelines and procedures define and introduce an element of uniformity in the practice of promotions to the TR-2 level.

15.150 This policy is relatively new and, at the time of our audit, the Bureau had not yet had occasion to apply it, because of the staffing freeze intended to reduce the size of the Public service.

The Canada Student Loans Program

15.151 Correction of data in current system. The Secretary of State reviewed the data in its computer files and, where possible, corrected or purged them as necessary. A final sifting will be conducted before the data are converted for use in the new information system. The Secretary of State recognizes that internal control of its current system is weak and expects that the system now being developed will correct this.

15.152 Monitoring progress on system development project. After completing the conceptual stage for the new system at a cost of five years' effort and $2 million, the Secretary of State began actual development of the new student loans system in April 1984.

15.153 Numerous problems encountered in the development period have caused delays and cost increases. The sources of these problems are several, some of them outside the control of the managers responsible for the projects. We identified the following as the most significant sources of problems:

    - The Secretary of State did not take full advantage of the planning and monitoring tools at its disposal. As a result, those concerned consistently underestimated the complexity and scope of the system being developed. For example, the project completion date was pushed back five times, without any thought being given to the adequacy of overall planning of the project, and the programming stage was begun before the detailed conceptualization had been fully approved by system users.
    - Communication between users and the development group was not entirely effective. On several occasions, the development team considered that intermediate decisions had been approved by users when in fact this was not the case.
15.154 This had the following effects in practice:

    - The date for putting the system into operation was February 1985, but the system was still not operational by June 1986. We were told that the new target date was October 1986, which, according to the original schedule, is a delay of 20 months.
    - The actual cost of the system will surpass $3 million, a 172 per cent increase over the original estimate of $1.1 million.

Program Evaluation - 1983, Chapter 3

Purpose

15.155 In 1983 we observed that the quality of program evaluation studies needed to be improved. We have followed up on some key aspects of this matter.

Developments Since 1983

15.156 In 1985 we followed up on two of the issues raised in 1983. First we found that, overall, the annual number of corporate program evaluations had remained stable for the 18 departments examined. Second, we found that 11 of 15 departments examined had reported one or more of their studies in Part III of the Estimates.

Scope

15.157 This year's follow-up looked at program evaluation reports and focused on the quality of evaluation studies. We examined the presentation of evaluation studies, the coverage of evaluation issues, the quality of the evaluation methodology and the trustworthiness of the reported findings.

15.158 Our audit was restricted to an analysis of the report documents supplied to us by the departments. This was sufficient to arrive at a judgement of the reliability of the information. A basic standard for preparing reports is that they should contain sufficient detail, including discussion of limitations and constraints, to allow the reader to make an independent assessment of the value of the information.

15.159 We did not examine utility - the extent to which these studies contributed to significant program change or gave management assurance that change was unnecessary. For many of these studies, it is too early for this kind of empirical data to be available. Utility is the ultimate test of the value of a study; however, reasonable technical quality and trustworthy reporting are essential prerequisites to it.

Criteria

15.160 The basic criteria we use in auditing program evaluations are based on the criteria approved by the Public Accounts Committee in 1980 and the Guide on the Program Evaluation Function issued by the Comptroller General in May 1981.

Sample

15.161 A probability sample of 25 studies was selected for audit from a list of corporate evaluation studies completed in 1984-85. An initial review of this list found a number that did not meet all of the criteria for inclusion in the audit. A review of the remainder showed that some did not meet one of our criteria for program evaluation - that a material portion of the study dealt empirically with at least one of the four classes of evaluation issues. The result was a non-probability sample of 16 studies that were audited.

Quality of Program Evaluations

15.162 Based on our audit experience, it is our opinion that, since 1983, there has been an improvement in the quality of study methodology and reporting program evaluations.

15.163 We found 6 of the 16 to be fully satisfactory in both the appropriate application of technique and the clarity and probity of reporting. Two of these studies were particularly well done. There was a second group of six that, while varying significantly in quality, represented acceptable responses to the problems that they tackled. In general, they were studies that provided reasonable information on most of the matters addressed, but they were deficient in differing degrees because of failures in methodology, reporting, or, in some instances, both. Thus, 12 of the 16 studies met a minimal standard of adequacy in their application of evaluation technique and in the way in which they were reported. Many were much better than that on either or both counts.

15.164 Four of the studies were unsatisfactory in their evaluation methods and/or reporting. Although there was also variation in quality within this group, all were found to have serious deficiencies.

15.165 The six fully satisfactory studies were well designed and carefully implemented. Any implementation problems were clearly stated. They were organized so the reader was led clearly from issues through the data to the conclusions about the issues. They were well documented. Typically, these studies presented results that could be related back to the program and offered information well targeted to the study issues. They brought together data in response to each issue, providing a full, reliable, and highly informative answer to each of the study questions. When methodological problems were observed in these studies, they were minor. These studies were unbiased and their conclusions were logical and based on sound argument.

15.166 The six acceptable studies in the middle group each had at least one serious deficiency. However, because they contained much reliable information potentially useful to departments we considered them as at least minimally satisfactory. But they did not meet one or more of the important criteria set out for quality evaluations and therefore could not be accepted as within the standard desirable for government program evaluations. At one end were the studies that were well designed and implemented. Their most typical fault lay in a biased presentation of their findings. Their summaries tended to be selective in the findings highlighted (often bringing forward only the positive findings). At the other end were studies that provided some useful information but missed providing reasonably complete answers to questions. These tended to address only superficially one or more of the issues assigned them.

15.167 The four unsatisfactory reports were poorly organized, poorly documented and poorly written. They often lacked such essentials as a program description or a context for the study. It was difficult to know what points were being made; often the only way to find out was to divine the information from examining the data. The findings in the unsatisfactory studies were often unreliable and invalid, because of poor methodology.

15.168 In 1983, we recommended that departmental evaluation units should ensure that evaluation reports fully present study findings in a balanced manner and discuss the implications of limitations in the methods used for the interpretation of the findings. The unsatisfactory studies show that these problems still exist.

Use to Outside Readers

15.169 In assessing the quality of these reports, we were particularly concerned with the value of the report to an external reader. Increasingly it is expected that these documents will be used by Members of Parliament and other groups. Accordingly, the way in which a study is reported will determine its credibility. To achieve credibility, a report should disclose fully the significant elements of the study process and discuss the impact of any deviations from its intended design. The design used has to be plausible in relation to the problem addressed. Sufficient, carefully organized information is necessary to allow the reader to form an independent assessment of the results. Finally, assertions should be supported by evidence and be fully consistent with it.

15.170 It is recognized that, while undesirable, studies not meeting the standards described here could possibly be useful within a department when the content of the program and the study are well known. However, such a report contributes little internally to corporate memory and has little value outside a department.

The Comptroller General's Role

15.171 In 1983, we observed that a strong central monitoring and quality assurance function would be required to support departmental efforts and to ensure the integrity of program evaluation information. Even though the quality has improved, in light of some of the problems identified, including the selection of inappropriate or superficial methodologies and inadequate reporting, there is evidence of a continued need for the Office of the Comptroller General to further strengthen its efforts in these areas.

National Museums of Canada - 1981, Chapter 11

15.172 The major recommendations we made in 1981 dealt with policies, planning and museum functions. We focused our follow-up on these subjects to assess the steps that National Museums has taken since 1981.

Overall Conclusion

15.173 Although some progress has been made, important problems still exist. Corporate policies on research, conservation and certain aspects of human resources management are still to be approved. The collections policy has not yet been defined.

15.174 With respect to planning, clear objectives for each museum and strategies to achieve the overall objective of the Corporation have not been set out.

15.175 In the area of museum functions, there are still serious backlogs in cataloguing collections. There is not always written evidence that acquisition criteria are being followed. Examining the state of collections and taking inventory are not done systematically. Several collections are still housed in inadequate facilities where storage conditions cannot be controlled. Finally, the National Museums have not yet established an order of priority among the collections so that less important collections are not added to at the expense of other collections.

General Management

15.176 Policies The Corporation has developed a number of policies in the last five years, including a publication policy and several administrative policies. However, there is still no overall policy on research, conservation, or on certain aspects of human resource management. The collections policy still does not specify an order of priority among the collections, as well as within each collection, to guide acquisitions, documentation and conservation.

15.177 Planning. The Corporation has developed a strategic document, multi-year and medium-term plans and annual work plans. But it has not developed strategies to achieve its overall legal objective, nor has it developed sub-objectives that take into account the raison d'être and individual character of the museums that make up the Corporation.

Museums Functions

15.178 Collections policy. The management principles governing National Museums collections are set out in its Collections Policy, published in 1981. The policy still does not assign an order of priority among the collections or among elements of them to ensure that less important collections do not grow at the expense of others.

15.179 Acquisitions. The Collections Policy lists the following acquisition criteria: quality; rarity; historical, scientific or technical value; cultural importance; condition; cost; availability of space; and the use to which an acquisition can be put. We found no written documentation to show that the National Museum of Natural Sciences and the Museum of Man (now the Canadian Museum of Civilization) have applied these criteria. This makes it difficult to ensure that all criteria have been considered and that their collections have been assembled in accordance with established objectives.

15.180 Recording. There are still serious backlogs in cataloguing some collections. Overall, about 600,000 of the 5 million objects in the collections remain to be catalogued. The Canadian War Museum, for example, has catalogued only 10 per cent of its collection. At the current pace of about l per cent a year, it will be 90 years before the Museum's present holdings are catalogued. As for entering the items in the Canadian Heritage Information Network, discussed later, the current backlog for National Museums collections alone numbers about 3.5 million entries. Parts of the collections are unavailable for research or display because they have not been catalogued.

15.181 Controlling. Objects must be examined periodically to see whether they are in good condition or in need of restoration. The National Museum of Natural Sciences and the Museum of Man do not systematically check on the state of their collections. Good management also requires that holdings be checked periodically against inventory records to ensure that collections are complete. None of the museums takes regular inventory of its collections.

15.182 Protection. Several collections are housed in inadequate facilities where storage conditions cannot be controlled. Although three new museums are under construction, the new buildings will be only a partial solution to the problem of adequate storage. Projections based on the rate of growth in the collections suggest that by the year 2000, the Corporation will still need an additional 164,000 square metres of space - that is, twice the area of the three museums now being built.

15.183 National Museums Corporation has developed an accommodation strategy that sets out criteria for evaluating buildings and includes an evaluation using these criteria. As well it has established multi-year interim and permanent accommodation plans.

15.184 Analysis. Since the Corporation has not established an order of priority among its collections and among the parts of each collection, it continues to acquire all sorts of objects through purchases, donations and excavations. The collections have grown by more than 600,000 items in the past five years. The Museums rarely refuse or reject items donated to them or found at excavations. These then have to be catalogued, restored if necessary, stored, preserved and protected. Items from the collections are seldom disposed of, with the result that as the collections grow, so does the space required to house them and the number of people needed to preserve and protect them. The present situation in National Museums illustrates the point: significant backlogs in cataloguing, insufficient storage and exhibit space, the danger that objects will be lost because of inadequate means of preserving, restoring and protecting them. In effect, it is a vicious circle: the larger the collections, the more inadequate the available resources to catalogue and preserve them, conduct research on them and display them to the public.

15.185 According to Corporation management, it is a lack of resources that is to blame for the cataloguing backlogs, the inadequate control over the state and completeness of collections, and the poor state of buildings. An internal study, whose conclusions were supported in part by a task force chaired by someone from outside the museums, showed that the National Museums Corporation is labouring under a severe shortage of the financial and human resources necessary to achieve its objectives.

15.186 Given the lack of resources, we believe that this situation could be improved by establishing an order of priority among the collections, identifying the collections with the highest priority, and disposing of those of less importance.

15.187 The imminent move of some of the museums' collections to new homes has made Corporation management more conscious of the urgent task of completing the cataloguing of objects already in its possession, restoring some of them, and completing research on them so that they can be moved without risk and new exhibits can be mounted. The move is a good opportunity to begin rationalizing the management of the collections.

15.188 Research. We observed that the National Museums Corporation has not established a corporate research policy defining research directions and objectives and overall priorities and setting out project management principles and responsibilities. A draft policy has, however, been prepared for discussion by the Board of Trustees.

15.189 In the past two years, the Corporation has set research priorities, an important first step in improving research project management. But priorities are stated in language so general that many projects could qualify. For example, 'The Arctic', a priority identified by the Museum of Natural Sciences, could include projects in the fields of zoology, botany, geology and so on. In our view, research priority areas should be identified more precisely.

15.190 Once approved, a research project can stretch out over several years. We noted that the Corporation does little in the way of monitoring progress on research projects. For instance, projects are not evaluated periodically by independent experts.

15.191 Finally, it is difficult to determine the amount of human and financial resources devoted to research, because the planning and management information systems do not identify them separately. As a result, the overall research budget is not monitored.

15.192 Exhibitions. Exhibitions are well planned. Proposals take the expected audience into account, set out costs and schedules, and, after discussion and approval, are incorporated into work plans.

15.193 However, we found that cost projections were not complete because they did not include internal costs such as employees' salaries. In the case of one large exhibition, figures supplied by National Museums showed that the actual costs were as much as four times the amount shown in the budget.

15.194 Finally, the Corporation conducts no systematic evaluation of each exhibition to see whether it has reached its intended audience and achieved its objectives.

15.195 Publications. In 1984, National Museums issued a Print Publication Policy. A process was also established to improve publications management - better documented proposals, more clearly defined selection criteria, review and approval of proposals, and use of a management information system. Our examination showed that employees were following established procedures.

15.196 Additional improvements are possible. For example, the Corporation has not set criteria for determining the size of press runs for each publication. It is important to do this because National Museums currently has to store 265,000 unsold publications (350 titles) issued before 1980, some of them dating back to 1924. Obviously, the surplus raises storage costs.

15.197 As well, budgeted amounts do not include costs such as the salaries of employees involved in preparing publications, marketing and so on. The Corporation therefore does not have complete information on which to base decisions about the proposed selling price of publications and their profitability. In the past two years, sales have covered only 33 per cent of the cost charged to publications intended for sale.

15.198 Finally, we observed that the National Museums Corporation does not monitor actual costs against budgeted costs. As a result, it is difficult to tell whether discrepancies are justified and whether they could have been prevented.

The Canadian Heritage Information Network

15.199 The Canadian Heritage Information Network (CHIN) is an automated management information system for Canadian Museum collections. Its purpose is to create a national inventory of collections and facilitate the exchange of information. Participating organizations receive consulting services to help them manage their collections and choose the appropriate technical resources. CHIN now serves 150 Canadian museums.

15.200 Considerable progress in managing CHIN has been made since 1981. Following the recommendations of several studies, the policy governing the program was changed, allowing the Network to offer consulting services concerning the management of collections and information.

15.201 In co-operation with private enterprise, CHIN has installed new software better suited to the needs of the program. A new computer with greater capacity was also acquired to serve clients better. The client list was rationalized by establishing an order of priority among the museums served, based on the significance of their collections. Finally, CHIN has succeeded in recording 25 per cent of the most significant Canadian museum collections.