This Web page has been archived on the Web.

1997 December Report of the Auditor General of Canada

Main Points

35.2 The following significant areas of concern remain:

• Five years after noting our concerns, much remains to be done to ensure that Canada is prepared for major oil and chemical spill emergencies.
• Parliament continues to need adequate information on the risks and clean-up costs of federal contaminated sites. While some progress has been made, environmental liabilities are likely quite substantial and could materially affect the government's reported financial condition.
• The issues reported in this chapter and in Other Audit Observations, Chapter 36, regarding on-reserve facilities funded by Indian and Northern Affairs Canada, continue to raise concerns about the Department's cost effectiveness in addressing on-reserve needs. These concerns involve deficiencies in project planning, implementation, evaluation and related matters.

35.4 In two cases included in this chapter, the departments concerned performed studies or reviews in response to our recommendations, but corrective action is yet to be implemented.

35.5 We encourage departments and agencies to continue addressing our concerns through further corrective action.

Introduction

35.7 Our purpose is to report to Parliament on the progress made until corrective action has been implemented or more recent events make further action unnecessary.

35.8 Follow-up does not provide the same high level of assurance as an audit. Rather, it provides a moderate level of assurance that departments have acted on our recommendations, unless we note otherwise. This assurance is based on information obtained from interviews with departmental officials, document reviews, and other evidence to allow us to conclude on the plausibility of the information provided to us.

35.9 We normally follow up on observations and recommendations and report on their status two years after we publish our original chapter. Some follow-up activity is reported as a separate chapter or reported with subsequent audits of departments, or grouped to provide an overall measure of progress on numerous recommendations, such as in Chapter 18 of our October 1997 Report - Revenue Canada and Department of Finance. Exhibit 35.1 provides a list of audits due for follow-up that are not included in this chapter.

35.10 This chapter reports on progress toward implementing our recommendations made in audits conducted between 1992 and 1995. All the follow-up reports note that progress has been made toward addressing our recommendations, and in some cases they have been fully implemented. But a large number also report that further work needs to be done before the recommendations have been dealt with satisfactorily. In some cases, the delays have been attributed to program changes, but in other cases there is no obvious reason for the delays.

Emergency Preparedness in the Federal Government - Accidents Involving Oil and Chemicals - 1992, Chapter 24

Background

35.12 This year we followed up on the section of the 1992 Report that dealt with accidents involving oil and chemicals. We examined the progress made in three important areas - marine oil spills, marine chemical emergencies, and non-marine chemical emergencies at industrial sites. We did not review preparedness for chemical emergencies that occur during transportation by land.

Conclusions

35.14 Progress has been slow in establishing a system to respond to marine chemical emergencies. There are no national or regional plans to deal with marine chemical accidents. However, the Coast Guard is now in the preliminary stage of developing a chemical emergency response regime in conjunction with industry.

35.15 In our 1992 Report, we indicated that Canada did not have a national legislative framework for land-based chemical accidents. Such a framework is still not in place. However, since our 1992 audit, Environment Canada has drafted proposed amendments to the Canadian Environmental Protection Act (Bill C-74) that include a section dealing with environmental matters related to emergencies. Bill C-74 did not reach second reading in Parliament.

35.16 The Major Industrial Accidents Council of Canada (MIACC) has made progress but now needs to focus on increasing awareness and encouraging action, particularly on the risk assessment of hazardous installations.

35.17 To date, MIACC has identified a list of hazardous substances and developed an initial inventory of hazardous sites. It has identified about 1,350 hazardous sites containing substances that could, if improperly handled, lead to a major accident. These sites contain substances such as vinyl chloride, benzene, and gasoline. The inventory of hazardous sites has not yet been completed. MIACC is sending a survey to hazardous site operators for self-assessment of their safety management practices. The results of the survey are expected in the spring of 1998.

Observations

Marine oil spills

35.19 The September 1990 report of the Public Review Panel was highly critical of preparedness for oil spills. The shortfall in spill response capability was seen by the Coast Guard to be the most pressing priority among the report's many findings. The report stated:

• The capability to respond effectively to a marine spill of any significant magnitude does not presently exist anywhere in Canada.
• Each year, Canada can expect over 100 small spills (less than a tonne), about 10 moderate spills (about 100 tonnes), and at least one major spill (100 to 10,000 tonnes) from oil tankers. A catastrophic spill (greater than 10,000 tonnes), for which we are totally unprepared, can be expected once every 15 years.

35.21 In October 1991, the Treasury Board approved an increase of $100 million over a six-year period for the Coast Guard, Environment Canada and Fisheries and Oceans to implement a strategy that responded to the key recommendations of the Panel. In 1993 an additional $15.9 million was provided to the Coast Guard and $1.2 million was provided to Environment Canada. In our 1992 Report, we noted that initiatives were only in the beginning stages or had not yet started.

35.22 Since 1992, the Coast Guard has done a significant amount of work to establish a marine oil spill preparedness and response regime. Amendments to the Canada Shipping Act enabling the implementation of the new regime were proclaimed on 15 August 1995. Among other things, amendments to the Act require that vessels and oil-handling facilities have emergency response plans. Vessels and designated oil-handling facilities are also required to have an arrangement with a response organization that has been certified by the Coast Guard to respond to a spill equal to the total amount of oil on board, up to a maximum of 10,000 tonnes.

35.23 The Coast Guard assessed two alternatives for funding the required upgrade of Canada's oil spill response capability:

• industry funding through a direct levy on oil imports and exports by ship; or
• a partnership approach involving joint government-industry arrangements for a response capability funded and operated by the private sector.

35.25 A Panel was established to investigate the fairness of fees for assuring oil spill response capability. The cost of establishing the response organizations was to be paid by fees collected from companies that could be involved in oil spills. The Coast Guard did not determine the fee to be charged; this was left to each response organization. The payment of the fee is for assurance of response capability in the event of a spill, not for actual response. If a ship or an oil-handling facility chooses to use the services of a response organization in the event of a spill, those services would be charged separately.

35.26 Before the fees can be approved by the Minister, they must be published in the Canada Gazette . On 16 September 1995, the fees proposed by the five Coast Guard-certified response organizations were published in the Canada Gazette . The proposed fee structure for four of the five response organizations consists of an initiation fee, a registration fee and a bulk oil cargo fee. One of the five response organizations excludes the initiation fee but charges a bunker fee. The proposed fees are:

• Initiation fee. This is a one-time payment of $200 by both ships and designated oil-handling facilities. In the case of one of the response organizations, a bunker fee applies instead of the initiation fee. The bunker fee is to be paid by ships at a rate of $100 for each transit through the response organization's area of response.
• Registration fee. This is an annual fee of $450 charged to both ships and designated oil-handling facilities.
• Bulk oil cargo fee. This is a levy based on the basis of the volume of oil handled multiplied by the bulk oil rate. The fee applies to designated oil-handling facilities for all oil that is unloaded or oil destined for foreign locations that is loaded at the facility. Where non-designated oil-handling facilities that do not have an arrangement with a response organization are involved, the fee is payable by the ship. The fee is a single rate expressed on a per tonne basis. It ranges across response organizations from 22.5 cents to $1.85 per tonne.

35.28 On 4 March 1996, the Minister of Fisheries and Oceans announced an investigation into the fairness and equity of the proposed fees and appointed a three-member Oil Spill Response Organization Fee Structure Investigation Panel. The Panel submitted its final report on 27 August 1996 to the Minister of Fisheries and Oceans.

35.29 The Panel criticized the overall approach to oil spills adopted by the Coast Guard. The Panel noted that the Coast Guard "continued to have prime responsibility for spill response, but had basically become a licensing authority for private sector response capability, but without any residual controls." The Panel stated that the approach adopted "appeared to have been driven by the prevailing policies developed by the previous federal government intent on conveniently abrogating certain government responsibilities and simply maintaining a minimum of control." It called for the Coast Guard to re-establish its principal role in the area of oil spill response and recommended that this be done by establishing a "National Spill Response Agency" that would be a "truly integrated public-private partnership between the Canadian Coast Guard, the oil and shipping industry and response organizations."

35.30 Preparedness for oil spills has been enhanced but a complete assessment is needed. The Panel's report concluded that the response organizations are professionally managed and staffed and their equipment and resources are considered fully adequate to meet the standards set down by the Coast Guard. We note that since our 1992 audit:

• ships and designated oil-handling facilities are now required to have oil pollution emergency plans as well as an arrangement in place with a response organization certified to provide a response capability; and
• a national marine spill response exercise program has been developed.

35.32 The Canada Shipping Act now requires that the Minister review the "capability of response organizations to meet the arrangements required by ships and oil-handling facilities" and report to Parliament. The first Report to Parliament (1996) was mainly a description of the development of the new regime. The report indicates that the 1998 Report to Parliament will contain a more detailed account of the effectiveness of the regime. However, to make an assessment of the adequacy of the current Canadian oil spill response regime, the Coast Guard would have to carry out a review to verify the oil spill response capacity of the response organizations. Without such a review, the Coast Guard cannot conclude that the response capability for oil spills is satisfactory.

35.33 Exercises are one means of assessing the effectiveness of the response organizations. Under the Canada Shipping Act regulations, response organizations are required to evaluate the effectiveness of all aspects of the procedures, equipment and resources that are identified in the response plan through an exercise program. The program must be carried out at least once during a three-year period that begins on the day on which the certificate of designation is issued. The Coast Guard is responsible for monitoring the exercise program. Officials are concerned that they may not have the resources to fully monitor and assess response organization exercises.

35.34 The Panel concluded that the oil spill response fee structure is unworkable. The Fee Structure Investigation Panel criticized the Coast Guard for withdrawing from active control of the new regime by not setting the fee structure and concluded that the proposed oil spill response fee structure is unworkable since it is neither fair nor equitable. With respect to the major objections, the Panel concluded:

• The objectors' assertion that response organizations are unregulated monopolies is valid. This is compounded by the collusive relationship in the system, that is, oil companies are the major shippers of oil and owners of many of the oil-handling facilities and, at the same time, are the owners of the response organizations .
• Multiple fees are unfair and inequitable because a ship traversing several regions with the same risk could potentially be charged different fees.
• The "one fee fits all" fee structure is clearly inconsistent with the concept of a risk-based fee.
• Objections regarding unfair costing procedures are conjectural based on limited information obtained by objectors; however, concern about the high rate of return on investment and certain inconsistent practices in establishing the cost base is legitimate. The report concluded that the proposed fees are set to generate a level of revenue that will cover all costs plus a return on investment, and that the majority of the response organizations have built into the fee structure a rate of return on capital invested of almost 20 percent on a before-tax basis.
• The objection on the ground that the proposed fees would place a greater burden on some companies than on others is plausible.

35.36 The Coast Guard is also awaiting the results of another consultant's report that is intended to develop options on:

• institutional arrangements for the Coast Guard within the existing policy framework;
• fee structure; and
• ensuring effectiveness of the response regime.

35.38 We recognize that extensive effort has gone into amending the Canada Shipping Act and establishing a marine oil spill response regime. However, given the Panel's conclusions that the current fee structure is unworkable, we are concerned about the viability of this regime. Since Canada's response capability is now to be based on private arrangements between ships and oil-handling facilities and response organizations for which a fee is paid, an inability to resolve the issues related to the proposed fee structures and broader policy issues means that the regime will not be financially viable.

Marine chemical emergencies

• There are a limited number of oil commodities, whereas more than 300 chemical products are shipped in bulk and thousands are shipped in containers.
• Oil products have similar physical properties, while there is a wide range of physical properties, such as flammability, toxicity and corrosiveness, among chemicals.
• In the case of oil, toxicity to humans is low. With chemical products there is a wide range of toxicity, with some being carcinogenic or immediately fatal.
• Oil has an immediate and visible impact on birds and marine life. Chemicals may have delayed and chronic effects.
• The impact of an oil spill is generally proportional to the size of the spill. The impact of a chemical spill is a function of the intrinsic properties of the chemical and is compounded by spill size.
• Oil does not react violently if mixed with water and/or other products. Chemical cargoes may react violently if mixed or upon contact with water or other materials.

35.41 The absence of good statistics on oil and chemical incidents is an area of concern and the Coast Guard indicates that it is in the process of pilot testing a Pollution Incident Reporting System to provide more comprehensive information on the nature and frequency of spills.

35.42 There has been limited progress in developing a marine chemical response capability. The report of the Public Review Panel on Tanker Safety and Marine Spills Response Capability concluded that Canada's ability to respond to marine spills of hazardous materials is, at best, limited. Although the Coast Guard has done some work in this area - for example, a review of practices of other countries, adoption of fundamental principles for design of the system and consultation with industry - progress in resolving this problem has been slow. At present, there is no national or regional integrated plan to deal with major marine chemical accidents.

35.43 The limited progress in this area is due, in part, to the difficulties in trying to first establish the marine oil spill preparedness and response regime. The oil spill regime was seen to be a priority because of the Public Review Panel's report and the two highly visible oil spills involving the Nestucca in 1988 and the Exxon Valdez in 1989.

35.44 The Coast Guard has indicated that it is now in the preliminary stage of developing a chemical emergency response regime in conjunction with industry. A fundamental requirement in developing this regime will be to clearly identify the roles, responsibilities and capabilities of government and industry.

Non-marine chemical emergencies

35.46 In 1996 Environment Canada assessed the progress by all levels of government and industry on the report's 21 recommendations, which were broken down into a total of 49 sub-recommendations for the analysis. The assessment rated the implementation of about half of the 49 sub-recommendations as "fair", meaning that most of the target audience is aware of what is needed but little or no action has taken place. "Good" ratings were given to 29 percent of the sub-recommendations, indicating that action steps have been implemented but that coverage is uneven, with critical gaps. A further 16 percent of the sub-recommendations were rated as "excellent", indicating full implementation. The report concludes that "good to excellent progress has been made on the development of products but the dissemination of these products to the industrial and municipal communities has only just begun."

35.47 A legislative framework for chemical accidents at industrial sites is not in place. In our 1992 Report, we indicated that Canada, unlike most modern industrialized countries, did not have a national legislative framework for chemical accidents at industrial sites that would provide for requirements on the reporting of chemicals used by industry, preparation of emergency plans, safety audits and reporting of spills. A legislative framework is still not in place and we believe that this is needed to provide a sound regulatory basis to complement voluntary initiatives.

35.48 A 1994 Environment Canada issues paper, prepared for the review of the Canadian Environmental Protection Act , noted that the existing approach to environmental emergencies has tended to be reactive and has not focussed on public concerns about prevention and the right to know. With a shift in focus to the prevention of emergencies, it is important to first know the location of sites holding quantities of hazardous substances that pose a potential threat of off-site damage to the environment and the public. Once the sites are known, it would then be possible to assess the risks posed and raise awareness about these risks. Legislation would help implement these preventive measures; for example, it could provide the authority to have sites registered and to ensure that site operators assess the extent to which preventive and preparedness measures are in place.

35.49 Since our 1992 audit, Environment Canada drafted proposed amendments to the Canadian Environmental Protection Act, which the government introduced as Bill C-74 in December 1996. The Bill included a part of the Act dealing with environmental matters related to emergencies. This part of the Act would have given the Governor in Council, on the recommendation of the Minister, enabling authority to make regulations related to:

• establishment of a list of hazardous substances and minimum quantity related to each substance;
• identification of places where substances are located (in the prescribed quantities);
• the prevention of, preparedness for, response to and recovery from environmental emergencies;
• notification and reporting of environmental emergencies; and
• notification and reporting of preventive measures or measures to repair, reduce or mitigate any negative effects of environmental emergencies.

35.51 MIACC has made progress but now needs to focus on risk assessment. In our 1992 Report, we noted that one of the important recommendations of the Bhopal Aftermath Review that had been implemented was the establishment of the Major Industrial Accidents Council of Canada (MIACC). Environment Canada sees the partnership approach, promoted through MIACC, as the best means of preventing accidents and building preparedness in an area of fragmented jurisdiction. MIACC, which is viewed as an alternative to a strictly governmental regulatory approach, receives an annual contribution from the federal government, which will be about $200,000 in 1997-98. It also receives funding from provinces, municipalities and industry. The mission of MIACC is to reduce the frequency and severity of major industrial accidents involving hazardous substances and to achieve harmonization in prevention, preparedness and response programs.

35.52 MIACC has made progress but now needs to focus on encouraging action, particularly on the risk assessment of hazardous installations. Until recently, MIACC has concentrated on developing new tools, such as training courses, guides and standards, for those working in the areas of prevention, preparedness and response. MIACC now sees itself shifting to a focus on "sensitization" - making target groups such as site operators and relevant community officials aware of risks and gaining their commitment to act to minimize risk. This change in focus is reflected by the work MIACC is currently doing with respect to hazardous installations and community preparedness. This work relates to observations made in our 1992 Report.

35.53 In 1986, the Bhopal Aftermath Review recommended that a site-specific risk assessment for each industrial plant be carried out to determine the potential for a major industrial accident. The 1990 Green Plan also highlighted the need for an inventory of hazardous installations. In our 1992 Report, we indicated that Environment Canada did not have such an inventory and had not conducted reviews of accident prevention activities at potentially hazardous installations.

35.54 In 1995, the Report of the Standing Committee on Environment and Sustainable Development recommended that all sites be identified and registered, including sites owned by the federal government, where hazardous substances are found in quantities exceeding specified threshold amounts. The Committee suggested that the list of hazardous substances could be drawn up based on those already identified by MIACC and that the list could then be made available to local fire stations, those in charge of emergency preparedness and the general public if necessary.

35.55 MIACC is developing an inventory of hazardous sites and encouraging site operators to undertake risk assessments. Environment Canada informed us that it does not have the mandate to undertake assessments of hazardous sites, but it is working with MIACC in taking steps to address the issues of hazardous site identification and assessment.

35.56 The first step in assessing risks posed by hazardous sites is to identify the priority substances. MIACC has developed lists of hazardous substances and the threshold quantities above which the substances could present hazards to on-site workers, the general public, property and the environment if not handled properly. The presence of the substances on these lists at the threshold quantities does not necessarily mean there is an unacceptable level of risk. The risk becomes unacceptable when measures are not in place to manage the substances properly. The substances on "MIACC List 1" are those that could lead to major accidents if improperly handled, for example, vinyl chloride, benzene, and gasoline.

35.57 The second step is to establish an inventory of hazardous sites. MIACC is currently preparing an inventory of sites that have "MIACC List 1" substances in such quantity that they pose a risk of injury or death to people outside the site in the community. MIACC informed us that approximately 1,350 sites have been identified to date.

35.58 The third step is to assess the risk at each hazardous site. MIACC has begun the process of conducting a survey of process safety management. This is a self-assessment designed to determine the current level of awareness and use of major accident prevention and preparedness techniques at sites in Canada with potential for major hazards. Results of the survey are expected in the spring of 1998.

35.59 MIACC informed us that there are "MIACC List 1" hazardous sites located in approximately 450 communities. It is conducting a survey of the state of community emergency preparedness in these communities. This is being done, in most cases, through provincial agencies with emergency preparedness responsibilities.

35.60 MIACC's study of hazardous sites does not cover federal sites. Environment Canada indicated that it does not have responsibility for establishing an inventory of federal sites and that departments are responsible for their own sustainable development strategies and environmental management systems. However, Environment Canada has recently completed a study that identifies some federal sites where "MIACC List 1" substances are used or stored. The purpose of this study was to assess the actions that some federal departments have instituted in the areas of accident prevention, preparedness and emergency response. It should be noted that of the 20 sites studied, only one had stored a substance in a quantity exceeding the MIACC threshold limit. The study proposed a number of action items, including the development, in the federal government, of accurate inventories of federal facilities, the development of training programs regarding hazardous substances, increasing the awareness of MIACC tools and the development of a generic risk assessment document.

Correctional Service Canada - Custody of Inmates - 1994, Chapter 16

Background

35.62 Our follow-up consisted of reviewing status reports prepared by Correctional Service on the progress it has made on our recommendations. We also reviewed the supporting documentation prepared by the Service and had discussions with its officials.

Conclusion

35.64 Correctional Service has acted on all our recommendations pertaining to accommodation planning, including using cost/benefit analysis in evaluating future accommodation projects, incorporating double-bunking and shared-accommodation strategies in its accommodation planning, and appointing a full-time senior executive responsible for all accommodation-related functions. We noted that the Service has clarified its definition of double-bunking to exclude custom-designed, shared accommodation and is planning to review its overall accommodation policy. These changes, in our opinion, need to be carefully addressed in its accommodation planning process.

Observations

Security classification

35.66 Overall, an objective system uses instruments that allow for consistency in decision making and for decisions to be more easily monitored. These benefits are difficult, if not impossible, to realize if the classification instrument relies only on subjective, qualitative information (commonly referred to as "gut feel") that allows individual staff members to introduce biases that may or may not be valid.

35.67 At the time of our 1994 audit, Correctional Service had a two-step classification process, not unlike many other jurisdictions. Inmates entering the federal correctional system upon sentencing were given an initial custody rating followed by a reassessment on an annual basis. As we noted in 1994, we were concerned that although Correctional Service has an initial classification instrument (the Custody Rating Scale) that is quantitative and objective, reclassifications (the Security Classification Review) are qualitative in nature and based almost totally on the individual judgment of case management officers. As a result, inmates with similar risk factors have been reclassified and placed in institutions with different and, possibly, inappropriate security levels.

35.68 From these observations, we, and later the Public Accounts Committee, recommended that Correctional Service improve its security classification system to make it more objective and allow for monitoring of results.

35.69 Follow-up audit observations. Our follow-up review confirmed that immediately following our audit, the Service required all case management staff to attend a three-day risk assessment course. This course emphasized the use of statistically derived instruments over that of subjective methods, as well the importance of always taking into consideration the risk to public safety when making case management decisions. The Service estimates that between 2,000 and 3,000 of its staff have completed the course. As well, the Service has continued to affirm these messages at training workshops and staff conferences and during routine queries and site visits.

35.70 By the end of 1994, Correctional Service began to apply the Custody Rating Scale to all offenders entering the system. However, the Service also required its staff to immediately apply its Security Classification Review guidelines, which had been used to change or override inmates' security classifications as determined by the Scale. In our opinion, this practice at offender intake assessment has been a significant factor in the large number of Scale overrides. In August, 1997, the Service issued a directive to stop this practice.

35.71 In 1996, Correctional Service completed a validation study of the Custody Rating Scale that confirmed that the Scale is well designed. This same study noted a 26 percent override rate and estimated that only about half of the overrides were for legitimate reasons. This level of overrides, in our opinion, indicates a continuing need for the Service to better train its staff on what constitutes a valid override.

35.72 The Service has not yet implemented a process to monitor regularly the extent of overrides. It is therefore unable to respond to the Public Accounts Committee's request that the Service assess, on an annual basis, the needs and level of understanding of its staff with regard to the Custody Rating Scale, and provide additional training as needed. The Service told us that beginning in December 1997, it will be able to report the number of and reasons for overrides.

35.73 Since the beginning of 1996, a considerable amount of work has gone into developing a new, more quantitative reclassification instrument used for security classification review. Correctional Service's Executive Committee is expected to approve the new instrument for field testing in November 1997. Following approval, additional field consultations, testing and validation will be required in addition to automating the instrument on its Offender Management System. The Service estimates that this instrument will be in place for use, at the earliest, by the end of 1998.

35.74 From the Offender Management System, Correctional Service now has better information to evaluate periodically the extent of overplacements or underplacements, that is, the number of inmates who are placed in institutions of a security level greater or lesser than the classification assigned to them. This information has been presented to and reviewed by the Service's Executive Committee and, subsequently, has been used by the Service to make periodic reviews of minimum security institutions to ensure that the risk from these institutions is minimized.

35.75 We noted that Correctional Service intends to use these security classification data for population forecasting and accommodation planning as recommended in our audit. This will begin January 1999 because four to five years of data are normally required to begin accurate forecasting.

Inmate accommodation

35.77 Follow-up audit observations. Our follow-up review confirmed that all our recommendations in this area have been implemented. Correctional Service and the Treasury Board had agreed on a specific cost/benefit model in 1995, training for its use was provided to staff in both national and regional headquarters, and the model was used in the Service's 1996-97 and 1997-98 capital and accommodation planning. The position of Assistant Commissioner, Corporate Development was created and given responsibility for co-ordinating strategic and operational planning, of which accommodation planning is one aspect.

35.78 Correctional Service guidelines further indicate that double-bunking and shared-accommodation strategies have been incorporated as part of the planning process. Shared accommodation in housing other than cells has been developed and built by the Service. We noted that Correctional Service has clarified its definition of double bunking and is planning to review its inmate accommodation policy because, in its view, double occupancy is "an inappropriate permanent accommodation measure for good corrections."

Headquarters costs

National Parole Board - 1994, Chapter 17

Background

Conclusion

Observations

35.83 For the purposes of our follow-up audit, we selected at random two appointments made in 1996 (one full-time member and one part-time member) to determine how the new process works. We observed that there is a standard process for choosing candidates with a background in criminal justice and an ability to apply risk assessment theory and knowledge to parole decisions. Our review confirmed that the process is applied to both full-time and part-time members and consists of the following elements:

• Board member vacancies are advertised in the Canada Gazette ;
• applications are reviewed against detailed, specific criteria pertinent to carrying out the duties of a Board member;
• candidates are interviewed by a selection committee of senior Board officials who use a candidate-completed case study and interview guide to assess qualifications and suitability;
• a list of qualified candidates is submitted to the Solicitor General for consideration and recommendation to the Governor in Council; and
• appointments are made by Governor in Council from this list.

35.85 As a result, current training for new members consists of:

• an introductory reading package pertaining to risk assessment and the correctional environment;
• an orientation pretest to determine the participants' threshold knowledge of risk assessment theory and application;
• a 15-day (formerly 10-day) orientation program consisting of classroom instruction, observation at parole hearings and visits to correctional facilities; and
• on-the-job training provided by the Regional Vice-Chair, senior regional staff and experienced Board members.

35.87 The Board has developed a variety of mechanisms to provide feedback to Board members, including information on successes and failures. Since our audit, the Board has developed a variety of mechanisms for assessing the performance of the Board as a whole. This includes a biannual Performance Monitoring Report that provides yearly data on decision trends, recidivism rates, success rates, rates of return and post-warrant expiry recidivism, by release type, offender risk profile and region. We acknowledge this performance assessment and reporting to be a major step forward from where the Board was in 1994. The Board recognizes, however, that while it is better able to collect and analyze information, it needs to continue to integrate this information into policy and practice.

35.88 Another major form of feedback the Board has implemented since our audit is a performance appraisal system to formally evaluate individual Board members' performance each year. For the purposes of this follow-up audit, we selected a total of five Board member performance appraisals (three full-time members and two part-time members). All of the appraisals followed the established framework and criteria developed by the Board in 1995 and incorporated a critical review of performance. They included examples of strengths and weaknesses as well as plans of action to remedy any weak areas of performance.

35.89 In 1995, the Board completed a comprehensive review of all reports concerning offenders on parole who committed further offences (usually violent) with an aim to identify areas that needed to be addressed through policy changes or training. In July 1996, a review of Board policy on inquiries and investigations was completed and new procedures subsequently distributed. In addition, we found evidence that the Board uses the results of investigations as training aids and in the appraisals of individual Board members. In March 1997, for example, a training session on lessons learned was given at the annual general meeting. These were drawn from the experience gained internally through the Board's appeal division case reviews, audit and investigation case reviews, and legal challenges presented to the Board.

35.90 The Board has carried out several reviews of its operations to determine whether it is concentrating its time and effort in the most appropriate areas. In 1995, as part of the government's program and agency review exercise, the Board decided, consistent with the regulations, to reduce the number of members required to render decisions for cases in which there is less risk to public safety. For example, now only one member may be assigned to carry out a paper decision on accelerated parole review and only two members are required to render appeal division decisions. The Board is considering more changes.

35.91 Immediately following the 1994 audit, the Board, with members of Correctional Service Canada, formed a working group to determine the kind of information that the Board needs for decision making and the best format in which to present it to help members make appropriate decisions. This initiative has resulted in the development of the Risk Assessment Report that Correctional Service uses to recommend whether or not parole for an inmate is appropriate. We did not carry out further examination to determine the extent of this new report's implementation. In our November 1996 chapter, Correctional Service Canada - Reintegration of Offenders, we noted that there have been problems in implementing the report. In our follow-up, we plan to conduct a more complete examination of the progress in implementing the Risk Assessment Report and the recommendations pertaining to ensuring that all relevant information is available for decision making.

Environment Canada - Managing the Legacy of Hazardous Wastes - 1995, Chapter 2

Background

35.93 We focussed on Environment Canada's management of its responsibilities under the $250 million joint federal-provincial National Contaminated Sites Remediation Program (NCSRP), which began in 1989-90. The objectives of the five-year program were to ensure the clean-up of high-risk orphan sites in Canada based on the "polluter pays" principle, to promote the development of the Canadian environmental technology industry and to clean up federal sites. We were concerned that when this program ended on 31 March 1995, adequate information on federal contaminated sites did not exist to properly inform Parliament of the health and environmental risks and clean-up costs.

35.94 We also examined the Department's administration of the PCB regulations on use, storage, export and treatment/destruction under the Canadian Environmental Protection Act . Under the Green Plan approved in 1990, the destruction of all federal PCBs was to be accomplished by 1996. However, the PCB Destruction Program ended on 31 March 1995 without this task having been completed.

Scope

35.96 We conducted interviews and reviewed supporting documentation as well as other material prepared since the completion of the status reports to further assess the extent of the progress up to our cut-off date of 30 May 1997.

Conclusion

Observations

Parliament needs adequate information on the risks and clean-up costs of federal contaminated sites

35.99 Since the May 1995 chapter, two committees have been formed - the Contaminated Sites Management Working Group (CSMWG) and the Environmental Accountability Partnership Policy Sub-committee (EAP Sub-committee) - to develop a common federal approach to dealing with federal contaminated sites.

35.100 The CSMWG committed itself to a workplan, including the task of Site Assessment and Associated Liability. As part of that task, the Working Group developed a definition for contaminated sites and a template to record summary inventory data. In addition, the CSMWG is supporting the Federal Committee on Environmental Management Systems in its efforts to assist with the development of environmental management systems and sustainable development strategies, both of which should contain components addressing contaminated sites. The EAP Sub-committee has developed a draft general policy on contaminated sites, which is currently under discussion; however, it has not established any departmental deadlines, including one for the implementation of the inventory template.

35.101 In Chapter 22 of our November 1996 Report, we looked at departments' progress in identifying, assessing and remediating federal contaminated sites. We concluded that accurate and complete summary data of departmental inventories and estimated costs, both by departments and centrally, will likely not be available for some time. This information is needed to ensure that the high-risk sites posing the greatest risk to public health, safety and the environment are among the first to be remediated.

Action plan needed for federal contaminated sites

Disclosure of potential federal environmental liabilities in the financial statements

35.104 There has been limited progress on the Auditor General's recommendations in the Observations on the Summary Financial Statements. In Chapter 22 of our November 1996 Report, the Treasury Board Secretariat stated that it "does not expect to issue and implement a policy before the 1998-99 fiscal year." We understand that the Secretariat is in the process of developing an environmental accounting policy. The draft policy is to be distributed to stakeholders for review and comment in 1997-98 and is to build upon the recommendations of the CSMWG for the consistent recording of information in the inventory of contaminated sites. The inventory is to be based upon the National Classification System for Contaminated Sites issued under the aegis of the Canadian Council of Ministers of the Environment. The assessment process for placing a site in the inventory includes the requirement to estimate the costs of remediation. The Treasury Board Secretariat states that when the assessment process is substantially complete, this inventory will facilitate the quantification of the government's liability for incorporation in the summary financial statements.

35.105 While we have no reason to doubt that the Treasury Board Secretariat will implement a policy by 1998-99, we remain concerned about departmental readiness, given the lack of progress to date.

Treasury Board Secretariat's response : In developing an appropriate accounting policy, the Secretariat is working through an interdepartmental committee that, in turn, is developing a policy for the management of contaminated sites. Through this process, the Secretariat is confident that all departments responsible for contaminated sites will be ready to implement the accounting policy by 31 March 1999.

Destruction of federal PCBs

• the failure of Environment Canada to report to Parliament on progress made to date on PCB destruction and to provide a realistic assessment of risks and costs needed to destroy the total federal inventory; and
• the termination of Environment Canada's leadership role in the management of PCB destruction with the end of the PCB Destruction Program at 31 March 1995.

35.108 As of January 1997, PWGSC reported that less than two percent of federal PCB waste remained in storage from Environment Canada's original 1995 waste estimate of 5,600 tonnes. Since the May 1995 Report, the number of PCB storage sites has been reduced from 495 (1993) to 227(April 1997). However, the amount of federal PCB waste in storage and the number of storage sites are dynamic. The numbers will change as PCBs come out of service and as PCBs in storage are destroyed. Storage sites are beginning to fill up again with PCBs coming out of service. As well, PWGSC's destruction program has recently been stalled due to difficulties at the Alberta Special Waste Treatment Centre (Swan Hills).

35.109 PWGSC would like to continue to offer PCB destruction services to federal departments and agencies because of the continuing need and to seek cost efficiencies. As of May 1997, PWGSC senior management was being briefed on the PCB destruction program and options for continuing this service.

35.110 Environment Canada has committed itself to providing a report to Parliament in 1997 that will address the progress made to date on PCB destruction and the risks and costs associated with the remaining federal PCBs. At 30 May 1997, the report had not been drafted, nor was there a target date for its completion.

Federal Radioactive Waste Management - 1995, Chapter 3

Background

35.112 Since we issued our Report, developments have occurred that move Canada closer to finding long-term solutions for its radioactive waste.

35.113 Natural Resources Canada (NRCan) has developed a policy framework for radioactive waste and has consulted with stakeholders on institutional and financial arrangements for radioactive waste disposal. It has also signed a memorandum of agreement with Ontario to assign responsibilities for decommissioning and long-term management of any "abandoned" uranium mine sites and uranium tailings. The Canadian Environmental Assessment Agency (CEAA) Panel on the Nuclear Fuel Waste Management and Disposal Concept completed its hearings and is developing its recommendations. NRCan has been negotiating for a site for historic low-level radioactive waste and has extended the mandate of the Low-Level Radioactive Waste Management Office.

35.114 This year we followed up on each of the recommendations in our May 1995 Report chapter. We reviewed NRCan's July 1996 status report to the Public Accounts Committee on action taken in response to our recommendations. Departmental officials provided us with an update on this status report. We discussed ongoing progress, obtained and reviewed supporting documentation and incorporated relevant information into this follow-up report up to 1 September 1997.

Conclusion

Observations

A policy framework has been developed for radioactive waste

35.117 In October 1996, NRCan issued a discussion paper to seek stakeholder views on the federal oversight role, on how the stakeholders would organize for waste disposal and on how they propose to fund disposal programs. At the time of our follow-up, the Department had received feedback, but indicated that it will wait for the CEAA Panel's recommendations before continuing work on the institutional and financial arrangements.

Decision on a long-term solution for nuclear fuel waste awaiting results of the CEAA Panel

Negotiations continue on a disposal facility for historic low-level radioactive waste

Mandate of Low-Level Radioactive Waste Management Office has been extended

AECB has initiated action to license the pre-1976 uranium mines and tailings sites

35.122 AECB believes that none of the pre-1976 tailings sites represents an immediate threat to health, safety or the environment but that appropriate steps must be taken to ensure their long-term acceptability.

Decommissioning and long-term management responsibilities of any "abandoned" uranium mine sites and uranium tailings are resolved for Ontario

35.124 NRCan officials have discussed with Saskatchewan officials the possibility of a similar initiative for their province.

NRCan and AECB have a common understanding about the underlying assumptions of volumes and cost data

Disclosure of potential federal environmental liabilities in the Notes to the Financial Statements includes liabilities for radioactive waste

Industry Canada - Business Assistance Programs in Transition - 1995, Chapter 14

Background

35.128 During our audit, the government announced that it was terminating a number of programs including almost all those that we had selected, such as the Sector Campaigns and the tourism sub-agreements. Tourism Canada was wound up but was replaced by the Canadian Tourism Commission in January 1995. Only the Business Service Centres remain today.

35.129 We recognize that Industry Canada's programs have been modified considerably and that, moreover, there is a considerable reduction in the grants and contributions budget of the Department. However, we believe that the recommendations formulated in our 1995 chapter are still relevant. On the one hand, Industry Canada must continue to pay attention to its old programs. Even though they have been terminated or wound up, they are still making payments to businesses that will be repayable to the government over a number of years. On the other hand, the good management principles we recommended also apply to the new contribution programs managed by the Department.

35.130 Our follow-up consisted of a review of the Department's reports on its progress in response to our 1995 recommendations. We also examined supporting documentation and held discussions with the responsible managers.

Conclusion

Observations

Financial contributions programs

35.133 Following the 1995 Budget, Industry Canada decided to centralize program delivery in Ottawa. A program and repayment section was established in the fall of 1995 to handle repayments for almost all terminated programs.

35.134 Departmental representatives indicated to us that the policy and operational improvements made during 1996-97 were instrumental in convincing businesses that the Department is serious in its efforts to ensure repayments of contributions. For example, businesses that are awarded new contributions under other departmental programs must first repay any amounts they owe before receiving additional payments. The representatives also informed us that additional improvements are being made. They recently put into place a program to audit the sales reports of businesses, to ensure the accuracy of the sales figures on which the royalties owing are calculated. According to the Department, $69 million was recovered in 1996-97, which represents twice the historic average of past years. The Department hopes to collect $74 million in 1997-98.

Assessment of applications

35.136 Technology Partnerships Canada is a special operating agency that was established in March 1996. Its budget is $200 million for 1997-98. By making contributions available to the private sector, Industry Canada shows that it wants to share the risks with private industry, but also to profit from its investments.

35.137 Unlike previous programs, all project decisions are reviewed by the Program Services Directorate. This section is separate from the decision-making process and reports to the Executive Director. We believe that the establishment of a separate unit to handle quality control within TPC should help respond to the observations in our 1995 Report. However, the limited work carried out as part of the follow-up does not permit us to make more detailed comments on this subject.

Direction of government assistance programs

35.139 In February 1996, the Program and Service Board approved changes to the departmental policy, limiting situations where contributions are no longer repayable. These changes included the elimination of the $100,000 threshold, below which contributions were rarely repayable. The policy also states that possible exemptions are no longer generally applicable but are instead to be used only when fully justified. We are satisfied with the actions taken by the Department to amend its policy.

Canada Business Service Centres

35.141 In 1995, we recommended the development of a common vision, national guiding principles, and a limited number of basic service standards. In February 1996, the CBSC Committee of Managing Partners approved a vision statement for the CBSC initiative. That document set out the mission, guiding principles and service standards for the CBSC network. The next phase involved applying the standards throughout the network.

35.142 We believe that there is still work to be done on the publication of standards. Only two CBSCs had published their service standards on the Internet at the time of our follow-up. Departmental representatives informed us that a CBSC Internet Working Group is currently developing national service standards to be added to the CBSC main Web site.

35.143 We noted in 1995 that the information given on certain programs was out-of-date or inadequate in some cases. We recommended that the Secretariat help the CBSCs to resolve these problems. As a result, standards for the quality and timeliness of federal information have been used by the National Secretariat since April 1996. They involve monthly updating of the most frequently used 20 percent of federal information and updating the remaining information within a period of not more than six months. There is a system to track the number of times users access information about each program on the Internet. The National Secretariat is planning to extend the compilation of data by incorporating inquiries from sources other than the Internet. According to departmental representatives, the Secretariat maintains frequent contact with its partners and monitors all government announcements to identify new programs.

Travel and Hospitality - 1995, Chapter 7

Background

• consider providing public information on travel;
• pursue the implementation of automated travel systems and help specify the information needed to manage travel better;
• assess and further clarify the application of policy for business-class travel and travel bonus points;
• consider publishing principles of behaviour in government travel; and
• assess possible changes to the Travel Directive and Travel Guide.

Scope

Conclusion

35.148 Work is continuing on the development of an improved Travel Expert System that can be used by managers and travellers to reduce the cost of travel administration and provide better information for improved accountability.

35.149 The Secretariat cannot unilaterally change the Travel Directive, and discussions are ongoing with employee unions to address the terms and conditions of the Directive. Reduction of administrative costs and improved technology and systems are the goals of the Secretariat throughout these negotiations. The impact of initiatives that may result from these negotiations is not known at this time.

35.150 The current travel service contract expires 31 December 1997. PWGSC has indicated that the new contract will focus on service to clients and reducing costs through the use of technology. As the final contract had not been awarded at the time of our follow-up, we are not yet able to assess the impact of this initiative.

Observations

Public information on travel

Automated travel systems

Business-class travel and travel bonus points

Publishing principles of behaviour for government travel

Changes to the Travel Directive and Travel Guide

35.156 Proposals for changes to the Directive require agreement by public service unions, and discussions on those proposals have commenced. The Travel Guide is a tool used to explain the Directive and it should reflect, in a timely manner, changes made to the Directive.

Measuring the use of travel toward a goal of least-cost travel

• A new automated travel management reporting system, "Reportlink", was implemented across 68 government entities and provides standard, as well as customized, travel reports to meet clients' specific needs.
• Random reviews of airfares were introduced.
• A new Low Fare Guarantee process was negotiated with the current Government Travel Service contractor in April 1996.
• As the current contract expires in December 1997, PWGSC issued a Request for Proposal (RFP) for a new travel service contractor in July 1997. The RFP aims at maximizing service to travellers, reducing costs through the use of technology (for example, automated booking) and providing a distinct international service. The RFP is based on a fee-for-service regime, which will enable the government to negotiate preferred airfares with air carriers.

Atlantic Canada Opportunities Agency - 1995, Chapter 18

Background

35.159 We recommended that, during the project approval process, the Agency ensure that expected project results are specified and clearly linked to the objectives of the programs. The risks of projects not achieving their goals needed to be considered in the type of assessment procedures employed. We also had concerns about the adequacy of monitoring of project progress and results.

35.160 Since the 1995 audit, the Agency has replaced one of its major programs, the Action Program, with the Business Development Program. In addition, the Agency has implemented the government's policies covering the repayability of support to business.

Scope

Conclusion

Measuring and reporting results

35.163 The Agency has made progress in improving some of its results measurement and reporting processes. The 1996 Performance Report to Parliament clearly indicates what results are being reported and the means by which those results are being reported. However, we did not audit the information presented to us by the Agency; therefore, we were unable to determine whether the Agency has made improvements in the measurement of the numbers of jobs created and maintained. The processes used for measuring results of the COOPERATION Program remain largely unchanged from 1995. Further, the program objectives for both the Business Development and COOPERATION programs remain broad, making subsequent program evaluation difficult.

Project management and decision making

Observations

Measuring and reporting results

35.166 Our review of the studies that support the performance measures reported in the Agency's 1996 Performance Report to Parliament and the 1997-98 Main Estimates found that the Agency has attempted to address the concerns raised in our 1995 audit. Without conducting an audit of the information presented to us by the Agency, we were unable to determine whether the Agency has made improvements in the measurement of the numbers of jobs created and maintained. In its econometric modelling exercise, which converts the number of jobs created and maintained into the total economic impact on the Atlantic economy, the Agency continues to use the assumption that all of the jobs created by the program will last for a period of 10 years. As in 1995, we were not able to find support for this assumption.

35.167 The Agency has recognized some of the inherent difficulties associated with the use of a client survey as the primary source of results information. To address this matter, the Agency is in the process of establishing a benefit monitoring program. This program is designed to ensure that Agency staff actively monitor key expected results, such as jobs created and maintained, for a sample of Agency projects and thereby provide an estimate of the results achieved for all of the Agency's program activity. These results will be corroborated through ongoing client surveys. We believe that these planned changes could greatly benefit the results measurement processes of the Agency if they are implemented appropriately.

35.168 The evolution of the Agency's performance measurement practices has occurred at a time when several important program changes, such as repayability of assistance to business, have been implemented. There has been an apparent lack of emphasis on program evaluation in favour of performance measurement. Although the latter is a very useful tool, it is not a substitute for program evaluations that provide assessments of the rationale for programs, the impacts and effects that can be attributed to them (including their success in achieving program objectives) and the cost-effectiveness of alternative means for achieving objectives. In this context, we note that the objectives for both the new Business Development and the COOPERATION programs remain very broad, which will make it difficult for future program evaluations to assess the extent to which objectives have been met.

Project management and decision making

35.170 The Agency has indicated that two of its regions are employing new means of monitoring Business Development Program projects. For the one process that we reviewed, the monitoring of projects was based on the risks associated with effective implementation (for example, financial and human resources available to support the project) and the attainment of the project objectives. Account managers are expected to have an understanding of client operations and the progress and results of projects based on the risks associated with the project.

35.171 The COOPERATION Program now funds only non-commercial projects. Thus, the Agency considers the individual eligibility criteria for COOPERATION agreements to be more relevant than the key economic development factors mentioned earlier. While the Agency has moved to streamline the number of COOPERATION agreements with individual provinces, agreement objectives are still not stated in a clear, measurable and results-oriented manner. The Agency has indicated that it will monitor COOPERATION projects through the benefit monitoring program in the future. Results measurement will be through program evaluation of individual agreements and surveys. With the exception of the planned implementation of the benefit monitoring program, the assessment and monitoring processes used by the Agency are not significantly different from those found in the 1995 audit. The Agency had developed the means to acquire information on recipients and activities for COOPERATION agreements. Recently, it has had to revise its approach to information gathering because of systems changes on the part of several of its provincial partners.

Federal Office of Regional Development-Quebec - 1995, Chapter 19

Background

35.173 Our 1995 chapter highlighted a number of major challenges that FORD-Q would have to meet in order to successfully implement its new program and achieve the expected results. We also made a number of observations and formulated recommendations on the delivery of financial assistance programs, ongoing performance measurement and program evaluation.

35.174 Our follow-up work consisted primarily of discussions with representatives from the Department regarding measures taken in response to our recommendations, and a review of the supporting documents. We did not attempt to evaluate the measures taken.

Conclusion

35.176 However, there is still much to do. A large number of measures have been implemented recently or will be over the course of the next year. The Department will need to persevere in its efforts to bring to completion all the projects that have been initiated.

35.177 The measures taken over the last two years demonstrate a willingness to put into place mechanisms that will help bring about a performance-oriented management culture. Changing management methods and attitudes takes considerable work. It will therefore require a great deal of time and a sustained effort to ensure the success of this initiative.

Observations

The challenges of implementing the new program

35.179 FORD-Q is currently reviewing its activities and services in the light of the past two years' experience, with a view to specifying them and determining the most efficient delivery methods. In our opinion, the Department needs to take advantage of this opportunity to update the management framework for the IDEA-SME program in order to ensure that its objectives, priorities, services and conditions remain relevant.

35.180 Computerized systems. During our follow-up, we noted that FORD-Q had experienced problems with the installation of two computer systems that are important to the smooth operation and sound management of the program. These were the system used to deliver services and collect data on results achieved (AIDER) and the financial information and contribution management system (G-MAX).

35.181 In the area of program operation, the delivery of non-financial services could be improved, especially in the provision of strategic information and advisory services. Accessing the information is difficult because the computer equipment is not powerful enough. To overcome these deficiencies, the Department has taken a number of steps over the past year. It has added to its computer infrastructure, determined the equipment requirements for its regional offices and begun the process of acquiring equipment. However, it has experienced difficulties in fine-tuning the AIDER system, and this has resulted in several delays in implementation since 1995-96. It is now expected to come on-line by the spring of 1998. The Department appears to have taken the appropriate steps to avoid additional delays and problems, by developing an implementation plan and carrying out periodic follow-up.

35.182 The expectations of management and staff for the AIDER system are very high because of its importance to the efficient delivery of services and the improvement of management information. In light of the considerable sums invested to date to get the system up and running and the difficulties experienced in the past, departmental management will need to continue giving special attention to the implementation of this system. Among other things, management needs to ensure that the required computer equipment is in place at the proper time, that users receive adequate training and that appropriate controls are in place to ensure the accuracy and completeness of the information.

35.183 With respect to program management, FORD-Q has set up a number of mechanisms and systems to manage activities related to IDEA-SME but management does not have available complete and regular management information. In 1995, FORD-Q replaced its contribution management system and experienced various difficulties in extracting data and preparing reports. These problems seem to have been resolved now. Further, the lack of non-financial services data in the system and delays in implementing AIDER have meant that management until now has received little information on the provision of strategic information and advisory services.

The criteria used in analyzing projects remain general

35.185 The Department has taken a first step in determining its risk by setting a range for the loss rates on its repayable contributions, although this range is quite broad (between 35 and 50 percent). The Department needs to specify to a greater degree its expectations in this respect and ensure that project analysis takes into account the possibility of recovering the contributions.

Important steps toward results-oriented management

35.187 An evaluation policy was to be approved by the fall of 1997. The Department has conducted evaluations of a number of its programs since our audit. We have noted improvements in that both negative and positive consequences have been examined, thereby providing a balanced picture of the effects of the programs that are being evaluated.

35.188 As well, the Department is now presenting performance information in its Part III Estimates, which constitutes an important addition. However, improvements are still required. The reader now sees data but these data are not linked to any pre-established objectives. The reader is therefore unable to judge whether or not the results are acceptable.

35.189 These initiatives require a major cultural change and a long-term effort in order to ensure success. It is too early to conclude that FORD-Q is in a position to analyze the results of its programs and make decisions based on the results obtained. However, the elements that have been put in place reflect major progress toward the implementation of results-oriented management.

Western Economic Diversification Canada - 1995, Chapter 20

Background

35.191 We began our audit at a time when the Department had many activities, but its Western Diversification Program was its primary focus. By the completion of our audit, the Program's termination had been announced. Consequently, wherever possible, our recommendations focussed on future programs and the challenges of administering the Western Diversification Program as a significant "legacy" program where both payments for previously approved assistance and collections of repayable assistance would continue for years.

35.192 Our follow-up involved reviewing Western Diversification's status report on action taken. We also reviewed supporting documents and conducted interviews to discuss and assess the extent of progress.

Conclusion

Observations

Organizational design

35.195 The Department indicated, at the time of our audit, that an examination of administrative functions then under way would help to address our recommendation. The Department more recently indicated that the Deputy Minister's office will be located in Edmonton effective 1 November 1997, and that the future design of the Department should be finalized by the fall of 1997.

Collecting repayments from legacy programs

35.197 In 1995 the Department used a combination of its own and full-time contract staff to collect repayments. We recommended that the Department seek ways to minimize collection costs. The challenge remains to collect repayments in a cost-effective manner.

35.198 The Department has considered three collection options:

1. continue to collect repayments internally;

2. sell the portfolio, or have a third party collect it on the Department's behalf; or

3. use some combination of the two.

35.200 As the Department has apparently decided to move to third-party collection of repayments, it needs to finalize the collection arrangements.

Performance measurement and reporting

35.202 Since 1995, performance reporting in government has assumed a higher profile with the new annual report by the President of the Treasury Board. This report focusses on the importance of measuring and reporting the results of government programs, on individual performance reports by 16 pilot departments and agencies in fall 1996, and on performance reports by all departments including Western Diversification in the fall of 1997.

35.203 Western Diversification faces the challenge of having to assess the performance of a new portfolio of programs. Furthermore, many of these new programs are delivered by third parties on the Department's behalf, which can complicate the process.

35.204 We reviewed a number of departmental documents to determine the Department's readiness to measure and report performance, and guidelines prepared by its Audit and Evaluation Branch that provide a basis for that effort. Other documents, such as a draft Service Partnerships Business Plan and a report on corporate realignment, contribute to its readiness.

35.205 Overall, the Department is moving toward strengthening its capacity to assess and report on performance, but actual performance measurement and reporting activities to date have been few. In our view, however, given the fundamental changes in the Department's programs over the last two years, progress has been reasonable.

35.206 Procurement advocacy. Our 1995 Report also made specific recommendations concerning the Department's reporting of its efforts in advocating the procurement interests of Western Canada in the government's major Crown projects. The Department had reported quantitative information on the West's increase in the share of the "industrial regional benefits" from major Crown projects since Western Diversification's inception.

35.207 We recommended a fuller discussion elaborating on quantitative measures, and enhancing the methodology used to measure and report the benefits of procurement advocacy. Subsequent to our recommendation, Western Diversification ceased publicizing this type of information. Currently, a group of departments involved with procurement are participating in an overall evaluation of the government's industrial regional benefits policy. The evaluation is led by Industry Canada in two phases, with completion of phases one and two planned for fall 1997 and spring 1998 respectively. When completed, the evaluation will presumably allow Western Diversification to report better information on the results of its procurement advocacy efforts.

Industry Canada - Regional Development Programs - 1995, Chapter 21

Background

35.209 Our follow-up work consisted of a review of the Department's reports on its progress in response to our 1995 recommendations. We also reviewed and discussed supporting documentation with the responsible managers.

Conclusion

Observations

FedNor

35.212 Program management. In 1995 we recommended development of a streamlined set of guidelines and criteria for assessing applications, differentiated by their size and risk. We also recommended that application documentation be simplified, especially for small projects. Since the restructuring, FedNor now works through partnerships with stakeholders such as financial institutions, community development corporations and the private sector.

35.213 Assistance provided to businesses takes two forms: contributions and loan/investment fund loss reserves. Programs are available both for private companies and for community organizations. FedNor has also set up a number of programs to meet the needs of certain target groups. Accordingly, we are satisfied with the efforts made by FedNor to better segment clients according to their needs.

35.214 Monitoring of projects. We recommended establishment of clear guidelines for monitoring project performance and for reporting project results in the final project report. In view of the creation of new programs and the use of new program delivery mechanisms, we believe it is still too early to determine whether FedNor is adequately monitoring its projects.

35.215 Performance and program evaluation. The recent development of an evaluation framework for the FedNor Business Incentives Program has addressed the recommendation we made in 1995. Management has informed us that it will begin collecting information identified in the evaluation framework to assess performance in late autumn 1997.

Canada-Quebec Subsidiary Agreements on Industrial Development

35.217 As a result of cuts and other adjustments, federal funding authorized under the two agreements now amounts to nearly $350 million. At the time of our follow-up, the government had committed nearly all of its funds.

35.218 Documenting support for decisions. In 1995, we looked at documentation in support of funding decisions. We found cases where there was little analysis of key issues. In view of the large sums invested, we recommended that the Department ensure that decisions to fund projects are supported by persuasive assessments in project files.

35.219 Shortly before our audit, Industry Canada was assigned responsibility for managing both of the subsidiary agreements. It had assumed this responsibility previously, from 1985 to 1991, when the Federal Office of Regional Development-Quebec (FORD-Q) took it over.

35.220 Industry Canada has added another level to the project approvals process. This level, the Program and Service Board, is a compulsory step for major projects undertaken under the various departmental programs. The Board, made up of senior departmental officials, is responsible for ensuring that certain specific elements have been examined before approval is given.

35.221 In its July 1997 progress report, Industry Canada stated it is continuing to ensure that arguments in support of the funding recommendations are clearly documented in project files. Departmental representatives told us that the Board's concerns are taken into consideration when the recommendation for project approval is being prepared. Our review of a small sample of files suggests that the documentation supporting project funding recommendations has improved.

35.222 Evaluation of the subsidiary agreements. During our audit, we noted that the first agreement had been renewed and the second agreement signed without a full evaluation of the first agreement having been conducted. We recommended that the Department complete an evaluation of both subsidiary agreements.

35.223 In autumn 1995, the Management Committee responsible for the subsidiary agreements agreed to conduct a joint evaluation of the two agreements in 1996-1997. The evaluation was assigned to a private sector company that was to present a first draft by 31 March 1997. At the time of our follow-up, the final version of the evaluation was about to be approved by the federal and provincial members of the Management Committee.

35.224 Even if the Canada-Quebec agreement is not renewed after 31 March 1998, we believe that the conclusions of the evaluation can be used, as appropriate, for developing other similar programs.

Human Resources Development Canada - Support for Training - 1995, Chapter 22

Background

35.226 The objective of the 1995 chapter on support for training was to provide Parliament with reasonable assurance that the Department had established satisfactory procedures to measure and report the effectiveness of its training programs, where procedures could appropriately and reasonably be implemented. We focussed on the training measures found mainly under two employment programs administered by the Department: Employability and Labour Market Adjustment. We also reviewed labour market information activities.

35.227 The chapter expressed some concerns regarding the limited information on current demand/supply conditions in the labour market and on occupational shortages/skills gaps. It also identified shortcomings in the collection of data on training outcomes and in the resource allocation models for training support. In the area of program evaluation, we noted that the majority of evaluations examined were of acceptable quality within their scope. However, we recommended that more emphasis be put on measuring the effects of the training programs in helping to meet the skill needs of the economy, on surplus occupations and on worker mobility. We also highlighted the importance of measuring interprogram effects. Finally, we stressed the need to ensure that departmental evaluations of developmental initiatives address the full range of key evaluation issues and use the most reliable measurement methods possible.

Conclusion

35.229 We have concluded that the Department has made substantial progress in many of the areas that we audited in 1995. For instance, in the area of results measurement, it has implemented new systems that will generate more information on the performance and outcomes of active labour market programs. The Department has also taken significant steps to improve labour market information and is now collecting and publishing information on current labour market conditions. Finally, the Department has indicated that it would undertake an assessment of projections of labour market demand and supplies and that it would make these measurements on a regular basis.

Observations

Labour market information

35.231 Under the new arrangements, the federal government retains overall responsibility for the development of labour market information, although some provinces have now assumed some responsibilities in this area. Until all the Agreements are finalized and implemented, how these responsibilities will be shared will remain unclear. However, HRDC will likely remain a significant producer and provider of labour market information. For instance, it is committed to publishing, on a more frequent basis, Job Futures , a detailed compendium of information on economic conditions that face newcomers to various occupational groups. This product is one of the significant outputs of the Canadian Occupational Projection System (COPS).

35.232 In 1995, we noted the lack of comprehensive ongoing information on current occupational demand and supply conditions in the Canadian labour market, especially in terms of prevailing trends in skill shortages. Since 1995, the Department has revised the COPS methodological framework, in order to make it capable of providing labour market information on prospective future occupational imbalances between demand and supply. Together with this important first step, another positive development in this area is the production by the Department of information on current labour conditions. HRDC publishes this information in Job Futures in the form of qualitative ratings.

35.233 In our 1995 audit, we identified the need for the Department to conduct an analysis of causal factors related to skill shortages. The Department still does not produce such analysis. We also noted that the Department intends to establish a link between the experience of those who have recently left school and the current occupational labour market situation for the occupations in which graduates generally find jobs. We therefore encourage the Department to do the same for graduates of federally sponsored active labour market programs in order to help determine the effectiveness of such programs.

35.234 Finally, in 1995, we also concluded that COPS forecasts should be compared with what actually occurred and an analysis of the variances between the two should be undertaken. The purpose in this instance was to provide a tracking of the reliability of COPS, with a view to improving its usefulness and more clearly identifying labour market adjustment processes. The Department has stated its intention to proceed with an assessment of its forecasting record sometime past the midpoint of its first forecasting period (using 1995-98 data) and to implement these performance procedures on a regular basis thereafter.

35.235 Although there has been encouraging progress, we are concerned about the limitations of current information on the above-noted occupational imbalances in the economy. Our 1995 audit stated that a program evaluation undertaken by the Department in 1991 estimated, from an employer-based survey, that during the 1988 to 1990 period there were 300,000 hard-to-fill vacancies annually. We also noted in our 1995 audit that the lack of causal analysis in this area generates a risk that the federally sponsored training activities may not adequately respond to the economy's skill needs; as a result, the effectiveness of such active labour market programs in improving the employability of workers and lowering unemployment levels may be reduced.

Results measurement

35.237 The Department has taken some steps to address our concerns and has developed information systems that will collect data with the intention of facilitating client documentation, increasing the capacity to collect and report data on employment results, increasing efficiency and improving the reliability of information. For this purpose, the Department has developed several projects.

35.238 One project that has been implemented is Contact IV, a case management service delivery microcomputer system that is operational in approximately 860 third-party agencies providing employment services to HRDC clients. It facilitates documentation of client information and employment data for accountability and reporting purposes. A similar system, called Window Ness (NESS-GUI), has been developed for internal and provincial staff clients. The Department is also in the process of developing the Human Resources Investment (HRI) Client Data Set, which will provide national headquarters, regions and Human Resources Centres of Canada with consistent management information on clients who participated in various HRI activities. The integration with financial systems is presently under development. Once completed, it will allow the monitoring of unit costs per trainee.

National allocation of program resources

35.240 The Department is pursuing discussions with provinces and territories in this regard. Most of the Labour Market Development Agreements reference the possibility of a multilateral process for the review of the Employment Insurance resource allocation formula. Under the federal-provincial/territorial Forum of Labour Market Ministers (FLMM), jurisdictions have agreed to place the review of the Employment Insurance formula on the Forum's 1997-98 workplan. HRDC is the federal co-chair for the multilateral review process, while the FLMM has designated Ontario as provincial co-chair. The Department indicated that parties will be working together starting in fall 1997 to determine an appropriate process, options and timetable for implementation.

Program evaluation

35.242 Under the new Labour Market Development Agreements, HRDC and its provincial partners have agreed to conduct evaluations of program effectiveness based on the Agreements. The Agreements outline the obligation to develop evaluation frameworks that will guide the development of evaluation processes adhering to recognized evaluation practices. The Department has already developed guidelines outlining the core federal issues for the formative and summative evaluations and shared these with the provinces to initiate a collaborative evaluation process. These guidelines address the relevance, design, delivery, success and cost-effectiveness of the programs and services provided under the Agreements.

Indian and Northern Affairs Canada - On-Reserve Capital Facilities and Maintenance - 1995, Chapter 23

Background

35.244 Capital facilities and maintenance continue to be a stated priority of the Department in its efforts to improve living conditions on reserves across Canada. The applicable 1997-98 budget is $806 million.

35.245 The objectives of our 1997 follow-up were to determine the status of the 1995 recommendations, to identify emerging improvements, and to note any significant other matters that came to our attention.

35.246 Accordingly, we reviewed and tested the May 1997 status report provided by the Department on its progress in implementing recommendations. We considered all the reported action and focussed on selected aspects of project planning, implementation and evaluation in a major region of the Department. These aspects offer good opportunities to achieve better value for money while improving living conditions on reserves. We also inquired into current maintenance practices.

35.247 The follow-up included a review of 13 capital facilities projects having a cumulative value of approximately $43 million. The region classified these as "major capital" projects.

Conclusion

35.249 Furthermore, additional concerns were noted in the region about the lack of agreement between the Department and First Nations on whether bid tendering should be used to select construction contractors.

35.250 We encourage the Department to implement the applicable recommendations fully and expeditiously and to resolve the issue of bid tendering. This should help the Department obtain assurance that good value is achieved for the funds provided and that improvements in living conditions on reserves are realized.

Observations

Results of project reviews

35.252 Overall, the audit sample disclosed no significant change from the findings reported in 1995. For example, improvements are still needed in such areas as project risk assessments, monitoring, evidence of project completion and evaluation of project results.

35.253 A case example of additional issues is reported separately in Chapter 36 - Other Audit Observations.

Maintenance problems remain

35.255 Up to an estimated $270 million, or one third of the annual capital facilities and maintenance budget, is allocated to maintenance. Proper maintenance is needed to obtain appropriate levels of service from the asset and to avoid premature and costly recapitalization. We therefore expected timely action from the Department to address the 1995 audit concerns.

35.256 An early 1997 internal assessment of operations and maintenance on reserves by a regional office of the Department raised several concerns, including:

• the lack of training and expertise in some First Nations to maintain assets;
• frequent turnover of maintenance personnel;
• the lack of incentive for First Nations to properly maintain assets; and
• the lack of a requirement for maintenance plans under certain funding arrangements with First Nations.

35.258 In June 1997, the Department issued draft compliance guidelines for the operation and maintenance (O & M) of capital assets on reserves. The purpose of the guidelines is to help ensure that O & M funds are used for the intended purpose, that the condition of the facilities is assessed annually and actioned where required, and that advice and assistance on maintenance are provided to First Nations. Full implementation of the guidelines is targeted for fiscal year 1998-99. In our view, the Department has not acted quickly enough to address this important issue.

Departmental action to May 1997

35.260 According to the Department, various efforts to address the audit concerns and management needs are in progress or have been completed, including the following:

• An internal audit was conducted to assist managers in implementing improvements.
• Training programs were developed and expanded for First Nations water and sewer operators.
• A national workshop for departmental officials was held to discuss several capital management issues.
• Service standard levels are being revised.
• Operational targets are being updated.
• A study is under way to revise the Department's Cost Reference Manual to ensure comparability between on-reserve and other communities.
• Draft compliance guidelines have been issued to regions to address operating and maintenance issues.

Bid tendering issues need resolution

35.263 The devolution of program delivery to First Nations has been increasing for many years. In construction of capital facilities, First Nations generally select the contractors to be used. The current policy of the Department requires that First Nations use public tenders where the estimated construction costs of a project exceed $500,000. However, this approach is not supported by a First Nations technical group that believes that value for money, including spinoff benefits to the community, can be achieved through means other than public tendering.

35.264 We are concerned that unresolved tendering issues can lead to unnecessary increased costs in the delivery of capital projects. A case illustration of tendering issues is reported separately in Chapter 36 - Other Audit Observations.

35.265 The Department has indicated that general principles to guide a First Nations tendering policy will be issued by March 1998.

Department's response: Following the release of the November 1995 audit on the On-Reserve Capital Facilities and Maintenance Program, Indian and Northern Affairs Canada developed an action plan that resulted in progress on each area of the recommendations made by the Auditor General. There are numerous initiatives currently under way, most of which will be completed by 31 March 1998. The Department will continue to work expeditiously toward the effective implementation of the applicable recommendations.

In particular, the Department has articulated its strategy for ensuring that on-reserve assets are properly operated and maintained through the issuance of an operations and maintenance compliance regime. The regional staff will work with First Nations to ensure full implementation by 31 March 1998. The Department believes that the time required to develop and implement this regime was appropriate due to the need to undertake regional consultation and collect the necessary data. In addition, the Department is working with First Nations in order to strengthen accountability to both First Nations communities and to Parliament. To achieve this, each First Nation and Tribal Council funded by the Department is required to complete an assessment of its accountability and management system by 31 March 1998.

The Department recognizes the need to work with First Nations to implement the policy requirement for First Nations to use a tendering process for capital projects over $500,000, excluding housing. In implementing this policy, the Department is faced with the challenge of achieving a balance between the concepts of achieving best value for federal funds and increased socio-economic benefits for First Nations. The Department is working on a paper to define general principles for guiding First Nations tendering policies.

Important progress has been made toward improving living conditions on-reserve and the Department is well positioned to continue this progress while at the same time ensuring value for money.