1989 Report of the Auditor General of Canada

Main Points

16.1 The Government of Canada spends more than five billion dollars annually on a variety of products. To ensure the quality of these products it has traditionally relied on product standards, inspection and warranty. These methods have been adequate for low technology and high volume production, but appear to be inadequate when dealing with higher technology and complex products that require a more comprehensive approach to quality assurance (paragraphs 16.7 and 16.19).

16.2 More is at stake than the quality of the products themselves. In a number of cases, we noted that deficiencies in quality assurance practices have had a detrimental effect on the operational efficiency of equipment purchased for government departments (16.36, 16.41, 16.47 and 16.52).

16.3 Since the 1960s, a more comprehensive approach, commonly known as "Quality Assurance", has been used by purchasers to require suppliers to ensure that quality is built into the product. National and international standards for quality assurance have been adopted as requirements by an increasing number of Canadian companies. However, the government has not developed a position on the use of these standards in government purchases. As a result, in may of the contracts we examined, DSS and its customer departments had no specified requirements for the suppliers' quality systems in the tender documents (16.22 and 16.23).

16.4 The government is the largest single purchaser of Canadian goods. Its position on issues related to quality assurance is critical to Canadian companies. The government can, through its procurement procedures, take positive steps to encourage a more rigorous approach to quality and thereby enhance the reputation and competitiveness of Canadian industry, both at home and in the international market (16.24).

16.5 DSS has not had a corporate focus on quality assurance since 1979 when the Quality Assurance Branch was dissolved. However, we noted that this functional responsibility has recently been assigned to the Canadian General Standards Board. The Board will be involved in the assessment and application of quality assurance standards (16.26 and 16.27).

16.6 Because the contracts we audited were limited and not randomly selected, government wide conclusions were not drawn based on the contracts examined. However, our examination of systems and procedures uncovered issues which need to be addressed if a full quality assurance program is to be instituted (16.15).

Introduction

16.7 The Government of Canada spends more than five billion dollars annually on a variety of products. To ensure the quality of these products it has traditionally relied on product standards, inspection and warranty. These methods have worked well for goods involving low-technology and high-volume production.

16.8 Because today's products are becoming more complex, prudent purchasers are expected to participate in various stages of design and production, although traditionally these tasks have been the responsibility of suppliers (including manufacturers, contractors and assemblers). Through a specific program, commonly known as "Quality Assurance", the purchaser must first clearly specify his technical requirements (including the requirements for quality assurance) and select suppliers who can meet the specified requirements. Then the purchaser must monitor the supplier's quality assurance system and inspect the product to ensure that all contractual requirements are met.

16.9 Relying on inspection to identify faulty items at or near the end of the production process is costly. It takes time, effort and money to inspect the products and to correct identified deficiencies. In addition, there are costs for repair, maintenance, liability, etc. when the products fail to perform satisfactorily. Experience in the United States has shown that the costs of not "doing the right things right the first time" total over 20 percent of gross sales for manufacturing companies. These costs are included in the prices that customers pay.

16.10 More is at stake than the quality of the products themselves. Product quality can affect the efficiency of government services and the effectiveness of its programs. In addition, there are direct effects on the ability of Canadian industry to compete in an increasingly difficult international market where quality is becoming a major factor.

16.11 Today's customers demand quality. Recently, more and more Canadian companies have applied new methods to ensure consistent quality of their products to meet customer requirements. As the single largest purchaser of Canadian goods, the Government of Canada has an important leadership role to play in this pursuit of quality.

Audit Objective and Scope

16.12 The objective of our audit was to determine whether the quality assurance systems and practices in selected departments and contracts conformed with generally recognized national and international standards for quality assurance.

16.13 The Department of Supply and Services (DSS) has the legislative mandate for "assuring the quality of materiel". We reviewed the history of quality assurance development in DSS and the related policies and procedures the Department has established.

16.14 We also examined the quality assurance practices followed in 18 selected contracts for 10 identified product areas in DSS, the Royal Canadian Mounted Police (RCMP), the Departments of the Environment (DOE) and Transport (DOT), and the Canadian International Development Agency (CIDA). We first determined the existence and adequacy of quality assurance systems and procedures in related branches of the departments and agencies. We then assessed whether these systems and procedures had been followed in the 18 contracts we examined. We also reviewed the relationships between DSS and the customer departments in exercising quality assurance in the selected contracts.

16.15 The contracts (see Exhibit 16.1) represent a wide range of product types purchased for government programs -- from simple, traditional items to technically complex equipment needed for major Crown projects (projects over $100 million). Because the contracts we audited were relatively few and not randomly selected, we could not draw broad government-wide or department-wide conclusions. However, our examination of systems and procedures uncovered issues which need to be addressed if a full quality assurance program is to be instituted. We plan to do additional work, wherever appropriate, on quality assurance in procurement of both goods and services. This will be done as part of our comprehensive audits in the future.

(Exhibit not available)

16.16 Our audit findings are presented below under Observations and Recommendations. Under Case Studies we show some of the deficiencies in quality assurance found during our audit of the 18 contracts.

Observations and Recommendations

Inspection alone cannot always assure quality

16.17 Quality assurance is more than inspection. Inspection has traditionally been an important element in ensuring the quality of goods but, particularly since the 1960s, quality assurance is understood to encompass much more. DSS directives agree with definitions by national and international standards associations. These stipulate that, depending on the value and the nature of the product to be purchased, quality assurance includes some, often all, of the following actions:

• the proper definition of requirements and specifications;
• appropriate requirements for the supplier's quality assurance system;
• judicious sourcing and thorough evaluation of suppliers;
• proper quality assurance clauses in contract documents;
• surveillance of the supplier's quality assurance system;
• a specified level of product inspection; and
• formal feedback on product performance in service.

16.18 Despite inspection there are quality deficiencies. A 1986 DSS internal study revealed that, by and large, departments used inspection extensively to assure quality, and that this had failed to ensure consistent results. In our examination of the selected contracts, we confirmed that in many cases departments had continued to rely heavily on inspection for quality assurance.

16.19 We found that the inspection approach worked well for products involving low-technology and high-volume production, such as RCMP uniforms and ammunition, but inspection alone was not appropriate for more complex government purchases, such as airport emergency vehicles and satellite station equipment (see case studies for details). We found that even when inspections were properly planned and carried out, product quality could still be unsatisfactory if other quality assurance deficiencies existed.

16.20 In the contracts we examined, quality problems stemmed from the supplier's lack of an adequate system for quality assurance. DSS and/or its customer departments had not specified requirements for the supplier's quality assurance system, had not evaluated the supplier's quality assurance capability before awarding the contract, or had not monitored the supplier's quality control system during the execution of the contract to ensure its effective functioning.

16.21 Standards for supplier's quality assurance are inadequate. DSS directives, and its customer manual on quality assurance, stipulate that requirements for a supplier's quality system should be considered under certain conditions, for example when a contract involves: a dollar value exceeding $10,000; product safety impact; newness of design; technical complexity; or past failure by the supplier to conform with specifications. All contracts we examined met one or more of the above conditions, but most did not contain specific requirements for suppliers' quality assurance. This could be attributed to the fact that specification of such requirements is not mandatory even under the prescribed conditions; customer departments lacked the expertise to specify the requirements; and DSS did not ensure that adequate requirements were specified in contract documents. Furthermore, procedures for selecting an appropriate level of quality assurance standards had not been developed by DSS.

16.22 In 1975 the Canadian Standards Association (CSA) published a series of quality assurance program standards -- consisting of four levels of increasingly stringent requirements. These CSA standards were developed by industry representatives using a consensus approach and were updated in 1979 and 1985. A guide for selecting and implementing quality assurance standards was published in 1986. The CSA standards and guide have been used by an increasing number of Canadian companies as the basis for their own quality assurance systems and as requirements with which their suppliers must comply. Since 1987, quality assurance standards developed by the International Standards Organization have also received increasing recognition internationally and are being adopted universally in Europe. In addition, more and more companies, both in Canada and abroad, now require that the quality assurance systems of their suppliers be subjected to third-party assessment and certification to a recognized standard prior to awarding a contract.

16.23 In 1984 a request by the Deputy Minister of DSS to investigate the potential use of CSA standards in government led to the internal study mentioned in paragraph 16.18. However, the task force did not specify the extent to which the CSA quality assurance standards should be applied to government purchases. As well, the government had not decided whether it should subject its suppliers to this independent certification.

16.24 Because the government is the largest single purchaser of Canadian goods, its position on issues related to requirements for suppliers' quality assurance systems will have a significant impact on Canadian industry. By promoting the importance of quality and by encouraging the use of quality assurance standards, the government, in concert with major industrial associations, could help increase the competitiveness of its suppliers in the market at home and abroad. The costs of improving quality systems in Canadian companies would be more than offset by the reduction in the amount of rejects, rework and warranty repairs.

16.25 DSS should consider, in consultation with central agencies, the clarification of government policy on requirements for suppliers' quality assurance systems, including the adoption of Canadian or international quality assurance standards and related independent assessment and certification. DSS should develop and promulgate procedures for selecting and specifying supplier quality assurance requirements.

DSS's response: Supply and Services agrees that this is an area which should be further addressed, in consultation with central agencies and departments of government. The clarification of the government's policy on quality assurance systems for suppliers cannot be done in isolation from the supplier marketplace, nor can it be done in haste, and thus the department recognizes the need to involve industry in all sectors in a planned approach to the development of policy.

There are systemic weaknesses

16.26 DSS had no corporate quality assurance focus. Through the Department of Supply and Services Act, Parliament has assigned to the Minister responsibility for purchasing and, explicitly, for "the assuring of quality of materiel". Until 1979 the Quality Assurance Branch of DSS administered a general quality assurance policy and published a series of directives. The Branch was dissolved in 1979 and the responsibility for quality assurance was distributed to the various DSS procurement branches.

16.27 The 1986 internal study by DSS found that this had contributed to many major deficiencies in quality assurance systems and practices. DSS formed a Product Quality Management Committee to implement the recommendations of the study. The Committee completed a public awareness campaign, provided training for procurement officers and amended quality assurance directives. However, a number of the study's key recommendations have not as yet been implemented.

16.28 In October 1988 the responsibility for the Committee was transferred to the Canadian General Standards Board (CGSB). The functional responsibility for quality assurance mandated to DSS was assigned to CGSB. CGSB has played an important role for some years in the development of standards and quality control. We were informed that this new corporate focus on quality assurance is being made to involve CGSB further in the assessment and application of the quality assurance standards. The line branches will be responsible for the application in the procurement process of the standards set by the CGSB.

16.29 Departmental procedures and organization need to be strengthened. Although DSS has been given statutory responsibility for quality assurance in the Government of Canada, in practice the funding department must share this responsibility with DSS. This is important to ensuring effective quality assurance and to establishing accountability for achieving it.

16.30 In the late 1970s, DSS signed Memoranda of Understanding (MOUs) with DOT and CIDA delegating to them its procurement responsibilities, including quality assurance, for certain products. We noted that these departments had not established appropriate quality assurance procedures and organizations to assume the delegated responsibility.

16.31 The 1978 DSS - DOT MOU was valid for one year but was never renewed. Of the three branches of the Department that we looked at, only Flight Services had maintained some quality assurance capability. In the other two branches, Airports Authority and Canadian Coast Guard, the division of responsibilities for quality assurance between DSS and DOT was not clear. In addition, we found that, although DOT technical people were involved in some quality assurance work, there were no written policies and procedures.

16.32 DOT should develop appropriate quality assurance policies and procedures for use by the Airports Authority and the Canadian Coast Guard branches.

DOT's response : The Department concurs with the recommendation. The Marine Group has already commenced development of quality assurance policies and procedures.

16.33 The DSS - CIDA MOU was renegotiated in 1986. We found that the importance of product quality to the Agency's aid projects was well recognized. The president of CIDA started a campaign for quality improvement in 1987, but there was no Agency-wide policy on quality assurance and this functional responsibility had not been clearly assigned to a single organizational unit in CIDA. In addition, although quality assurance guidelines for procurement had been developed for Canadian executing agencies of CIDA projects, these procedures were not mandatory and did not cover contracts for which DSS or recipient countries assume purchasing authority.

16.34 CIDA should assign the functional responsibility for quality assurance to a senior Agency executive and charge him or her with developing and implementing an Agency-wide quality assurance policy and procedures to cover all CIDA-funded contracts.

CIDA's response. It is agreed that the Agency will consider the possibility of assigning the functional responsibility for Quality Assurance to a Senior Agency Executive. This can be more adequately considered subsequent to the establishment of an overall government policy on Quality Assurance.

16.35 DSS should, in consultation with its customer departments, clarify and agree to an acceptable division of responsibilities for quality assurance. DSS and central agencies should provide policy and functional guidance on quality assurance to departments wherever necessary. DOT and CIDA should ensure that their policies and procedures are consistent with central policy and directives on quality assurance developed and issued by DSS and central agencies.

DSS's response: The division of responsibilities between Supply and Services and customer departments is described in the Supply Policy Manual, and the DSS Customer Manual. As recently as 1988, the department revisited and re-articulated the division of responsibilities to clients through Deputy Minister communication. Nevertheless, the department is in a review process of the aforementioned document which will include consultation with client departments to determine if there are areas which require further expansion or clarification.

CIDA's response: DSS has the overall responsibility to establish a government policy on Quality Assurance applicable to all procurement projects. Subsequent to this policy being established, CIDA will prepare its own policy within the framework set by DSS for application to CIDA funded projects.

DOT's response : The Department supports the recommendation that consultation should be held to clarify the division of responsibilities with customer departments of DSS.

CASE STUDIES

Case 1. Buoy-handling Systems for Coast Guard Ships

16.36 In 1983, the government entered into contracts with four shipyards to construct six Type 1100 ships for the Canadian Coast Guard (CCG) to be used for buoy tending and ice breaking. CCG selected a British design, "speedcrane", as the buoy-handling system. The shipyards subcontracted the work to a British firm, which in turn engaged a Canadian agent to construct the speedcranes, in order to meet the level of Canadian content required by the government.

16.37 A serious problem with the buoy-handling system for these ships was noted in our 1987 Report. We concluded at that time that the buoy-handling equipment did not meet specifications and required extensive modification.

16.38 In January 1988 CCG implemented a corrective action plan to carry out the necessary modifications, but the costs of the modification were substantial. For example, the refit and repair costs for the crane on one of the six ships exceeded $300,000. The vessel worked none of the 1278 workable hours available for the navigational aids program, because its speedcrane was being modified (refer to Chapter 22 Department of Transport - Canadian Coast Guard - Protecting Mariners' and the Public Interest).

16.39 In addition to the corrective action plan, an internal review of the speedcrane system was carried out in 1988 at the request of CCG management, to determine what had gone wrong. The review revealed many factors that had contributed to the unsatisfactory result, several of them deficiencies in quality assurance practices. For instance, a Statement of Operational Requirements for the buoy-handling system had not been developed by CCG. As a result, there was no basis for the parties involved to assess the crane design. Furthermore, changes were made to the original speedcrane design to satisfy Canadian content requirements, without adequate analysis, testing and approval. The operational ability of the prototype design was not verified.

16.40 We found an anomaly in the sharing of quality assurance responsibilities between DOT and DSS. In 1978 a Memorandum of Understanding (MOU) was signed between DSS and DOT, delegating procurement responsibilities (including quality assurance) for technical products to DOT. The arrangement was not renewed after one year as had been stipulated in the MOU. Unlike other branches of the Department where the complete quality assurance process was handled by departmental officials, CCG relied on DSS to carry out inspection of speedcranes. This division of responsibility, in our opinion, contributed to the deficiencies in quality assurance practices noted by the internal review.

Case 2. Emergency Vehicles for Canadian Airports

16.41 In 1983 the Treasury Board approved a Department of Transport submission to procure a fleet of 68 airport emergency vehicles, at a total cost of approximately $22 million. These vehicles were intended to be the primary units for crash, fire-fighting and rescue services at airports owned and operated by DOT, to meet the airport safety standards of the International Civil Aviation Organization.

16.42 The specifications for the vehicles were developed by DOT and the responsibility for detailed design and construction was left with the contractor. DOT retained the right to approve a prototype. When the prototype was delivered for testing it was found to have several deviations from specification, which were eventually corrected and allowed production to proceed.

16.43 A requirement to establish an adequate quality assurance program was critical, given the size of the purchase, the newness of the design and the importance of emergency operations. Nevertheless, this requirement was not specified in the contract, nor was the manufacturer's existing quality assurance system verified by the department.

16.44 Departmental documentation indicates that DOT inspectors noticed deficiencies in the vehicles delivered for acceptance testing and were concerned therefore about the contractor's quality assurance system. The contractor reinforced its inspection system but was unable to satisfy DOT. The number of deficiencies reported by DOT, and the fact that their deficiencies were dealt with at the final inspection point resulted in increased time and cost being incurred during the acceptance process for both the manufacturer and the department. In the final analysis all 68 vehicles were accepted by DOT.

16.45 Acceptance tests cannot, and are not designed to, identify all defects. Although the deficiencies reported by DOT inspectors were rectified by the contractor before the vehicles were accepted, many other deficiencies surfaced after delivery. In a survey conducted by DOT soon after all vehicles had been delivered, 21 were reported to have various operational problems with the central tire deflation system (CTDS). CTDS, which costs more than $30,000 per vehicle, is a critical feature specified by the DOT design authority, to provide the emergency vehicles with additional traction for operating in difficult terrain. The prime contractor agrees with the assessment of the CTDS system and notes that DOT was informed of possible serious operational problems in April 1987.

(Photo not available)

16.46 We were also informed by the vehicle operators that problems continue with the computerized instrument display system and with the transmission.

Case 3. Satellite Ground Station Upgrade for Bangladesh

16.47 A satellite ground station in Bangladesh was constructed in 1975 with CIDA funding, to be the key telecommunications link between Bangladesh and the rest of the world. It was upgraded in 1983 through local funding and again in 1986 under CIDA funding. Both upgrades were contracted to the original supplier. The focus of our audit was the 1986 upgrade, funded at a cost of $2.8 million through a Memorandum of Understanding on Agro-Industrial Support Program (MOU-AISP), signed in 1983 by Canada and Bangladesh.

16.48 Under the MOU-AISP arrangement, Canada delegated purchasing authority to Bangladesh. Although the contract was signed between the Bangladesh government and the Canadian supplier, CIDA was accountable for the funds expended on this project. The MOU-AISP recognized the need for consultant services to assist the Bangladesh government in procurement activities. CIDA engaged a number of technical consultants to assist in preparation of specifications, inspections, acceptance tests, training and project monitoring.

16.49 However, the requirement for the contractor to establish and maintain an adequate quality assurance system, which would normally be stipulated in procurement of this kind of complex equipment, was not specified in the Bangladesh contract. Instead, CIDA relied on the contractor's own quality assurance system and the provision of warranty. Furthermore, the CIDA project monitor attended all major in-plant tests as well as field commissioning and while he reported deficiencies to CIDA, he did not have direct authority to initiate corrective action.

16.50 Several unusable parts were shipped to the project site and a number of system modules failed during the on-site acceptance tests. During our inspection of the site three months after the acceptance tests, many deficiencies remained. A major sub-system was not operational, due to defective parts and water damage during transit. Since then, many of the deficiencies have been corrected by the use of spares. However, faulty components that were returned to the contractor for repair or replacement under warranty have not yet been shipped back to Bangladesh. This delay was due to an ongoing contract dispute.

16.51 During our examination we also noted that the station was operating under a waiver on the antenna system, issued by the International Telecommunication Satellite (INTELSAT) after the 1983 upgrade. This CIDA contract was not meant to address antenna problems related to the waiver. However, if the station's antenna deteriorates further and causes interference with other stations, it could be shut down by INTELSAT, in effect cutting off satellite communication in Bangladesh.

(Photo not available)

Case 4. Purchase of Railcars for Senegal

16.52 In 1986 CIDA purchased, on a sole source basis through DSS, 100 railcars for Senegal from a Canadian rail equipment company, at a cost of $8.3 million.

16.53 Railway development has been a major focal point for CIDA in its aid program. An internal evaluation of the Railway Aid Program, completed in 1986, found that Canadian rail equipment supplied by CIDA to foreign operations had a shorter life than its normal design life. The evaluation identified several quality deficiencies, such as non-conformance to specifications and defects in the original design. It suggested that responsibility for the problem rested jointly with CIDA, the recipient countries and Canadian suppliers.

16.54 For this project, a requirement for supplier quality assurance was specified in the contract. Although the company had developed a quality assurance manual, it had not implemented the program at the time of the contract. Recognizing this shortcoming and given the urgency of the contract, CIDA engaged technical consultants to carry out inspections, as had been its practice previously.

16.55 During in-plant inspections, inspectors hired by CIDA found numerous defects and welding deficiencies in the railcars and recommended a review of the company's quality control procedures. All the defects found by the inspectors were subsequently rectified before the cars were shipped to Senegal.

16.56 As part of the corrective action, the company and CIDA agreed that a field inspection would be performed after the railcars had been in service for approximately a year, to determine whether the corrective measures taken had been adequate. The planned inspection had not been carried out more than 24 months after delivery of the railcars. We were informed by CIDA that this inspection would take place before January 1990.

Case 5. Contracts with Appropriate Quality Assurance Practices

16.57 Based on our examination of the selected contracts, we have concluded that quality assurance procedures in procuring modular office furniture by DSS, uniforms and ammunition by RCMP, and water bombers by DOT have been adequate. Some of the reasons for this, we found, are:

• Responsibility for quality assurance for the product is clearly assigned, either formally or informally. For example, the contract between DSS and the water bomber supplier specified DOT as the quality assurance authority. The quality assurance responsibility for uniforms and ammunition has traditionally been assumed by RCMP, with informal agreement by DSS. Modular office furniture is one of the stocked items for which DSS retains the procurement and quality assurance responsibility.
• In each of these product areas, the department involved has established and maintained a technical and quality assurance capability. Appropriate quality assurance policies and procedures have also been developed.

16.58 DSS and the respective customer departments have been able to maintain a co-operative, professional working relationship with suppliers.