1998 December Report of the Auditor General of Canada
Chapter 20—Preparedness for Year 2000: Government-Wide Mission-Critical Systems
20.2 In 1998, we audited some key computer systems and devices that support six mission-critical functions in government. The functions were inspection services to contribute to food safety; emergency assistance and support; income security services; First Nations transfer payments and trust funds; border crossing services for commercial goods; and law enforcement services. We concluded that as of 30 June 1998, several key systems supporting three of those functions remained at risk of not becoming fully compliant before 2000.
20.3 We observed that the government has accelerated the pace of its efforts since our 1997 audit and has made progress in various areas, including:
- ranking Year 2000 as a top priority;
- monitoring government-wide progress of systems and devices that support mission-critical functions in government;
- advancing certain common horizontal initiatives; and
- engaging the concern of ministers and advising them of the urgency of Year 2000 efforts.
20.5 We concluded that as of June 1998, various government systems supporting mission-critical functions remained at risk. With 18 months left, we are very concerned that some essential government services may be interrupted at the start of 2000.
20.6 To ensure that mission-critical functions will continue, we recommend that the focus on Year 2000 be sustained, that the Treasury Board Secretariat consider strategic intervention as appropriate, and that contingency plans be prepared and tested for those systems that remain at risk. We also recommend that further action be taken to advance common horizontal initiatives and that reporting of Year 2000 information to Parliament be improved.
20.8 Given the ever-increasing reliance on information technology, Year 2000 represents a business risk, not just a technical problem. It is a threat to all organizations, including businesses, non-profit organizations, schools, hospitals and all levels of government.
20.9 In 1997, we audited the federal government's overall state of preparedness for Year 2000. We concluded that as of 30 April 1997, the government's rate of progress in addressing the Year 2000 threat had generally been slow.
20.10 We were concerned that if progress were to continue at the same rate, it would likely be too slow to ensure that government systems would be ready in time. We concluded that Year 2000 remained a serious threat to essential programs and services and that urgent and aggressive action was required.
20.11 We recommended that the Treasury Board Secretariat give top priority to monitoring progress. In particular, we recommended that it identify the systems considered to be mission-critical to the government as a whole and intervene strategically as appropriate, to ensure that they would be ready before 2000. We also emphasized the need for contingency plans. Other recommendations included:
- ranking Year 2000 as a top priority;
- affirming the sustained commitment and support of senior management;
- informing Parliament of the potential impact of Year 2000 and reporting on progress toward making systems compliant; and
- accelerating common initiatives that support Year 2000 work in departments and agencies.
Focus of the audit20.12 The current audit focussed on the government's progress in making its mission-critical systems compliant for Year 2000.
20.13 We used the results of the Treasury Board Secretariat's 30 June 1998 survey to obtain an update of the government's progress in making compliant the systems that support its mission-critical functions. We examined several key systems and applications that support six government-wide mission-critical functions. We also followed up on action taken by the government to address our 1997 recommendations.
20.14 Further information about the audit objective, scope and criteria can be found at the end of the chapter in About the Audit .
Observations and Recommendations
Examination of Systems That Support Government-Wide Mission-Critical Functions20.15 In July 1997, the Treasury Board Secretariat prepared an initial list of government-wide mission-critical functions - those considered essential to delivering programs and providing services. The list was circulated to all deputy ministers and heads of agencies for feedback. The departments and agencies responsible for the functions were also requested to provide a brief impact statement on each of the functions.
20.16 In the fall of 1997, the list included 19 departments and agencies that provide 44 mission-critical functions of the government. Those functions became the focus of the Secretariat's subsequent survey efforts. Year 2000 work on them receives priority access to resources such as funding, and to procurement arrangements. As of June 1998, 48 government-wide mission-critical functions had been identified; they are provided by 21 departments and agencies and two Crown corporations (Exhibit 21.1) .
20.17 In assessing the government's progress, we selected for examination six government-wide mission-critical functions by applying the criteria used in our 1997 audit. These functions include:
- inspection services to contribute to food safety;
- emergency aid to civil powers and assistance and support in the event of natural disaster or loss of civilian infrastructure;
- income security services to seniors;
- transfer payments to First Nations and management of their trust funds;
- border crossing services for commercial goods, including collection of and accounting for customs duties and the goods and services tax; and
- law enforcement services for the safety and security of the public.
20.19 In some functions we audited the lead system; in others, we selected and audited up to three key applications and systems. All the selected systems and devices are considered necessary to support the mission-critical functions but not always sufficient to ensure that the functions will continue. Our findings pertain only to the systems and devices that we audited and are not representative of overall progress in the organizations where they operate. Our findings reflect the status of progress at 30 June 1998 (31 August 1998 for emergency assistance and support).
Canadian Food Inspection Agency - inspection services for food safety and animal and plant health20.20 Food inspection services help maintain Canada's high standards of quality and safety for food, agricultural inputs, and animal and plant health. The Canadian Food Inspection Agency depends extensively on information technology to carry out inspection, testing and certification of food products. Its systems that support food inspection are mission-critical because their potential failure or malfunction could result in health and safety concerns.
20.21 The Canadian Food Inspection Agency was created in 1997 to consolidate the food inspection and animal and plant health activities of three federal departments. To address the Year 2000 threat, the Agency decided to develop new information systems, including the Multi-Commodity Audit Program (MCAP) and Laboratory Sample Tracking Systems (LSTS), in the place of existing systems from three departments (Exhibit 20.2) . Further, the Agency needed to test and replace laboratory equipment where necessary to make it Year 2000 compliant.
20.22 We noted that implementation dates for the MCAP systems were delayed by about four months from earlier plans. The Year 2000 effort for laboratory equipment also experienced delays.
20.23 We found that the Agency had identified interdependencies among its Year 2000 efforts on individual systems. However, analyses of these interdependencies for completion timeline and critical path modelling were incomplete. We were advised by the Agency that it had attempted to conduct these analyses but had not been able to complete the task, due to a number of factors such as time and costs. As a result, there is no critical path for the Agency's overall Year 2000 project. Further, as of June 1998, it had yet to begin reviewing its interdependencies with other countries.
20.24 Although the Agency has a defined protocol for testing and certification of its information systems and technology infrastructure and has developed an overall test strategy, it has not prepared test plans for the individual systems. As of 30 June 1998, it had yet to finish establishing a Year 2000 test environment. These environments are established to simulate processing in 2000. Systems are not considered fully compliant for Year 2000 purposes until they have been tested successfully in this type of environment.
20.25 Some application systems can have failure dates prior to 2000. We observed that the Agency's analysis of the possibility of early failure of its application systems was insufficient and required further work. We noted that one application in LSTS that experienced failure due to the date code had not been identified in the Agency's analysis.
20.26 The Canadian Food Inspection Agency has over 2,000 pieces of laboratory equipment. Many of them contain microprocessors that can be difficult to detect and test, complicating the work of making those systems compliant for Year 2000. In most cases, communication with and guidance from the suppliers will be required. The Agency contacted suppliers of its laboratory equipment but, when we completed our audit, it had not received responses from about two thirds of them.
20.27 The target completion date for the Agency's Multi-Commodity Audit Program (MCAP) and Laboratory Sample Tracking Systems (LSTS) is June 1999. While this meets the milestone date stipulated by the Treasury Board Secretariat, it does not provide room for further slippage or for unexpected problems with various data interfaces among systems. The testing and replacement of laboratory equipment were planned for completion at 30 October 1998. Given that the needed vendor responses on most of the Agency's laboratory equipment had yet to be received by the end of the audit, completion by October 1998 was unlikely and no new target date had been set. Thus, in our view, the MCAP and LSTS systems and the laboratory equipment remained at risk of not being fully compliant in time for Year 2000.
20.28 The Year 2000 project office at Treasury Board Secretariat requested departments and agencies with systems that support government-wide mission-critical functions to complete and submit a Year 2000 risk assessment by 31 August 1998, to determine if contingency plans would be necessary. At the time of our audit, the Canadian Food Inspection Agency was planning to assess its Year 2000 risks and prepare detailed contingency plans by the end of August 1998. Subsequent to the audit, the Agency advised us that it expected the two systems and its laboratory equipment to be fully compliant in time for Year 2000.
National Defence - emergency assistance and support20.29 National Defence is responsible for five government-wide mission-critical functions. In early 1998, as part of its Operational Readiness program, it commissioned a series of impact analyses for 12 representative missions that it needs to undertake from time to time. One such national mission is to provide emergency aid to civil powers and assistance and support in the event of natural disaster or loss of infrastructure. The 1997 Red River flood in western Canada and the 1998 eastern Canada ice storm are examples of the events that this type of mission would support.
20.30 The impact analysis for this representative mission was completed in late June 1998. The analysis identified 31 essential systems, 17 of which were categorized as "very high priority". We selected three of the 17 systems for examination - Joint Establishment and Movement Management System (JEMMS), Automated Defence Data Network (ADDN) and National Materiel Distribution System (NMDS). The JEMMS generates task force movement tables to support the movement of staff and equipment in a planned mission. The ADDN provides messaging capability seven days a week and 24 hours a day to facilitate secure and reliable communication in times of peace and war. The NMDS produces documents for distribution of materiel, and tracks shipments (see Exhibit 20.3) .
20.31 During the planning phase, the Department found that the Joint Establishment and Movement Management System (JEMMS) had been developed in 1996 using programming tools that were Year 2000 compliant. The JEMMS operates on personal computers. Although the personal computers at National Defence Headquarters are Year 2000 compliant, in August 1998 those at the Department's bases and stations were being tested and replaced as necessary. Also in August 1998, the systems manager and the director certified that JEMMS does not use dates. Since the audit, the Department has advised us that JEMMS will be considered compliant after the hardware at its bases and stations are confirmed or made compliant.
20.32 By August 1998, programming codes in the Automated Defence Data Network (ADDN) had been verified and were undergoing testing. National Defence has requested and received confirmation of Year 2000 compliance from suppliers of the core software and hardware. About 15 older personal computers have been identified as non-compliant. The Department plans to apply a software solution to address the problem and to not replace the computers at this time. At the end of our audit, it was expecting to complete the software correction by November 1998. According to departmental officials, the ADDN system and other messaging systems in the Department are expected to be phased out and replaced by a new departmental message-handling system in 2001.
20.33 The technology platform for the National Materiel Distribution System (NMDS), including hardware, software and the data base, has been assessed as non-compliant and needs to be replaced. In addition, the existing hardware that supports NMDS has experienced more frequent breakdowns in recent years. The Department plans in April 1999 to phase out its integrated data network, the network used by NMDS.
20.34 The Department prepared detailed project plans to redevelop NMDS and reinstall it on a compliant technology platform; at the end of our audit, the redevelopment work was on schedule. Test plans for NMDS are not scheduled to be completed before February 1999.
20.35 Some risks of delay are involved in replacing non-compliant data terminals and procuring and installing work stations to access the Department's wide area network. This could delay in turn the completion of Year 2000 conversion and testing of NMDS.
20.36 In our view, the risks facing the three systems that we examined are not significant in comparison with the risks to other systems in the Department or to others we examined in other departments and agencies. Year 2000 risks are higher in the Department's large information technology projects and the large number of embedded devices. Over 80 percent of its systems are embedded systems; when we completed our audit, its inventory list was still being updated.
20.37 Under the Department's Operational Readiness program, impact and risk analyses for representative missions were to continue in the fall of 1998. As of 31 August 1998, the Department considered that no contingency plans were necessary for the three systems we audited. Subsequent to the audit, the Department indicated that it was planning a number of activities to reduce risk in its embedded systems. It also advised us that it would prepare contingency plans for the three systems and develop business continuity plans under the Operational Readiness program.
Human Resources Development Canada - income security services20.38 Income security services at Human Resources Development Canada include the Canada Pension Plan (CPP) and the Old Age Security program. For many seniors who depend on benefit payments as part of their retirement income, services that provide those payments are essential. The CPP provides 3.7 million benefit payments to eligible contributors and to surviving spouses and dependent children (Exhibit 20.4) .
20.39 Over the past few years, under an initiative to redesign the information systems for its income security programs, Human Resources Development Canada compiled a full inventory of the CPP system. While the redesign project was under development, the Department started Year 2000 repair work on the existing CPP system as a contingency measure. When the redesign project was halted in late 1997, the Year 2000 strategy focussed on repairing the CPP system. The conversion strategy involved using a combination of a four-digit year code and program logic for two-digit codes. The repair work on the three subsystems was concurrent, and completion was planned in sequence - Earnings, Benefits and then Payments. The Department also used a phased approach to repairing and testing the subsystems.
20.40 As of June 1998, repair work on the Earnings subsystem was almost complete and the repaired codes were undergoing tests for quality assurance and acceptance. We noted a minor slippage in the repair work for the Benefits subsystem: 73 percent of the programming codes had been converted by the end of June 1998. Code conversion for the Payments subsystem was about one month behind. At 30 June 1998, just under half the codes had been converted and the completion date had been rescheduled to mid-November 1998.
20.41 The Department completed its analysis phase of the three CPP subsystems in February 1998. According to its plans prepared in late February, quality assurance and acceptance testing of the Benefits and Payments subsystems was to occur over approximately 14 months. The planned 14 months included testing for CPP legislative changes passed by Parliament in December 1997. As work on those two subsystems progressed, the plans were revised each month. As of June 1998, about 20 percent of the Benefits subsystem had been tested for quality assurance and acceptance and those tests had not begun for the Payments subsystem.
20.42 The testing of systems in a Year 2000 test environment is a critical step in ensuring compliance and readiness. Thus, once established, such a test environment becomes an important resource and often its use is scheduled to full capacity. Human Resources Development Canada has arranged to test its three CPP subsystems in a Year 2000 test environment (called MVSK), which is managed by another department. The testing of the CPP system in that environment is scheduled from mid-October to 31 December 1998. In addition to supporting its own Year 2000 testing, the department managing MVSK has set up a tight schedule to support its other client departments and agencies.
20.43 Before testing in the Year 2000 environment, code conversion and quality assurance and acceptance testing need to be completed. There is a risk that those steps may not be finished in time for the MVSK testing. In the event that the CPP subsystems are not fully ready, Human Resources Development Canada could miss its scheduled MVSK testing and have difficulty rescheduling. As of June 1998, it appeared that the Department would need to compress its quality assurance and acceptance testing process to meet the MVSK schedule. Compressed testing could increase the risk of failure in the Year 2000 test environment; it could also cause functionality problems after the converted subsystems are implemented.
20.44 Quality assurance and acceptance testing is an essential step. We noted that as of June 1998 the testing team was not fully staffed and the quality assurance and acceptance testing of some functions in the CPP subsystems had yet to be assigned. Although detailed plans for quality assurance and acceptance testing were in place, plans for testing in the Year 2000 environment (MVSK) were not complete.
20.45 System interfaces for the CPP system were identified during the redesign project and were subsequently confirmed as part of the Department's Year 2000 project. Work on the interface of the Earnings subsystem with data on CPP contributions has been co-ordinated with another department. Human Resources Development Canada has requested and received a copy of the Year 2000 plans for the government's payment system, which interfaces with its Payments subsystem; periodic meetings have been held since December 1997 to co-ordinate testing activities between the two systems.
20.46 Human Resources Development Canada's plan is to make the CPP system fully compliant by December 1998, leaving up to one year to address any slippage or problems encountered during testing. As of 30 June 1998, quality assurance and acceptance testing was ongoing; it was uncertain whether testing in MVSK would be completed successfully by 31 December 1998. In our view, there remains a risk that some of the CPP subsystems may not be fully compliant before 2000. If the CPP subsystems fail to pass MVSK testing in 1998, the Department has indicated that it will retest the changes and corrections for quality assurance and acceptance and then retest in MVSK. In that event, it will implement a detailed contingency plan to ensure that the essential service of providing CPP benefit payments will continue in 2000.
20.47 As we completed our audit, Human Resources Development Canada was intending to prepare a risk assessment by the end of August 1998, as requested by the Treasury Board Secretariat. The Department indicated that it was confident that the CPP system would be compliant by the end of December 1998. It also indicated that, in any event, it would finalize detailed contingency plans.
Indian and Northern Affairs Canada - First Nations transfer payments and trust funds20.48 First Nations transfer payments support basic services on reserves, such as education, social assistance, housing and other capital facilities. Transfer payments are also made pursuant to comprehensive land claims and specific claims. In fulfilling the government's fiduciary responsibilities, Indian and Northern Affairs Canada also provides trust services, including the management of Indian moneys as stipulated in the Indian Act . These are essential services for Aboriginal peoples and a mission-critical function of the government.
20.49 The Transfer Payment Management System (TPMS) and the Trust Fund Management System (TFMS) are the primary systems supporting this function. We also examined the Departmental Accounting System (DAS) because both TPMS and TFMS interface with it (Exhibit 20.5) .
20.50 Treaties with First Nations date back to the 18th century; funding arrangements span many years. Consequently, application systems like TPMS and TFMS were developed using an eight-digit date code. Indian and Northern Affairs Canada determined that some conversion work was needed for data display and reporting. Independent of its Year 2000 work, the Department was also changing its operating system and data base standard. The change involved a conversion of the Departmental Accounting System (DAS), which was considered Year 2000 compliant by the supplier. After conversion, the system would need to be tested using the new technology platform to ensure that it would be Year 2000 compliant.
20.51 We found that the Department had not addressed the Year 2000 problem on a department-wide, business-oriented basis. Under the direction of its Information Management Branch, the Department's Year 2000 project activities at the time of our audit focussed primarily on managing changes in the areas of its corporate information systems and technology. Only at the end of June 1998 did the Department formally appoint a senior sponsor to direct its Year 2000 project and take on a department-wide perspective.
20.52 Indian and Northern Affairs Canada prepared a master list of software systems - an inventory list of corporate systems - during the planning phase of its Year 2000 project. However, we found that the list did not identify all the significant systems that interface with those in the inventory list. The Resource Management Information System (RIMS) was omitted - a local system that serves as the primary source of data input to the Trust Fund Management System (TFMS). In addition, the list of interfaces for DAS did not include the linkage with the government's payment system or the central accounting system. As a result of our audit observation, the Department has added RIMS and those linkages to its inventory of Year 2000 work.
20.53 We found that work plans have been prepared for all three mission-critical systems. However, they do not contain details of the activities to be performed and the corresponding resources needed. Thus, it will be difficult for the Department to monitor progress and assess the adequacy of the estimated resource requirement. We also noted that test plans for the systems have not been prepared.
20.54 In our view, the Year 2000 risks to TPMS, TFMS and DAS are lower than the risks to some of the other mission-critical systems we examined. In comparison with others, these systems are less complex. When we completed our audit, the Department had six months remaining to complete its conversion work and test the systems by its target date of December 1998. Both TPMS and TFMS now use an eight-digit date code and DAS has been upgraded to a version that is considered Year 2000 compliant.
20.55 At the end of our audit, Indian and Northern Affairs Canada was planning to complete a risk assessment by 31 August 1998 for submission to the Secretariat. According to the Department, preparation of an information technology contingency plan and a business contingency plan would follow.
Revenue Canada - border crossing services for commercial goods20.56 Revenue Canada's customs service administers over 80 acts and regulations on the government's behalf at border crossings. For commercial goods, the responsibilities vary from facilitating shipment and import of commercial goods to intercepting illicit drugs and other banned substances. Customs processes over 11 million commercial entries annually, accounting for about $20 billion in customs duties and the goods and services tax. It is a mission-critical function that not only provides an essential service to importers but can also affect public health and safety and Canada's finances.
20.57 The Customs Commercial System (CCS) operates on a continuous basis. It maintains an inventory of all commercial entries and directly supports the processing of and accounting for the goods. Further, it interfaces with the Accelerated Commercial Release Operating Support System (ACROSS), a departmental system that supports customs officers in making decisions on the release of goods (Exhibit 20.6) .
20.58 The Year 2000 conversion strategy for CCS involved repairing existing codes. The Department decided to implement the repaired codes in a single release.
20.59 Over the years, in meeting the need to update the tax code annually, Revenue Canada has developed a structured methodology for testing code changes. In preparing for the testing phase of its Year 2000 project, it augmented its regular testing process with a Year 2000 testing environment (called LPAR). This initial testing strategy was reviewed by an information technology firm, which recommended adding a test environment called WT. The WT environment is capable of automated date manipulation to age data. It can help identify major problems in an application before the Department tests it in its LPAR environment.
20.60 The recommendation was accepted, bringing the testing process for Year 2000 to four phases of regular testing as well as testing in the two Year 2000 environments. Revenue Canada has a department-wide definition of Year 2000 compliance. Systems are certified for Year 2000 only if they have met the compliance definition, tested successfully for user acceptance and in the LPAR environment, and been reintroduced in the normal operating environment.
20.61 We found that repair work on CCS had been completed, and the system was in testing at the time of our audit. Testing in the LPAR environment was scheduled for October 1998.
20.62 We note that there has been some slippage in the deadlines for testing and that the time allowed for the testing phases has been compressed. In particular, as of 30 June 1998, the WT integration testing had not been completed and the objective of conducting WT testing - to identify and correct major problems early in the process - had not been achieved.
20.63 Revenue Canada's Information Technology Branch noted in February 1998 that testing was becoming the bottleneck for Year 2000 projects in industry as well as in the Department. It stated that its LPAR testing schedules were not negotiable. As a result, at the time of our audit there was a risk that the CCS might not undergo sufficient testing prior to being tested in the LPAR environment in October 1998 and that problems could arise during that critical phase. Thus, there was a risk that CCS would not achieve Year 2000 certification by the planned date.
20.64 We noted that two releases of CCS have been planned for 1999; the first is scheduled for January. In the event that CCS failed to achieve Year 2000 certification in October 1998, the Department was planning to certify the system in early 1999.
20.65 Although some risks exist for CCS, in our view they are lower than the risks to some of the other systems we examined in this audit. If CCS failed to achieve Year 2000 certification in October 1998, it would have to be given priority for LPAR testing in 1999 and the Department would need to consider undertaking risk assessment and contingency planning. Subsequent to the audit, the Department advised us that its certification testing for CCS was near completion and the system was on schedule to be implemented in October 1998 as planned.
Royal Canadian Mounted Police - law enforcement services20.66 Law enforcement services support law and order and provide for the safety and security of the public. Thus, they form part of the government's mission-critical functions. The Canadian Police Information Centre (CPIC), a system that supports policing service across Canada, is maintained by the Royal Canadian Mounted Police (RCMP) and is used by its 700 detachments and over 500 other agencies (Exhibit 20.7) .
20.67 The Year 2000 strategy for CPIC involved two phases:
- converting the application and operating systems from COBOL to MVS COBOL; and
- repairing application systems using a four-digit year code.
20.69 We found that the RCMP has an overall Year 2000 project plan for the entire CPIC system. Activities in that plan range from one week to many months in duration. In the absence of detailed plans, it is difficult for the Year 2000 project office to assess whether scheduled dates for task completion are reasonable, and to monitor progress. In these circumstances, any delays could go undetected for months.
20.70 As of June 1998, there had been only minor slippage in the first phase of converting to MVS COBOL. But phase two, year code conversion, had experienced a three-month delay. Although the overall project plan had been updated accordingly, which changed the target date for full CPIC compliance from 31 December 1998 to 31 March 1999, target dates for some key activities had not yet been set.
20.71 The RCMP has established a Year 2000 test environment (called LPAR), and some guidelines exist on testing applications for specific dates. A Year 2000 test plan and a test strategy, including co-ordinated testing with external users, were still being developed at the time of our audit.
20.72 The CPIC system is used extensively by the RCMP and other organizations involved in law enforcement. Outside users who have data interfaces with CPIC are categorized as follows:
- Category 1, police departments;
- Category 2, departments and agencies with some law enforcement roles; and
- Category 3, departments and agencies with limited law enforcement roles.
20.73 The RCMP has identified all data interface partners in its inventory phase for Year 2000. In January 1998, it communicated with all Category 1 users to advise them of its plans and the date when CPIC modules would start using a four-digit year code. It has also communicated with the FBI and both parties have agreed to have their respective systems accept a two-digit and a four-digit year code from January until June 1999, when they expect both systems to be ready.
20.74 However, as of 30 June 1998, the response from Category 1 users had been limited. As a result, the RCMP had little information on whether its plans or time frame are acceptable to those users.
20.75 The prime use of the CPIC system is to provide information to law enforcement personnel on demand. Thus, the ability to continue to exchange data and provide information is of paramount importance. In our view, follow-up with Category 1 users is required and, further, the RCMP needs to extend its communication to users in categories 2 and 3. Subsequent to the audit, the RCMP advised us that it had started to follow up with its Category 1 users and it intended to communicate with users in the other two categories.
20.76 We also noted a Year 2000 project risk associated with the availability of technical resources. The lack of technical resources was a primary cause of the delays noted in paragraph 20.70 . We found the resource situation in the application development and the quality control and testing areas to be of particular concern. For example, at the time of the audit only one individual had been identified to conduct testing for the entire CPIC system.
20.77 We have concluded that the CPIC system continues to be at risk. In particular, if end-to-end testing and co-ordination with the user community are not addressed adequately, the use of the CPIC system could be greatly curtailed.
20.78 At the time of our audit, the RCMP was planning to prepare a Year 2000 risk assessment by 31 August 1998 and submit it to the Treasury Board Secretariat. The RCMP has indicated that it has a continuous risk management process in place using outside consultants to assure an independent and objective review, and has made continuous and significant progress. It has also noted that contingency planning is actively under way with a very comprehensive checklist that will be used at every detachment across the country, and that it has played a lead role in educating many police forces and municipalities across Canada on the issue of Year 2000.
Progress Monitoring by Treasury Board Secretariat20.79 In May 1996, the Treasury Board Secretariat established a project office within its Chief Information Officer Branch to provide leadership for the Year 2000 issue in government. The project office has been charged with the responsibility to oversee Year 2000 efforts in the government and to address common needs of departments and agencies. In our 1997 audit, we reported that the Secretariat had helped to raise awareness across the government and had facilitated the exchange of experiences and views on Year 2000 efforts.
20.80 In response to our 1997 recommendations, the Secretariat stated that it had started to examine the state of readiness of all systems that the government considers mission-critical, in order to confirm that they would continue to function properly after 31 December 1999.
Survey results for systems supporting government-wide mission-critical functions20.81 The project office conducted a survey of 76 departments and agencies in the fall of 1997. It tabulated separately the survey results in the 19 departments responsible for systems that support government-wide mission-critical functions, in 16 other large departments and agencies, and in the remaining 41 smaller departments and agencies.
20.82 In its report of January 1998, the Secretariat noted that although almost all departments and agencies had made significant progress, only a few had carried out substantial repair work or testing of systems by the fall of 1997. It concluded that there was sufficient time to complete the conversion of mission-critical systems but that some non-essential systems might not be converted at the same pace.
20.83 The project office conducted a further survey in the spring of 1998, focussing primarily on the mission-critical functions of the government. Those results were updated in June 1998 and the project office has monitored progress on a monthly basis since then.
20.84 In order to assess progress, the Secretariat used a model from a major information technology research firm. The firm assigns to each of its seven phases of a Year 2000 project an estimate of level of effort required to complete the work. Exhibit 20.8 compares the phases in the generic model we used in 1997 with those in the firm's model; the firm's estimate of level of effort is shown as a percentage.
20.85 In the fall of 1997, the research firm considered that an organization should by then have completed 45 percent of its Year 2000 efforts in order to have fully compliant systems in place by 31 December 1999. On that premise, the Secretariat set targets for percentage completion based on elapsed time. According to the targets, departments and agencies should have completed 76 percent of their Year 2000 work by 30 June 1998.
20.86 As of 30 June 1998, however, the Secretariat's summary results showed that overall, the 21 departments and agencies had completed about 50 percent of their Year 2000 efforts for systems that support 46 government-wide mission-critical functions. Two Crown corporations responsible for two such functions were added recently and information on their progress was not available for the June 1998 summary. The results also showed that the pace of Year 2000 action had accelerated in recent months.
20.87 In the spring of 1998, the Secretariat announced that the target date for full implementation of all systems that support mission-critical functions was summer 1999, later confirmed as June 1999 and subsequently endorsed by Treasury Board ministers. Given that this extends the original model by six months, and assuming the same pace of action as seen in the spring of 1998, the Secretariat takes the position that completion by June 1999 is achievable (Exhibit 20.9) .
20.88 However, in adapting the research firm's model, the Secretariat had set its target using elapsed time as the main proxy for level of effort. Its indicator, percentage completed, did not take into account other measures such as percentage of staff hours used or quantity of work done in comparison with the overall budget. The quantity of work done is often measured in lines of programming code. For some departments and agencies, the Secretariat used the number of lines of programming code as a weighting factor in assessing progress.
20.89 In addition, the model was intended for situations that primarily involved repairing and testing existing systems. The methodology may not be as applicable by departments that have decided as their conversion strategy to replace non-compliant systems. In those cases, progress reporting based on elapsed time may not always be meaningful. In the Secretariat's view, the model is adaptable to situations involving the replacement of systems. However, as of June 1998, it had not adapted the model to that use in assessing the progress of departments and agencies.
20.90 In 1997, we reported that the testing and implementation phase took more effort than all other phases combined. It is now widely acknowledged in the information technology industry that the testing phase of Year 2000 projects generally has proved even more time-consuming than anticipated. Thus, a simple linear projection of percentage completion against a target elapsed time can prove to be optimistic.
20.91 The Secretariat did not validate data and information that it collected in its survey. Although some departments and agencies used internal audit or consultants to validate their Year 2000 progress, others did not. Best practices in industry have Year 2000 project offices seeking independent assurance to validate progress reports that they have received from units in their organizations. Some project offices go to the extent of using automated tools to verify the converted programming codes. Experience in some private sector firms showed that upon validation the reported percentage completed was often found to be overstated.
20.92 We compared our findings on the systems that support the six mission- critical functions we audited with information that the same departments and agencies had provided to the Secretariat. In most cases, the start dates of work and the completion dates that we had noted in the departments were consistent with those in the Secretariat's files. In general, its files did not contain sufficient details to use the number of lines of programming code in support of analysis of quantity of work done and assessment of progress toward making systems Year 2000 compliant.
20.93 While the Secretariat's survey asked for information on data interfaces and embedded systems, these were not used in the calculation of percentage completed. Consequently, the assessments of departments and agencies with significant data dependencies or numerous embedded devices may not adequately reflect their Year 2000 progress.
20.94 We reported last year that 31 December 1998 was emerging as the norm in the information technology industry for full implementation of compliant systems. We are concerned that the Secretariat's target date for full implementation allows only six months to address slippage and unexpected problems. Further, some systems supporting the 48 government-wide mission-critical functions have target dates beyond June 1999 for full implementation. Most of the systems we examined this year have experienced delays, some by up to four months.
Many mission-critical systems remain at risk20.95 An overall rate of completion for systems supporting government-wide mission-critical functions can provide no more than a general indication of Year 2000 progress. Given that the average rate of completion was 50 percent at 30 June 1998, some departments and agencies were progressing at below that rate.
20.96 According to the Secretariat's summary results at the end of June, 12 departments and agencies responsible for 28 mission-critical functions had a more than 50 percent completion rate for applications, systems and devices supporting those functions. The Secretariat also noted that information for two departments was not complete and that this could affect their reported percentage completion.
20.97 Overall, the Secretariat's summary of the progress made at 30 June 1998 by the 21 departments and agencies with mission-critical functions was as follows:
- 4 departments and agencies with good progress (over 60 percent complete);
- 10 departments and agencies with satisfactory progress (46 to 60 percent complete); and
- 7 departments and agencies with difficulties (45 percent complete or less).
20.98 Although the Secretariat's monitoring of progress showed that Year 2000 action in the government had accelerated, as of 30 June 1998 only four departments and agencies, responsible for 7 of the 48 mission-critical functions, were rated as having made good progress; work on 18 functions in nine departments and agencies had a score of 50 percent complete or less.
20.99 We are very concerned that as of 30 June 1998, various of these government systems supporting mission-critical functions remained at risk of not being ready in time.
20.100 In August 1998, the Secretariat advised us that some of its analysts had started attending Year 2000 project meetings in the 21 departments and agencies and the two Crown corporations. In addition, its Year 2000 project director had met with senior staff in some departments to discuss their progress.
20.101 The Secretariat had also written to departments and agencies responsible for systems that support the mission- critical functions to request that risk assessments be completed by 31 August 1998. For systems considered to be at risk of not meeting the implementation target date, detailed contingency plans will be requested by 31 December 1998.
20.102 In our 1997 audit chapter, we recommended that the Secretariat intervene strategically as appropriate to ensure the successful implementation of the government's mission-critical systems. Near the end of this year's audit, although it had held some meetings with certain departments to discuss Year 2000 progress, there were no documented plans to indicate what would trigger intervention, at what dates, and what form it would take.
20.103 Our 1997 chapter also raised the issue of triage, which provides for the redeployment of resources to those systems that are most critical to an organization. While the Secretariat acknowledged last year that triage may become necessary at some point, to date it is not evident what criteria would be used to judge this and how resources could be redeployed among departments and agencies.
20.104 If systems that support mission-critical functions were to remain at risk, it would be essential that contingency plans be developed and tested prior to 2000. Overall, the government's objective for Year 2000 readiness is to ensure that essential programs and services continue for its beneficiaries and stakeholders. The ideal would be fully compliant systems that would support mission-critical functions; interim work-around arrangements may be acceptable for the short term.
20.105 The government should continue to rank Year 2000 as a top priority and further accelerate its efforts on those systems that support its mission-critical functions, to reduce the risk that they may not be fully compliant before 2000.
20.106 The Treasury Board Secretariat should determine when and how it ought to exercise strategic intervention to ensure Year 2000 compliance of systems that support mission-critical functions in government. It should also develop plans for triage and put them in place as necessary.
20.107 Where systems that support mission-critical functions remain at risk, the Secretariat should ensure that contingency plans are developed and tested prior to 2000 so that essential government programs and services continue after 31 December 1999.
20.108 The Secretariat should examine ways to further improve the quality of its survey data and information, including the use of other measures in analyzing and assessing progress and the independent verification of progress information.
Treasury Board Secretariat's response: The Treasury Board Secretariat agrees with and supports the main points of the chapter. The chapter highlights areas of both concern and progress, areas with which we have widely been reported to agree.
We feel that it is critical to note that progress is an ongoing fact during the Year 2000 remediation programs in the departments and agencies we are monitoring. Not only is the percentage of completion (as reported monthly) steadily increasing, but the pace of change is also increasing. In fact, since the audit was completed at the end of June 1998, the overall government readiness has moved from 52 to 64 percent.
We agree with the audit recommendations that the government continue to rank Year 2000 readiness as a top priority. The Treasury Board will continue to receive regular monthly reports on progress. Year 2000 will also be the topic of correspondence from the President of the Treasury Board to his Cabinet colleagues, and from the Secretary to deputy ministers.
The rate of compliance continues to improve; nonetheless, we agree that prudent project management acknowledges the fact that slippage can occur, and triage may have to occur. The Treasury Board Secretariat will be working with departments and agencies in the development of plans for the triage of systems that support government-wide mission-critical functions.
The Secretariat is working with departments and agencies to assist, where possible, in the development or refinement of contingency plans to ensure that government-wide mission-critical functions continue.
The Secretariat does not rely exclusively on elapsed time as an indicator of progress. It is willing to consider changes to improve the quality of reporting, provided that work presently being done on remediation is not jeopardized. We have recommended to departments and agencies the engagement of their own internal audit staff to provide independent verification and validation of the results our analysts collect and report.
Initiatives to Address Common Needs
Some progress made in addressing exposure for Year 200020.109 In the 1997 audit, we identified a number of areas in which Year 2000 projects faced significant exposure to risk:
- competing priorities and developments;
- insufficient technical resources;
- failure or delay in obtaining compliant upgrades from vendors;
- data interface exposure; and
- risk of funding delay.
20.110 For example, the Secretariat wrote on several occasions to deputy heads and heads of agencies to emphasize the significance of the Year 2000 threat and the high priority that it ought to be assigned. In March 1998, the government also directed departments and agencies to consider Year 2000 implications in all policy proposals.
20.111 We noted in 1997 that departments were expressing concerns about availability of technical resources; many were experiencing higher than normal attrition rates among computer specialists. Since that time, efforts have been under way to identify and attract information technology professionals into government. The conclusion of contract negotiations with the labour union for the government's computer specialist community has helped to retain staff. In addition, an omnibus contract has been awarded to seven private sector firms to support Year 2000 remediation and testing work on systems that support government-wide mission-critical functions.
20.112 In 1997, the Secretariat estimated the government's total costs for Year 2000 work at $1 billion. During our 1998 audit, its Year 2000 project office advised us that the cost estimates had been revised to $1.4 billion. To address the risk of delay in some Year 2000 projects in the event that funds could not be secured on a timely basis, the Secretariat made provisions to make up to a total of $400 million available for loans to support the departments and agencies responsible for mission-critical functions. As of July 1998, the Secretariat had approved loans totalling about $365 million to 15 departments and agencies.
Work plan needed for remaining horizontal issues20.113 While some progress has been made in addressing common needs, exposure remains in the areas we have noted. There is also significant exposure in other areas.
20.114 For example, the availability of technical resources continues to be an area of significant exposure for all Year 2000 projects. Although departments indicate that attrition rates have stabilized since the conclusion of contract negotiations for the government's computer specialists, staff turnover has continued. In examining mission-critical systems, we noted a specific case of slippage for which shortage of technical resources was cited as the reason. Further, the contract for computer specialists in government will expire in 1999, when Year 2000 work will still be critical.
20.115 Data interface also represents an area of significant exposure. Information technology research firms have emphasized the need for end-to-end testing among organizations that interact electronically. In describing its model for comparing Year 2000 progress among organizations, one research firm indicated that such testing would be time-consuming and in some cases difficult to accomplish.
20.116 Legal liability is another area of exposure. In response to our 1997 recommendations, the government assigned a lawyer to co-ordinate its legal efforts. Subsequently, legal co-ordinators were identified in various departments and agencies. The efforts helped to raise Year 2000 awareness in the government's legal community and resulted in a Year 2000 issue paper. In March 1998, the issue paper was released to departments and agencies to help them in assessing the legal risks to their respective organizations as a result of Year 2000. Nevertheless, given that the government has many regulatory responsibilities, the implications of the Year 2000 threat have yet to be fully analyzed and addressed.
20.117 Embedded systems and devices are emerging as a significant concern. Numbering in the billions, they perform a vast range of duties serving many domains such as science, engineering, manufacturing and health care. Experts have estimated that only a small percentage of these micro processors are date-sensitive. Even so, an estimated 20 million to 250 million could fail as a result of the millennium bug.
20.118 In January 1998, the Treasury Board Secretariat tasked a small team of assistant deputy ministers to review Year 2000 readiness in government. In February 1998, the team reported the issues and areas of exposure we have noted, as well as other impediments to success. It concluded that the Year 2000 challenge could be overcome only if decisions and actions were taken to address the issues it had identified.
20.119 As of August 1998, the Secretariat had started to define broad roles for some departments and agencies to address a number of those issues. In early October we were advised by the Secretariat that it had identified two initiatives to assist departments and agencies in dealing with embedded technology, and that it planned to hold a symposium on embedded devices and technology later that month. The Secretariat indicated that the initiatives would proceed once funding became available. However, a work plan that includes the following has yet to be developed:
- a full list of the issues to be addressed;
- roles and responsibilities of each organization assigned to address them;
- the resources required;
- the desired deliverables; and
- a time frame for their completion.
20.120 The Treasury Board Secretariat should develop a work plan and take steps to facilitate the resolution of common horizontal issues that departments and agencies face in addressing Year 2000.
Treasury Board Secretariat's response: Plans and efforts have been put into place for monitoring and assessing departmental readiness, and identifying horizontal readiness issues through interdepartmental working groups and other means. With the need for a more frequent and detailed reporting schedule to Treasury Board ministers, and for addressing overall issues related to government readiness, Treasury Board Secretariat agrees that additional and more precise plans must be prepared. A master plan for reporting to ministers has already been completed and the integrating of additional activities is under way.
Reporting Information to Ministers and Parliament
Information on Year 2000 urgency and progress has been provided to ministers20.121 In 1997, we recommended that departments and agencies work to engage the involvement and concern of ministers and advise them of the urgency of the Year 2000 threat. We also recommended that the Secretariat advise the Treasury Board periodically of the government's progress. In addition, we recommended that Parliament be kept informed about the potential impact of Year 2000 on the government and about the government's progress in making its systems compliant.
20.122 We found, in this audit, that the government has kept ministers informed about the priority and significance of Year 2000. Further, since our 1997 audit the Secretariat has appeared several times before the Treasury Board to discuss the Year 2000 issue and provide information on the government's progress, and to seek approval for specific measures and related funding as necessary. As we neared the completion of our audit work in 1998, the Secretariat advised us that it would be providing Treasury Board with monthly updates of Year 2000 progress.
Information to Parliament needs to be improved20.123 Subsequent to the tabling of our October 1997 Report to Parliament, two standing committees of the House of Commons held hearings on the subject of Year 2000. The Standing Committee on Industry studied industry readiness; the Standing Committee on Public Accounts focussed on Chapter 12 of our Report, on the government's preparedness for Year 2000. The Chief Information Officer of the Treasury Board Secretariat appeared as a witness at hearings of both committees.
20.124 In November 1997 the Public Accounts Committee tabled its Second Report to the House of Commons, on Chapter 12 of our Report. It recommended, among other things, that the Treasury Board Secretariat prepare semi-annual reports on Year 2000 progress and submit them to the Committee, commencing January 1998.
20.125 The government accepted the recommendations and the Secretariat submitted its first progress report to the Committee in early February 1998. The progress report was dated January 1998 and reflected status as of the fall of 1997. As of 31 August 1998, no further progress reports had been submitted. Consequently, the information that the Public Accounts Committee had on Year 2000 progress was almost one year out of date. The Secretariat has since advised us that in late September 1998 it submitted a second progress report to the Committee.
20.126 Although the Committee also recommended that information on departments' Year 2000 progress be included in their Performance Reports, we found no evidence that the Secretariat had communicated this to departments. Four departments and agencies that we audited this year referred to Year 2000 in their 1998-99 Estimates documents and only one department reported on the status of its Year 2000 project in its 1997 Performance Report to Parliament.
20.127 The government should improve its reporting of information to Parliament on its progress in making systems compliant for Year 2000.
Treasury Board Secretariat's response: Treasury Board Secretariat has reported to the Public Accounts Committee, the Industry Committee, and to Cabinet. We will continue to examine ways of keeping Parliament informed.
20.129 However, in reviewing the June 1998 survey results summarized by the Treasury Board Secretariat, we observed that only four departments responsible for 7 of the 48 mission-critical functions in the government were rated as having made good progress; nine departments and agencies responsible for 18 mission- critical functions had a score for work completed of 50 percent or less. The results of our examination also showed that, as of June 1998, several key systems supporting three of six government-wide mission-critical functions remained exposed to the Year 2000 threat.
20.130 We have concluded that with 18 months left, many systems that support mission-critical functions remain at risk of not being ready before 2000. We are very concerned that some essential government services could be interrupted at the start of 2000.
20.131 Notwithstanding the accelerated pace of work and the progress made since our last audit, more needs to be done. There needs to be a continued focus on Year 2000, including strategic intervention by the Treasury Board Secretariat as appropriate, and preparation and testing of contingency plans by departments and agencies to ensure that those mission-critical functions continue without interruption into 2000.
20.132 As we reported in 1997, other areas remain exposed to risk and merit common, horizontal efforts to address them. Reporting of information to Parliament on Year 2000 issues needs to be improved.
About the Audit
Objective and ScopeThe 1998 audit focussed on selected government-wide mission-critical functions and selected systems that support them. Our objective was to assess the progress the government has made in identifying and mitigating the risks to its systems that support mission-critical functions as a result of the millennium date code problem, and to report on this to Parliament.
We examined actions taken by the government to address our October 1997 Report recommendations. We obtained an update of the government's state of readiness for Year 2000 at 30 June 1998 by reviewing survey results collected by the Treasury Board Secretariat. We have not audited the representations made by departments and agencies in the surveys.
In addition, we assessed Year 2000 risks to certain systems and applications that support six mission-critical functions of the government. The functions were selected on the basis of their significance to health and safety, to financial and legal implications and to essential services and support for program beneficiaries. Those systems and applications operate in the following departments and agencies:
- Canadian Food Inspection Agency;
- National Defence;
- Human Resources Development Canada;
- Indian and Northern Affairs Canada;
- Revenue Canada; and
- Royal Canadian Mounted Police.
Our findings pertain only to the systems and applications that we examined and are not representative of general progress in those departments and agencies.
CriteriaThe general criteria used in this audit followed those used in our 1997 audit of the government's overall state of Year 2000 preparedness:
- There should be ongoing leadership initiatives to sustain commitment and support from departmental and/or agency senior management.
- Systems and applications that are critical to government as a whole should be identified, and the progress of their Year 2000 conversion, testing and implementation should be monitored and reported periodically. Where necessary, strategic intervention should be exercised to ensure that these systems and applications are successfully implemented before 2000.
- Where risk of failure remains a threat to critical functions of government, contingency plans should be developed and measures put in place to provide for appropriate transition to 2000.
- Initiatives should be in place to co-ordinate and facilitate Year 2000 efforts of departments and agencies to maximize their cost effectiveness.
- Parliament should be kept informed of matters of significance arising from the Year 2000 challenge and its effects on government programs and service delivery, including the government's progress in making its systems Year 2000 compliant.
Audit TeamAssistant Auditor General: Douglas Timmins
Principal: Nancy Cheng
Director: Maria Wisniowski
For information, please contact Nancy Cheng.